断电重启异常4070

AntTech_KKDF8Q · 2026 年7 月 15 日 20:24

OceanBase 4.4.1 CE 单机启动失败 -4070，clog block checksum 损坏

环境
OceanBase 4.4.1.0 CE，OBD 单节点部署，express_oltp 场景
Kylin V10 SP1，x86_64，31GB 内存
数据目录：/root/oceanbase-ce/store/
现象
observer 启动后 10 秒自动退出，报错：

ERROR [SERVER] start (ob_server.cpp:1185) [server_start 9/18] observer instance start fail
EDIAG [STORAGE] handle_tenant_create_commit … errcode=-4070 fail to replay create tenant
EDIAG [CLOG] OB_LOG_CHECKSUM_MISMATCH … “invalid data or checksum mismatch”

根因定位
系统崩溃前内存不足（Free: 259M, Need: 6G），observer 写入 clog/log_pool/2981 时被中断，导致该文件 checksum 损坏
SSTable/slog 中的 super_block 硬编码指向 file_id=106, ls_meta_entry[2nd=2981]，observer 启动时强制回放到此处
读取 block 2981 → checksum 校验失败 → -4070 → observer 退出
ob_admin log_tool dump_meta 确认：ERROR|CLOG|OB_LOG_CHECKSUM_MISMATCH|-4070|block 2981

尝试过的修复
–mode standalone / recovery / normal — 全部 -4070
–scn 0 闪回 — 无效
删除损坏 slog 文件（106, server/14-17）— 级联回退到更老的 checkpoint，对应的 clog 块已被回收
Python 文本替换 block_file 中 super_block（file_id=106→105, 2nd=2981→2980）— slog 中为二进制编码，文本替换无效
清空所有 slog + 删 clog/tenant_1 — log_pool integrity 失败（-4016）
填零 block 2981 — observer 仍从 slog 读取原始 super_block
ob_admin 工具 — 仅支持读取，不支持写入 super_block
obdiag — 集群未运行无法连接
重命名 observer.config.bin — 不相关
核心矛盾
SSTable block_file (8GB) 数据完好，但 super_block 以二进制格式多层存储（SSTable / slog/tenant_1/106 / slog/server/），CE 版没有工具可以修改。clog 归档未开启，无副本可恢复。

求助
CE 版是否有隐藏参数（如 _skip_replay、_bypass_checkpoint）可以跳过 super_block 的 replay 检查？或者是否有内部工具可以修改 SSTable 中的 super_block 元数据指针？

AntTech_KKDF8Q · 2026 年7 月 15 日 20:28

请问官方有工具吗，收费的工具也行！

Sunshining · 2026 年7 月 16 日 05:09

突然断电确实会带来很多异常

旭辉 · 2026 年7 月 16 日 10:20

你发下这个完整的observer.log 看下

耶一耶 · 2026 年7 月 16 日 10:21

学一下

AntTech_RFKTWX · 2026 年7 月 16 日 10:22

说是pg断电重启啥事没有

AntTech_KKDF8Q · 2026 年7 月 16 日 10:44

EDIAG [STORAGE] handle tenant create comit (ob server checkpoint s log handler.cpp:848) [1337221tobserver][To][Yo-00000000000001-0-0] [lt-1l][errcode-4070] fail to replay create tenant(ret-4070, tenant meta-funit: ftenan id:1, unit id:1, has memstore:true, unit status:“NoRMAL”, config:funit config id:1, name:“sys unit config”, resource:(min cpu:2, max cpu:2, memory size: “16B”, Log disk size:“26B”, data disk size:0, min iops:9223372036854775807, max iops:9223372036854775807, 1ops weight:2, max net bandwidth: INT64 MAX, net bandvidth veight:2, }, mode:0, create timestamp:176228715523748, istarint 6Lgus0rfite-10 c00g -102050ct0rsct-144201 9 mct ntrver-1.,b=-0cg-5401312nd-2901wait gc tablet entry: ([ver-1,mode-0,seq-0][2nd-18446744073709551615]}, tablet meta entry: {[ver-1, mode-0,seq-0][2nd-1844674407309516151), 1s hidden:fal se, version4, snapshot cnt:0, preallocated segs: object seq:60000, tmp fite seq:60000, write seq:60
09194ab0 0x9194587 091944dd 8x9194308 0x1acoc087 0x1ac5ec4e 0x10752269x12027f0c 0xe99e26r bxe93002 5x23e1aca0oxe90032 0x
1522076b50b3 0x96682aa
2026-07- 15 19:5756.129159] EDIAG [SToRAGE] apply replay result (ob server checkpoint stog handLer.cpp:788) [133722][observer
[To] Y0-000000000000001-0-0] [1t=79errcode-4070] fail to handle tenant create commit(ret=-4070, tenant meta-(unit: (tenantid:1, unit id:1, has memstore:true, unit status:“NoRMAL”, config: funit config id:1, name:“sys unit config”, resource: (min cpu2, max cpu:2, memory size:“1GB”, 1og disk size:“2GB”, data disk size:0, min iops:9223372036854775807, max iops:922337203685475807, 1ops weight:2, max net bandvidth:INT64 MAX, net bandvidth weight:2, , mode:0, create timestamp:1762228715523748, isIemoved:false, hidden sys data disk config size:0, actual data disk size:0, replica type:0), super block:tenant id:1, replay start point:0bLogcursorffile id-10, 1og id=17620501, offset=14423695), 15 meta entry:[ver=1,mode-0,seg-54107131[2nd-2981]), "ait gC tablet entry: ([ver=1, mode-0,seq-0][2nd-18446744073709551651), tablet meta entry:([ver-1, mode-0,seq-0][2nd-184467440737095516151), is hidden:false, version:4, snaphot cnt:, preallocated segs:fobject seq: 60000, tmp file seq: 60000, write seq:6000), auto inc 1s epoch:0, 1s cnt:0, min file id:0, max file id:0), create status:1, epoch:0}) BACKTRACE: 0x91dffd8 ex90e38cd 0
9194ab0 0x9194587 0x91944dd 0x9194308 0x1ac6b9d 0x1ac5ec79 0x1a7522e7 0x12027f6c 0xe99e2f 0xe983d02 0x23elaca0 0xe9a0527 0x1
522076b50b3 0x96682aa
12026-07- 15 19:57:56.129214] wDIAG [SoRAGE] start (ob server checkpoint stog handler.cpp:108) [133722] [observer] [T0][Y0 -000000000000001-0-0] [lt=6][errcode=-4070] fail to apply replay result(ret=-4070)12026-07- 15 19:57:56.129224] WDIAG [STORAGE] start (ob server storage meta service.cpp:73) [133722]observer][T0] (Y0-000000000000001-0-0] [lt=7][errcode=-4070] fail to start replay(ret=-4070)2026-07-15 19:57:56.129238] EDIAG [SERVER] start (ob server.cpp:1027) [133722]observerl[T0][Y0-000000000000001-0-0] [Lt-61rrcode-- 4070] fail to Start server Storage meta service (ret–4070, ret=“0B INVALID DATA”) BACKTRACE:0x91dffd8 Ox90e38cd Ox9191ab0 0x9194587 0x91944dd 0x9194308 0x1202ddb4 0x120294af 0xe99e2bf 0xe9a3d02 0x23e1aca0 0xe90527 0x1522076b50b3 0x96682aa2026-07- 15 19:57:56.129276] ERROR [SERVER] start (ob server.cp:185) [133722][observer][T0][Y0 -0000000000001-0-0] [Lt-121errcode–4070] [server start 9/18] observer instance start fail. you may find solutions in previous error logs or seek help
rom official technicians

旭辉 · 2026 年7 月 16 日 10:45

这个observer.log压缩上传一下吧

AntTech_KKDF8Q · 2026 年7 月 16 日 10:48

新用户无法上传附件

AntTech_KKDF8Q · 2026 年7 月 16 日 10:49

[2026-07-15 19:57:56.092800] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] mtl is nullptr before destroy(type=“PN9oceanbase8tmp_file22ObTenantTmpFileManagerE”)
[2026-07-15 19:57:56.092805] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=4] finish destroy mtl20(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase8tmp_file22ObTenantTmpFileManagerE”)
[2026-07-15 19:57:56.092811] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] mtl is nullptr before destroy(type=“PN9oceanbase7storage26ObTenantStorageMetaServiceE”)
[2026-07-15 19:57:56.092815] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=4] finish destroy mtl19(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase7storage26ObTenantStorageMetaServiceE”)
[2026-07-15 19:57:56.092821] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] mtl is nullptr before destroy(type=“PN9oceanbase7storage11ObLSServiceE”)
[2026-07-15 19:57:56.092825] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=4] finish destroy mtl18(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase7storage11ObLSServiceE”)
[2026-07-15 19:57:56.092830] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] mtl is nullptr before destroy(type=“PN9oceanbase10logservice18ObGarbageCollectorE”)
[2026-07-15 19:57:56.092835] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=4] finish destroy mtl17(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase10logservice18ObGarbageCollectorE”)
[2026-07-15 19:57:56.092840] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] mtl is nullptr before destroy(type=“PN9oceanbase10logservice12ObLogServiceE”)
[2026-07-15 19:57:56.092845] INFO delete_tenant_log_allocator (ob_tenant_mutil_allocator_mgr.cpp:59) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=4] delete_tenant_mutil_allocator_ success(tenant_id=1)
[2026-07-15 19:57:56.092851] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=6] finish destroy mtl16(cost_time_us=7, mtl_ptr=NULL, type=“PN9oceanbase10logservice12ObLogServiceE”)
[2026-07-15 19:57:56.092857] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] mtl is nullptr before destroy(type=“PN9oceanbase10logservice11coordinator17ObFailureDetectorE”)
[2026-07-15 19:57:56.092862] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5] finish destroy mtl15(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase10logservice11coordinator17ObFailureDetectorE”)
[2026-07-15 19:57:56.092867] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] mtl is nullptr before destroy(type=“PN9oceanbase10logservice11coordinator19ObLeaderCoordinatorE”)
[2026-07-15 19:57:56.092871] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=4] finish destroy mtl14(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase10logservice11coordinator19ObLeaderCoordinatorE”)
[2026-07-15 19:57:56.092877] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] mtl is nullptr before destroy(type=“PN9oceanbase11transaction14ObTransServiceE”)
[2026-07-15 19:57:56.092881] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=4] finish destroy mtl13(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase11transaction14ObTransServiceE”)
[2026-07-15 19:57:56.092886] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] mtl is nullptr before destroy(type=“PN9oceanbase5share19ObSharedMemAllocMgrE”)
[2026-07-15 19:57:56.092891] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5] finish destroy mtl12(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase5share19ObSharedMemAllocMgrE”)
[2026-07-15 19:57:56.092897] WDIAG [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=5][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth]
[2026-07-15 19:57:56.092902] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=4] finish destroy mtl11(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase12blocksstable21ObSharedMacroBlockMgrE”)
[2026-07-15 19:57:56.092908] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=0] finish destroy mtl10(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase7storage3mds18ObTenantMdsServiceE”)
[2026-07-15 19:57:56.092914] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=0] finish destroy mtl9(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase6common17ObTenantIOManagerE”)
[2026-07-15 19:57:56.092920] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=0] finish destroy mtl8(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase6common18ObServerObjectPoolINS_7storage19ObTableScanIteratorEEE”)
[2026-07-15 19:57:56.092926] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=0] finish destroy mtl7(cost_time_us=1, mtl_ptr=NULL, type=“PN9oceanbase6common18ObServerObjectPoolINS_11transaction14ObPartTransCtxEEE”)
[2026-07-15 19:57:56.092932] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=0] finish destroy mtl5(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase7storage18ObTenantMetaMemMgrE”)
[2026-07-15 19:57:56.092937] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=0] finish destroy mtl4(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase3sql21ObTenantSQLSessionMgrE”)
[2026-07-15 19:57:56.092943] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=0] finish destroy mtl3(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase3omt13ObSharedTimerE”)
[2026-07-15 19:57:56.092949] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=0] finish destroy mtl2(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase6common14ObTimerServiceE”)
[2026-07-15 19:57:56.092955] INFO [SHARE] operator() (ob_tenant_base.cpp:336) [133722][observer][T1][Y0-0000000000000001-0-0] [lt=1] finish destroy mtl1(cost_time_us=0, mtl_ptr=NULL, type=“PN9oceanbase6common25ObDiagnosticInfoContainerE”)
[2026-07-15 19:57:56.093028] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=4] set idle finish(ret=0, tenant_id=1, ctx_id=0, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093039] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=11] set idle finish(ret=0, tenant_id=1, ctx_id=1, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093048] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=2, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093055] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=7] set idle finish(ret=0, tenant_id=1, ctx_id=3, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093061] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=4, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093067] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=5, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093075] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=5] set idle finish(ret=0, tenant_id=1, ctx_id=6, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093083] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=7] set idle finish(ret=0, tenant_id=1, ctx_id=7, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093090] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=8, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093096] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=9, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093102] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=5] set idle finish(ret=0, tenant_id=1, ctx_id=10, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093109] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=11, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093114] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=5] set idle finish(ret=0, tenant_id=1, ctx_id=12, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093121] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=13, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093128] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=14, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093134] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=15, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093139] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=5] set idle finish(ret=0, tenant_id=1, ctx_id=16, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093145] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=5] set idle finish(ret=0, tenant_id=1, ctx_id=17, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093152] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=18, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093157] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=5] set idle finish(ret=0, tenant_id=1, ctx_id=19, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093163] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=20, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093169] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=21, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093177] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=22, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093183] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=5] set idle finish(ret=0, tenant_id=1, ctx_id=23, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093189] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=24, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093195] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=25, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093202] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] set idle finish(ret=0, tenant_id=1, ctx_id=26, idle_size=0, chunk_cnt=0)
[2026-07-15 19:57:56.093209] INFO [CLOG] abort_create_tenant (ob_server_log_block_mgr.cpp:361) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] ObServerLogBlockMGR abort_create_tenant success(this={dir::"/root/oceanbase-ce/store/clog/log_pool", dir_fd:19, meta_fd:20, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:2797, max_block_id:2996, min_log_disk_size_for_all_tenants_:0, is_inited:true}, log_disk_size=2147483648)
[2026-07-15 19:57:56.093254] INFO [COMMON] mark_tenant_delete (ob_kvcache_inst_map.cpp:369) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=13] mark delete details(ret=0, tenant_id=1)
[2026-07-15 19:57:56.109114] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [133872][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.109609] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [133874][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.109664] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [133871][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.109858] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [133877][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.109981] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [133867][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.109986] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [133873][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.109990] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [133868][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.110091] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [133869][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.110099] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [133870][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.110457] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [133879][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.110890] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [133875][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
[2026-07-15 19:57:56.129003] INFO [COMMON] retire (ob_kvcache_hazard_version.cpp:288) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=8] erase tenant hazard map node details(ret=0, tenant_id=1)
[2026-07-15 19:57:56.129050] INFO [COMMON] erase_tenant (ob_kvcache_inst_map.cpp:417) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=16] erase tenant cache inst details(ret=0, tenant_id=1)
[2026-07-15 19:57:56.129059] INFO [COMMON] sync_flush_tenant (ob_kv_storecache.cpp:501) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=9] erase tenant cache details(ret=0, tenant_id=1)
[2026-07-15 19:57:56.129067] INFO [SERVER.OMT] create_tenant (ob_multi_tenant.cpp:1294) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6] finish create new tenant(ret=-4070, tenant_id=1, write_slog=false, create_step=5, bucket_lock_idx=9780)
[2026-07-15 19:57:56.129079] EDIAG [STORAGE] handle_tenant_create_commit (ob_server_checkpoint_slog_handler.cpp:848) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=11][errcode=-4070] fail to replay create tenant(ret=-4070, tenant_meta={unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:“NORMAL”, config:{unit_config_id:1, name:“sys_unit_config”, resource:{min_cpu:2, max_cpu:2, memory_size:“1GB”, log_disk_size:“2GB”, data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:2, }}, mode:0, create_timestamp:1762228715523748, is_removed:false, hidden_sys_data_disk_config_size:0, actual_data_disk_size:0, replica_type:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=106, log_id=17620501, offset=14423695}, ls_meta_entry:{[ver=1,mode=0,seq=5410713][2nd=2981]}, wait_gc_tablet_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0, min_file_id:0, max_file_id:0}, create_status:1, epoch:0}) BACKTRACE:0x91dffd8 0x90e38cd 0x9194ab0 0x9194587 0x91944dd 0x9194308 0x1ac6c087 0x1ac5ec4e 0x1a7522e7 0x12027f6c 0xe99e2bf 0xe9a3d02 0x23e1aca0 0xe9a0527 0x1522076b50b3 0x96682aa
[2026-07-15 19:57:56.129159] EDIAG [STORAGE] apply_replay_result (ob_server_checkpoint_slog_handler.cpp:788) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=79][errcode=-4070] fail to handle tenant create commit(ret=-4070, tenant_meta={unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:“NORMAL”, config:{unit_config_id:1, name:“sys_unit_config”, resource:{min_cpu:2, max_cpu:2, memory_size:“1GB”, log_disk_size:“2GB”, data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:2, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:2, }}, mode:0, create_timestamp:1762228715523748, is_removed:false, hidden_sys_data_disk_config_size:0, actual_data_disk_size:0, replica_type:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=106, log_id=17620501, offset=14423695}, ls_meta_entry:{[ver=1,mode=0,seq=5410713][2nd=2981]}, wait_gc_tablet_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0, min_file_id:0, max_file_id:0}, create_status:1, epoch:0}) BACKTRACE:0x91dffd8 0x90e38cd 0x9194ab0 0x9194587 0x91944dd 0x9194308 0x1ac6b90d 0x1ac5ec79 0x1a7522e7 0x12027f6c 0xe99e2bf 0xe9a3d02 0x23e1aca0 0xe9a0527 0x1522076b50b3 0x96682aa
[2026-07-15 19:57:56.129207] INFO [STORAGE] apply_replay_result (ob_server_checkpoint_slog_handler.cpp:813) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=47] finish replay create tenants(ret=-4070, tenant_count=1)
[2026-07-15 19:57:56.129214] WDIAG [STORAGE] start (ob_server_checkpoint_slog_handler.cpp:108) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6][errcode=-4070] fail to apply replay result(ret=-4070)
[2026-07-15 19:57:56.129224] WDIAG [STORAGE] start (ob_server_storage_meta_service.cpp:73) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=7][errcode=-4070] fail to start replay(ret=-4070)
[2026-07-15 19:57:56.129232] INFO [STORAGE] start (ob_server_storage_meta_service.cpp:84) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=7] finish start server storage meta service(ret=-4070, cost_time_us=10850538)
[2026-07-15 19:57:56.129238] EDIAG [SERVER] start (ob_server.cpp:1027) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=6][errcode=-4070] fail to start server storage meta service(ret=-4070, ret=“OB_INVALID_DATA”) BACKTRACE:0x91dffd8 0x90e38cd 0x9194ab0 0x9194587 0x91944dd 0x9194308 0x1202ddb4 0x120294af 0xe99e2bf 0xe9a3d02 0x23e1aca0 0xe9a0527 0x1522076b50b3 0x96682aa
[2026-07-15 19:57:56.129276] ERROR [SERVER] start (ob_server.cpp:1185) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=12][errcode=-4070] [server_start 9/18] observer instance start fail. you may find solutions in previous error logs or seek help from official technicians.
[2026-07-15 19:57:56.152074] INFO destroy_tg (thread_mgr.cpp:89) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=5] destroy tg(tg_id=278, tg=0x1521fd237cb0, tg->attr_={name:StartupAccelHandler, type:4})
[2026-07-15 19:57:56.152090] EDIAG [SERVER] start (ob_server.cpp:1263) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=14][errcode=-4070] failure occurs, try to set stop and wait(ret=-4070, ret=“OB_INVALID_DATA”) BACKTRACE:0x91dffd8 0x90e38cd 0x9194ab0 0x9194587 0x91944dd 0x9194308 0x1202fc0a 0x1202aabb 0xe99e2bf 0xe9a3d02 0x23e1aca0 0xe9a0527 0x1522076b50b3 0x96682aa
[2026-07-15 19:57:56.152122] ERROR [SERVER] start (ob_server.cpp:1267) [133722][observer][T0][Y0-0000000000000001-0-0] [lt=9][errcode=-4070] [server_start 10/18] observer start fail, the stop status is true. you may find solutions in previous error logs or seek help from official technicians.

旭辉 · 2026 年7 月 16 日 10:55

find /root/oceanbase-ce/store/clog -type f | sort | tail -30
ls -la /root/oceanbase-ce/store/clog/log_pool/ | sort -k9 -n | tail -20

查询下

AntTech_KKDF8Q · 2026 年7 月 16 日 11:07

AntTech_KKDF8Q · 2026 年7 月 16 日 11:08

lQLPJxYCKv_KkV_NBNDNB96wFNRZx-p64EEKK9bG4L19AA_2014_12321920×1174 177 KB

旭辉 · 2026 年7 月 16 日 11:28

clog 是 OceanBase 的 WAL（ redo 日志），palf 要求 block 严格连续、可校验 。现在 block 2981 的 info block checksum 损坏，且它不是尾块，后面还有 2982~2995

删 2981 留后面 → palf 读到不连续日志，仍然会报错。
删 2981~2995 → 会截断 WAL，但 slog 里的 super_block.replay_start_point 仍然硬编码指向 file_id=106, ls_meta_entry[2nd=2981]，启动时还会尝试回放 2981。
要同时改 slog super_block + palf manifest + 截断 clog，没有官方工具，手工二进制改风险极高，极易把状态搞得更乱。

这个节点已经很难低成本修复

AntTech_AF3WIR · 2026 年7 月 16 日 11:52

是否还有别的方式恢复？或者是能够把几张关键表的数据导出来也可以。

AntTech_KKDF8Q · 2026 年7 月 16 日 11:54

可以通过其他方式恢复数据吗，或者购买其他版本兼容啥的

旭辉 · 2026 年7 月 16 日 14:05

确认下没有做过备份是吧？

wkk2025 · 2026 年7 月 16 日 14:06

学到了

AntTech_AF3WIR · 2026 年7 月 16 日 14:44

是的没有做过备份

随遇而安 · 2026 年7 月 16 日 14:54

数据文件和日志没丢失的话是可以恢复的