OceanBase4.3.5.5在ocp执行备份失败，显示推进检查点超时

OceanBase 技术问题
#1

版本: OceanBase4.3.5.5
近期异常：10.8日出现了clog磁盘达到阈值95%写入不进去数据，扩充了clog磁盘空间解决
问题：10.9日手动在ocp上备份，等待2小时后报错：

2026-10-09 12:41:10.873 ERROR 2892065 — [manual-subtask-executor-16,fa79d71f08536524,f9574136c4d98da4] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Data backup task failed.

com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_DATA_BACKUP_TASK_FAILED, args=
at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.checkDataBackupTaskFinished(WaitDataBackupSuccessTask.java:170)
at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.lambda$run$5(WaitDataBackupSuccessTask.java:211)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.run(WaitDataBackupSuccessTask.java:206)
at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64)
at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32)
at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26)
at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:204)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:198)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$0(SubtaskExecutor.java:124)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:30)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Set state for subtask: 1028873, operation:EXECUTE, state: FAILED

寻求解决方案

1 个赞
#3

sys租户查一下

SELECT * FROM oceanbase.CDB_OB_BACKUP_JOB_HISTORY where TENANT_ID=xxx order by START_TIMESTAMP desc limit 20\G;
SELECT * FROM oceanbase.CDB_OB_BACKUP_TASK_HISTORY where TENANT_ID=xxx order by START_TIMESTAMP desc limit 20\G;

SELECT TENANT_NAME, LOG_MODE FROM oceanbase.DBA_OB_TENANTS\G;
SELECT * FROM oceanbase.CDB_OB_ARCHIVELOG where TENANT_ID=xxx\G;
#4

SELECT * FROM oceanbase.CDB_OB_BACKUP_JOB_HISTORY where TENANT_ID=xxx order by START_TIMESTAMP desc limit 20\G;

1002 2394 1 45 1 126 1002 OFF FULL USER_TENANT NONE 2026-10-09 13:41:16.089519 2026-10-09 15:43:34.458035 FAILED -9086 (SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595B684F-0-0 ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2026-10-09 13:41:15.0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2393 1 44 1 123 1002 OFF FULL USER_TENANT NONE 2026-10-09 13:31:36.408187 2026-10-09 13:40:15.265098 CANCELED -4072 ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2026-10-09 13:31:36.0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2392 1 43 1 122 1002 OFF FULL USER_TENANT NONE 2026-10-09 13:29:18.260823 2026-10-09 13:31:23.355017 CANCELED -4072 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2391 1 42 1 121 1002 OFF FULL USER_TENANT NONE 2026-10-09 13:26:35.352413 2026-10-09 13:28:53.213492 CANCELED -4072 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2390 1 41 1 118 1002 OFF FULL USER_TENANT NONE 2026-10-09 13:11:01.141888 2026-10-09 13:25:14.956670 CANCELED -4072 ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2026-10-09 13:11:00.0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2386 1 40 1 115 1002 OFF FULL USER_TENANT NONE 2026-10-09 10:39:15.531813 2026-10-09 12:41:01.864255 FAILED -9086 (SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595AE95B-0-0 ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2026-10-09 10:39:15.0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2362 1 39 1 112 1002 OFF FULL USER_TENANT NONE 2026-10-08 10:30:06.229131 2026-10-08 12:31:56.833682 FAILED -9086 (SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659A9446D90F4-0-0 ALTER BACKUP CLUSTER FULL_BACKUP BY SCHEDULED AT 2026-10-08 10:30:05.0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2313 1 38 1 109 1002 OFF FULL USER_TENANT NONE 2026-10-06 10:30:06.264364 2026-10-06 12:32:05.013884 FAILED -9086 (SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659A87EFE4F15-0-0 ALTER BACKUP CLUSTER FULL_BACKUP BY SCHEDULED AT 2026-10-06 10:30:05.0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2264 1 37 1 106 1002 OFF FULL USER_TENANT NONE 2026-10-04 10:30:06.199173 2026-10-04 12:32:25.642826 FAILED -9086 (SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659A87EFC1B9D-0-0 ALTER BACKUP CLUSTER FULL_BACKUP BY SCHEDULED AT 2026-10-04 10:30:05.0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2191 1 36 1 103 1002 OFF FULL USER_TENANT NONE 2026-10-01 10:30:06.231642 2026-10-01 10:49:12.700811 COMPLETED 0 ALTER BACKUP CLUSTER FULL_BACKUP BY SCHEDULED AT 2026-10-01 10:30:05.0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data
1002 2142 1 35 1 100 1002 OFF FULL USER_TENANT NONE 2026-09-29 10:30:13.353139 2026-09-29 11:18:48.287447 COMPLETED 0 ALTER BACKUP CLUSTER FULL_BACKUP BY SCHEDULED AT 2026-09-29 10:30:07.0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data

SELECT * FROM oceanbase.CDB_OB_BACKUP_TASK_HISTORY where TENANT_ID=xxx order by START_TIMESTAMP desc limit 20\G;

1002 168 2394 1 45 2026-10-09 13:41:16.089519 2026-10-09 15:43:34.458035 FAILED 1791524505356343000 0 1791524518347112000 NONE 2100338 78286 10.6680 0 3 3 1 1 0 1 0 -9086 backup advance checkpoint by flush timeout file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 167 2393 1 44 2026-10-09 13:31:36.408187 2026-10-09 13:40:15.256563 CANCELED 1791523913400554000 0 0 NONE 0 0 0.0000 0 0 0 0 0 0 1 0 -4072 Operation canceled file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 166 2392 1 43 2026-10-09 13:29:18.260823 2026-10-09 13:31:23.351034 CANCELED 1791523763262359000 0 0 NONE 0 0 0.0000 0 0 0 0 0 0 1 0 -4072 Operation canceled file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 165 2391 1 42 2026-10-09 13:26:35.352413 2026-10-09 13:28:53.206788 CANCELED 1791523605058710000 0 1791523613118947000 NONE 2100339 90574 657.0267 0 3 3 1 1 0 1 0 -4072 Operation canceled file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 164 2390 1 41 2026-10-09 13:11:01.141888 2026-10-09 13:25:14.950160 CANCELED 1791522673792641000 0 1791522687888263000 NONE 65028667 60815334 71228.3261 0 3 3 31 31 0 1 0 -4072 Operation canceled file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 162 2386 1 40 2026-10-09 10:39:15.531813 2026-10-09 12:41:01.864255 FAILED 1791513556004824000 0 1791513565756670000 NONE 8393139 6595183 902.6667 0 3 3 4 4 0 1 0 -9086 backup advance checkpoint by flush timeout file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 160 2362 1 39 2026-10-08 10:30:06.229131 2026-10-08 12:31:56.833682 FAILED 1791426609533009001 0 1791426623109254000 NONE 6295895 2458165 336.2465 0 3 3 3 3 0 1 0 -9086 backup advance checkpoint by flush timeout file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 158 2313 1 38 2026-10-06 10:30:06.264364 2026-10-06 12:32:05.013884 FAILED 1791253817210673000 0 1791253830316415000 NONE 146835003 129187794 17651.6212 0 3 3 70 70 0 1 0 -9086 backup advance checkpoint by flush timeout file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 154 2264 1 37 2026-10-04 10:30:06.199173 2026-10-04 12:32:25.642826 FAILED 1791081035970052000 0 1791081050120710000 NONE 73418702 66771132 9097.5740 0 3 3 35 35 0 1 0 -9086 backup advance checkpoint by flush timeout file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 150 2191 1 36 2026-10-01 10:30:06.231642 2026-10-01 10:49:12.659450 COMPLETED 1790821815219416000 1790822947015015000 1790821830540573000 NONE 130854618847 100243983129 87440292.7332 0 6396 6396 62388 62388 0 1 0 0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1
1002 145 2142 1 35 2026-09-29 10:30:13.353139 2026-09-29 11:18:48.255893 COMPLETED 1790649024291367000 1790651920905357000 1790649039739123000 NONE 132882619922 102015506085 34997910.6318 0 6396 6396 63355 63355 0 1 0 0 file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data 1 1

SELECT TENANT_NAME, LOG_MODE FROM oceanbase.DBA_OB_TENANTS\G;

sys NOARCHIVELOG
META$1002 NOARCHIVELOG
my_tenant ARCHIVELOG
META$1004 NOARCHIVELOG
ocp_meta ARCHIVELOG
META$1006 NOARCHIVELOG
ocp_monitor ARCHIVELOG

SELECT * FROM oceanbase.CDB_OB_ARCHIVELOG where TENANT_ID=1002;

1002 1004 5 1 0 DOING 1783390013644389000 2026-07-07 10:06:53.644389 1791532677889649000 2026-10-09 15:57:57.889649 1 18 112 86400000000 1 none 786139659408 732.15GB 786139659408 732.15GB 1.00 0 0.00MB 0 0.00MB file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/clog

#5

在192.168.8.241节点

grep Y8072C0A808F1-000659AA595B684F-0-0 observer.log*
grep Y8072C0A808F1-000659AA595B684F-0-0 rootservice.log*
#6

(base) [root@192-168-8-241 log]#grep Y8072C0A808F1-000659AA595B684F-0-0 observer.log*

684F-0-0] [lt=22][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth]
observer.log.20261009152926993:[2026-10-09 15:28:29.128045] WDIAG [STORAGE] advance_checkpoint_by_flush (ob_backup_task.cpp:142) [3388490][T1002_BACKUP_ME][T1002][Y8072C0A808F1-000659AA595B684F-0-0] [lt=22][errcode=0] clog checkpoint scn has not passed start scn, retry again(i=17, tenant_id=1002, ls_id={id:1001}, clog_checkpoint_scn={val:1790911160321499000, v:0}, start_scn={val:1791524518347112000, v:0}, need_advance_checkpoint=false, advance_checkpoint_interval=60000000, cur_ts=1791530909128032, last_advance_checkpoint_ts=1791530894126411)
observer.log.20261009152926993:[2026-10-09 15:28:30.128150] WDIAG [STORAGE] advance_checkpoint_by_flush (ob_backup_task.cpp:142) [3388490][T1002_BACKUP_ME][T1002][Y8072C0A808F1-000659AA595B684F-0-0] [lt=22][errcode=0] clog checkpoint scn has not passed start scn, retry again(i=17, tenant_id=1002, ls_id={id:1001}, clog_checkpoint_scn={val:1790911160321499000, v:0}, start_scn={val:1791524518347112000, v:0}, need_advance_checkpoint=false, advance_checkpoint_interval=60000000, cur_ts=1791530910128142, last_advance_checkpoint_ts=1791530894126411)
observer.log.20261009152926993:[2026-10-09 15:28:31.128269] WDIAG [STORAGE] advance_checkpoint_by_flush (ob_backup_task.cpp:142) [3388490][T1002_BACKUP_ME][T1002][Y8072C0A808F1-000659AA595B684F-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth]
observer.log.20261009152926993:[2026-10-09 15:28:32.128401] WDIAG [STORAGE] advance_checkpoint_by_flush (ob_backup_task.cpp:142) [3388490][T1002_BACKUP_ME][T1002][Y8072C0A808F1-000659AA595B684F-0-0] [lt=31][errcode=0] clog checkpoint scn has not passed start scn, retry again(i=17, tenant_id=1002, ls_id={id:1001}, clog_checkpoint_scn={val:1790911160321499000, v:0}, start_scn={val:1791524518347112000, v:0}, need_advance_checkpoint=false, advance_checkpoint_interval=60000000, cur_ts=1791530912128394, last_advance_checkpoint_ts=1791530894126411)
observer.log.20261009152926993:[2026-10-09 15:28:33.128519] WDIAG [STORAGE] advance_checkpoint_by_flush (ob_backup_task.cpp:142) [3388490][T1002_BACKUP_ME][T1002][Y8072C0A808F1-000659AA595B684F-0-0] [lt=28][errcode=0] clog checkpoint scn has not passed start scn, retry again(i=17, tenant_id=1002, ls_id={id:1001}, clog_checkpoint_scn={val:1790911160321499000, v:0}, start_scn={val:1791524518347112000, v:0}, need_advance_checkpoint=false, advance_checkpoint_interval=60000000, cur_ts=1791530913128508, last_advance_checkpoint_ts=1791530894126411)
observer.log.20261009152926993:[2026-10-09 15:28:34.128639] WDIAG [STORAGE] advance_checkpoint_by_flush (ob_backup_task.cpp:142)

(base) [root@192-168-8-241 log]# grep ‘Y8072C0A808F1-000659AA595B684F-0-0’ rootservice.log*

rootservice.log.20261009161502281:[2026-10-09 15:43:34.450649] INFO [RS] process (ob_backup_data_ls_task_mgr.cpp:83) [9661][T1001_BackupDat][T1001][Y8072C0A808F1-000659A13EAD0747-0-0] [lt=28] [DATA_BACKUP]schedule backup ls task(ls_attr_={task_id:168, tenant_id:1002, ls_id:{id:1001}, job_id:2394, backup_set_id:45, backup_type:{type:1}, task_type:{type:0}, status:{status:3}, start_ts:1791524518360309, end_ts:1791531814427016, backup_date:20261009, black_servers:[“192.168.8.241:32882”], dst:“192.168.8.241:32882”, task_trace_id:Y8072C0A808F1-000659A13E8AB75C-0-0, stats:{input_bytes:0, output_bytes:0, tablet_count:0, finish_tablet_count:0, macro_block_count:0, finish_macro_block_count:0, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:3, result:-9086, comment:"(SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595B684F-0-0", max_tablet_checkpoint_scn:{val:0, v:0}})

rootservice.log.20261009161502281:[2026-10-09 15:43:34.450669] WDIAG [RS] finish_ (ob_backup_data_ls_task_mgr.cpp:487) [9661][T1001_BackupDat][T1001][Y8072C0A808F1-000659A13EAD0747-0-0] [lt=17][errcode=-9086] ls task failed, backup can’t not continue(ret=-9086, ls_attr_={task_id:168, tenant_id:1002, ls_id:{id:1001}, job_id:2394, backup_set_id:45, backup_type:{type:1}, task_type:{type:0}, status:{status:3}, start_ts:1791524518360309, end_ts:1791531814427016, backup_date:20261009, black_servers:[“192.168.8.241:32882”], dst:“192.168.8.241:32882”, task_trace_id:Y8072C0A808F1-000659A13E8AB75C-0-0, stats:{input_bytes:0, output_bytes:0, tablet_count:0, finish_tablet_count:0, macro_block_count:0, finish_macro_block_count:0, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:3, result:-9086, comment:"(SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595B684F-0-0", max_tablet_checkpoint_scn:{val:0, v:0}})

rootservice.log.20261009161502281:[2026-10-09 15:43:34.450697] WDIAG [RS] process (ob_backup_data_ls_task_mgr.cpp:96) [9661][T1001_BackupDat][T1001][Y8072C0A808F1-000659A13EAD0747-0-0] [lt=26][errcode=-9086] [DATA_BACKUP]failed to finish task(ret=-9086, ls_attr_={task_id:168, tenant_id:1002, ls_id:{id:1001}, job_id:2394, backup_set_id:45, backup_type:{type:1}, task_type:{type:0}, status:{status:3}, start_ts:1791524518360309, end_ts:1791531814427016, backup_date:20261009, black_servers:[“192.168.8.241:32882”], dst:“192.168.8.241:32882”, task_trace_id:Y8072C0A808F1-000659A13E8AB75C-0-0, stats:{input_bytes:0, output_bytes:0, tablet_count:0, finish_tablet_count:0, macro_block_count:0, finish_macro_block_count:0, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:3, result:-9086, comment:"(SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595B684F-0-0", max_tablet_checkpoint_scn:{val:0, v:0}})

rootservice.log.20261009161502281:[2026-10-09 15:43:34.457859] WDIAG [RS] do_backup_meta_ (ob_backup_data_set_task_mgr.cpp:1046) [9661][T1001_BackupDat][T1001][Y8072C0A808F1-000659A13EAD0747-0-0] [lt=15][errcode=-9086] [DATA_BACKUP]failed to process ls backup meta task(ret=-9086, ls_attr={task_id:168, tenant_id:1002, ls_id:{id:1001}, job_id:2394, backup_set_id:45, backup_type:{type:1}, task_type:{type:0}, status:{status:3}, start_ts:1791524518360309, end_ts:1791531814427016, backup_date:20261009, black_servers:[“192.168.8.241:32882”], dst:“192.168.8.241:32882”, task_trace_id:Y8072C0A808F1-000659A13E8AB75C-0-0, stats:{input_bytes:0, output_bytes:0, tablet_count:0, finish_tablet_count:0, macro_block_count:0, finish_macro_block_count:0, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:3, result:-9086, comment:"(SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595B684F-0-0", max_tablet_checkpoint_scn:{val:0, v:0}}, set_task_attr_={task_id:168, tenant_id:1002, incarnation_id:1, job_id:2394, backup_set_id:45, start_ts:1791524476089519, end_ts:0, start_scn:{val:1791524505356343000, v:0}, end_scn:{val:0, v:0}, user_ls_start_scn:{val:1791524518347112000, v:0}, data_turn_id:0, meta_turn_id:1, minor_turn_id:1, major_turn_id:1, status:{status:7}, encryption_mode:0, passwd:"", stats:{input_bytes:2100338, output_bytes:78286, tablet_count:776, finish_tablet_count:3, macro_block_count:1015, finish_macro_block_count:1, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, backup_path:“file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data”, retry_cnt:0, result:0, comment:""})

rootservice.log.20261009161502281:[2026-10-09 15:43:34.457920] WDIAG [RS] backup_user_meta_ (ob_backup_data_set_task_mgr.cpp:520) [9661][T1001_BackupDat][T1001][Y8072C0A808F1-000659A13EAD0747-0-0] [lt=58][errcode=-9086] [DATA_BACKUP]failed to do backup meta(ret=-9086, ls_task=[{task_id:168, tenant_id:1002, ls_id:{id:1}, job_id:2394, backup_set_id:45, backup_type:{type:1}, task_type:{type:0}, status:{status:3}, start_ts:1791524505386513, end_ts:1791524518326056, backup_date:20261009, black_servers:[], dst:“192.168.8.241:32882”, task_trace_id:Y8072C0A808F1-000659A13E8AB758-0-0, stats:{input_bytes:2100338, output_bytes:78286, tablet_count:776, finish_tablet_count:3, macro_block_count:1015, finish_macro_block_count:1, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:0, result:0, comment:"", max_tablet_checkpoint_scn:{val:1791524507003780000, v:0}}, {task_id:168, tenant_id:1002, ls_id:{id:1001}, job_id:2394, backup_set_id:45, backup_type:{type:1}, task_type:{type:0}, status:{status:3}, start_ts:1791524518360309, end_ts:1791531814427016, backup_date:20261009, black_servers:[“192.168.8.241:32882”], dst:“192.168.8.241:32882”, task_trace_id:Y8072C0A808F1-000659A13E8AB75C-0-0, stats:{input_bytes:0, output_bytes:0, tablet_count:0, finish_tablet_count:0, macro_block_count:0, finish_macro_block_count:0, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:3, result:-9086, comment:"(SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595B684F-0-0", max_tablet_checkpoint_scn:{val:0, v:0}}])

rootservice.log.20261009161502281:[2026-10-09 15:43:34.458007] WDIAG [RS] process (ob_backup_data_scheduler.cpp:1161) [9661][T1001_BackupDat][T1001][Y8072C0A808F1-000659A13EAD0747-0-0] [lt=18][errcode=-9086] [DATA_BACKUP]failed to backup data(ret=-9086, job_attr_={job_id:2394, tenant_id:1002, incarnation_id:1, backup_set_id:45, initiator_tenant_id:1, initiator_job_id:126, executor_tenant_id:[1002], plus_archivelog:false, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:“file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data”, description:“ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2026-10-09 13:41:15.0”, start_ts:1791524476089519, end_ts:0, status:{status:1}, result:0, can_retry:false, retry_count:0, comment:"(SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595B684F-0-0"})

rootservice.log.20261009161502281:[2026-10-09 15:43:34.458021] WDIAG [RS] process (ob_backup_data_scheduler.cpp:986) [9661][T1001_BackupDat][T1001][Y8072C0A808F1-000659A13EAD0747-0-0] [lt=13][errcode=0] [DATA_BACKUP]failed to schedule tenant backup job(tmp_ret=-9086, tenant_id=1001, job_attr={job_id:2394, tenant_id:1002, incarnation_id:1, backup_set_id:45, initiator_tenant_id:1, initiator_job_id:126, executor_tenant_id:[1002], plus_archivelog:false, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:“file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data”, description:“ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2026-10-09 13:41:15.0”, start_ts:1791524476089519, end_ts:0, status:{status:1}, result:0, can_retry:false, retry_count:0, comment:"(SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595B684F-0-0"})

rootservice.log.20261009161502281:[2026-10-09 15:43:34.488263] INFO [SHARE] advance_job_status (ob_backup_data_table_operator.cpp:1003) [9661][T1001_BackupDat][T1001][Y8072C0A808F1-000659A13EAD0747-0-0] [lt=1] [DATA_BACKUP]advance status(job_attr={job_id:2394, tenant_id:1002, incarnation_id:1, backup_set_id:45, initiator_tenant_id:1, initiator_job_id:126, executor_tenant_id:[1002], plus_archivelog:false, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:“file:///root/std-dig/OceanBase/data/backup/data/ob_standby/1781772829/tenant_incarnation_1/1002/data”, description:“ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2026-10-09 13:41:15.0”, start_ts:1791524476089519, end_ts:1791531814458035, status:{status:1}, result:-9086, can_retry:false, retry_count:0, comment:"(SERVER)ls_id: 1001, addr: 192.168.8.241:32882, module: BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout), trace_id: Y8072C0A808F1-000659AA595B684F-0-0"}, next_status={status:3})

#7

clog_checkpoint_scn=1790911160321499000 → 2026-10-02 13:38:40

不是备份的问题，是 LS 1001 的 clog checkpoint 从 10-02 起就冻住了 且 checkpoint 不推进，clog 就无法回收，10-02 以来的 clog 全堆在盘上 导致clog空间持续增长

这个节点继续


# 冻结起点时间窗：
grep -E 'freeze|flush|checkpoint|write stop|-4184|io hang|disk hang' observer.log.20261002* | grep -i 1001 | head -50

grep "handler flush failed" observer.log* | tail -20

-- sys租户执行
SELECT gmt_create, type, ret, retry_cnt, LEFT(warning_info, 200)
FROM oceanbase.__all_virtual_dag_warning_history
WHERE tenant_id=1002 AND svr_ip='192.168.8.241'
ORDER BY gmt_create DESC LIMIT 30;

SELECT tablet_id, start_time, end_time, ret_code, state, LEFT(diagnose_info,200), LEFT(memtables_info,200)
FROM oceanbase.__all_virtual_minor_freeze_info
WHERE tenant_id=1002 AND ls_id=1001 AND svr_ip='192.168.8.241'
ORDER BY start_time DESC LIMIT 20;

SELECT * FROM oceanbase.GV$OB_MEMSTORE
WHERE TENANT_ID=1002 AND SVR_IP='192.168.8.241';
#8

grep -E ‘freeze|flush|checkpoint|write stop|-4184|io hang|disk hang’ observer.log.20261002* | grep -i 1001 | head -50
这个文件已经被自动清理了

grep “handler flush failed” observer.log* | tail -20
没有结果

image
image1332×132 68.1 KB

SELECT gmt_create, type, ret, retry_cnt, LEFT(warning_info, 200)
FROM oceanbase.__all_virtual_dag_warning_history
WHERE tenant_id=1002 AND svr_ip=‘192.168.8.241’
ORDER BY gmt_create DESC LIMIT 30;

gmt_create type ret retry_cnt LEFT(warning_info, 200)
2026-10-02 18:53:23.545362 MINI_MERGE OB_SIZE_OVERFLOW 119352 ob_basic_tablet_merge_ctx.cpp:1088(update_tablet),MINI_MERGE;ls_id=1001;tablet_id=200229;compaction_scn=0;exec_mode=EXEC_MODE_LOCAL,concurrent_cnt=1

SELECT tablet_id, start_time, end_time, ret_code, state, LEFT(diagnose_info,200), LEFT(memtables_info,200)
FROM oceanbase.__all_virtual_minor_freeze_info
WHERE tenant_id=1002 AND ls_id=1001 AND svr_ip=‘192.168.8.241’
ORDER BY start_time DESC LIMIT 20;

tablet_id start_time end_time ret_code state LEFT(diagnose_info,200) LEFT(memtables_info,200)
200229 2026-10-09 16:42:14.726351 2026-10-09 16:42:14.726351 -4263 FINISH no need to freeze tablet

SELECT * FROM oceanbase.GV$OB_MEMSTORE
WHERE TENANT_ID=1002 AND SVR_IP=‘192.168.8.241’;

SVR_IP SVR_PORT TENANT_ID ACTIVE_SPAN FREEZE_TRIGGER FREEZE_CNT MEMSTORE_USED MEMSTORE_LIMIT
192.168.8.241 32882 1002 49899634688 9502642776 3998 2141192192 43486543850
#9 
1.定位 tablet 200229 属于哪张表（sys 租户执行）
SELECT TENANT_ID, DATABASE_NAME, TABLE_NAME, TABLE_ID, TABLET_ID, LS_ID
FROM oceanbase.CDB_OB_TABLE_LOCATIONS
WHERE TENANT_ID=1002 AND TABLET_ID=200229;

2.拿完整告警内容
SELECT gmt_create, module, type, status, retry_cnt, warning_info
FROM oceanbase.__all_virtual_dag_warning_history
WHERE tenant_id=1002 AND svr_ip='192.168.8.241' AND type='MINI_MERGE'
ORDER BY gmt_create DESC LIMIT 3;

3.在 241 当前 observer.log 
grep "200229" observer.log | grep -- "-4019\|SIZE_OVERFLOW" | tail -30

4.10-02 13:38 前后对 1002 租户（尤其 LS 1001 上的表）做过什么操作——ALTER TABLE 改列、导入、批量写入？当时的第一现场日志已被清理
#10

1.定位 tablet 200229 属于哪张表（sys 租户执行）
SELECT TENANT_ID, DATABASE_NAME, TABLE_NAME, TABLE_ID, TABLET_ID, LS_ID
FROM oceanbase.CDB_OB_TABLE_LOCATIONS
WHERE TENANT_ID=1002 AND TABLET_ID=200229;

TENANT_ID DATABASE_NAME TABLE_NAME TABLE_ID TABLET_ID LS_ID
1002 std_main jgh_pdf_term 500838 200229 1001

2.拿完整告警内容
SELECT gmt_create, module, type, status, retry_cnt, warning_info
FROM oceanbase.__all_virtual_dag_warning_history
WHERE tenant_id=1002 AND svr_ip=‘192.168.8.241’ AND type=‘MINI_MERGE’
ORDER BY gmt_create DESC LIMIT 3;

gmt_create module type status retry_cnt warning_info
2026-10-02 18:53:23.545362 COMPACTION MINI_MERGE RETRYED 119738 ob_basic_tablet_merge_ctx.cpp:1088(update_tablet),MINI_MERGE;ls_id=1001;tablet_id=200229;compaction_scn=0;exec_mode=EXEC_MODE_LOCAL,concurrent_cnt=1

(base) [root@192-168-8-241 log]# grep “200229” observer.log | grep – “-4019|SIZE_OVERFLOW” | tail -30
[2026-10-09 17:16:36.733034] WDIAG [STORAGE] update_tablet_table_store_without_lock_ (ob_ls.cpp:1843) [4188930][T1002_MINI_MERG][T1002][Y8072C0A808F1-000659AA595BBFF7-0-0] [lt=15][errcode=-4019] failed to update tablet table store(ret=-4019, tablet_id={id:200229}, param={sstable:0x7f206c93e710, snapshot_version:1790964003447317001, multi_version_start:1781799424465202000, storage_schema:{this:0x7f206c93e060, storage_schema_version:5, version:0, is_use_bloomfilter:0, column_info_simplified:0, compat_mode:0, table_type:3, index_type:0, row_store_type:1, schema_version:1781799522110216, is_cs_replica_compat:0, is_column_table_schema:0, enable_macro_block_bloom_filter:0, column_cnt:22, store_column_cnt:15, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:“BIGINT”, collation:“binary”, coercibility:“NUMERIC”}, order:0}], column_cnt:22, column_array:[{meta_type:{type:“BIGINT”, collation:“binary”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:1, is_rowkey_column:1, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“CHAR”, collation:“utf8mb4_general_ci”, coercibility:“INVALID”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“BIGINT”, collation:“binary”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“VARCHAR”, collation:“utf8mb4_unicode_ci”, coercibility:“INVALID”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“VARCHAR”, collation:“utf8mb4_unicode_ci”, coercibility:“INVALID”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“VARCHAR”, collation:“utf8mb4_unicode_ci”, coercibility:“INVALID”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“MEDIUMTEXT”, collation:“utf8mb4_unicode_ci”, coercibility:“INVALID”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“INT”, collation:“binary”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“INT”, collation:“binary”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“INT”, collation:“binary”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“VARCHAR”, collation:“utf8mb4_unicode_ci”, coercibility:“INVALID”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“MYSQL_DATETIME”, collation:“binary”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“VARCHAR”, collation:“utf8mb4_unicode_ci”, coercibility:“INVALID”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“MYSQL_DATETIME”, collation:“binary”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“BIT”, collation:“binary”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:1, is_rowkey_column:0, is_generated_column:0, orig_default_value:{“NULL”:“NULL”}}, {meta_type:{type:“VARCHAR”, collation:“binary”, coercibility:“INVALID”}, is_column_stored_in_sstable:0, is_rowkey_column:0, is_generated_column:1, orig_default_value:{“VARCHAR”:“DOC_ID()”, collation:“utf8mb4_bin”, coercibility:“IMPLICIT”}}, {meta_type:{type:“VARCHAR”, collation:“utf8mb4_unicode_ci”, coercibility:“INVALID”}, is_column_stored_in_sstable:0, is_rowkey_column:0, is_generated_column:1, orig_default_value:{“VARCHAR”:“WORD_SEGMENT(title)”, collation:“utf8mb4_bin”, coercibility:“IMPLICIT”}}, {meta_type:{type:“BIGINT UNSIGNED”, collation:“utf8mb4_general_ci”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:0, is_rowkey_column:0, is_generated_column:1, orig_default_value:{“VARCHAR”:“WORD_COUNT(title)”, collation:“utf8mb4_bin”, coercibility:“IMPLICIT”}}, {meta_type:{type:“BIGINT UNSIGNED”, collation:“utf8mb4_general_ci”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:0, is_rowkey_column:0, is_generated_column:1, orig_default_value:{“VARCHAR”:“DOC_LENGTH(title)”, collation:“utf8mb4_bin”, coercibility:“IMPLICIT”}}, {meta_type:{type:“VARCHAR”, collation:“utf8mb4_unicode_ci”, coercibility:“INVALID”}, is_column_stored_in_sstable:0, is_rowkey_column:0, is_generated_column:1, orig_default_value:{“VARCHAR”:“WORD_SEGMENT(word)”, collation:“utf8mb4_bin”, coercibility:“IMPLICIT”}}, {meta_type:{type:“BIGINT UNSIGNED”, collation:“utf8mb4_general_ci”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:0, is_rowkey_column:0, is_generated_column:1, orig_default_value:{“VARCHAR”:“WORD_COUNT(word)”, collation:“utf8mb4_bin”, coercibility:“IMPLICIT”}}, {meta_type:{type:“BIGINT UNSIGNED”, collation:“utf8mb4_general_ci”, coercibility:“NUMERIC”}, is_column_stored_in_sstable:0, is_rowkey_column:0, is_generated_column:1, orig_default_value:{“VARCHAR”:“DOC_LENGTH(word)”, collation:“utf8mb4_bin”, coercibility:“IMPLICIT”}}], skip_index_cnt:0, skip_idx_attr_array:[], column_group_cnt:1, column_group_array:[{version:1, type:1, compressor_type:6, row_store_type:1, block_size:16384, schema_column_cnt:15, rowkey_column_cnt:3, schema_rowkey_column_cnt:1, column_cnt:17, column_idxs:null}], has_all_column_group:true, merge_engine_type:0}, rebuild_seq:0, compaction_info:{clog_checkpoint_scn:{val:1790964003919505000, v:0}, need_report:false, merge_type:“MINI_MERGE”, major_ckm_info:{is_empty:true}, has_truncate_info:false}, ha_info:{transfer_seq:0, need_check_sstable:false, need_check_transfer_seq:true, need_replace_remote_sstable:false, is_only_replace_major:false}, ddl_info:{keep_old_ddl_sstable:true, update_with_major_flag:false, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0, ddl_redo_callback:null, ddl_finish_callback:null, ddl_replay_status:16, slice_sstables:[]}, allow_duplicate_sstable:false, upper_trans_param:{new_upper_trans:null, last_minor_end_scn:{val:0, v:0}}})
[2026-10-09 17:16:36.733386] WDIAG [STORAGE.COMPACTION] update_tablet (ob_tablet_merge_ctx.cpp:157) [4188930][T1002_MINI_MERG][T1002][Y8072C0A808F1-000659AA595BBFF7-0-0] [lt=169][errcode=-4019] failed to update tablet(ret=-4019, ret=“OB_SIZE_OVERFLOW”, param={skip_get_tablet:false, merge_type:“MINI_MERGE”, merge_version:0, ls_id:{id:1001}, tablet_id:{id:200229}, exec_mode:“EXEC_MODE_LOCAL”, need_swap_tablet_flag:false, is_reserve_mode:true, schedule_transfer_seq:-1})
[2026-10-09 17:16:36.733406] WDIAG [STORAGE.COMPACTION] update_tablet_after_merge (ob_basic_tablet_merge_ctx.cpp:1136) [4188930][T1002_MINI_MERG][T1002][Y8072C0A808F1-000659AA595BBFF7-0-0] [lt=19][errcode=-4019] failed to update tablet(ret=-4019, ret=“OB_SIZE_OVERFLOW”, dag_param={skip_get_tablet:false, merge_type:“MINI_MERGE”, merge_version:0, ls_id:{id:1001}, tablet_id:{id:200229}, exec_mode:“EXEC_MODE_LOCAL”, need_swap_tablet_flag:false, is_reserve_mode:true, schedule_transfer_seq:-1}, new_tablet_handle={obj:null, obj_pool:null, allocator:null, wash_priority:2, allow_copy_and_assign:true})

  1. 2号当天2026-10-02 13:38:54写入一条数据，当天一共对这张表写入过12条数据，请问现在是需要处理掉异常数据才能正常备份吗
#11

原因还在定位，处理方案还不确定

#12

老师，兜底方案的话如果停止重新开启日志归档并修改路径和备份路径，能否解决该问题呢

#13

这个方案走不通， 这不是备份的问题，是 LS 1001 的 clog checkpoint 从 10-02 起就冻住了
转储侧卡死 ：LS 1001 下 tablet 200229 的 memtable freeze/dump 自 10-02 起没完成（IO hang、数据盘满/write stop、转储 DAG 反复失败），callback 不上来，checkpoint 永远推不动。所以备份时报了：BACKUP_DATA, result: -9086(backup advance checkpoint by flush timeout)

#14

好的老师，请问什么时候能给出处理方案呢

#15 
1.grep "Too Many sstables" observer.log* | tail -20

-- 以下在1002 租户内执行

2.谁卡住了 clog checkpoint（tenant 1002 的 MySQL 会话执行）——按最小 rec_log_scn 排序，最老的行就是阻塞者：

SELECT * FROM oceanbase.__all_virtual_transaction_freeze_checkpoint
WHERE ls_id = 1001 ORDER BY rec_log_scn ASC LIMIT 5;

SELECT * FROM oceanbase.__all_virtual_transaction_checkpoint
WHERE ls_id = 1001 ORDER BY rec_log_scn ASC LIMIT 5;

3.找挂住的事务：
SELECT trans_id, session_id, ctx_create_time, state, start_scn, is_decided
FROM oceanbase.__all_virtual_trans_stat
WHERE ls_id = 1001 ORDER BY ctx_create_time ASC;

4.compaction 自我诊断
SELECT * FROM oceanbase.GV$OB_COMPACTION_DIAGNOSE_INFO
WHERE svr_ip = '192.168.8.241' AND tablet_id = 200229;
#16

grep “Too Many sstables” observer.log* | tail -20
以下是片段部分
(base) [root@192-168-8-241 log]# grep “Too Many sstables” observer.log* | tail -20
grep: observer.log.wf.20261009131225126: No such file or directory
observer.log.20261009175558196:[2026-10-09 17:55:22.325697] WDIAG [STORAGE] check_ready_for_read (ob_tablet_table_store.cpp:2175) [429542][T1002_MINI_MERG][T1002][Y8072C0A808F1-000659AA595BD7B6-0-0] [lt=129][errcode=-4019] Too Many sstables, cannot add another sstable any more(ret=-4019, this={this:0x7f206c980af0, version:259, major_tables:{cnt:126, sstable_array:0x7f206c980c90, serialize_table_type:false, is_inited:true}, minor_tables:{cnt:3, sstable_array:0x7f206c9464b0, serialize_table_type:false, is_inited:true}, ddl_sstables:{cnt:0, sstable_array:null, serialize_table_type:false, is_inited:false}, meta_major_tables:{cnt:0, sstable_array:null, serialize_table_type:false, is_inited:false}, mds_sstables:{cnt:1, sstable_array:0x7f206c947010, serialize_table_type:false, is_inited:true}, memtables:{count:14}, major_ckm_info:{is_empty:true}, is_ready_for_read:false, [{i:0, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1564672,size=6800,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1781799275472897000, v:0}}, snapshot_version:1781799275472897000, max_merge_version:1781799275472897000}, {i:1, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1571472,size=6789,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1781805604252392000, v:0}}, snapshot_version:1781805604252392000, max_merge_version:1781799275472897000}, {i:2, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1578261,size=6789,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1781892000957173000, v:0}}, snapshot_version:1781892000957173000, max_merge_version:1781799275472897000}, {i:3, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1585050,size=6789,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1781978403545356000, v:0}}, snapshot_version:1781978403545356000, max_merge_version:1781799275472897000}, {i:4, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1591839,size=6789,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1782064800429101000, v:0}}, snapshot_version:1782064800429101000, max_merge_version:1781799275472897000}, {i:5, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1598628,size=6827,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1782151202355621000, v:0}}, snapshot_version:1782151202355621000, max_merge_version:1782117406543548000}, {i:6, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1605455,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1782237604372336000, v:0}}, snapshot_version:1782237604372336000, max_merge_version:1782176456721136000}, {i:7, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1612769,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1782324001864237000, v:0}}, snapshot_version:1782324001864237000, max_merge_version:1782176456721136000}, {i:8, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1620083,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1782410402281757000, v:0}}, snapshot_version:1782410402281757000, max_merge_version:1782176456721136000}, {i:9, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1627397,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1782496803180472000, v:0}}, snapshot_version:1782496803180472000, max_merge_version:1782176456721136000}, {i:10, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1634711,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1782756002339338000, v:0}}, snapshot_version:1782756002339338000, max_merge_version:1782176456721136000}, {i:11, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1642025,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1782842400037620000, v:0}}, snapshot_version:1782842400037620000, max_merge_version:1782176456721136000}, {i:12, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1649339,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1782928802678365000, v:0}}, snapshot_version:1782928802678365000, max_merge_version:1782176456721136000}, {i:13, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1656653,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783015200418136000, v:0}}, snapshot_version:1783015200418136000, max_merge_version:1782176456721136000}, {i:14, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1663967,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783101603156412000, v:0}}, snapshot_version:1783101603156412000, max_merge_version:1782176456721136000}, {i:15, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1671281,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783188001214138000, v:0}}, snapshot_version:1783188001214138000, max_merge_version:1782176456721136000}, {i:16, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1678595,size=7314,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783274404412842000, v:0}}, snapshot_version:1783274404412842000, max_merge_version:1782176456721136000}, {i:17, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1685909,size=7784,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783360802653425000, v:0}}, snapshot_version:1783360802653425000, max_merge_version:1783298827942844000}, {i:18, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1693693,size=7345,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783447200663831000, v:0}}, snapshot_version:1783447200663831000, max_merge_version:1783412184670390000}, {i:19, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1701038,size=7345,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783533603847730000, v:0}}, snapshot_version:1783533603847730000, max_merge_version:1783412184670390000}, {i:20, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1708383,size=7345,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783620002125982000, v:0}}, snapshot_version:1783620002125982000, max_merge_version:1783412184670390000}, {i:21, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1715728,size=7345,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783706400403608000, v:0}}, snapshot_version:1783706400403608000, max_merge_version:1783412184670390000}, {i:22, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1723073,size=7345,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783792803582303000, v:0}}, snapshot_version:1783792803582303000, max_merge_version:1783412184670390000}, {i:23, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1730418,size=7345,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783879202003186000, v:0}}, snapshot_version:1783879202003186000, max_merge_version:1783412184670390000}, {i:24, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1737763,size=7345,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1783965600371013000, v:0}}, snapshot_version:1783965600371013000, max_merge_version:1783412184670390000}, {i:25, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1745108,size=7345,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784052003785364000, v:0}}, snapshot_version:1784052003785364000, max_merge_version:1783412184670390000}, {i:26, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1752453,size=7345,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784138402386592000, v:0}}, snapshot_version:1784138402386592000, max_merge_version:1783412184670390000}, {i:27, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1759798,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784224801006477000, v:0}}, snapshot_version:1784224801006477000, max_merge_version:1784166362312226000}, {i:28, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1767629,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784311204896389000, v:0}}, snapshot_version:1784311204896389000, max_merge_version:1784166362312226000}, {i:29, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1775460,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784397603821659000, v:0}}, snapshot_version:1784397603821659000, max_merge_version:1784166362312226000}, {i:30, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1783291,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784484002629616000, v:0}}, snapshot_version:1784484002629616000, max_merge_version:1784166362312226000}, {i:31, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1791122,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784570401506605000, v:0}}, snapshot_version:1784570401506605000, max_merge_version:1784166362312226000}, {i:32, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1798953,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784656800626176000, v:0}}, snapshot_version:1784656800626176000, max_merge_version:1784166362312226000}, {i:33, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1806784,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784743204830599000, v:0}}, snapshot_version:1784743204830599000, max_merge_version:1784166362312226000}, {i:34, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1814615,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784829603791417000, v:0}}, snapshot_version:1784829603791417000, max_merge_version:1784166362312226000}, {i:35, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1822446,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1784916003409695000, v:0}}, snapshot_version:1784916003409695000, max_merge_version:1784166362312226000}, {i:36, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1830277,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785002402974740000, v:0}}, snapshot_version:1785002402974740000, max_merge_version:1784166362312226000}, {i:37, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1838108,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785088802180188000, v:0}}, snapshot_version:1785088802180188000, max_merge_version:1784166362312226000}, {i:38, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1845939,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785175201797805000, v:0}}, snapshot_version:1785175201797805000, max_merge_version:1784166362312226000}, {i:39, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1853770,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785261601394342000, v:0}}, snapshot_version:1785261601394342000, max_merge_version:1784166362312226000}, {i:40, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1861601,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785348001615171000, v:0}}, snapshot_version:1785348001615171000, max_merge_version:1784166362312226000}, {i:41, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1869432,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785434401610744000, v:0}}, snapshot_version:1785434401610744000, max_merge_version:1784166362312226000}, {i:42, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1877263,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785520801659949000, v:0}}, snapshot_version:1785520801659949000, max_merge_version:1784166362312226000}, {i:43, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1885094,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785607201528732000, v:0}}, snapshot_version:1785607201528732000, max_merge_version:1784166362312226000}, {i:44, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1892925,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785693601529539000, v:0}}, snapshot_version:1785693601529539000, max_merge_version:1784166362312226000}, {i:45, addr:{[ver=1,mode=0,seq=12261836][2nd=520934][3rd=0][offset=1900756,size=7831,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, is_loaded:false, type:“MAJOR”, tablet_id:{id:200229}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1785780001614658000, v:0}}, snapshot_version:1785780001614658000,

SELECT * FROM oceanbase.__all_virtual_transaction_freeze_checkpoint
WHERE ls_id = 1001 ORDER BY rec_log_scn ASC LIMIT 5;

image
image1380×243 11.6 KB

SELECT * FROM oceanbase.__all_virtual_transaction_checkpoint
WHERE ls_id = 1001 ORDER BY rec_log_scn ASC LIMIT 5;

image
image1395×245 11.8 KB

3.找挂住的事务：
SELECT trans_id, session_id, ctx_create_time, state, start_scn, is_decided
FROM oceanbase.__all_virtual_trans_stat
WHERE ls_id = 1001 ORDER BY ctx_create_time ASC;

image
image903×140 3.18 KB

4.compaction 自我诊断
SELECT * FROM oceanbase.GV$OB_COMPACTION_DIAGNOSE_INFO
WHERE svr_ip = ‘192.168.8.241’ AND tablet_id = 200229;

SVR_IP SVR_PORT TENANT_ID TYPE LS_ID TABLET_ID STATUS CREATE_TIME DIAGNOSE_INFO
192.168.8.241 32882 1002 MAJOR_MERGE 1001 200229 FAILED 2026-10-09 17:56:35.315447 major not schedule for long time=major wait for freeze,max_receive_medium_snapshot=1790964003858672000,compaction_scn=1790964003858672000,tablet_snapshot=1790877603834731000,last_major_scn=1790877603202556000
#17

sys租户下执行

select *
FROM oceanbase.__all_reserved_snapshot
WHERE tenant_id = 1002
ORDER BY snapshot_version;
#18

查询结果为空