2025-01-16 17:01:34.285 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=30278, context=Context{parallelIdx=-1, stringMap={ob_tenant_id=1002, tenant_id=5, checkBeforeDataBackup=true, startClusterCompact=false, task_instance_id=30707, dataBackupTriggerInfo={"async":true,"backupDimension":"TENANT","dataBackupMode":"FULL_BACKUP","triggerTime":"2025-01-16T09:01:33Z","triggerType":"BACKUP_NOW"}, task_operation=execute, triggerTime=1737018093000000, startLogBackup=true, cluster_id=2, storageBaseInfo=******, tenantName=szjg_default, backupMode=physical, sub_task_instance_name=Wait data backup task successful, sub_task_instance_id=30278, dataBackupMode=FULL_BACKUP, tenant_name=szjg_default, ob_version=4.3.4.0, ob_cluster_id=1731558683, backupDim=TENANT, dataSetIncIdMap={"1002":"2"}, triggerType=BACKUP_NOW, latest_execution_start_time=2025-01-16T17:01:34.261+08:00, backupNow=true, obClusterName=szjg_cluster}, listMap={}}, executor=XXX.XXX.X.227 2025-01-16 17:01:34.291 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=1 2025-01-16 17:01:34.329 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-01-16 17:01:34.337 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2025-01-16 17:01:34.345 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:01:34.351 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-01-16 17:01:34.357 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2025-01-16 17:01:34.373 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:01:34.378 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:01:34.380 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:01:34.388 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:01:34.393 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:01:34.399 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-01-16 17:01:34.436 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= '2025-01-16 17:01:33', args: [ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2025-01-16 17:01:33.0] 2025-01-16 17:01:34.467 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=2, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=2, backupDest=file:///data/oceanbase-ce/oceanbase/backup/szjg_cluster/1731558683/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-01-16T09:01:33.419991Z, finishTime=null, status=INITIALIZING, progress=null, errorMsg=}] 2025-01-16 17:01:34.475 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : task status is UNKNOWN 2025-01-16 17:01:34.480 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : count running backup task size: 1 2025-01-16 17:02:04.483 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=2 2025-01-16 17:02:04.511 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=szjg_cluster, tenantName=sys, dbUser=ocp_monitor 2025-01-16 17:02:04.520 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-01-16 17:02:04.525 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2025-01-16 17:02:04.531 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:02:04.537 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-01-16 17:02:04.541 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2025-01-16 17:02:04.560 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:02:04.564 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:02:04.569 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:02:04.572 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:02:04.577 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:02:04.581 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-01-16 17:02:04.619 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= '2025-01-16 17:01:33', args: [ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2025-01-16 17:01:33.0] 2025-01-16 17:02:04.661 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=2, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=2, backupDest=file:///data/oceanbase-ce/oceanbase/backup/szjg_cluster/1731558683/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-01-16T09:01:33.419991Z, finishTime=2025-01-16T09:01:48.189065Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: XXX.XXX.X.31:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0B4071F-00062BA60952B108-0-0}] 2025-01-16 17:02:04.672 WARN 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task failed: [DataBackupHistoryWrapperTask{taskId=2, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=2, backupDest=file:///data/oceanbase-ce/oceanbase/backup/szjg_cluster/1731558683/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-01-16T09:01:33.419991Z, finishTime=2025-01-16T09:01:48.189065Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: XXX.XXX.X.31:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0B4071F-00062BA60952B108-0-0}] 2025-01-16 17:02:04.677 ERROR 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,0c0bc434816d7e6b] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Data backup task failed. com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_DATA_BACKUP_TASK_FAILED, args= at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.checkDataBackupTaskFinished(WaitDataBackupSuccessTask.java:172) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.lambda$run$5(WaitDataBackupSuccessTask.java:218) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.run(WaitDataBackupSuccessTask.java:213) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Set state for subtask: 30278, operation:EXECUTE, state: FAILED 2025-01-16 17:12:00.039 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=30278, context=Context{parallelIdx=-1, stringMap={ob_tenant_id=1002, tenant_id=5, checkBeforeDataBackup=true, startClusterCompact=false, task_instance_id=30707, dataBackupTriggerInfo={"async":true,"backupDimension":"TENANT","dataBackupMode":"FULL_BACKUP","triggerTime":"2025-01-16T09:01:33Z","triggerType":"BACKUP_NOW"}, task_operation=retry, triggerTime=1737018093000000, startLogBackup=true, cluster_id=2, storageBaseInfo=******, tenantName=szjg_default, backupMode=physical, sub_task_instance_name=Wait data backup task successful, sub_task_instance_id=30278, dataBackupMode=FULL_BACKUP, tenant_name=szjg_default, ob_version=4.3.4.0, ob_cluster_id=1731558683, backupDim=TENANT, dataSetIncIdMap={"1002":"2"}, triggerType=BACKUP_NOW, latest_execution_start_time=2025-01-16T17:12:00.005+08:00, backupNow=true, obClusterName=szjg_cluster}, listMap={}}, executor=XXX.XXX.X.227 2025-01-16 17:12:00.045 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : do nothing in rollback! 2025-01-16 17:12:00.052 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=1 2025-01-16 17:12:00.091 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-01-16 17:12:00.099 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2025-01-16 17:12:00.105 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:12:00.132 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-01-16 17:12:00.166 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2025-01-16 17:12:00.190 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:12:00.197 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:12:00.202 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:12:00.207 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:12:00.213 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2025-01-16 17:12:00.224 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-01-16 17:12:00.265 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= '2025-01-16 17:01:33', args: [ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2025-01-16 17:01:33.0] 2025-01-16 17:12:00.317 INFO 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=2, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=2, backupDest=file:///data/oceanbase-ce/oceanbase/backup/szjg_cluster/1731558683/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-01-16T09:01:33.419991Z, finishTime=2025-01-16T09:01:48.189065Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: XXX.XXX.X.31:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0B4071F-00062BA60952B108-0-0}] 2025-01-16 17:12:00.356 WARN 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task failed: [DataBackupHistoryWrapperTask{taskId=2, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=2, backupDest=file:///data/oceanbase-ce/oceanbase/backup/szjg_cluster/1731558683/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-01-16T09:01:33.419991Z, finishTime=2025-01-16T09:01:48.189065Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: XXX.XXX.X.31:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0B4071F-00062BA60952B108-0-0}] 2025-01-16 17:12:00.365 ERROR 4181880 --- [manual-subtask-executor14,dbbf4b6ff68d305a,ca2c225544aabbf8] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Data backup task failed. com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_DATA_BACKUP_TASK_FAILED, args= at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.checkDataBackupTaskFinished(WaitDataBackupSuccessTask.java:172) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.lambda$run$5(WaitDataBackupSuccessTask.java:218) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.run(WaitDataBackupSuccessTask.java:213) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Set state for subtask: 30278, operation:RETRY, state: FAILED