2025-03-19 15:02:10.157 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-03-19 15:02:10.159 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1016589, context=Context{parallelIdx=-1, stringMap={checkBeforeDataBackup=true, ob_version=4.3.5.0, startClusterCompact=false, task_instance_id=1016229, dataBackupTriggerInfo={"async":true,"backupDimension":"CLUSTER","dataBackupMode":"FULL_BACKUP","triggerTime":"2025-03-19T07:02:09Z","triggerType":"BACKUP_NOW"}, task_operation=execute, triggerTime=1742367729000000, ob_cluster_id=1740541070, startLogBackup=true, backupDim=CLUSTER, dataSetIncIdMap={"1004":"51"}, cluster_id=1000001, storageBaseInfo=******, tenantName=*, backupMode=physical, triggerType=BACKUP_NOW, latest_execution_start_time=2025-03-19T15:02:10.147+08:00, sub_task_instance_name=Wait data backup task successful, backupNow=true, sub_task_instance_id=1016589, dataBackupMode=FULL_BACKUP, obClusterName=shanxi_ob_second}, listMap={}}, executor=10.104.106.167 2025-03-19 15:02:10.161 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=1 2025-03-19 15:02:10.177 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-19 15:02:10.178 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2025-03-19 15:02:10.181 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-19 15:02:10.183 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2025-03-19 15:02:10.193 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-19 15:02:10.220 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= '2025-03-19 15:02:09', args: [ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-03-19 15:02:09.0] 2025-03-19 15:02:10.243 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=2, backupType=FULL, clusterId=null, obTenantId=1004, backupSetId=2, backupDest=file:///data/back_up/shanxi_ob_second/1740541070/tenant_incarnation_1/1004/data, snapshotVersionTime=null, startTime=2025-03-19T07:02:09.233949Z, finishTime=null, status=INITIALIZING, progress=null, errorMsg=}] 2025-03-19 15:02:10.245 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : task status is UNKNOWN 2025-03-19 15:02:10.246 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : count running backup task size: 1 2025-03-19 15:02:40.247 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=2 2025-03-19 15:02:40.260 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=shanxi_ob_second, tenantName=sys, dbUser=ocp_monitor 2025-03-19 15:02:40.262 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-19 15:02:40.264 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2025-03-19 15:02:40.267 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-19 15:02:40.269 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2025-03-19 15:02:40.279 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-19 15:02:40.306 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= '2025-03-19 15:02:09', args: [ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-03-19 15:02:09.0] 2025-03-19 15:02:40.330 INFO 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=2, backupType=FULL, clusterId=null, obTenantId=1004, backupSetId=2, backupDest=file:///data/back_up/shanxi_ob_second/1740541070/tenant_incarnation_1/1004/data, snapshotVersionTime=null, startTime=2025-03-19T07:02:09.233949Z, finishTime=2025-03-19T07:02:24.046106Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 10.104.106.173:2882, result: -9120(cannot find object on object storage), trace_id: YB420A686AAD-00062F087091B94B-0-0}] 2025-03-19 15:02:40.333 WARN 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task failed: [DataBackupHistoryWrapperTask{taskId=2, backupType=FULL, clusterId=null, obTenantId=1004, backupSetId=2, backupDest=file:///data/back_up/shanxi_ob_second/1740541070/tenant_incarnation_1/1004/data, snapshotVersionTime=null, startTime=2025-03-19T07:02:09.233949Z, finishTime=2025-03-19T07:02:24.046106Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 10.104.106.173:2882, result: -9120(cannot find object on object storage), trace_id: YB420A686AAD-00062F087091B94B-0-0}] 2025-03-19 15:02:40.334 ERROR 3137618 --- [manual-subtask-executor16,ce413c6b136c4588,09e7b7790593a14d] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Data backup task failed. com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_DATA_BACKUP_TASK_FAILED, args= at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.checkDataBackupTaskFinished(WaitDataBackupSuccessTask.java:172) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.lambda$run$5(WaitDataBackupSuccessTask.java:218) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.run(WaitDataBackupSuccessTask.java:213) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Set state for subtask: 1016589, operation:EXECUTE, state: FAILED