2026-05-18 17:49:39.075 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2026-05-18 17:49:39.080 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9004525, context=Context{parallelIdx=-1, stringMap={ob_tenant_id=1004, tenant_id=3, checkBeforeDataBackup=true, startClusterCompact=false, task_instance_id=9004363, logBackupRecordList=[{"clusterId":1,"obTenantId":1004,"status":"RUNNING","tenantId":3,"tenantName":"ocp_monitor"}], dataBackupTriggerInfo={"async":true,"backupDimension":"TENANT","dataBackupMode":"FULL_BACKUP","triggerTime":"2026-05-18T09:49:38Z","triggerType":"BACKUP_NOW"}, description=ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2026-05-18 17:49:38.0, task_operation=execute, triggerTime=1779097778000000, startLogBackup=true, cluster_id=1, storageBaseInfo=******, tenantName=ocp_monitor, backupMode=physical, sub_task_instance_name=Wait data backup task successful, sub_task_instance_id=9004525, dataBackupMode=FULL_BACKUP, tenant_name=ocp_monitor, ob_version=4.2.1.8, ob_cluster_id=1771206186, backupDim=TENANT, dataSetIncIdMap={"1004":"6000017"}, keepLogRunningAfterBackup=true, triggerType=BACKUP_NOW, latest_execution_start_time=2026-05-18T17:49:39.060684084+08:00, backupNow=true, obClusterName=myocp}, listMap={}}, executor=172.16.18.34 2026-05-18 17:49:39.085 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=1 2026-05-18 17:49:39.103 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2026-05-18 17:49:39.108 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= '2026-05-18 17:49:38', args: [ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2026-05-18 17:49:38.0] 2026-05-18 17:49:39.125 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=48, backupType=FULL, clusterId=null, obTenantId=1004, backupSetId=48, backupDest=file:///mnt/ob_backup/myocp/1771206186/tenant_incarnation_1/1004/data, snapshotVersionTime=null, startTime=2026-05-18T09:49:38.130775Z, finishTime=null, status=INITIALIZING, progress=null, errorMsg=, dataProgress=null}] 2026-05-18 17:49:39.129 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : task status is UNKNOWN 2026-05-18 17:49:39.132 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : count running backup task size: 1 2026-05-18 17:50:09.135 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=2 2026-05-18 17:50:09.147 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myocp, tenantName=sys, dbUser=ocp_monitor 2026-05-18 17:50:09.152 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2026-05-18 17:50:09.156 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= '2026-05-18 17:49:38', args: [ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2026-05-18 17:49:38.0] 2026-05-18 17:50:09.173 INFO 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=48, backupType=FULL, clusterId=null, obTenantId=1004, backupSetId=48, backupDest=file:///mnt/ob_backup/myocp/1771206186/tenant_incarnation_1/1004/data, snapshotVersionTime=null, startTime=2026-05-18T09:49:38.130775Z, finishTime=2026-05-18T09:49:49.700570Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 172.16.18.35:2882, result: -9080(the format file does not exist under the destination), trace_id: YB42AC101223-0006519FE2BCE6F0-0-0, dataProgress=null}] 2026-05-18 17:50:09.179 WARN 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task failed: [DataBackupHistoryWrapperTask{taskId=48, backupType=FULL, clusterId=null, obTenantId=1004, backupSetId=48, backupDest=file:///mnt/ob_backup/myocp/1771206186/tenant_incarnation_1/1004/data, snapshotVersionTime=null, startTime=2026-05-18T09:49:38.130775Z, finishTime=2026-05-18T09:49:49.700570Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 172.16.18.35:2882, result: -9080(the format file does not exist under the destination), trace_id: YB42AC101223-0006519FE2BCE6F0-0-0, dataProgress=null}] 2026-05-18 17:50:09.183 ERROR 5987 --- [manual-subtask-executor14,cf6bd781dbad8b0d,14e8c35cbd8763a0] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Data backup task failed. com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_DATA_BACKUP_TASK_FAILED, args= at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.checkDataBackupTaskFinished(WaitDataBackupSuccessTask.java:170) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.lambda$run$5(WaitDataBackupSuccessTask.java:211) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.run(WaitDataBackupSuccessTask.java:206) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:234) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:228) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$3(SubtaskExecutor.java:154) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.base/java.lang.Thread.run(Unknown Source) Set state for subtask: 9004525, operation:EXECUTE, state: FAILED