2025-03-12 14:06:39.631 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-03-12 14:06:39.636 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=6250, context=Context{parallelIdx=-1, stringMap={checkBeforeDataBackup=true, ob_version=4.3.5.0, startClusterCompact=false, task_instance_id=5924, logBackupRecordList=[{"clusterId":2,"obTenantId":1002,"status":"RUNNING","tenantId":5,"tenantName":"exam"}], dataBackupTriggerInfo={"async":true,"backupDimension":"CLUSTER","dataBackupMode":"FULL_BACKUP","triggerTime":"2025-03-12T06:06:38Z","triggerType":"BACKUP_NOW"}, description=ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-03-12 14:06:38.0, task_operation=execute, triggerTime=1741759598000000, ob_cluster_id=1741079861, startLogBackup=true, backupDim=CLUSTER, dataSetIncIdMap={"1002":"8"}, cluster_id=2, storageBaseInfo=******, tenantName=*, backupMode=physical, triggerType=BACKUP_NOW, latest_execution_start_time=2025-03-12T14:06:39.616+08:00, sub_task_instance_name=Wait data backup task successful, backupNow=true, sub_task_instance_id=6250, dataBackupMode=FULL_BACKUP, obClusterName=test}, listMap={}}, executor=10.1.46.60 2025-03-12 14:06:39.641 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=1 2025-03-12 14:06:39.661 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.161, port=2881, database=oceanbase) 2025-03-12 14:06:39.665 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.124, port=2881, database=oceanbase) 2025-03-12 14:06:39.672 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.161, port=2881, database=oceanbase) 2025-03-12 14:06:39.676 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.124, port=2881, database=oceanbase) 2025-03-12 14:06:39.680 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-12 14:06:39.687 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE ? 2025-03-12 14:06:39.692 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.161, port=2881, database=oceanbase) 2025-03-12 14:06:39.696 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.124, port=2881, database=oceanbase) 2025-03-12 14:06:39.701 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-12 14:06:39.706 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2025-03-12 14:06:39.721 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.161, port=2881, database=oceanbase) 2025-03-12 14:06:39.726 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.124, port=2881, database=oceanbase) 2025-03-12 14:06:39.730 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-12 14:06:39.761 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= '2025-03-12 14:06:38', args: [ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-03-12 14:06:38.0] 2025-03-12 14:06:39.818 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=12, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=12, backupDest=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-03-12T06:06:38.763213Z, finishTime=null, status=INITIALIZING, progress=null, errorMsg=}] 2025-03-12 14:06:39.826 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : task status is UNKNOWN 2025-03-12 14:06:39.831 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : count running backup task size: 1 2025-03-12 14:07:09.836 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=2 2025-03-12 14:07:09.856 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test, tenantName=sys, dbUser=ocp_monitor 2025-03-12 14:07:09.859 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.161, port=2881, database=oceanbase) 2025-03-12 14:07:09.863 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.124, port=2881, database=oceanbase) 2025-03-12 14:07:09.868 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.161, port=2881, database=oceanbase) 2025-03-12 14:07:09.872 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.124, port=2881, database=oceanbase) 2025-03-12 14:07:09.877 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-12 14:07:09.881 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE ? 2025-03-12 14:07:09.887 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.161, port=2881, database=oceanbase) 2025-03-12 14:07:09.891 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.124, port=2881, database=oceanbase) 2025-03-12 14:07:09.895 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-12 14:07:09.901 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2025-03-12 14:07:09.917 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.161, port=2881, database=oceanbase) 2025-03-12 14:07:09.921 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=test, obClusterId=1741079861, tenantName=sys, username=ocp_monitor, address=10.1.46.124, port=2881, database=oceanbase) 2025-03-12 14:07:09.925 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-03-12 14:07:09.954 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= '2025-03-12 14:06:38', args: [ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-03-12 14:06:38.0] 2025-03-12 14:07:09.988 INFO 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=12, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=12, backupDest=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-03-12T06:06:38.763213Z, finishTime=2025-03-12T06:07:06.218438Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 10.1.46.161:2882, result: -9120(cannot find object on object storage), trace_id: YB420A012EA1-0006300C870CBF7A-0-0}] 2025-03-12 14:07:10.003 WARN 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task failed: [DataBackupHistoryWrapperTask{taskId=12, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=12, backupDest=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-03-12T06:06:38.763213Z, finishTime=2025-03-12T06:07:06.218438Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 10.1.46.161:2882, result: -9120(cannot find object on object storage), trace_id: YB420A012EA1-0006300C870CBF7A-0-0}] 2025-03-12 14:07:10.008 ERROR 113014 --- [manual-subtask-executor15,d21d2b5f7e1cea81,604f9187b9898583] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Data backup task failed. com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_DATA_BACKUP_TASK_FAILED, args= at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.checkDataBackupTaskFinished(WaitDataBackupSuccessTask.java:170) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.lambda$run$5(WaitDataBackupSuccessTask.java:211) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.run(WaitDataBackupSuccessTask.java:206) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Set state for subtask: 6250, operation:EXECUTE, state: FAILED