使用OCP部署了一个集群，集群中创建好租户后，再租户中新建租户级备份，执行到Wait data backup task successful时失败
2 个赞
发下OCP版本，OBServer版本
右上角任务日志下载发下
############{EXECUTE}{2025-08-26T15:40:28.775+08:00}############2025-08-26 15:40:28.823 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000
2025-08-26 15:40:28.836 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=26286, context=Context{parallelIdx=-1, stringMap={checkBeforeDataBackup=true, startClusterCompact=false, task_instance_id=26405, logBackupRecordList=[{“clusterId”:2,“obTenantId”:1002,“status”:“STOPPED”,“tenantId”:5,“tenantName”:“htcl”},{“clusterId”:2,“obTenantId”:1004,“status”:“STOPPED”,“tenantId”:6,“tenantName”:“oms”},{“clusterId”:2,“obTenantId”:1006,“status”:“STOPPED”,“tenantId”:7,“tenantName”:“ocean_test”},{“clusterId”:2,“obTenantId”:1008,“status”:“STOPPED”,“tenantId”:8,“tenantName”:“ttt”},{“clusterId”:2,“obTenantId”:1010,“status”:“STOPPED”,“tenantId”:9,“tenantName”:“qq”}], dataBackupTriggerInfo={“async”:true,“backupDimension”:“CLUSTER”,“dataBackupMode”:“FULL_BACKUP”,“triggerTime”:“2025-08-26T07:40:27Z”,“triggerType”:“BACKUP_NOW”}, description=ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-08-26 15:40:27.0, task_operation=execute, triggerTime=1756194027000000, startLogBackup=true, cluster_id=2, storageBaseInfo=****, tenantName=, backupMode=physical, sub_task_instance_name=Wait data backup task successful, sub_task_instance_id=26286, dataBackupMode=FULL_BACKUP, ob_version=4.3.5.2, ob_cluster_id=1753235999, backupDim=CLUSTER, dataSetIncIdMap={“1008”:“20”,“1006”:“17”,“1004”:“18”,“1002”:“16”,“1010”:“19”}, keepLogRunningAfterBackup=true, triggerType=BACKUP_NOW, latest_execution_start_time=2025-08-26T15:40:28.654+08:00, backupNow=true, obClusterName=ky10arrch64}, listMap={}}, executor=192.168.102.29
2025-08-26 15:40:28.911 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=1
2025-08-26 15:40:29.065 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2025-08-26 15:40:29.087 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE ?
2025-08-26 15:40:29.106 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2025-08-26 15:40:29.128 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /+ ocp_query / tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ?
2025-08-26 15:40:29.177 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2025-08-26 15:40:29.225 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /+ ocp_query / * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, ‘TENANT’ AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= ‘2025-08-26 15:40:27’, args: [ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-08-26 15:40:27.0]
2025-08-26 15:40:29.315 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=3, backupType=FULL, clusterId=null, obTenantId=1008, backupSetId=3, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1008/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.915506Z, finishTime=null, status=INITIALIZING, progress=null, errorMsg=}, DataBackupHistoryWrapperTask{taskId=3, backupType=FULL, clusterId=null, obTenantId=1010, backupSetId=3, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1010/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.828828Z, finishTime=null, status=INITIALIZING, progress=null, errorMsg=}, DataBackupHistoryWrapperTask{taskId=3, backupType=FULL, clusterId=null, obTenantId=1004, backupSetId=3, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1004/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.749530Z, finishTime=null, status=INITIALIZING, progress=null, errorMsg=}, DataBackupHistoryWrapperTask{taskId=4, backupType=FULL, clusterId=null, obTenantId=1006, backupSetId=4, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1006/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.673087Z, finishTime=null, status=INITIALIZING, progress=null, errorMsg=}, DataBackupHistoryWrapperTask{taskId=7, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=7, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.580608Z, finishTime=null, status=INITIALIZING, progress=null, errorMsg=}]
2025-08-26 15:40:29.334 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : task status is UNKNOWN
2025-08-26 15:40:29.354 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : task status is UNKNOWN
2025-08-26 15:40:29.369 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : task status is UNKNOWN
2025-08-26 15:40:29.381 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : task status is UNKNOWN
2025-08-26 15:40:29.390 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : task status is UNKNOWN
2025-08-26 15:40:29.397 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : count running backup task size: 5
2025-08-26 15:40:59.405 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : try to check data backup task finished, retryTimes=2
2025-08-26 15:40:59.438 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=ky10arrch64, tenantName=sys, dbUser=ocp_monitor
2025-08-26 15:40:59.449 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2025-08-26 15:40:59.457 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE ?
2025-08-26 15:40:59.469 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2025-08-26 15:40:59.477 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /+ ocp_query / tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ?
2025-08-26 15:40:59.527 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2025-08-26 15:40:59.574 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /+ ocp_query */ * from (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, task.data_progress AS data_progress, status, comment, description, ‘TENANT’ AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, data_progress FROM CDB_OB_BACKUP_TASKS) task ON job.tenant_id = task.tid AND job.backup_set_id = task.backup_set_id WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC) where start_time >= ‘2025-08-26 15:40:27’, args: [ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-08-26 15:40:27.0]
2025-08-26 15:40:59.662 INFO 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task: [DataBackupHistoryWrapperTask{taskId=3, backupType=FULL, clusterId=null, obTenantId=1008, backupSetId=3, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1008/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.915506Z, finishTime=2025-08-26T07:40:53.201209Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A97EFF55863-0-0}, DataBackupHistoryWrapperTask{taskId=3, backupType=FULL, clusterId=null, obTenantId=1010, backupSetId=3, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1010/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.828828Z, finishTime=2025-08-26T07:40:48.744036Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A9811C5552C-0-0}, DataBackupHistoryWrapperTask{taskId=3, backupType=FULL, clusterId=null, obTenantId=1004, backupSetId=3, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1004/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.749530Z, finishTime=2025-08-26T07:40:58.688242Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A903006B0A4-0-0}, DataBackupHistoryWrapperTask{taskId=4, backupType=FULL, clusterId=null, obTenantId=1006, backupSetId=4, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1006/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.673087Z, finishTime=2025-08-26T07:40:43.234226Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A9753D56E26-0-0}, DataBackupHistoryWrapperTask{taskId=7, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=7, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.580608Z, finishTime=2025-08-26T07:40:47.302133Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A8F7C26CB46-0-0}]
2025-08-26 15:40:59.739 WARN 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.b.i.t.s.WaitDataBackupSuccessTask : check backup task failed: [DataBackupHistoryWrapperTask{taskId=3, backupType=FULL, clusterId=null, obTenantId=1008, backupSetId=3, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1008/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.915506Z, finishTime=2025-08-26T07:40:53.201209Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A97EFF55863-0-0}, DataBackupHistoryWrapperTask{taskId=3, backupType=FULL, clusterId=null, obTenantId=1010, backupSetId=3, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1010/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.828828Z, finishTime=2025-08-26T07:40:48.744036Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A9811C5552C-0-0}, DataBackupHistoryWrapperTask{taskId=7, backupType=FULL, clusterId=null, obTenantId=1002, backupSetId=7, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1002/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.580608Z, finishTime=2025-08-26T07:40:47.302133Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A8F7C26CB46-0-0}, DataBackupHistoryWrapperTask{taskId=3, backupType=FULL, clusterId=null, obTenantId=1004, backupSetId=3, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1004/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.749530Z, finishTime=2025-08-26T07:40:58.688242Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A903006B0A4-0-0}, DataBackupHistoryWrapperTask{taskId=4, backupType=FULL, clusterId=null, obTenantId=1006, backupSetId=4, backupDest=file:///obbackup/ky10arrch64/1753235999/tenant_incarnation_1/1006/data, snapshotVersionTime=null, startTime=2025-08-26T07:40:27.673087Z, finishTime=2025-08-26T07:40:43.234226Z, status=FAILED, progress=null, errorMsg=(ROOTSERVICE)addr: 192.168.102.27:2882, result: -9120(cannot find object on object storage), trace_id: YB42C0A8661B-00063A9753D56E26-0-0}]
2025-08-26 15:40:59.764 ERROR 75261 — [manual-subtask-executor15,a949ad7d2b23b40a,fa55da6e3fe1f819] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Data backup task failed.
com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_DATA_BACKUP_TASK_FAILED, args=
at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.checkDataBackupTaskFinished(WaitDataBackupSuccessTask.java:170)
at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.lambda$run$5(WaitDataBackupSuccessTask.java:211)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.run(WaitDataBackupSuccessTask.java:206)
at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64)
at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32)
at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26)
at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:212)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:206)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Set state for subtask: 26286, operation:EXECUTE, state: FAILED
OCP版本号: 4.3.6-20250709105610
oceanbase版本号 4.3.5.2
日志放在下面了
集群是单节点还是多节点？有配置NFS备份目录吗？
你那个NFS是3.0还是4.0？
学习
围观一下
学习了