OCP备份失败

OceanBase 技术问题
#1

024-04-02 16:24:15.801 ERROR 13883 — [pool-manual-subtask-executor16,84b0d9b11ab046f4,5a58d90a396f] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Data backup task failed.

98

99

com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_DATA_BACKUP_TASK_FAILED, args=

100

at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.checkDataBackupTaskResult(WaitDataBackupSuccessTask.java:153)

101

at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.run(WaitDataBackupSuccessTask.java:204)

102

at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64)

103

at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32)

104

at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26)

105

at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76)

106

at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:203)

107

at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:197)

108

at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:134)

109

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

110

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

111

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

112

at java.lang.Thread.run(Thread.java:750)

113

114

115

Set state for subtask: 20007444, operation:EXECUTE, state: FAILED

116

#3

— 看下归档状态
SELECT * FROM oceanbase.CDB_OB_ARCHIVELOG\G

— 看下数据备份进度
SELECT * FROM oceanbase.CDB_OB_BACKUP_JOBS\G

使用什么备份介质备份的？

#4

image
image1946×200 112 KB

9011可能是目录访问权限问题或者找不到目录。

如果是集群环境需要配置nfs备份介质。

如果已经配置nfs，可以去rs节点 rootservice.log 和observer.log 过滤下上面截图的trace_id.

#5

root@172.23.88.131 09:37:50 [(none)]>SELECT * FROM oceanbase.CDB_OB_ARCHIVELOG\G
*************************** 1. row ***************************
TENANT_ID: 1004
DEST_ID: 1001
ROUND_ID: 1
INCARNATION: 1
DEST_NO: 0
STATUS: DOING
START_SCN: 1712046180799219000
START_SCN_DISPLAY: 2024-04-02 16:23:00.799219
CHECKPOINT_SCN: 1712108203007883004
CHECKPOINT_SCN_DISPLAY: 2024-04-03 09:36:43.007883
COMPATIBLE: 1
BASE_PIECE_ID: 1
USED_PIECE_ID: 1
PIECE_SWITCH_INTERVAL: 86400000000
UNIT_SIZE: 1
COMPRESSION: none
INPUT_BYTES: 435056147
INPUT_BYTES_DISPLAY: 414.90MB
OUTPUT_BYTES: 435056147
OUTPUT_BYTES_DISPLAY: 414.90MB
COMPRESSION_RATIO: 1.00
DELETED_INPUT_BYTES: 0
DELETED_INPUT_BYTES_DISPLAY: 0.00MB
DELETED_OUTPUT_BYTES: 0
DELETED_OUTPUT_BYTES_DISPLAY: 0.00MB
COMMENT:
PATH: file:///home/tenant/myoceanbase/1711703621/tenant_incarnation_1/1004/clog
1 row in set (0.042 sec)

root@172.23.88.131 09:37:59 [(none)]>SELECT * FROM oceanbase.CDB_OB_BACKUP_JOBS\G
Empty set (0.367 sec)

root@172.23.88.131 09:39:38 [(none)]>

#6

存储类型 File

没有配置nfs备份介质

后端节点都没有过滤出这个trace_id: YB42AC175881-000614C97C6C28FF-0-0

#7

单机还是集群？

如果是多节点的集群是必须配置备份介质的。

如果单机的话，可以后台手动触发全量备份，看下日志。

业务租户下先执行
SET ob_enable_show_trace=‘ON’;

执行全量备份
ALTER SYSTEM BACKUP DATABASE;

再执行，会打印trace_id信息。
select last_trace_id();

#8

集群，如何配置备份介质，我用OBD方式安装的

#9

参看文档
https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000000639729