【 使用环境 】生产环境
【 OB or 其他组件 】OB
【 使用版本 】4.2.10
【问题描述】NAS设备故障，备份任务无法退出
【复现路径】使用umount -l 强制卸载，然后ALTER SYSTEM CANCEL BACKUP，ALTER SYSTEM NOARCHIVELOG ，OCP和黑屏命令查询归档任务还在。
【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接（右键跳转查看）：
【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试，在发帖时输入 [@论坛小助手] 即可召唤小助手，欢迎试用！
MySQL [oceanbase]> SELECT * FROM oceanbase.CDB_OB_ARCHIVELOG;
±----------±--------±---------±------------±--------±-------±--------------------±---------------------------±--------------------±---------------------------±-----------±--------------±--------------±----------------------±----------±------------±------------±--------------------±-------------±---------------------±------------------±--------------------±----------------------------±---------------------±-----------------------------±--------±--------------------------------------------------------------------------------+
| TENANT_ID | DEST_ID | ROUND_ID | INCARNATION | DEST_NO | STATUS | START_SCN | START_SCN_DISPLAY | CHECKPOINT_SCN | CHECKPOINT_SCN_DISPLAY | COMPATIBLE | BASE_PIECE_ID | USED_PIECE_ID | PIECE_SWITCH_INTERVAL | UNIT_SIZE | COMPRESSION | INPUT_BYTES | INPUT_BYTES_DISPLAY | OUTPUT_BYTES | OUTPUT_BYTES_DISPLAY | COMPRESSION_RATIO | DELETED_INPUT_BYTES | DELETED_INPUT_BYTES_DISPLAY | DELETED_OUTPUT_BYTES | DELETED_OUTPUT_BYTES_DISPLAY | COMMENT | PATH |
±----------±--------±---------±------------±--------±-------±--------------------±---------------------------±--------------------±---------------------------±-----------±--------------±--------------±----------------------±----------±------------±------------±--------------------±-------------±---------------------±------------------±--------------------±----------------------------±---------------------±-----------------------------±--------±--------------------------------------------------------------------------------+
| 1002 | 1001 | 1 | 1 | 0 | DOING | 1733256058452024335 | 2024-12-04 04:00:58.452024 | 1751471218838801396 | 2025-07-02 23:46:58.838801 | 1 | 1 | 211 | 86400000000 | 1 | none | 96284381810 | 89.67GB | 96284381810 | 89.67GB | 1.00 | 0 | 0.00MB | 0 | 0.00MB | | file:///nas_backup/obbak_ce/obce_xc01/1732845643/tenant_incarnation_1/1002/clog |
| 1006 | 1001 | 1 | 1 | 0 | DOING | 1750907790006214810 | 2025-06-26 11:16:30.006214 | 1751471159265768932 | 2025-07-02 23:45:59.265768 | 1 | 1 | 7 | 86400000000 | 1 | none | 14476722763 | 13.48GB | 14476722763 | 13.48GB | 1.00 | 0 | 0.00MB | 0 | 0.00MB | | file:///nas_backup/obbak_ce/obce_xc01/1732845643/tenant_incarnation_1/1006/clog |
| 1008 | 1001 | 1 | 1 | 0 | DOING | 1750907763245675223 | 2025-06-26 11:16:03.245675 | 1751471168565366256 | 2025-07-02 23:46:08.565366 | 1 | 1 | 7 | 86400000000 | 1 | none | 22035703261 | 20.52GB | 22035703261 | 20.52GB | 1.00 | 0 | 0.00MB | 0 | 0.00MB | | file:///nas_backup/obbak_ce/obce_xc01/1732845643/tenant_incarnation_1/1008/clog |
±----------±--------±---------±------------±--------±-------±--------------------±---------------------------±--------------------±---------------------------±-----------±--------------±--------------±----------------------±----------±------------±------------±--------------------±-------------±---------------------±------------------±--------------------±----------------------------±---------------------±-----------------------------±--------±--------------------------------------------------------------------------------+
MySQL [oceanbase]> SELECT TENANT_NAME, LOG_MODE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE=‘USER’\G
*************************** 1. row ***************************
TENANT_NAME: ********
LOG_MODE: NOARCHIVELOG
*************************** 2. row ***************************
TENANT_NAME: ********
LOG_MODE: NOARCHIVELOG
*************************** 3. row ***************************
TENANT_NAME: ********
LOG_MODE: NOARCHIVELOG
3 rows in set (0.026 sec)
ALTER SYSTEM SET data_backup_dest=‘file:////ob_bkpnew/obce_xc01’ tenant=********;
ERROR 9036 (HY000): Backup is in progress
针对您的问题：“NAS设备故障导致OceanBase归档任务无法停止”，我们建议使用obdiag工具来收集相关日志信息，以便于进一步的分析。这里我们主要关注的是日志归档问题。
日志采集场景: 日志归档问题
日志采集命令: obdiag gather scene run --scene=observer.log_archive
请您执行上述命令，并将生成的日志文件提供给我们，这将有助于我们更准确地定位并解决问题。
为了能够更加有效地解决您的问题，请回答以下几个问题：
感谢您的配合，我们将尽快为您解决问题。
附上敏捷诊断工具 obdiag 使用帮助链接
使用obdiag可以巡检归档问题，然后根据下面方法
1）设置trace信息
SET ob_enable_show_trace=‘ON’;
2）执行sql。（关闭归档sql
3）获取上个命令的trace
select last_trace_id();
4）获取trace对应的节点
select query_sql,svr_ip from gv$ob_sql_audit where trace_id=‘第三步获取的trace信息’；
5）取对应的svr_ip节点 过滤日志
grep “第三步获取的trace信息” observer.log*
grep “第三步获取的trace信息” rootservice.log*
6）提供日志信息即可。
现在是能执行alter system noarichivelog；但是归档日志进程没有正常退出，异常卡死了。
[2025-07-14 10:38:39.617600] INFO [RS] handle_archive_log (ob_root_service.cpp:9959) [369566][T1_L0_G0][T1][Y10B4251115123-000628168668C5E1-0-0] [lt=15] handle_archive_log(arg={enable:false, tenant_id:1, archive_tenant_ids:[1002, 1008, 1006]})
[2025-07-14 10:38:39.619195] INFO [STORAGE] ~ObStorageTableGuard (ob_storage_table_guard.cpp:153) [369566][T1_L0_G0][T1001][Y10B4251115123-000628168668C5E1-0-0] [lt=11] throttle statics(need_speed_limit=false, last_throttle_status=false, last_print_log_t
ime=1752460191293747, stat={total_throttle_time_us:0, total_skip_throttle_time_us:0, last_log_timestamp:1752460719618376, last_throttle_status:false, 0=0, 1=0, 2=0, 3=0})
[2025-07-14 10:38:39.620647] INFO [ARCHIVE] close_archive_mode (ob_tenant_archive_scheduler.cpp:477) [369566][T1_L0_G0][T1][Y10B4251115123-000628168668C5E1-0-0] [lt=10] close archive mode(tenant_id=1002)
[2025-07-14 10:38:39.620686] INFO [SHARE] add_event (ob_event_history_table_operator.h:295) [369566][T1_L0_G0][T1][Y10B4251115123-000628168668C5E1-0-0] [lt=11] event table add task(ret=0, event_table_name="__all_rootservice_event_history", sql=INSERT INT
O __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1752460719620660), ‘log_archive’, ‘close_archive_mode’, ‘tenant_id’, 1002, ‘result’, 0, ‘81.17.81.33’, 2882))
[2025-07-14 10:38:39.623676] INFO [ARCHIVE] close_archive_mode (ob_tenant_archive_scheduler.cpp:477) [369566][T1_L0_G0][T1][Y10B4251115123-000628168668C5E1-0-0] [lt=10] close archive mode(tenant_id=1008)
[2025-07-14 10:38:39.623703] INFO [SHARE] add_event (ob_event_history_table_operator.h:295) [369566][T1_L0_G0][T1][Y10B4251115123-000628168668C5E1-0-0] [lt=10] event table add task(ret=0, event_table_name="__all_rootservice_event_history", sql=INSERT INT
O __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1752460719623687), ‘log_archive’, ‘close_archive_mode’, ‘tenant_id’, 1008, ‘result’, 0, ‘81.17.81.33’, 2882))
[2025-07-14 10:38:39.626731] INFO [ARCHIVE] close_archive_mode (ob_tenant_archive_scheduler.cpp:477) [369566][T1_L0_G0][T1][Y10B4251115123-000628168668C5E1-0-0] [lt=10] close archive mode(tenant_id=1006)
[2025-07-14 10:38:39.626758] INFO [SHARE] add_event (ob_event_history_table_operator.h:295) [369566][T1_L0_G0][T1][Y10B4251115123-000628168668C5E1-0-0] [lt=11] event table add task(ret=0, event_table_name="__all_rootservice_event_history", sql=INSERT INT
O _all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1752460719626742), ‘log_archive’, ‘close_archive_mode’, ‘tenant_id’, 1006, ‘result’, 0, ‘81.17.81.33’, 2882))
[2025-07-14 10:38:39.626771] INFO [RS] process (ob_rs_rpc_processor.h:230) [369566][T1_L0_G0][T1][Y10B4251115123-000628168668C5E1-0-0] [lt=11] [DDL] execute ddl like stmt(ret=0, cost=9172, ddl_arg=NULL)
只有rootservice有信息，是不是遇到备份任务异常退出BUG了
4.2.11### 关键缺陷修复
告警详情：[OBServer 非预期的内部错误] 集群：obce_xc01，主机：...，日志类型：observer，日志文件：/home/admin/oceanbase/log/observer.log，日志级别：ERROR，关键字=Unexpected internal error happen，错误码=4388，日志详情=[2025-07-15 17:26:58.565661] ERROR issue_dba_error (ob_log.cpp:1875) [994374][T1006_ArcSeq][T1006][YB4251115122-000639F4645B5BD9-0-0] [lt=9][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file=“ob_archive_round_mgr.cpp”, line_no=235, info=“archive mark_fatal_error”)。
最后还是重启解决 重启节点1的时候失败，OCP监控集群不可用，强制重启系统后问题解决，应该是归档任务HANG死在里面了，疑似4.2.1.10版本的BUG。
