节点报【服务已停止】,报错:[errcode=-4006]、[errcode=-4743]

【 使用环境 】
测试环境
3节点
【 OB or 其他组件 】
【 使用版本 】 社区版 4.1.0
【问题描述】
1、管理平台

             2、日志:
                节点报【服务已停止】,报错:[errcode=-4006]、[errcode=-4743]


[lt=39][errcode=-4006] traversal ref obj version map failed(ret=-4006)
[2023-12-12 08:27:23.784621] WDIAG foreach_refactored (ob_hashtable.h:1308) [31046][T1002_FrzInfoDe][T1002][YB42AC1D010F–0-0] [lt=84][errcode=0] hashtable not init
[2023-12-12 08:27:23.784645] WDIAG reset (ob_dependency_info.cpp:855) [31046][T1002_FrzInfoDe][T1002][YB42AC1D010F-00060C[lt=24][errcode=-4006] traversal ref obj version map failed(ret=-4006)
[2023-12-12 08:27:23.790947] WDIAG foreach_refactored (ob_hashtable.h:1308) [31046][T1002_FrzInfoDe][T1001][YB42AC1D010F–0-0] [lt=49][errcode=0] hashtable not init
[2023-12-12 08:27:23.790993] WDIAG reset (ob_dependency_info.cpp:855) [31046][T1002_FrzInfoDe][T1001][YB42AC1D010F-00060C[lt=47][errcode=-4006] traversal ref obj version map failed(ret=-4006)
[2023-12-12 08:27:23.796922] WDIAG foreach_refactored (ob_hashtable.h:1308) [31046][T1002_FrzInfoDe][T1001][YB42AC1D010F–0-0] [lt=19][errcode=0] hashtable not init
[2023-12-12 08:27:23.796960] WDIAG reset (ob_dependency_info.cpp:855) [31046][T1002_FrzInfoDe][T1001][YB42AC1D010F-00060C[lt=38][errcode=-4006] traversal ref obj version map failed(ret=-4006)
[2023-12-12 08:27:25.876407] WDIAG [RS] run3 (ob_freeze_info_detector.cpp:160) [30893][T1001_FrzInfoDe][T1001][YB42AC1D0118F-0-0] [lt=9][errcode=-4743] fail to try_idle(ret=-4743, ret=“OB_FREEZE_SERVICE_EPOCH_MISMATCH”, tmp_ret=-4743, tmp_retICE_EPOCH_MISMATCH")
[2023-12-12 08:27:25.878111] WDIAG foreach_refactored (ob_hashtable.h:1308) [30893][T1001_FrzInfoDe][T1001][YB42AC1D010F–0-0] [lt=70][errcode=0] hashtable not init
[2023-12-12 08:27:25.878146] WDIAG reset (ob_dependency_info.cpp:855) [30893][T1001_FrzInfoDe][T1001][YB42AC1D010F-00060C[lt=36][errcode=-4006] traversal ref obj version map failed(ret=-4006)
[2023-12-12 08:27:25.882395] WDIAG foreach_refactored (ob_hashtable.h:1308) [30893][T1001_FrzInfoDe][T1001][YB42AC1D010F–0-0] [lt=69][errcode=0] hashtable not init
[2023-12-12 08:27:25.882425] WDIAG reset (ob_dependency_info.cpp:855) [30893][T1001_FrzInfoDe][T1001][YB42AC1D010F-00060C[lt=31][errcode=-4006] traversal ref obj version map failed(ret=-4006)
[2023-12-12 08:27:25.886922] WDIAG foreach_refactored (ob_hashtable.h:1308) [30893][T1001_FrzInfoDe][T1001][YB42AC1D010F–0-0] [lt=17][errcode=0] hashtable not init
[2023-12-12 08:27:25.886958] WDIAG reset (ob_dependency_info.cpp:855) [30893][T1001_FrzInfoDe][T1001][YB42AC1D010F-00060C[lt=37][errcode=-4006] traversal ref obj version map failed(ret=-4006)
[2023-12-12 08:27:25.887623] WDIAG [RS] check_freeze_service_epoch (ob_zone_merge_manager.cpp:170) [30893][T1001_FrzInfoD1D010F-00060C4502999190-0-0] [lt=37][errcode=-4743] freeze service epoch mismatch(ret=-4743, ret="OB_FREEZE_SERVICE_EPOCHcted_epoch=537, persistent_epoch=536)
[2023-12-12 08:27:25.887641] WDIAG [RS] try_update_zone_merge_info (ob_zone_merge_manager.cpp:879) [30893][T1001_FrzInfoD1D010F-00060C4502999190-0-0] [lt=17][errcode=-4743] fail to check freeze_service_epoch(ret=-4743, ret="OB_FREEZE_SERVICE_ expected_epoch=537)
[2023-12-12 08:27:25.888118] WDIAG foreach_refactored (ob_hashtable.h:1308) [30893][T1001_FrzInfoDe][T1001][YB42AC1D010F–0-0] [lt=82][errcode=0] hashtable not init
[2023-12-12 08:27:25.888128] WDIAG reset (ob_dependency_info.cpp:855) [30893][T1001_FrzInfoDe][T1001][YB42AC1D010F-00060C[lt=10][errcode=-4006] traversal ref obj version map failed(ret=-4006)
[2023-12-12 08:27:25.888200] WDIAG [RS] try_update_zone_info (ob_freeze_info_manager.cpp:639) [30893][T1001_FrzInfoDe][T1F-00060C4502999190-0-0] [lt=9][errcode=-4743] fail to try update zone_merge_info(ret=-4743, ret="OB_FREEZE_SERVICE_EPOCH_t_id=1001, expected_epoch=537)
[2023-12-12 08:27:25.888235] WDIAG [RS] try_update_zone_info (ob_freeze_info_detector.cpp:240) [30893][T1001_FrzInfoDe][T0F-00060C4502999190-0-0] [lt=34][errcode=-4743] fail to try update zone info(ret=-4743, ret="OB_FREEZE_SERVICE_EPOCH_MISM=1001, expected_epoch=537)
[2023-12-12 08:27:25.888247] WDIAG [RS] run3 (ob_freeze_info_detector.cpp:154) [30893][T1001_FrzInfoDe][T1001][YB42AC1D01190-0-0] [lt=12][errcode=-4743] fail to try update zone info(ret=-4743, ret=“OB_FREEZE_SERVICE_EPOCH_MISMATCH”, tenant_idid=537)

【复现路径】
重启节点,alter system start server “172.29.1.15:2882”;
仍报错

【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接(右键跳转查看):

【SOP系列 22 】——故障诊断第一步(自助诊断和诊断信息收集)

observer.zip (9.0 MB)

检查下几台机器之间的时间同步是否正常,clockdiff -o ip