部署集群后节点observer进程相继中断，使用obdiag check提示如下，这是什么问题导致的？

AntTech_UCR3NB · 2024 年12 月 20 日 18:13

17:44启动
17:47第一个节点172.69.0.90就挂了
18:03第二个节点172.69.0.92挂了
172.69.0.90-observer.7z (2.8 MB)
172.69.0.92-observer.7z (2.0 MB)
172.69.0.94-observer.7z (295.7 KB)

旭辉 · 2024 年12 月 20 日 22:45

节点2（92）和节点3（94）都将节点1（90）拉入黑名单了，节点1报错误 IO error，io handle wait failed，大概率节点1磁盘有问题，建议更换节点1试下；节点2被节点3加入了黑名单，检查下节点2是否有网络层面的限制，例如iptables

节点2:

[2024-12-20 17:52:15.183479] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [147411][AutoLSLocRpc][T0][YB42AC45005C-000629B07FF74F03-0-0] [lt=19][errcode=-4122] address in blacklist(ret=-4122, addr="172.69.0.90:2882")
[2024-12-20 17:52:15.183574] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [147411][AutoLSLocRpc][T0][YB42AC45005C-000629B07FF74F03-0-0] [lt=90][errcode=-4122] check_blacklist failed(addr="172.69.0.90:2882")

节点3:

将节点1加入黑名单
[2024-12-20 18:12:22.320657] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [112607][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4122] address in blacklist(ret=-4122, addr="172.69.0.90:2882")
[2024-12-20 18:12:22.320760] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [112607][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4122] check_blacklist failed(ret=-4122)

将节点2加入黑名单
[2024-12-20 18:12:34.343942] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [112184][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4122] address in blacklist(ret=-4122, addr="172.69.0.92:2882")
[2024-12-20 18:12:34.344015] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [112184][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=70][errcode=-4122] check_blacklist failed(ret=-4122)
[2024-12-20 18:12:34.344055] WDIAG [CLOG] post_sync_request_to_server_ (log_net_service.h:347) [112184][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4122] ObLogRpc post_sync_request failed(ret=-4122, palf_id=1, req={get_type:1}, server="172.69.0.92:2882")

节点1:

[2024-12-20 17:44:26.755591] WDIAG [COMMON] wait (ob_io_define.cpp:1877) [113551][observer][T0][Y0-0000000000000001-0-0] [lt=56][errcode=-4224] IO error, (ret=-4224, *result_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:0, offset_:0, size_:66060288, timeout_us_:10000000, result_ref_cnt_:1, out_ref_cnt_:1, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:3, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4224, fs_errno_:0}, tenant_id_:500, tenant_io_mgr_:{ptr:0x2b09efdf8030}, user_data_buf_:0x2b0a1da05000, buf_:null, io_callback_:null, time_log:{begin_ts:1734687866755248, enqueue_used:-1, dequeue_used:-1, submit_used:1734687866755308, return_used:46, callback_enqueue_used:-1, callback_dequeue_used:-1, callback_finish_used:-1, end_used:1734687866755563}})
[2024-12-20 17:44:26.755692] WDIAG [COMMON] read (ob_io_manager.cpp:473) [113551][observer][T0][Y0-0000000000000001-0-0] [lt=97][errcode=-4224] io handle wait failed(ret=-4224, info={tenant_id_:500, fd_:{first_id:-1, second_id:186, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2b09eec4a080}, offset_:0, size_:66060288, timeout_us_:10000000, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:3, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, callback_:null, buf_:null, user_data_buf_:0x2b0a1da05000, part_id:-1}, info.timeout_us_=10000000)