【 使用环境 】测试环境
【 使用版本 】4.3.5
【问题描述】重启服务器后，执行obd cluster restart myoceanbase连接服务失败，报错码1005，1006，使用 obd cluster start myoceanbase --wop 无参启动仍然报错1006
【复现路径】仅部署在一台服务器上，此服务器重启后，重启oceanbase服务无法连接
【附件及日志】
执行obd cluster restart myoceanbase日志信息：
[2025-07-10 11:03:12.605] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p
[2025-07-10 11:03:15.608] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p******
[2025-07-10 11:03:18.612] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p
[2025-07-10 11:03:21.615] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p******
[2025-07-10 11:03:24.619] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p
[2025-07-10 11:03:27.623] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p******
[2025-07-10 11:03:30.625] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p
[2025-07-10 11:03:33.629] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p******
[2025-07-10 11:03:36.632] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p
[2025-07-10 11:03:39.636] [DEBUG] — connect 10.34.17.169 -P2881 -uroot -p******
[2025-07-10 11:03:39.637] [ERROR] Traceback (most recent call last):
[2025-07-10 11:03:39.637] [ERROR] File “pymysql/connections.py”, line 613, in connect
[2025-07-10 11:03:39.637] [ERROR] File “socket.py”, line 808, in create_connection
[2025-07-10 11:03:39.637] [ERROR] File “socket.py”, line 796, in create_connection
[2025-07-10 11:03:39.637] [ERROR] ConnectionRefusedError: [Errno 111] Connection refused
[2025-07-10 11:03:39.637] [ERROR]
[2025-07-10 11:03:39.637] [ERROR] During handling of the above exception, another exception occurred:
[2025-07-10 11:03:39.637] [ERROR]
[2025-07-10 11:03:39.637] [ERROR] Traceback (most recent call last):
[2025-07-10 11:03:39.637] [ERROR] File “core.py”, line 2756, in restart_cluster
[2025-07-10 11:03:39.638] [ERROR] File “core.py”, line 2831, in _restart_cluster
[2025-07-10 11:03:39.638] [ERROR] File “core.py”, line 2295, in _start_cluster
[2025-07-10 11:03:39.638] [ERROR] File “core.py”, line 246, in run_workflow
[2025-07-10 11:03:39.638] [ERROR] File “core.py”, line 288, in run_plugin_template
[2025-07-10 11:03:39.638] [ERROR] File “core.py”, line 336, in call_plugin
[2025-07-10 11:03:39.638] [ERROR] File “_plugin.py”, line 348, in call
[2025-07-10 11:03:39.638] [ERROR] File “_plugin.py”, line 304, in _new_func
[2025-07-10 11:03:39.638] [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 68, in connect
[2025-07-10 11:03:39.638] [ERROR] cursor = Cursor(ip=server.ip, port=server_config.get(‘mysql_port’, 2881), tenant=’’, password=****** if password is not None else ‘’, stdio=stdio)
[2025-07-10 11:03:39.638] [ERROR] File “_stdio.py”, line 1089, in wrapper
[2025-07-10 11:03:39.638] [ERROR] File “_stdio.py”, line 1076, in func_wrapper
[2025-07-10 11:03:39.638] [ERROR] File “tool.py”, line 783, in init
[2025-07-10 11:03:39.638] [ERROR] File “tool.py”, line 813, in _connect
[2025-07-10 11:03:39.638] [ERROR] File “pymysql/connections.py”, line 353, in init
[2025-07-10 11:03:39.638] [ERROR] File “pymysql/connections.py”, line 664, in connect
[2025-07-10 11:03:39.638] [ERROR] pymysql.err.OperationalError: (2003, “Can’t connect to MySQL server on ‘10.34.17.169’ ([Errno 111] Connection refused)”)
[2025-07-10 11:03:39.638] [ERROR]
[2025-07-10 11:03:42.666] [ERROR] OBD-1006: Failed to connect to oceanbase-ce
[2025-07-10 11:03:42.666] [DEBUG] – sub connect ref count to 0
[2025-07-10 11:03:42.666] [DEBUG] – export connect
[2025-07-10 11:03:42.666] [DEBUG] – plugin oceanbase-ce-py_script_connect-3.1.0 result: False
[2025-07-10 11:03:42.666] [ERROR] OBD-1005: Some of the servers in the cluster have been stopped
[2025-07-10 11:03:42.671] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2025-07-10 11:03:42.671] [INFO] Trace ID: b6796bae-5d39-11f0-9535-000c29081250
[2025-07-10 11:03:42.671] [INFO] If you want to view detailed obd logs, please run: obd display-trace b6796bae-5d39-11f0-9535-000c29081250
[2025-07-10 11:03:42.671] [DEBUG] - unlock /root/.obd/lock/global
[2025-07-10 11:03:42.671] [DEBUG] - unlock /root/.obd/lock/deploy_myoceanbase
[2025-07-10 11:03:42.671] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
Failed to connect to oceanbase-ce。连接ob报错，需要提供一份observer日志看看
您好，这是截取的部分observe.log日志信息
[2025-07-10 15:42:26.307336] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [36264][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, cluster_id=1751427365, tenant_id=1, ls_id={id:1})
[2025-07-10 15:42:26.334839] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [36600][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:“post cluster heartbeat rpc failed, tenant_id=1001”, warnings:[]}, tenant_id_=1001, dst=“10.34.17.169:2882”, cluster_service_tablet_id={id:226})
[2025-07-10 15:42:26.334872] WDIAG [SHARE] check_write_limited (ob_local_device.cpp:1315) [36531][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] The ObLocalDevice has not been marked(ret=-4006)
[2025-07-10 15:42:26.334886] WDIAG [COORDINATOR] detect_data_disk_full_ (ob_failure_detector.cpp:486) [36531][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4006] check space full failed(ret=-4006)
[2025-07-10 15:42:26.334894] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:865) [36600][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret=“OB_NEED_WAIT”, tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1752133346334833)
[2025-07-10 15:42:26.334904] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:875) [36600][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret=“OB_NEED_WAIT”, tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1752133346234631, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master=“0.0.0.0:0”)
[2025-07-10 15:42:26.334922] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [36600][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001)
[2025-07-10 15:42:26.334927] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [36600][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001)
[2025-07-10 15:42:26.334931] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:593) [36600][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret=“OB_EAGAIN”, tenant_id=1001, index=0x7f06daa97d70, server_version_epoch_tstamp_=1752133346334910)
[2025-07-10 15:42:26.338374] INFO [COMMON] retire (ob_kvcache_hazard_version.cpp:289) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=5] erase tenant hazard map node details(ret=0, tenant_id=1002)
[2025-07-10 15:42:26.338402] WDIAG [COMMON] erase_tenant (ob_kvcache_inst_map.cpp:399) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] Still can not destroy cache inst(ret=-4016, inst={cache_id:15, tenant_id:1002, is_delete:true, status:{config:0x55f9943bba68, kv_cnt:0, store_size:0, map_size:0, lru_mb_cnt:0, lfu_mb_cnt:0, base_mb_score:“0.000000000000000000e+00”, hold_size:0}, is_block_cache:false, ref_cnt:1}, inst->status_.store_size_=0, inst->status_.kv_cnt_=0, inst->status_.lfu_mb_cnt_=0, inst->status_.lru_mb_cnt_=0)
[2025-07-10 15:42:26.338450] INFO [COMMON] erase_tenant (ob_kvcache_inst_map.cpp:417) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=19] erase tenant cache inst details(ret=-4016, tenant_id=1002)
[2025-07-10 15:42:26.338454] WDIAG [COMMON] sync_flush_tenant (ob_kv_storecache.cpp:496) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4016] Fail to erase tenant from insts(ret=-4016, tenant_id=1002)
[2025-07-10 15:42:26.338458] INFO [COMMON] sync_flush_tenant (ob_kv_storecache.cpp:499) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] erase tenant cache details(ret=-4016, tenant_id=1002)
[2025-07-10 15:42:26.338462] WDIAG [SERVER.OMT] create_tenant (ob_multi_tenant.cpp:1215) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4052] Fail to sync flush tenant cache(tmp_ret=-4016)
[2025-07-10 15:42:26.338534] INFO [COMMON] operator() (ob_diagnostic_info_container.cpp:663) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=16] target di collector need to be purged(tenant_id=1002, key={tenant_id:1002, group_id:0}, collector={tenant_id:1002, group_id:0, slot_mask:7, is_inited:true, get_uref():1073741824, get_href():2})
[2025-07-10 15:42:26.338553] INFO [COMMON] ~ObDiagnosticInfoCollector (ob_diagnostic_info_summary.cpp:72) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=15] destroy current di collector(this={tenant_id:1002, group_id:0, slot_mask:7, is_inited:true, get_uref():0, get_href():2})
[2025-07-10 15:42:26.338628] INFO [COMMON] purge_tenant_summary (ob_diagnostic_info_container.cpp:671) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=5] success to remove summary collects(tenant_id=1002, this={is_inited:true, tenant_id:1, runnings:{tenant_id:1, slot_count:8, slot_mask:7, is_inited:true}, summarys:{cpu_cnt:4}, cache:{di_cache_queue_.capacity():0, di_cache_queue_.get_total():0, di_cache_queue_.get_free():0}})
[2025-07-10 15:42:26.338662] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=31] set idle finish(ret=0, tenant_id=1002, ctx_id=0, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338668] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=6] set idle finish(ret=0, tenant_id=1002, ctx_id=1, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338671] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=2, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338674] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] set idle finish(ret=0, tenant_id=1002, ctx_id=3, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338677] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] set idle finish(ret=0, tenant_id=1002, ctx_id=4, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338681] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=5, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338684] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=6, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338687] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=7, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338690] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=8, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338693] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=9, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338696] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=10, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338699] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] set idle finish(ret=0, tenant_id=1002, ctx_id=11, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338702] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=12, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338705] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] set idle finish(ret=0, tenant_id=1002, ctx_id=13, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338708] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] set idle finish(ret=0, tenant_id=1002, ctx_id=14, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338726] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=17] set idle finish(ret=0, tenant_id=1002, ctx_id=15, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338730] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] set idle finish(ret=0, tenant_id=1002, ctx_id=16, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338733] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=17, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338746] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=12] set idle finish(ret=0, tenant_id=1002, ctx_id=18, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338750] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] set idle finish(ret=0, tenant_id=1002, ctx_id=19, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338753] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] set idle finish(ret=0, tenant_id=1002, ctx_id=20, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338756] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] set idle finish(ret=0, tenant_id=1002, ctx_id=21, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338761] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] set idle finish(ret=0, tenant_id=1002, ctx_id=22, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338766] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=5] set idle finish(ret=0, tenant_id=1002, ctx_id=23, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338769] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] set idle finish(ret=0, tenant_id=1002, ctx_id=24, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338774] INFO [LIB] set_idle (ob_tenant_ctx_allocator.cpp:390) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] set idle finish(ret=0, tenant_id=1002, ctx_id=25, idle_size=0, chunk_cnt=0)
[2025-07-10 15:42:26.338781] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] wait tenant ctx allocator success(tenant_id=1002, ctx_id=0, ctx_name=“DEFAULT_CTX_ID”)
[2025-07-10 15:42:26.338788] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=6] wait tenant ctx allocator success(tenant_id=1002, ctx_id=1, ctx_name=“DO_NOT_USE_ME”)
[2025-07-10 15:42:26.338790] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=2, ctx_name=“MEMSTORE_CTX_ID”)
[2025-07-10 15:42:26.338793] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] wait tenant ctx allocator success(tenant_id=1002, ctx_id=3, ctx_name=“EXECUTE_CTX_ID”)
[2025-07-10 15:42:26.338795] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=4, ctx_name=“TRANS_CTX_MGR_ID”)
[2025-07-10 15:42:26.338798] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] wait tenant ctx allocator success(tenant_id=1002, ctx_id=5, ctx_name=“PLAN_CACHE_CTX_ID”)
[2025-07-10 15:42:26.338802] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] wait tenant ctx allocator success(tenant_id=1002, ctx_id=6, ctx_name=“WORK_AREA”)
[2025-07-10 15:42:26.338805] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=7, ctx_name=“GLIBC”)
[2025-07-10 15:42:26.338809] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] wait tenant ctx allocator success(tenant_id=1002, ctx_id=8, ctx_name=“CO_STACK”)
[2025-07-10 15:42:26.338811] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=9, ctx_name=“LIBEASY”)
[2025-07-10 15:42:26.338815] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=10, ctx_name=“LOGGER_CTX_ID”)
[2025-07-10 15:42:26.338818] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=11, ctx_name=“KVSTORE_CACHE_ID”)
[2025-07-10 15:42:26.338822] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] wait tenant ctx allocator success(tenant_id=1002, ctx_id=12, ctx_name=“META_OBJ_CTX_ID”)
[2025-07-10 15:42:26.338824] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] wait tenant ctx allocator success(tenant_id=1002, ctx_id=13, ctx_name=“TX_CALLBACK_CTX_ID”)
[2025-07-10 15:42:26.338828] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=14, ctx_name=“LOB_CTX_ID”)
[2025-07-10 15:42:26.338830] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=15, ctx_name=“PS_CACHE_CTX_ID”)
[2025-07-10 15:42:26.338834] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=16, ctx_name=“RPC_CTX_ID”)
[2025-07-10 15:42:26.338837] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] wait tenant ctx allocator success(tenant_id=1002, ctx_id=17, ctx_name=“PKT_NIO”)
[2025-07-10 15:42:26.338840] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] wait tenant ctx allocator success(tenant_id=1002, ctx_id=18, ctx_name=“TX_DATA_TABLE”)
[2025-07-10 15:42:26.338843] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=2] wait tenant ctx allocator success(tenant_id=1002, ctx_id=19, ctx_name=“STORAGE_LONG_TERM_META_CTX_ID”)
[2025-07-10 15:42:26.338847] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4] wait tenant ctx allocator success(tenant_id=1002, ctx_id=20, ctx_name=“MDS_DATA_ID”)
[2025-07-10 15:42:26.338849] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=21, ctx_name=“MDS_CTX_ID”)
[2025-07-10 15:42:26.338853] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=22, ctx_name=“SCHEMA_SERVICE”)
[2025-07-10 15:42:26.338856] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=23, ctx_name=“UNEXPECTED_IN_500”)
[2025-07-10 15:42:26.338859] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=24, ctx_name=“MERGE_RESERVE_CTX_ID”)
[2025-07-10 15:42:26.338862] INFO [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:713) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] wait tenant ctx allocator success(tenant_id=1002, ctx_id=25, ctx_name=“MERGE_NORMAL_CTX_ID”)
[2025-07-10 15:42:26.338883] EDIAG [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:757) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] tenant memory leak!!!(tenant_id=1002, ctx_id=0, ctx_name=“DEFAULT_CTX_ID”, label=“CACHE_INST”, backtrace=“0x26453000 0x9335d95 0xfc78476 0x94e4ea0 0xfba123d 0xfba04cf 0xfabe593 0xfb0707a 0xfb06157 0xfb05f64 0xfbce200 0xfbc6aeb 0xfbb8487 0xfdd1a15 0x1f36a96f 0x13199f39”) BACKTRACE:0x2644d88d 0x973ff35 0x960ef1a 0x960e934 0x960e86b 0x960e680 0xf9b3705 0xf9b2227 0x1319c5ad 0x1c1ee666 0x1c1edd27 0x139a9c9e 0xf9909ac 0x2791ab70 0xf98c82d 0x7f06da97b555 0x9d0975a
[2025-07-10 15:42:26.338993] WDIAG [LIB] recycle_tenant_allocator (ob_malloc_allocator.cpp:773) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4000] failed to recycle tenant allocator immediately(ret=-4000, tenant_id=1002)
[2025-07-10 15:42:26.339010] INFO [SERVER.OMT] create_tenant (ob_multi_tenant.cpp:1224) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=16] finish create new tenant(ret=-4052, tenant_id=1002, write_slog=false, create_step=5, bucket_lock_idx=3968)
[2025-07-10 15:42:26.339026] EDIAG [STORAGE] handle_tenant_create_commit_ (ob_server_storage_meta_replayer.cpp:172) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4052] fail to replay create tenant(ret=-4052, tenant_meta={unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:“NORMAL”, config:{unit_config_id:1001, name:“ocp_meta_unit”, resource:{min_cpu:1, max_cpu:1, memory_size:“1GB”, log_disk_size:“5.4GB”, data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1751427502887439, is_removed:false, hidden_sys_data_disk_config_size:0, actual_data_disk_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}) BACKTRACE:0x2644d88d 0x973ff35 0x983a0f4 0x9839ba5 0x9839af4 0x9839986 0x1c1f5c64 0x1c1ee695 0x1c1edd27 0x139a9c9e 0xf9909ac 0x2791ab70 0xf98c82d 0x7f06da97b555 0x9d0975a
[2025-07-10 15:42:26.339068] EDIAG [STORAGE] apply_replay_result_ (ob_server_storage_meta_replayer.cpp:112) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4052] fail to handle tenant create commit(ret=-4052, tenant_meta={unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:“NORMAL”, config:{unit_config_id:1001, name:“ocp_meta_unit”, resource:{min_cpu:1, max_cpu:1, memory_size:“1GB”, log_disk_size:“5.4GB”, data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:1, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1751427502887439, is_removed:false, hidden_sys_data_disk_config_size:0, actual_data_disk_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}) BACKTRACE:0x2644d88d 0x973ff35 0x983a0f4 0x9839ba5 0x9839af4 0x9839986 0x1c1f54e0 0x1c1ee6c4 0x1c1edd27 0x139a9c9e 0xf9909ac 0x2791ab70 0xf98c82d 0x7f06da97b555 0x9d0975a
[2025-07-10 15:42:26.339097] INFO [STORAGE] apply_replay_result_ (ob_server_storage_meta_replayer.cpp:137) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=27] finish replay create tenants(ret=-4052, tenant_count=3)
[2025-07-10 15:42:26.339105] WDIAG [STORAGE] start_replay (ob_server_storage_meta_replayer.cpp:60) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4052] fail to apply repaly result(ret=-4052)
[2025-07-10 15:42:26.339115] WDIAG [STORAGE] start (ob_server_storage_meta_service.cpp:77) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4052] fail to start replayer(ret=-4052)
[2025-07-10 15:42:26.339119] INFO [STORAGE] start (ob_server_storage_meta_service.cpp:84) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] finish start server storage meta service(ret=-4052, cost_time_us=23450831)
[2025-07-10 15:42:26.339126] EDIAG [SERVER] start (ob_server.cpp:1017) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4052] fail to start server storage meta service(ret=-4052, ret=“OB_LAST_LOG_RUINNED”) BACKTRACE:0x2644d88d 0x973ff35 0x983a0f4 0x9839ba5 0x9839af4 0x9839986 0x139b15d4 0x139ad206 0xf9909ac 0x2791ab70 0xf98c82d 0x7f06da97b555 0x9d0975a
[2025-07-10 15:42:26.339159] ERROR [SERVER] start (ob_server.cpp:1172) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4052] [server_start 9/18] observer instance start fail. you may find solutions in previous error logs or seek help from official technicians.
[2025-07-10 15:42:26.373717] INFO destroy_tg (thread_mgr.cpp:89) [36198][observer][T0][Y0-0000000000000000-0-0] [lt=3] destroy tg(tg_id=277, tg=0x7f06c6a3dcb0, tg->attr_={name:StartupAccelHandler, type:4})
存在4052报错，这个一般可能是由于硬件或者磁盘错误，导致读取到的日志数据有误。
麻烦检查一下硬件问题看看操作系统的message日志是否有报错