observer启动时提示[errcode=-4034]错误

[AntTech_OBYO3G]

【 使用环境 】测试环境
【 OB or 其他组件 】OB
【 使用版本 】4.3.5
【问题描述】observer启动时提示[errcode=-4034]错误
【复现路径】通过命令重启observer：./bin/observer -I 192.168.3.73 -c 1745292874 -p 2881 -P 2882 -z zone3 -n obtest -d /home/admin/oceanbase/store/obtest -r ‘192.168.3.71:2882:2881,192.168.3.229:2882:2881,192.168.3.228:2882:2881,’ -l WARN -o “system_memory=10GB,datafile_size=1200G,log_disk_size=200G,memory_limit_percentage=80,config_additional_dir=/home/admin/data/1/obtest/etc3;/usr/local/data/log1/obtest/etc2”
【附件及日志】
[2025-07-22 16:03:20.805081] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [1658057][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=105] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-102009760972, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:3, 500, 508, 509])
[2025-07-22 16:03:20.829142] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [1658055][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=61] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=0)
[2025-07-22 16:03:20.830568] INFO do_work (ob_rl_mgr.cpp:710) [1658166][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=92] swc wakeup.(stat_period_=1000000, ready=false)
[2025-07-22 16:03:20.871558] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [1658198][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=74] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0)
[2025-07-22 16:03:20.874559] INFO eloop_run (eloop.c:145) [1658147][pnio1][T0][Y0-0000000000000000-0-0] [lt=53] PNIO [ratelimit] time: 1753171400874557, bytes: 23473, bw: 0.002072 MB/s, add_ts: 1007620, add_bytes: 2189
[2025-07-22 16:03:20.876552] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:450) [1658171][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4721] nonblock get location failed(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, cluster_id=1745292874, tenant_id=1, ls_id={id:1})
[2025-07-22 16:03:20.904921] INFO [COMMON] retire (ob_kvcache_hazard_version.cpp:289) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=60] erase tenant hazard map node details(ret=0, tenant_id=1)
[2025-07-22 16:03:20.904927] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:450) [1658056][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4721] nonblock get location failed(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, cluster_id=1745292874, tenant_id=1, ls_id={id:1})
[2025-07-22 16:03:20.905030] INFO [COMMON] erase_tenant (ob_kvcache_inst_map.cpp:415) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=55] erase tenant cache inst details(ret=0, tenant_id=1)
[2025-07-22 16:03:20.905043] INFO [COMMON] sync_flush_tenant (ob_kv_storecache.cpp:549) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=28] erase tenant cache details(ret=0, tenant_id=1)
[2025-07-22 16:03:20.905527] INFO [COMMON] ~ObDiagnosticInfoCollector (ob_diagnostic_info_summary.cpp:71) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=14] destoy current di collector(this={tenant_id:1, group_id:0, slot_mask:63, is_inited:true, get_uref():0, get_href():2})
[2025-07-22 16:03:20.905545] INFO [COMMON] purge_tenant_summary (ob_diagnostic_info_container.cpp:531) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=35] success to remove summary collects(tenant_id=1, this={is_inited:true, tenant_id:1, runnings:{tenant_id:1, slot_count:64, slot_mask:63, is_inited:true}, summarys:{cpu_cnt:48}})
[2025-07-22 16:03:20.905559] INFO [SERVER.OMT] create_tenant (ob_multi_tenant.cpp:1277) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=26] finish create new tenant(ret=-4034, tenant_id=1, write_slog=false, create_step=5, bucket_lock_idx=9780)
[2025-07-22 16:03:20.905571] EDIAG [STORAGE] handle_tenant_create_commit_ (ob_server_storage_meta_replayer.cpp:179) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=23][errcode=-4034] fail to replay create tenant(ret=-4034, tenant_meta={unit:{tenant_id:1, unit_id:3, has_memstore:true, unit_status:“NORMAL”, config:{unit_config_id:1, name:“sys_unit_config”, resource:{min_cpu:4, max_cpu:4, memory_size:“5GB”, log_disk_size:“9GB”, data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1745380741557563, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=2, log_id=518864, offset=63764372}, ls_meta_entry:{[ver=1,mode=0,seq=1247976][2nd=567079]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x18554982 0x1854dcb5 0x1854d3e7 0x109318ce 0xc6130bc 0x2081f770 0xc60ecdd 0x7f46efed7d90 0x7f46efed7e40 0x8437656
[2025-07-22 16:03:20.905667] EDIAG [STORAGE] apply_replay_result_ (ob_server_storage_meta_replayer.cpp:119) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=210][errcode=-4034] fail to handle tenant create commit(ret=-4034, tenant_meta={unit:{tenant_id:1, unit_id:3, has_memstore:true, unit_status:“NORMAL”, config:{unit_config_id:1, name:“sys_unit_config”, resource:{min_cpu:4, max_cpu:4, memory_size:“5GB”, log_disk_size:“9GB”, data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1745380741557563, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=2, log_id=518864, offset=63764372}, ls_meta_entry:{[ver=1,mode=0,seq=1247976][2nd=567079]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x185541fe 0x1854dce4 0x1854d3e7 0x109318ce 0xc6130bc 0x2081f770 0xc60ecdd 0x7f46efed7d90 0x7f46efed7e40 0x8437656
[2025-07-22 16:03:20.905712] INFO [STORAGE] apply_replay_result_ (ob_server_storage_meta_replayer.cpp:144) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=98] finish replay create tenants(ret=-4034, tenant_count=7)
[2025-07-22 16:03:20.905721] WDIAG [STORAGE] start_replay (ob_server_storage_meta_replayer.cpp:67) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=18][errcode=-4034] fail to apply repaly result(ret=-4034)
[2025-07-22 16:03:20.905746] WDIAG [STORAGE] start (ob_server_storage_meta_service.cpp:77) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=19][errcode=-4034] fail to start replayer(ret=-4034)
[2025-07-22 16:03:20.905756] INFO [STORAGE] start (ob_server_storage_meta_service.cpp:82) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=23] finish start server storage meta service(ret=-4034, cost_time_us=12903162)
[2025-07-22 16:03:20.905765] EDIAG [SERVER] start (ob_server.cpp:1061) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=13][errcode=-4034] fail to start server storage meta service(ret=-4034, ret=“OB_DESERIALIZE_ERROR”) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x1093871e 0x10934915 0xc6130bc 0x2081f770 0xc60ecdd 0x7f46efed7d90 0x7f46efed7e40 0x8437656
[2025-07-22 16:03:20.905798] ERROR [SERVER] start (ob_server.cpp:1216) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=26][errcode=-4034] [server_start 9/18] observer instance start fail. you may find solutions in previous error logs or seek help from official technicians.
[2025-07-22 16:03:20.919672] INFO eloop_run (eloop.c:145) [1658155][pnio2][T0][Y0-0000000000000000-0-0] [lt=93] PNIO [ratelimit] time: 1753171400919669, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0
[2025-07-22 16:03:20.939802] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:450) [1658220][Occam][T0][Y0-0000000000000000-0-0] [lt=132][errcode=-4721] nonblock get location failed(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, cluster_id=1745292874, tenant_id=1, ls_id={id:1})
[2025-07-22 16:03:20.939854] WDIAG [SHARE.LOCATION] get_leader_with_retry_until_timeout (ob_location_service.cpp:108) [1658220][Occam][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4721] fail to get log stream location leader with retry until_timeout(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, cluster_id=1745292874, tenant_id=1, ls_id={id:1}, leader=“0.0.0.0:0”, abs_retry_timeout=1753171401139192, retry_interval=200000)
[2025-07-22 16:03:20.939881] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:1937) [1658220][Occam][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4721] get leader with retry until timeout failed(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, tenant_id=1, ls_id={id:1}, leader=“0.0.0.0:0”, cluster_id=1745292874, tmp_abs_timeout_us=1753171401139192, retry_interval_us=200000)
[2025-07-22 16:03:20.939904] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1990) [1658220][Occam][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1)
[2025-07-22 16:03:20.939925] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:1928) [1658220][Occam][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] user tenant has been dropped(ret=0, ret=“OB_SUCCESS”, tenant_id=1)
[2025-07-22 16:03:20.939946] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:450) [1658220][Occam][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4721] nonblock get location failed(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, cluster_id=1745292874, tenant_id=1, ls_id={id:1})
[2025-07-22 16:03:20.941462] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:450) [1658172][Occam][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4721] nonblock get location failed(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, cluster_id=1745292874, tenant_id=1, ls_id={id:1})
[2025-07-22 16:03:20.941513] WDIAG [SHARE.LOCATION] get_leader_with_retry_until_timeout (ob_location_service.cpp:108) [1658172][Occam][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4721] fail to get log stream location leader with retry until_timeout(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, cluster_id=1745292874, tenant_id=1, ls_id={id:1}, leader=“0.0.0.0:0”, abs_retry_timeout=1753171401140946, retry_interval=200000)
[2025-07-22 16:03:20.941556] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:1937) [1658172][Occam][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4721] get leader with retry until timeout failed(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, tenant_id=1, ls_id={id:1}, leader=“0.0.0.0:0”, cluster_id=1745292874, tmp_abs_timeout_us=1753171401140946, retry_interval_us=200000)
[2025-07-22 16:03:20.941584] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1990) [1658172][Occam][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1)
[2025-07-22 16:03:20.941605] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:1928) [1658172][Occam][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] user tenant has been dropped(ret=0, ret=“OB_SUCCESS”, tenant_id=1)
[2025-07-22 16:03:20.941625] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:450) [1658172][Occam][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4721] nonblock get location failed(ret=-4721, ret=“OB_LS_LOCATION_NOT_EXIST”, cluster_id=1745292874, tenant_id=1, ls_id={id:1})
[2025-07-22 16:03:20.967841] INFO destroy_tg (thread_mgr.cpp:91) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=11] destroy tg(tg_id=284, tg=0x7f46e7907cb0, tg->attr_={name:StartupAccelHandler, type:4})
[2025-07-22 16:03:20.967871] EDIAG [SERVER] start (ob_server.cpp:1286) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=60][errcode=-4034] failure occurs, try to set stop and wait(ret=-4034, ret=“OB_DESERIALIZE_ERROR”) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x1093bac8 0x109347b4 0xc6130bc 0x2081f770 0xc60ecdd 0x7f46efed7d90 0x7f46efed7e40 0x8437656
[2025-07-22 16:03:20.967932] ERROR [SERVER] start (ob_server.cpp:1290) [1658044][observer][T0][Y0-0000000000000001-0-0] [lt=33][errcode=-4034] [server_start 10/18] observer start fail, the stop status is true. you may find solutions in previous error logs or seek help from official technicians.

【SOP系列 22 】——故障诊断第一步(自助诊断和诊断信息收集)

【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试，在发帖时输入 [@论坛小助手] 即可召唤小助手，欢迎试用！

[论坛小助手]

根据您提供的信息，似乎遇到了事务处理中的一个特定错误。errcode=-4034 表示遇到了与事务处理相关的错误，但具体的错误类型和原因需要进一步的诊断才能确定。为了更好地理解问题的原因并提供有效的解决方案，我建议您使用 obdiag 工具来收集详细的日志信息。

日志采集场景: 事务其他错误
日志采集命令: obdiag gather scene run --scene=observer.transaction_other_error

请您按照上述命令进行日志的采集，并将生成的日志文件提供给我们以便进一步分析。此外，为了更准确地定位问题，请回答以下几个问题：

1. 您能否提供触发该错误的具体操作或SQL语句？
2. 错误发生前后，系统是否有其他异常现象，例如高CPU、高内存使用率或磁盘空间不足等情况？
3. 测试环境中使用的OceanBase版本是否为最新版本？如果不是，是否有计划升级至最新版本？
4. 是否有其他组件或服务同时在该测试环境中运行，可能会影响OceanBase的正常工作？

这些问题的答案将帮助我们更快地找到问题的根源，并为您提供针对性的解决方案。感谢您的配合。

附上敏捷诊断工具 obdiag 使用帮助链接

[淇铭]

ob是怎么搭建的 是obd搭建的还是ocp搭建的

[AntTech_OBYO3G]

是通过OCP搭建的，现在是单台observer出现故障，OCP内重启失败，所以尝试通过命令重启的

[淇铭]

@AntTech_OBYO3G

192.168.3.71:2882:2881,192.168.3.229:2882:2881,192.168.3.228:2882:2881,’ -l W

ocp截图看一下 节点没有问题 查一下进程 ps -ef | grep observer | grep -v grep

[AntTech_OBYO3G]

是这一部分吗？

[淇铭]

启动这个节点 拿一下这个节点的observer.log日志信息

[AntTech_OBYO3G]

另外，我的一个名为CA的租户，今天上午突然出现不可用状态，OCP中出现两个告警，并且租户的服务名也没了，跟这个服务起不来有关系吗？

[淇铭]

ocp-server.log日志信息发一下

[AntTech_OBYO3G]

这个是-4034那个问题的observer.log，文件太大了，我删了一部分之前时间比较早的日志，留了最近半小时的
observer.log (4.2 MB)

[淇铭]

你在ocp上 重新启动有问题的这个节点 再把observer.log的日志上传一下

[旭辉]

这个OB集群是什么版本？OCP是什么版本？

show variables like '%version_comment%'

在OCP上启动失败 一般黑屏也会启动失败，不建议在黑屏操作启动

这个集群是1-1-1的拓扑结构吗？只有一个节点起不来吗？问题出现前有做过什么操作吗？

另外看到73这台服务器时钟同步服务不存在，可参考这个先解决下
https://www.oceanbase.com/docs/common-ocp-1000000003340101

[AntTech_OBYO3G]

刚试了一下用OCP启动没反应，一直转圈，也没报错

[AntTech_OBYO3G]

@旭辉

show variables like '%version_comment%'

OB版本是4.3.5，OCP版本是4.2.1，拓扑结构最开始是1-1-1，后来又扩展成2-2-2，目前就73一台observer起不来，之前这个服务是正常启动的，但是日志归档总是启动不起来，数据库里也搜不到日志归档的相关信息，所以就想通过重启服务试一下。重启之前为了清理服务器磁盘空间，删了block_file的，这就是出问题之前的全部操作了

[AntTech_OBYO3G]

这个是刚才提到的CA租户告警的ocp-server.log，因为文件有点大，压缩了一下
ocp-server.zip (6.8 MB)

[旭辉]

删了block_file 是指哪里的block_file？

[旭辉]

重启时的第一个observer.log 压缩发下

[AntTech_OBYO3G]

删的是这个下面的

[AntTech_OBYO3G]

这是刚才黑屏重启的那个observer.log，OCP重启一直不生效，没产生log
observer.zip (4.7 MB)

[旭辉]

sstable删掉了，OB的确会起不来