OceanBase 通过Docker容器独立部署后, 稳定运行一段时间后数据库连接超时, 重启容器后, 一直Server is initializing, 数据库起不来

OceanBase 技术问题
#1

【 使用环境 】生产环境
【 OB or 其他组件 】OceanBase
【 使用版本 】5.7.25-OceanBase_CE-v4.3.5.1
【问题描述】内网环境docker-compose部署OceanBase, 单机MODE=MINI独立部署
【复现路径】数据库通过容器部署已经半年, 期间正常运行, 昨天发现数据库连接超时, 也没找到原因, 就重启docker restart 重启容器, 重启后一直Server is initializing, 容器重启成功了, ob数据库没成功, 日志如下【附件及日志】中.

【附件及日志】

数据目录磁盘信息
root@LP086106:/home/app# df -h /home/oceanbase/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu–vg-ubuntu–lv 93G 37G 52G 42% /home/oceanbase/

容器日志如下:
repository/oceanbase-ce-libs/4.3.5.1/99da58cec7132bb3f538ef0a7dc844eb07ab6d05/
repository/oceanbase-ce-libs/4.3.5.1/99da58cec7132bb3f538ef0a7dc844eb07ab6d05/.data
repository/oceanbase-ce-libs/4.3.5.1/99da58cec7132bb3f538ef0a7dc844eb07ab6d05/libaio.so.1.0.1
repository/oceanbase-ce-libs/4.3.5.1/99da58cec7132bb3f538ef0a7dc844eb07ab6d05/libaio.so.1
find obd deploy information, skip configuring…
start ob cluster …
Get local repositories ok
Load cluster param plugin ok
Cluster status check ok
[WARN] OBD-1007: (127.0.0.1) The recommended number of core file size is unlimited (Current value: 0)
[WARN] OBD-1007: (127.0.0.1) The recommended number of stack size is unlimited (Current value: 8192)
cluster scenario: express_oltp
Start observer ok
observer program health check ok
Connect to observer x
[ERROR] OBD-1006: Failed to connect to oceanbase-ce
See https://www.oceanbase.com/product/ob-deployer/error-codes
Trace ID: 36eb0d58-3d25-11f1-9323-0242ac160002

If you want to view detailed obd logs, please run: obd display-trace 36eb0d58-3d25-11f1-9323-0242ac160002

boot success!

obd display-trace 36eb0d58-3d25-11f1-9323-0242ac160002 执行结果如下:

26-04-21 02:00:51.239 [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
26-04-21 02:00:51.241 [ERROR] Traceback (most recent call last):
26-04-21 02:00:51.241 [ERROR] File “core.py”, line 2089, in start_cluster
26-04-21 02:00:51.241 [ERROR] File “core.py”, line 2139, in start_cluster
26-04-21 02:00:51.241 [ERROR] File “core.py”, line 228, in run_workflow
26-04-21 02:00:51.241 [ERROR] File “core.py”, line 270, in run_plugin_template
26-04-21 02:00:51.241 [ERROR] File “core.py”, line 315, in call_plugin
26-04-21 02:00:51.241 [ERROR] File “plugin.py”, line 347, in call
26-04-21 02:00:51.241 [ERROR] File “plugin.py”, line 304, in new_func
26-04-21 02:00:51.242 [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 68, in connect
26-04-21 02:00:51.242 [ERROR] cursor = Cursor(ip=server_ip, port=server_config.get(‘mysql_port’, 2881), tenant=’’, password=’*****’ if password is not None else ‘’, stdio=stdio)
26-04-21 02:00:51.242 [ERROR] File “st dio.py”, line 1012, in wrapper
26-04-21 02:00:51.242 [ERROR] File “st dio.py”, line 999, in func_wrapper
26-04-21 02:00:51.242 [ERROR] File “tool.py”, line 763, in init
26-04-21 02:00:51.242 [ERROR] File “tool.py”, line 793, in _connect
26-04-21 02:00:51.242 [ERROR] File “pymysql/connections.py”, line 353, in init
26-04-21 02:00:51.242 [ERROR] File “pymysql/connections.py”, line 633, in connect
26-04-21 02:00:51.242 [ERROR] File “pymysql/connections.py”, line 907, in _request_authentication
26-04-21 02:00:51.242 [ERROR] File “pymysql/connections.py”, line 725, in _read_packet
26-04-21 02:00:51.242 [ERROR] File “pymysql/protocol.py”, line 222, in raise_for_error
26-04-21 02:00:51.242 [ERROR] File “pymysql/err.py”, line 143, in raise_mysql_exception
26-04-21 02:00:51.242 [ERROR] pymysql.err.OperationalError: (8001, ‘Server is initializing[0.0.0.0:0] [2026-04-21 02:00:51.240110] [Y0-00064FEEB9DA198D-0-0]’)

26-04-21 02:00:51.242 [ERROR] OBD-1006: Failed to connect to oceanbase-ce
26-04-21 02:00:54.244 [DEBUG] - sub connect ref count to 0
26-04-21 02:00:54.244 [DEBUG] - export connect
26-04-21 02:00:54.245 [DEBUG] - plugin oceanbase-ce-py script connect-3.1.0 result: False
26-04-21 02:00:54.251 [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
26-04-21 02:00:54.251 [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
26-04-21 02:00:54.251 [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
26-04-21 02:00:54.251 [DEBUG] - exclusive lock /root/.obd/lock/deploy_obcluster release, count 0
26-04-21 02:00:54.251 [DEBUG] - unlock /root/.obd/lock/deploy_obcluster
26-04-21 02:00:54.251 [DEBUG] - share lock /root/.obd/lock/global release, count 0
26-04-21 02:00:54.251 [DEBUG] - unlock /root/.obd/lock/global

【备注】docker-compose 配置如下:
version: ‘2.1’
services:
oceanbase-db:
networks:
backend:
image: Quay
volumes:
- /home/oceanbase/ob:/root/ob
- /home/oceanbase/obd/cluster:/root/.obd/cluster
restart: always
environment:
# 部署 mini 的独立实例
- MODE=MINI
- OB_SERVER_IP=127.0.0.1
- OB_TENANT_NAME=ob
- OB_TENANT_PASSWORD=iletek@ob
ports:
- “2881:2881”

1 个赞
#3

26-04-21 02:00:51.242 [ERROR] pymysql.err.OperationalError: (8001, ‘Server is initializing[0.0.0.0:0] [2026-04-21 02:00:51.240110] [Y0-00064FEEB9DA198D-0-0]’)
你把这个时间段的observer.log日志 发一下

#4

新用户无法上传附件, observer.log 最新片段如下:
026-04-21 02:51:56.517583] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=707393, *p_task_={this:0x7fb1ca4e10d0, caller:ob_event_history_table_operator.cpp:add_event_to_timer_:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca4bfc30, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:08.910, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.517934] INFO [CLOG] wait_replay_service_replay_done_ (ob_role_change_service.cpp:855) [348][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=24] wait replay done return false, need retry(ls_id={id:1}, end_lsn={lsn:108312203502}, start_ts=1776739914714497)
[2026-04-21 02:51:56.518694] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CA9FFD-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.518810] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] task executed too delay(delay_time=708618, *p_task={this:0x7fb1ca4bfd30, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca465c30, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:08.910, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.519075] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:118) [609][T1002_TimerWK0_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=79] success to notify_need_writing_throttling True
[2026-04-21 02:51:56.519793] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CA9FFE-0-0] [lt=50] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.519871] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=709676, *p_task={this:0x7fb1ca465d30, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca4c9bb0, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:08.911, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.519959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [241][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR=“cgroup”, ret=-4027)
[2026-04-21 02:51:56.520773] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CA9FFF-0-0] [lt=22] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.520859] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=710660, *p_task={this:0x7fb1ca4c9cb0, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca50a850, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:09.211, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.522906] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA000-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.522983] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=712782, *p_task={this:0x7fb1ca45fa30, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca4d2ad0, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:09.310, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.524037] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA001-0-0] [lt=21] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.524110] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] task executed too delay(delay_time=713907, *p_task={this:0x7fb1ca4d2bd0, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca4e3b70, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:09.311, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.525133] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA002-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.525206] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=715000, *p_task={this:0x7fb1ca4e3c70, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb2ad3e1550, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:09.311, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.526225] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA003-0-0] [lt=20] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.526314] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=716104, *p_task={this:0x7fb2ad3e1650, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb2ad3e1930, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:09.611, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.527035] EDIAG [STORAGE.BLKMGR] alloc_object (ob_block_manager.cpp:270) [610][T1002_TimerWK1_WriteCkpt][T1002][YB427F000001-00064FEEBBEA19A8-0-0] [lt=0][errcode=-4184] The data file disk space is exhausted. Please expand the capacity by resizing datafile!!!(ret=-4184) BACKTRACE:0x9510a36 0x90911a6 0x9160f76 0x9160886 0x91607ba 0x91605b1 0x191f7008 0x191f611f 0x19218751 0x96f6669 0x96fa418 0x1861ebfe 0x18614054 0x1860d295 0x1860aa43 0x1860a27f 0x192f36a0 0x192d2d10 0x192d1c94 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7fb2dfa081cf 0x7fb2df639dd3
[2026-04-21 02:51:56.527391] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA004-0-0] [lt=48] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.527482] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=717271, *p_task={this:0x7fb1ca41a7f0, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca41aad0, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:09.711, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.528838] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA005-0-0] [lt=159] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.528940] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:118) [498][T1001_TimerWK3_PalfGC][T1001][Y0-0000000000000000-0-0] [lt=36] success to notify_need_writing_throttling True
[2026-04-21 02:51:56.528947] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=718731, *p_task={this:0x7fb1ca5407f0, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca41ac30, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:09.711, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.529953] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA006-0-0] [lt=19] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.530040] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=719822, *p_task={this:0x7fb1ca540a70, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca540d50, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:09.711, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.530093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [241][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR=“cgroup”, ret=-4027)
[2026-04-21 02:51:56.531113] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA007-0-0] [lt=36] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.531203] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] task executed too delay(delay_time=720981, *p_task={this:0x7fb1ca5007f0, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca500ad0, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:05.961, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.531329] WDIAG [SHARE] avaliable_resource_after_dt_ (ob_throttle_unit.ipp:412) [397][T1_ReplaySrv11][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth]
[2026-04-21 02:51:56.532270] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA008-0-0] [lt=50] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.532350] WDIAG [SHARE] operator() (ob_event_history_table_operator.cpp:414) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] fail to schedule report event task cause retry too much times
[2026-04-21 02:51:56.532370] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] task executed too delay(delay_time=722146, *p_task={this:0x7fb1ca500bd0, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb1ca500eb0, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:05.962, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.533368] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA009-0-0] [lt=43] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.533440] WDIAG [SHARE] operator() (ob_event_history_table_operator.cpp:414) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] fail to schedule report event task cause retry too much times
[2026-04-21 02:51:56.533472] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] task executed too delay(delay_time=723246, *p_task={this:0x7fb2ad3c6030, caller:ob_event_history_table_operator.cpp:add_event_to_timer:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$2268, timer_running_flag:1, total_running_count:1963, func_shared_ptr.ptr:0x7fb2ad3c6310, time_wheel:0x7fb2d6ff7cf0, thread_pool:0x7fb2d6ff74d0, is_running:0, time_interval:15.00s, expected_run_time:02:52:05.963, task_priority:1, with_handle_protected:0})
[2026-04-21 02:51:56.534482] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [119][Occam][T1][YB427F000001-00064FEEF8CAA00A-0-0] [lt=47] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 02:51:56.534583] WDIAG [SHARE] operator() (ob_event_history_table_operator.cpp:414) [119][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] fail to schedule report event task cause retry too much times

#5

看着这日志不对呀 你在重启一下 把observer.log日志 发一下

#6

[2026-04-21 08:11:20.075958] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [634][T1002_LogRessvr][T1002][YB427F000001-00064FF3E0B0F254-0-0] [lt=4][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001)
[2026-04-21 08:11:20.075966] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [634][T1002_LogRessvr][T1002][YB427F000001-00064FF3E0B0F254-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1, tenant_id=1001)
[2026-04-21 08:11:20.075976] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [634][T1002_LogRessvr][T1002][YB427F000001-00064FF3E0B0F254-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f955f985ae0, start=1776759080075888, sql=select tenant_id, id, type, value, recovery_until_scn from all_log_restore_source where tenant_id=1002 and id=1)
[2026-04-21 08:11:20.075987] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [634][T1002_LogRessvr][T1002][YB427F000001-00064FF3E0B0F254-0-0] [lt=11][errcode=-5627] read failed(ret=-5627)
[2026-04-21 08:11:20.075991] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [634][T1002_LogRessvr][T1002][YB427F000001-00064FF3E0B0F254-0-0] [lt=4][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from all_log_restore_source where tenant_id=1002 and id=1)
[2026-04-21 08:11:20.076033] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [634][T1002_LogRessvr][T1002][YB427F000001-00064FF3E0B0F254-0-0] [lt=9][errcode=-5627] table_operator get_source failed(ret=-5627)
[2026-04-21 08:11:20.076041] WDIAG [CLOG] get_source (ob_remote_location_adaptor.cpp:146) [634][T1002_LogRessvr][T1002][YB427F000001-00064FF3E0B0F254-0-0] [lt=8][errcode=-5627] get source failed(ret=-5627)
[2026-04-21 08:11:20.076044] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [634][T1002_LogRessvr][T1002][YB427F000001-00064FF3E0B0F254-0-0] [lt=3][errcode=-5627] get source failed(tenant_id=1002)
[2026-04-21 08:11:20.076049] WDIAG [CLOG] do_thread_task (ob_log_restore_service.cpp:181) [634][T1002_LogRessvr][T1002][YB427F000001-00064FF3E0B0F254-0-0] [lt=3][errcode=-5627] update_upstream failed
[2026-04-21 08:11:20.077540] EDIAG [STORAGE.BLKMGR] alloc_object (ob_block_manager.cpp:270) [331][T1_TimerWK2_WriteCkpt][T1][YB427F000001-00064FF3DF70F277-0-0] [lt=0][errcode=-4184] The data file disk space is exhausted. Please expand the capacity by resizing datafile!!!(ret=-4184) BACKTRACE:0x9510a36 0x90911a6 0x9160f76 0x9160886 0x91607ba 0x91605b1 0x191f7008 0x191f611f 0x19218751 0x96f6669 0x96fa418 0x1861ebfe 0x18614054 0x1860d295 0x1860aa43 0x1860a27f 0x192f36a0 0x192d2d10 0x192d1c94 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f9568c081cf 0x7f9568839dd3
[2026-04-21 08:11:20.077853] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [240][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR=“cgroup”, ret=-4027)
[2026-04-21 08:11:20.085467] EDIAG [STORAGE.BLKMGR] alloc_object (ob_block_manager.cpp:270) [611][T1002_TimerWK2_WriteCkpt][T1002][YB427F000001-00064FF3E390F28D-0-0] [lt=0][errcode=-4184] The data file disk space is exhausted. Please expand the capacity by resizing datafile!!!(ret=-4184) BACKTRACE:0x9510a36 0x90911a6 0x9160f76 0x9160886 0x91607ba 0x91605b1 0x191f7008 0x191f611f 0x19218751 0x96f6669 0x96fa418 0x1861ebfe 0x18614054 0x1860d295 0x1860aa43 0x1860a27f 0x192f36a0 0x192d2d10 0x192d1c94 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f9568c081cf 0x7f9568839dd3
[2026-04-21 08:11:20.086221] INFO [CLOG] wait_replay_service_replay_done_ (ob_role_change_service.cpp:855) [630][T1002_RCSrv0][T1002][Y0-0000000000000000-0-0] [lt=31] wait replay done return false, need retry(ls_id={id:1}, end_lsn={lsn:33200823411}, start_ts=1776759079635377)
[2026-04-21 08:11:20.086452] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [534][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:“post cluster heartbeat rpc failed, tenant_id=1001”, warnings:[]}, tenant_id_=1001, dst=“127.0.0.1:2882”, cluster_service_tablet_id={id:226})
[2026-04-21 08:11:20.086476] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [534][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret=“OB_NEED_WAIT”, tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1776759080086448)
[2026-04-21 08:11:20.086485] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [534][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret=“OB_NEED_WAIT”, tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1776759079886410, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master=“0.0.0.0:0”)
[2026-04-21 08:11:20.086496] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [534][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret=“OB_SUCCESS”, tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0)
[2026-04-21 08:11:20.086519] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [534][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=781255689435, log_service_ts=1775977824396718, min_tx_service_ts=4611686018427387, timestamp={val:1775977824396718001, v:0})
[2026-04-21 08:11:20.086535] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [534][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1775977823394795001, v:0}", last_lsn={lsn:195382239638}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0})
[2026-04-21 08:11:20.087951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [240][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR=“cgroup”, ret=-4027)
[2026-04-21 08:11:20.090521] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:118) [609][T1002_TimerWK0_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=93] success to notify_need_writing_throttling True
[2026-04-21 08:11:20.090539] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1159) [609][T1002_TimerWK0_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:“Dummy”}, self:“127.0.0.1:2882”, log_dir:"/root/ob/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2560, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2560, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):2303, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}})
[2026-04-21 08:11:20.091192] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:118) [1310][T1_TimerWK5_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15] success to notify_need_writing_throttling True
[2026-04-21 08:11:20.091211] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1159) [1310][T1_TimerWK5_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:“Dummy”}, self:“127.0.0.1:2882”, log_dir:"/root/ob/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1796, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}})
[2026-04-21 08:11:20.092214] INFO [CLOG] wait_replay_service_replay_done_ (ob_role_change_service.cpp:855) [347][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=18] wait replay done return false, need retry(ls_id={id:1}, end_lsn={lsn:108312203502}, start_ts=1776759079992097)
[2026-04-21 08:11:20.092477] EDIAG [STORAGE.BLKMGR] alloc_object (ob_block_manager.cpp:270) [496][T1001_TimerWK2_WriteCkpt][T1001][YB427F000001-00064FF3E160F275-0-0] [lt=0][errcode=-4184] The data file disk space is exhausted. Please expand the capacity by resizing datafile!!!(ret=-4184) BACKTRACE:0x9510a36 0x90911a6 0x9160f76 0x9160886 0x91607ba 0x91605b1 0x191f7008 0x191f611f 0x19218751 0x96f6669 0x96fa418 0x1861ebfe 0x18614054 0x1860d295 0x1860aa43 0x1860a27f 0x192f36a0 0x192d2d10 0x192d1c94 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f9568c081cf 0x7f9568839dd3
[2026-04-21 08:11:20.094555] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [330][T1_TimerWK1_FreInfoReload][T1][YB427F000001-00064FF3DFA0F315-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1776759080027336]})
[2026-04-21 08:11:20.098059] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [240][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR=“cgroup”, ret=-4027)
[2026-04-21 08:11:20.099241] EDIAG [STORAGE.BLKMGR] alloc_object (ob_block_manager.cpp:270) [496][T1001_TimerWK2_WriteCkpt][T1001][YB427F000001-00064FF3E160F275-0-0] [lt=0][errcode=-4184] The data file disk space is exhausted. Please expand the capacity by resizing datafile!!!(ret=-4184) BACKTRACE:0x9510a36 0x90911a6 0x9160f76 0x9160886 0x91607ba 0x91605b1 0x191f7008 0x191f611f 0x19218751 0x96f6669 0x96fa418 0x1861ebfe 0x18614054 0x1860d295 0x1860aa43 0x1860a27f 0x192f36a0 0x192d2d10 0x192d1c94 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f9568c081cf 0x7f9568839dd3
[2026-04-21 08:11:20.100658] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:118) [497][T1001_TimerWK3_PalfGC][T1001][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True
[2026-04-21 08:11:20.100677] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1159) [497][T1001_TimerWK3_PalfGC][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:“Dummy”}, self:“127.0.0.1:2882”, log_dir:"/root/ob/store/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):422, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}})
[2026-04-21 08:11:20.100736] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [153][Occam][T1][YB427F000001-00064FF3E3213892-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 08:11:20.100794] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [153][Occam][T1][YB427F000001-00064FF3E3213892-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret=“OB_SUCCESS”)
[2026-04-21 08:11:20.101878] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [153][Occam][T1][YB427F000001-00064FF3E3213893-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=all_server_event_history, table_name.ptr()=“data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279”, ret=-5019)
[2026-04-21 08:11:20.101924] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [153][Occam][T1][YB427F000001-00064FF3E3213893-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret=“OB_SUCCESS”)
[2026-04-21 08:11:20.102385] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [329][T1_TimerWK0_LSMetaCh][T1][YB427F000001-00064FF3E350F25D-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=all_ls_meta_table, table_name.ptr()=“data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65”, ret=-5019)
[2026-04-21 08:11:20.102432] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [329][T1_TimerWK0_LSMetaCh][T1][YB427F000001-00064FF3E350F25D-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret=“OB_SUCCESS”)
[2026-04-21 08:11:20.106098] EDIAG [STORAGE.BLKMGR] alloc_object (ob_block_manager.cpp:270) [611][T1002_TimerWK2_WriteCkpt][T1002][YB427F000001-00064FF3E390F28D-0-0] [lt=0][errcode=-4184] The data file disk space is exhausted. Please expand the capacity by resizing datafile!!!(ret=-4184) BACKTRACE:0x9510a36 0x90911a6 0x9160f76 0x9160886 0x91607ba 0x91605b1 0x191f7008 0x191f611f 0x19218751 0x96f6669 0x96fa418 0x1861ebfe 0x18614054 0x1860d295 0x1860aa43 0x1860a27f 0x192f36a0 0x192d2d10 0x192d1c94 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f9568c081cf 0x7f9568839dd3
[2026-04-21 08:11:20.107364] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:290) [402][T1_ReplaySrv17][T1][Y0-0000000000000000-0-0] [lt=122] success to update reserved snapshot(ret=0, ls_id={id:1}, min_reserved_snapshot=1776112641834046000)
[2026-04-21 08:11:20.107387] INFO [CLOG] do_replay_task (ob_log_replay_service.cpp:994) [402][T1_ReplaySrv17][T1][Y0-0000000000000000-0-0] [lt=22] pre barrier log replay succ(replay_task={ls_id:{id:1}, log_type:17, log_type:“RESERVED_SNAPSHOT”, lsn:{lsn:107828236820}, scn:{val:1776114448812637001, v:0}, is_pre_barrier:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1776114448943612, first_handle_ts_:1776759080107354, replay_cost_:32, retry_cost_:-1, read_log_buf_:0x7f947081f790, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:34, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:18811, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1776759080107395, err_info_:{has_fatal_error_:false, fail_ts_:1776759080106903, fail_cost_:365192057, ret_code_:-4023}}, next_to_submit_lsn_:{lsn:107836335076}, next_to_submit_scn_:{val:1776116542594879002, v:0}, base_lsn_:{lsn:106495266816}, base_scn_:{val:1775831797661091000, v:0}, iterator_:{iterator_impl:{buf_:0x7f94fd205000, next_round_pread_size:3678208, curr_read_pos:3590617, curr_read_buf_start_pos:0, curr_read_buf_end_pos:3678208, log_storage_:{start_lsn:{lsn:107832744515}, end_lsn:{lsn:107836422723}, read_buf:{buf_len_:3747840, buf_:0x7f94fd205000}, block_size:67104768, log_storage_:0x7f9537d81bf0, log_storage_:{log_tail:{lsn:108312203502}, readable_log_tail:{lsn:108312203502}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:1614, palf_id:1}, block_mgr:{log_dir:"/root/ob/store/clog/tenant_1/1/log", dir_fd:124, min_block_id:1586, max_block_id:1615, curr_writable_block_id:1614}, logical_block_size_:67104768, curr_block_writable_size_:61996818, block_header_serialize_buf_:0x7f9537d828b8, flashback_version:0}, storage_type:“DISK_STORAGE”}, curr_entry_is_raw_write:false, curr_entry_size:66, prev_entry_scn:{val:1776116542594879002, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1776116542594879002, v:0}, data_checksum:2923900223, flag:50854}}, init_mode_version:0, accumulate_checksum:1293854875, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1776116542594879002, v:0}}, io_ctx:{user:“REPLAY”, palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:379, cache_read_size:0, hit_ratio:“0.000000000000000000e+00”}, cold_cache_stat:{hit_cnt:1, miss_cnt:379, cache_read_size:195, hit_ratio:“2.631578947368420993e-03”}, read_io_cnt:708, read_io_size:1353310208, read_disk_cost_ts:3414196, start_lsn:{lsn:106495266816}}}}}}, replay_queue_idx=28, ret=0)
[2026-04-21 08:11:20.108140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [240][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR=“cgroup”, ret=-4027)
[2026-04-21 08:11:20.113346] EDIAG [STORAGE.BLKMGR] alloc_object (ob_block_manager.cpp:270) [611][T1002_TimerWK2_WriteCkpt][T1002][YB427F000001-00064FF3E390F28D-0-0] [lt=0][errcode=-4184] The data file disk space is exhausted. Please expand the capacity by resizing datafile!!!(ret=-4184) BACKTRACE:0x9510a36 0x90911a6 0x9160f76 0x9160886 0x91607ba 0x91605b1 0x191f7008 0x191f611f 0x19218751 0x96f6669 0x96fa418 0x1861ebfe 0x18614054 0x1860d295 0x1860aa43 0x1860a27f 0x192f36a0 0x192d2d10 0x192d1c94 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f9568c081cf 0x7f9568839dd3
[2026-04-21 08:11:20.115661] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221504458, support_ssl=false)
[2026-04-21 08:11:20.115728] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221504458, sess.client_addr_=“192.111.81.106:57478”)
[2026-04-21 08:11:20.115791] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=58] accept one succ(*s={this:0x7f9457216fb0, session_id:3221504458, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:151}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false})
[2026-04-21 08:11:20.115996] INFO [SHARE.SCHEMA] get_tenant_info (ob_schema_getter_guard.cpp:2110) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] tenant not exist(tenant_name=ob)
[2026-04-21 08:11:20.116016] WDIAG [SHARE.SCHEMA] get_tenant_id (ob_schema_getter_guard.cpp:394) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5160] Can not find tenant(tenant_name=ob)
[2026-04-21 08:11:20.116022] WDIAG [SERVER] extract_tenant_id (obmp_connect.cpp:106) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5160] get_tenant_id failed(ret=-5160, tenant_name=ob)
[2026-04-21 08:11:20.116032] WDIAG [SERVER] deliver_mysql_request (ob_srv_deliver.cpp:758) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5160] extract tenant_id fail(ret=-5160, tenant_name=ob, tenant_id=18446744073709551615)
[2026-04-21 08:11:20.116067] WDIAG [SERVER] process (obmp_connect.cpp:326) [95][MysqlQueueTh1][T0][Y0-00064FF3DEB0F3DE-0-0] [lt=19][errcode=-8001] server is initializing(ret=-8001)
[2026-04-21 08:11:20.116100] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [95][MysqlQueueTh1][T0][Y0-00064FF3DEB0F3DE-0-0] [lt=24] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()=“0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340cbe 0x7f9568c081cf 0x7f9568839dd3”)
[2026-04-21 08:11:20.116113] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [95][MysqlQueueTh1][T0][Y0-00064FF3DEB0F3DE-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221504458, proxy_sessid=0)
[2026-04-21 08:11:20.116139] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [95][MysqlQueueTh1][T0][Y0-00064FF3DEB0F3DE-0-0] [lt=13][errcode=0] server close connection(sessid=3221504458, proxy_sessid=0, stack=“0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340cbe 0x7f9568c081cf 0x7f9568839dd3”)
[2026-04-21 08:11:20.116149] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [95][MysqlQueueTh1][T0][Y0-00064FF3DEB0F3DE-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221504458, proxy_sessid=0)
[2026-04-21 08:11:20.116153] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [95][MysqlQueueTh1][T0][Y0-00064FF3DEB0F3DE-0-0] [lt=3][errcode=-4016] session is null
[2026-04-21 08:11:20.116159] INFO [SERVER] process (obmp_connect.cpp:504) [95][MysqlQueueTh1][T0][Y0-00064FF3DEB0F3DE-0-0] [lt=4] MySQL LOGIN(direct_client_ip=“192.111.81.106”, client_ip=, tenant_name=ob, tenant_id=18446744073709551615, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221504458, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol=“OB_MYSQL_CS_TYPE”, autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0)
[2026-04-21 08:11:20.116236] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f9457216fb0, session_id:3221504458, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:151}, err:5, last_decode_time:1776759080115359, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:237, pending_flag:1, may_handling_flag:true, handler_close_flag:false})
[2026-04-21 08:11:20.116270] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] kill and revert session(conn.sessid_=3221504458, proxy_sessid=0, ret=0)
[2026-04-21 08:11:20.116280] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f9457216fb0, session_id:3221504458, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:151}, err:5, last_decode_time:1776759080115359, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:237, pending_flag:1, may_handling_flag:false, handler_close_flag:false})
[2026-04-21 08:11:20.116296] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [179][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221504458)

#7

看着报错信息是磁盘空间满了 检查一下磁盘空间

#8

4c7806ff-b5a8-418e-8820-8c5e6a649d7a
4c7806ff-b5a8-418e-8820-8c5e6a649d7a693×102 11.4 KB

磁盘空间还有的

#9

可能是你设置的datafile_size太小了 你看看datafile_size设置的多大 obd的yaml文件发一下 看看

#10

e2dff551-d524-4c74-8a95-82010cf40ba9
直接修改然后重启容器么

#11

obd cluster edit-config [cluster_name] 修改一下 log_disk_size也小 建议是内存的三倍

#12

cluster_name 在上面配置文件加一行么

#13

obd cluster list --查看一下集群名 不指定集群名 不知道要修改哪个集群 你可以看看文档了解一下命令的使用
https://www.oceanbase.com/docs/common-obd-cn-1000000005623684

#14

可以了, 谢谢