【 使用环境 】测试环境
【 使用版本 】4.2.
【问题描述】启动失败
obd cluster start watchpoints //vmare 虚拟机 启动很长时间
配置文件:obdeploy/example/mini-local-example.yaml at master · oceanbase/obdeploy · GitHub
Get local repositories ok
Search plugins ok
Open ssh connection ok
Load cluster param plugin ok
Check before start observer ok
Start observer ok
observer program health check ok
Connect to observer ok
observer need bootstarp x
See https://www.oceanbase.com/product/ob-deployer/error-codes .
Trace ID: 808f2a7a-0286-11ee-a0ef-000c29378c15
If you want to view detailed obd logs, please run: obd display-trace 808f2a7a-0286-11ee-a0ef-000c29378c15
日志
] failed to resolve insert filed(ret=-5019)
[2023-06-04 11:18:59.255827] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:95) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=3][errcode=-5019] resolve single table insert failed(ret=-5019)
[2023-06-04 11:18:59.255830] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:156) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3069)
[2023-06-04 11:18:59.255842] WDIAG [SQL] generate_stmt (ob_sql.cpp:2708) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019)
[2023-06-04 11:18:59.255846] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2829) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false)
[2023-06-04 11:18:59.255853] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4596) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false)
[2023-06-04 11:18:59.255856] WDIAG [SQL] handle_text_query (ob_sql.cpp:2422) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019)
[2023-06-04 11:18:59.255863] WDIAG [SQL] stmt_query (ob_sql.cpp:205) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event) VALUES (usec_to_time(1685848739254721), 'observice', 'register'), ret=-5019)
[2023-06-04 11:18:59.255867] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:672) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019)
[2023-06-04 11:18:59.255870] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:819) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event) VALUES (usec_to_time(1685848739254721), 'observice', 'register')"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1)
[2023-06-04 11:18:59.255884] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:869) [5523][EvtHisUpdTask][T1][YB427F000001-0005FD453F1A8554-0-0] [lt=6][errcode=-5019] [RETRY] check if need retr
[2023-06-04 11:18:40.951] [DEBUG] -- local execute: df --block-size=1024
[2023-06-04 11:18:40.956] [DEBUG] -- exited code 0
[2023-06-04 11:18:40.956] [DEBUG] -- get disk info for path /dev, total: 4621701120 avail: 4621701120
[2023-06-04 11:18:40.956] [DEBUG] -- get disk info for path /dev/shm, total: 4651732992 avail: 4651732992
[2023-06-04 11:18:40.956] [DEBUG] -- get disk info for path /run, total: 4651732992 avail: 4641632256
[2023-06-04 11:18:40.957] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 4651732992 avail: 4651732992
[2023-06-04 11:18:40.957] [DEBUG] -- get disk info for path /, total: 39675510784 avail: 10722549760
[2023-06-04 11:18:40.957] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 703868928
[2023-06-04 11:18:40.957] [DEBUG] -- get disk info for path /run/user/42, total: 930344960 avail: 930332672
[2023-06-04 11:18:40.957] [DEBUG] -- get disk info for path /run/user/0, total: 930344960 avail: 930344960
[2023-06-04 11:18:40.957] [DEBUG] -- get disk info for path /mnt/tidb, total: 20957446144 avail: 16855334912
[2023-06-04 11:18:40.957] [DEBUG] -- get disk info for path /mnt/oceanbase, total: 20957446144 avail: 3730812928
[2023-06-04 11:18:40.957] [DEBUG] -- get disk info for path /mnt/ob, total: 105089261568 avail: 72812974080
[2023-06-04 11:18:40.957] [DEBUG] -- disk: {'/dev': {'total': 4621701120, 'avail': 4621701120, 'need': 0}, '/dev/shm': {'total': 4651732992, 'avail': 4651732992, 'need': 0}, '/run': {'total': 4651732992, 'avail': 4641632256, 'need': 0}, '/sys/fs/cgroup': {'total': 4651732992, 'avail': 4651732992, 'need': 0}, '/': {'total': 39675510784, 'avail': 10722549760, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 703868928, 'need': 0}, '/run/user/42': {'total': 930344960, 'avail': 930332672, 'need': 0}, '/run/user/0': {'total': 930344960, 'avail': 930344960, 'need': 0}, '/mnt/tidb': {'total': 20957446144, 'avail': 16855334912, 'need': 0}, '/mnt/oceanbase': {'total': 20957446144, 'avail': 3730812928, 'need': 0}, '/mnt/ob': {'total': 105089261568, 'avail': 72812974080, 'need': 0}}
[2023-06-04 11:18:40.957] [DEBUG] -- local execute: date +%s%N
[2023-06-04 11:18:40.960] [DEBUG] -- exited code 0
[2023-06-04 11:18:40.961] [DEBUG] -- 127.0.0.1 time delta -0.491455078125
[2023-06-04 11:18:41.035] [DEBUG] - sub start_check ref count to 0
[2023-06-04 11:18:41.035] [DEBUG] - export start_check
[2023-06-04 11:18:41.035] [DEBUG] - Call oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.2.0.0-20230603214048-83515d6223e27ae54d23fa6cacbd89a5
[2023-06-04 11:18:41.035] [DEBUG] - import start
[2023-06-04 11:18:41.036] [DEBUG] - add start ref count to 1
[2023-06-04 11:18:41.037] [INFO] Start observer
[2023-06-04 11:18:41.037] [DEBUG] -- local execute: ls /mnt/ob/data/clog/tenant_1/
[2023-06-04 11:18:41.042] [DEBUG] -- exited code 0
[2023-06-04 11:18:41.042] [DEBUG] -- local execute: cat /mnt/ob/observer/run/observer.pid
[2023-06-04 11:18:41.046] [DEBUG] -- exited code 0
[2023-06-04 11:18:41.046] [DEBUG] -- local execute: ls /proc/3890
[2023-06-04 11:18:41.050] [DEBUG] -- exited code 2, error output:
[2023-06-04 11:18:41.050] [DEBUG] ls: 无法访问'/proc/3890': 没有那个文件或目录
[2023-06-04 11:18:41.050] [DEBUG]
[2023-06-04 11:18:41.050] [DEBUG] -- 127.0.0.1 start command construction
[2023-06-04 11:18:41.050] [DEBUG] -- starting 127.0.0.1 observer
[2023-06-04 11:18:41.051] [DEBUG] -- root@127.0.0.1 set env LD_LIBRARY_PATH to '/mnt/ob/observer/lib:'
[2023-06-04 11:18:41.051] [DEBUG] -- local execute: cd /mnt/ob/observer; /mnt/ob/observer/bin/observer -r '127.0.0.1:2882:2881' -p 2881 -P 2882 -z 'zone1' -c 1 -d '/mnt/ob/data' -i 'lo' -o __min_full_resource_pool_memory=2147483648,memory_limit='5G',system_memory='1G',datafile_size='15G',log_disk_size='10G',cpu_count=4,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4
[2023-06-04 11:18:41.101] [DEBUG] -- exited code 0
[2023-06-04 11:18:41.102] [DEBUG] -- root@127.0.0.1 delete env LD_LIBRARY_PATH
[2023-06-04 11:18:41.171] [INFO] observer program health check
[2023-06-04 11:18:44.181] [DEBUG] -- 127.0.0.1 program health check
[2023-06-04 11:18:44.181] [DEBUG] -- local execute: cat /mnt/ob/observer/run/observer.pid
[2023-06-04 11:18:44.186] [DEBUG] -- exited code 0
[2023-06-04 11:18:44.186] [DEBUG] -- local execute: ls /proc/5470
[2023-06-04 11:18:44.191] [DEBUG] -- exited code 0
[2023-06-04 11:18:44.191] [DEBUG] -- 127.0.0.1 observer[pid: 5470] started
[2023-06-04 11:18:44.301] [DEBUG] - sub start ref count to 0
[2023-06-04 11:18:44.301] [DEBUG] - export start
[2023-06-04 11:18:44.301] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.0.0-20230603214048-83515d6223e27ae54d23fa6cacbd89a5
[2023-06-04 11:18:44.301] [DEBUG] - import connect
[2023-06-04 11:18:44.322] [DEBUG] - add connect ref count to 1
[2023-06-04 11:18:44.322] [INFO] Connect to observer
[2023-06-04 11:18:44.323] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p2G7jX0hpcfXbm6pMU9op
[2023-06-04 11:18:47.414] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p
[2023-06-04 11:18:47.450] [DEBUG] - sub connect ref count to 0
[2023-06-04 11:18:47.451] [DEBUG] - export connect
[2023-06-04 11:18:47.451] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.0.0-20230603214048-83515d6223e27ae54d23fa6cacbd89a5
[2023-06-04 11:18:47.451] [DEBUG] - import display
[2023-06-04 11:18:47.451] [DEBUG] - add display ref count to 1
[2023-06-04 11:18:47.452] [INFO] Wait for observer init
[2023-06-04 11:18:47.452] [DEBUG] -- execute sql: select * from oceanbase.__all_server. args: None
[2023-06-04 11:18:47.454] [DEBUG] -- OBD-5000: select * from oceanbase.__all_server execute failed
[2023-06-04 11:18:47.455] [ERROR] Traceback (most recent call last):
[2023-06-04 11:18:47.455] [ERROR] File "core.py", line 1574, in start_cluster
[2023-06-04 11:18:47.455] [ERROR] File "core.py", line 1695, in _start_cluster
[2023-06-04 11:18:47.455] [ERROR] File "core.py", line 184, in call_plugin
[2023-06-04 11:18:47.455] [ERROR] File "_plugin.py", line 323, in __call__
[2023-06-04 11:18:47.455] [ERROR] File "_plugin.py", line 285, in _new_func
[2023-06-04 11:18:47.455] [ERROR] File "/root/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 37, in display
[2023-06-04 11:18:47.455] [ERROR] servers = cursor.fetchall('select * from oceanbase.__all_server', raise_exception=True, exc_level='verbose')
[2023-06-04 11:18:47.455] [ERROR] File "_stdio.py", line 885, in func_wrapper
[2023-06-04 11:18:47.455] [ERROR] File "/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 108, in fetchall
[2023-06-04 11:18:47.455] [ERROR] return self.execute(sql, args=args, execute_func='fetchall', raise_exception=raise_exception, exc_level=exc_level, stdio=stdio)
[2023-06-04 11:18:47.455] [ERROR] File "_stdio.py", line 885, in func_wrapper
[2023-06-04 11:18:47.455] [ERROR] File "/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 91, in execute
[2023-06-04 11:18:47.455] [ERROR] self.cursor.execute(sql, args)
【复现路径】问题出现前后相关操作
- 检测内存4c8g
- 更新系统时间