重启服务器，报错PRROR 881(88884):Server is initializing

AntTech_XU9FVQ · 2025 年1 月 10 日 13:44

【使用环境】生产环境
【 OB or 其他组件】
【使用版本】oceanbase-all-in-one-4.2.1_BP9_HF1_20241018
【问题描述】PRROR 881(88884):Server is initializing
【复现路径】服务器重启后无法启动数据库
【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接（右键跳转查看）：
详细日志：

[2025-01-10 12:45:04.267] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /boot, total: 1020702720 avail: 740462592
[2025-01-10 12:45:04.267] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /boot/efi, total: 627875840 avail: 621559808
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home, total: 864086933504 avail: 801760567296
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/storage/overlay/539d90e16aad0330bffafc6ae29c17aebe6cd2fce72aabb9bc9d13e8f420fdfc/merged, total: 864086933504 avail: 801760567296
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/engines/runc/539d90e16aad0330bffafc6ae29c17aebe6cd2fce72aabb9bc9d13e8f420fdfc/mounts/shm, total: 67108864 avail: 67108864
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/storage/overlay/d47f4a8d7cdead144d6c1c23ace2cd8a8803aa227754416a8d41a6f0c61274b9/merged, total: 864086933504 avail: 801760567296
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/engines/runc/d47f4a8d7cdead144d6c1c23ace2cd8a8803aa227754416a8d41a6f0c61274b9/mounts/shm, total: 67108864 avail: 67108864
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/engines/runc/f4ea5eaa04a3268f0a20558bea11f4daa8577810c1aa21edec4644d4ac2f12d3/mounts/shm, total: 67108864 avail: 67108864
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/engines/runc/8a95706feec598a503b132d77d723e9ec5d4d07eaed74efda07763975e26d92a/mounts/shm, total: 67108864 avail: 67108864
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/engines/runc/a9f4661bfb59f9c8d21f7fe1b258ca490191c8f35596daccbdac181abfa9a345/mounts/shm, total: 67108864 avail: 67108864
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/storage/overlay/a9f4661bfb59f9c8d21f7fe1b258ca490191c8f35596daccbdac181abfa9a345/merged, total: 864086933504 avail: 801760567296
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/storage/overlay/8a95706feec598a503b132d77d723e9ec5d4d07eaed74efda07763975e26d92a/merged, total: 864086933504 avail: 801760567296
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – get disk info for path /home/local/storage/overlay/f4ea5eaa04a3268f0a20558bea11f4daa8577810c1aa21edec4644d4ac2f12d3/merged, total: 864086933504 avail: 801760567296
[2025-01-10 12:45:04.268] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – disk: {’/’: {‘total’: 73391005696, ‘avail’: 16356503552, ‘need’: 0}, ‘/dev’: {‘total’: 4194304, ‘avail’: 4194304, ‘need’: 0}, ‘/dev/shm’: {‘total’: 134892752896, ‘avail’: 134892752896, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 4194304, ‘avail’: 4194304, ‘need’: 0}, ‘/sys/firmware/efi/efivars’: {‘total’: 131072, ‘avail’: 102400, ‘need’: 0}, ‘/run’: {‘total’: 53957103616, ‘avail’: 53921062912, ‘need’: 0}, ‘/tmp’: {‘total’: 134892756992, ‘avail’: 134805762048, ‘need’: 0}, ‘/boot’: {‘total’: 1020702720, ‘avail’: 740462592, ‘need’: 0}, ‘/boot/efi’: {‘total’: 627875840, ‘avail’: 621559808, ‘need’: 0}, ‘/home’: {‘total’: 864086933504, ‘avail’: 801760567296, ‘need’: 0}, ‘/home/local/storage/overlay/539d90e16aad0330bffafc6ae29c17aebe6cd2fce72aabb9bc9d13e8f420fdfc/merged’: {‘total’: 864086933504, ‘avail’: 801760567296, ‘need’: 0}, ‘/home/local/engines/runc/539d90e16aad0330bffafc6ae29c17aebe6cd2fce72aabb9bc9d13e8f420fdfc/mounts/shm’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/home/local/storage/overlay/d47f4a8d7cdead144d6c1c23ace2cd8a8803aa227754416a8d41a6f0c61274b9/merged’: {‘total’: 864086933504, ‘avail’: 801760567296, ‘need’: 0}, ‘/home/local/engines/runc/d47f4a8d7cdead144d6c1c23ace2cd8a8803aa227754416a8d41a6f0c61274b9/mounts/shm’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/home/local/engines/runc/f4ea5eaa04a3268f0a20558bea11f4daa8577810c1aa21edec4644d4ac2f12d3/mounts/shm’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/home/local/engines/runc/8a95706feec598a503b132d77d723e9ec5d4d07eaed74efda07763975e26d92a/mounts/shm’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/home/local/engines/runc/a9f4661bfb59f9c8d21f7fe1b258ca490191c8f35596daccbdac181abfa9a345/mounts/shm’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/home/local/storage/overlay/a9f4661bfb59f9c8d21f7fe1b258ca490191c8f35596daccbdac181abfa9a345/merged’: {‘total’: 864086933504, ‘avail’: 801760567296, ‘need’: 0}, ‘/home/local/storage/overlay/8a95706feec598a503b132d77d723e9ec5d4d07eaed74efda07763975e26d92a/merged’: {‘total’: 864086933504, ‘avail’: 801760567296, ‘need’: 0}, ‘/home/local/storage/overlay/f4ea5eaa04a3268f0a20558bea11f4daa8577810c1aa21edec4644d4ac2f12d3/merged’: {‘total’: 864086933504, ‘avail’: 801760567296, ‘need’: 0}}
[2025-01-10 12:45:04.269] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: date +%s%N
[2025-01-10 12:45:04.278] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 0
[2025-01-10 12:45:04.279] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – 127.0.0.1 time delta 3.66259765625
[2025-01-10 12:45:04.329] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - sub start_check ref count to 0
[2025-01-10 12:45:04.329] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - export start_check
[2025-01-10 12:45:04.329] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - plugin oceanbase-ce-py_script_start_check-4.2.1.4 result: True
[2025-01-10 12:45:04.329] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.1.4 for oceanbase-ce-4.2.1.9-109010012024101414.el7-67b8a2b699db96053d3f8d8ba3116b0e97a27722
[2025-01-10 12:45:04.330] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - import start
[2025-01-10 12:45:04.334] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - add start ref count to 1
[2025-01-10 12:45:04.335] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [INFO] Start observer
[2025-01-10 12:45:04.336] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: ls /root/observer/store/clog/tenant_1/
[2025-01-10 12:45:04.346] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 0
[2025-01-10 12:45:04.346] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2025-01-10 12:45:04.355] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 0
[2025-01-10 12:45:04.355] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: ls /proc/998383
[2025-01-10 12:45:04.363] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 0
[2025-01-10 12:45:04.467] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – start_obshell: True
[2025-01-10 12:45:04.467] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: cat /root/observer/run/obshell.pid
[2025-01-10 12:45:04.474] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 1, error output:
[2025-01-10 12:45:04.474] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] cat: /root/observer/run/obshell.pid: No such file or directory
[2025-01-10 12:45:04.474] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG]
[2025-01-10 12:45:04.475] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – root@127.0.0.1 export OB_ROOT_PASSWORD=‘’
[2025-01-10 12:45:04.475] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – start obshell: cd /root/observer; /root/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886
[2025-01-10 12:45:04.475] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: cd /root/observer; /root/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886
[2025-01-10 12:45:06.591] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 0
[2025-01-10 12:45:06.593] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [INFO] observer program health check
[2025-01-10 12:45:09.596] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – 127.0.0.1 program health check
[2025-01-10 12:45:09.597] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2025-01-10 12:45:09.605] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 0
[2025-01-10 12:45:09.605] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: ls /proc/998383
[2025-01-10 12:45:09.612] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 0
[2025-01-10 12:45:09.613] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – 127.0.0.1 observer[pid: 998383] started
[2025-01-10 12:45:09.720] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [INFO] obshell program health check
[2025-01-10 12:45:09.721] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: cat /root/observer/run/obshell.pid
[2025-01-10 12:45:09.728] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 0
[2025-01-10 12:45:09.728] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – local execute: ls /proc/1011734
[2025-01-10 12:45:09.735] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – exited code 0
[2025-01-10 12:45:09.736] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – 127.0.0.1 obshell[pid: 1011734] started
[2025-01-10 12:45:09.851] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – need_bootstrap: False
[2025-01-10 12:45:09.852] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - sub start ref count to 0
[2025-01-10 12:45:09.852] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - export start
[2025-01-10 12:45:09.852] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - plugin oceanbase-ce-py_script_start-4.2.1.4 result: True
[2025-01-10 12:45:09.852] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - Call oceanbase-ce-py_script_connect-4.2.1.4 for oceanbase-ce-4.2.1.9-109010012024101414.el7-67b8a2b699db96053d3f8d8ba3116b0e97a27722
[2025-01-10 12:45:09.852] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - import connect
[2025-01-10 12:45:10.716] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - add connect ref count to 1
[2025-01-10 12:45:10.716] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect obshell (127.0.0.1:2886)
[2025-01-10 12:45:10.717] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [INFO] Connect to observer
[2025-01-10 12:45:10.718] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:45:13.723] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:45:16.726] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:45:19.731] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:45:22.734] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:45:25.738] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:45:28.741] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:45:31.745] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:45:34.750] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:45:37.755] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:45:40.760] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:45:43.762] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:45:46.767] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:45:49.773] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:45:52.777] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:45:55.782] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:45:58.785] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:01.789] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:04.792] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:07.796] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:10.798] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:13.803] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:16.807] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:19.813] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:22.818] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:25.820] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:28.824] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:31.829] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:34.834] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:37.839] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:40.843] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:43.847] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:46.850] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:49.855] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:52.857] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:46:55.861] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:46:58.866] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:01.871] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:04.875] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:07.880] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:10.885] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:13.890] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:16.892] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:19.896] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:22.901] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:25.906] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:28.910] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:31.913] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:34.915] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:37.920] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:40.924] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:43.926] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:46.929] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:49.934] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:52.939] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:47:55.943] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:47:58.950] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:01.955] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:04.960] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:07.965] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:10.969] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:13.974] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:16.979] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:19.984] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:22.986] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:25.990] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:28.996] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:32.001] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:35.003] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:38.008] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:41.013] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:44.017] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:47.022] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:50.027] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:53.032] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:48:56.037] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:48:59.041] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:02.046] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:05.051] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:08.055] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:11.060] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:14.062] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:17.066] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:20.071] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:23.075] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:26.078] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:29.083] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:32.088] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:35.093] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:38.098] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:41.102] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:44.106] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:47.111] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:50.116] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:53.120] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:49:56.125] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:49:59.130] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:50:02.132] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:50:05.138] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:50:08.143] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2025-01-10 12:50:11.148] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p******
[2025-01-10 12:50:11.150] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] Traceback (most recent call last):
[2025-01-10 12:50:11.150] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “core.py”, line 2104, in start_cluster
[2025-01-10 12:50:11.150] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “core.py”, line 2204, in _start_cluster
[2025-01-10 12:50:11.150] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “core.py”, line 198, in call_plugin
[2025-01-10 12:50:11.150] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “_plugin.py”, line 348, in call
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “_plugin.py”, line 305, in _new_func
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “/root/.obd/plugins/oceanbase-ce/4.2.1.4/connect.py”, line 636, in connect
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] cursor = Cursor(ip=server.ip, port=server_config[‘mysql_port’], tenant=’’, password=password if password is not None else ‘’, stdio=stdio)
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “_stdio.py”, line 969, in wrapper
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “_stdio.py”, line 956, in func_wrapper
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “/root/.obd/plugins/oceanbase-ce/4.2.1.4/connect.py”, line 528, in init
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] self._connect()
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “/root/.obd/plugins/oceanbase-ce/4.2.1.4/connect.py”, line 558, in _connect
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] self.db = mysql.connect(host=self.ip, user=self.user, port=int(self.port), password=str(self.password),
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “pymysql/connections.py”, line 353, in init
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “pymysql/connections.py”, line 633, in connect
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “pymysql/connections.py”, line 907, in _request_authentication
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “pymysql/connections.py”, line 725, in _read_packet
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “pymysql/protocol.py”, line 221, in raise_for_error
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] File “pymysql/err.py”, line 143, in raise_mysql_exception
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] pymysql.err.OperationalError: (8001, ‘Server is initializing’)
[2025-01-10 12:50:11.151] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR]
[2025-01-10 12:50:14.228] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [ERROR] OBD-1006: Failed to connect to oceanbase-ce
[2025-01-10 12:50:14.228] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - sub connect ref count to 0
[2025-01-10 12:50:14.229] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - export connect
[2025-01-10 12:50:14.229] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - plugin oceanbase-ce-py_script_connect-4.2.1.4 result: False
[2025-01-10 12:50:14.236] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2025-01-10 12:50:14.236] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2025-01-10 12:50:14.237] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - exclusive lock /root/.obd/lock/deploy_local release, count 0
[2025-01-10 12:50:14.237] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - unlock /root/.obd/lock/deploy_local
[2025-01-10 12:50:14.237] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2025-01-10 12:50:14.237] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [DEBUG] - unlock /root/.obd/lock/global
[2025-01-10 12:50:14.237] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2025-01-10 12:50:14.237] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [INFO] Trace ID: a7c10a0a-cf0d-11ef-92be-80615f21f6df
[2025-01-10 12:50:14.237] [a7c10a0a-cf0d-11ef-92be-80615f21f6df] [INFO] If you want to view deta11ef-92be-80615f21f6dfyer/error-codes .icat

【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试，在发帖时输入 [@论坛小助手] 即可召唤小助手，欢迎试用！

论坛小助手 · 2025 年1 月 10 日 13:44

日志采集场景: 集群无法连接
日志采集命令: obdiag gather scene run --scene=observer.cluster_down

推荐使用 obdiag 工具进行日志采集，并且将采集到的日志提供给我们以便确认问题所在。此命令将会收集集群的基础信息和启动过程中的日志，有助于我们分析服务器重启后 OceanBase 数据库无法启动的原因。

另外，为了确定具体的问题，在此需要您提供一些信息：

请问您是否对服务器或 OceanBase 进行了任何配置更改或更新？
您能否提供更详细的错误日志片段或者具体的报错时间点？
请问您的 OceanBase 版本是多少？以及部署模式（如 OCP、OBD 或者手工部署）是怎样的？
在服务器重启之前，OceanBase 是否有出现过异常情况，例如性能下降、警告信息等？

附上敏捷诊断工具 obdiag 使用帮助链接

Giant · 2025 年1 月 10 日 13:48

配置信息发下吧。您的配置启动文件。

AntTech_XU9FVQ · 2025 年1 月 10 日 13:54

oceanbase-ce:
servers:
# Please don’t use hostname, only IP can be supported
- 127.0.0.1
global:
# The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
home_path: /root/observer
# The directory for data storage. The default value is $home_path/store.
# data_dir: /data
# The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
# redo_dir: /redo
# Starting from observer version 4.2, the network selection for the observer is based on the ‘local_ip’ parameter, and the ‘devname’ parameter is no longer mandatory.
# If the ‘local_ip’ parameter is set, the observer will first use this parameter for the configuration, regardless of the ‘devname’ parameter.
# If only the ‘devname’ parameter is set, the observer will use the ‘devname’ parameter for the configuration.
# If neither the ‘devname’ nor the ‘local_ip’ parameters are set, the ‘local_ip’ parameter will be automatically assigned the IP address configured above.
# devname: eth0
mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
obshell_port: 2886 # Operation and maintenance port for Oceanbase Database. The default value is 2886. This parameter is valid only when the version of oceanbase-ce is 4.2.2.0 or later.
zone: zone1
cluster_id: 1
# please set memory limit to a suitable value which is matching resource.
memory_limit: 6G # The maximum running memory for an observer
system_memory: 1G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
datafile_size: 2G # Size of the data file.
datafile_next: 2G # the auto extend step. Please enter an capacity, such as 2G
datafile_maxsize: 20G # the auto extend max size. Please enter an capacity, such as 20G
log_disk_size: 14G # The size of disk space used by the clog files.
cpu_count: 16
production_mode: false
enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
max_syslog_file_count: 4 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
root_password: admin1Qaz12 # root user password, can be empty

论坛小助手 · 2025 年1 月 10 日 13:56

根据您提供的配置文件信息，您的 OceanBase 数据库配置看起来基本正确。然而，您遇到了 PRROR 881(88884): Server is initializing 的错误，这通常意味着 OceanBase 数据库在启动过程中遇到了问题，并且尚未完成初始化。

分析与建议

1. 检查日志

您的日志显示数据库正在尝试连接但失败了，错误信息为 pymysql.err.OperationalError: (8001, 'Server is initializing')。这表明数据库确实正在初始化，但可能由于某些原因无法完成。
建议您查看 /root/observer/log 目录下的日志文件，特别是 observer.log 和 observer.err 文件，以获取更多关于初始化失败的详细信息。

2. 资源限制

您的配置文件中设置了 memory_limit: 6G 和 system_memory: 1G，这对于小型测试环境可能是合理的，但在生产环境中，这些值可能过低。确保服务器有足够的内存来支持这些配置。
如果服务器内存不足，可能会导致初始化失败。您可以尝试增加 memory_limit 和 system_memory 的值，或者减少其他资源占用。

3. 端口冲突

确保没有其他服务占用了您配置的端口（如 mysql_port: 2881, rpc_port: 2882, obshell_port: 2886）。可以使用以下命令检查端口占用情况：
```
netstat -tuln | grep 2881
netstat -tuln | grep 2882
netstat -tuln | grep 2886
```

4. 权限问题

确保 /root/observer 目录及其子目录具有正确的权限，允许 OceanBase 进程读写这些目录。可以使用以下命令检查和设置权限：
```
chown -R observer:observer /root/observer
chmod -R 755 /root/observer
```

5. 重启顺序

确保在服务器重启后，OceanBase 数据库是最后启动的服务之一。有时，依赖的服务（如网络服务）未完全启动可能导致 OceanBase 初始化失败。

6. 检查磁盘空间

确保磁盘空间充足，特别是在 /root/observer/store 和 /root/observer/log 目录下。磁盘空间不足也可能导致初始化失败。

7. 重置或重新部署

如果以上步骤都无法解决问题，您可以尝试使用 obd cluster destroy 命令销毁当前集群，然后重新部署。请注意，这将删除所有数据，请确保在此之前备份重要数据。

示例命令

如果您决定重新部署，可以参考以下步骤：

销毁现有集群：
```
obd cluster destroy <cluster_name>
```
清理残留文件（如果需要）：
```
rm -rf /root/observer/*
```

重新部署集群：

obd cluster deploy <cluster_name> -c <config_file.yaml>

启动集群：
```
obd cluster start <cluster_name>
```

总结

根据文档库中的信息，PRROR 881(88884): Server is initializing 错误通常是由于初始化过程中的某些问题引起的。通过检查日志、调整资源配置、确保端口和权限正确，以及考虑重新部署，应该能够解决这个问题。如果问题仍然存在，建议您提供更详细的日志信息以便进一步分析。

更多信息请查看:

(小助手的答复已结束，如未能解决您的问题，请继续提问并等待其他同学的回复，谢谢！)

Giant · 2025 年1 月 10 日 14:05

memory_limit: 6G
system_memory: 1G

修改如下参数
memory_limit: 10G
system_memory: 3G

如果内存资源不足的话，一点一点网上添加

辞霜 · 2025 年1 月 10 日 15:05

ps -ef|grep observer看一下进程存在么
发一份启动期间的observer日志 ~/oceanbase/log

AntTech_XU9FVQ · 2025 年1 月 10 日 15:13

有进程，日志在帖子上

辞霜 · 2025 年1 月 10 日 15:22

有进程，有测试黑屏化登录么。你发的是obd日志，这边需要您提供一下observer日志
obd版本是多少，老版本存在conn超时bug实际ob已经启动成功问题

AntTech_XU9FVQ · 2025 年1 月 10 日 15:32

感谢回复，已经排查解决了，datafile_maxsize的配置扩大就好了

辞霜 · 2025 年1 月 10 日 15:36

麻烦您提供一下observer日志。datafile_maxsize参数应该不会导致conn超时这边帮忙分析下