【 使用环境 】测试环境
【 使用版本 】4.2.0
【问题描述】docker部署oceanbase,由于磁盘满了，停止服务后，启动不了
【日志】[2024-01-11 04:06:40.938] [DEBUG] - cmd: [‘obcluster’]
[2024-01-11 04:06:40.939] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2024-01-11 04:06:40.939] [DEBUG] - mkdir /root/.obd/lock/
[2024-01-11 04:06:40.940] [DEBUG] - unknown lock mode
[2024-01-11 04:06:40.942] [DEBUG] - try to get share lock /root/.obd/lock/global
[2024-01-11 04:06:40.942] [DEBUG] - share lock /root/.obd/lock/global, count 1
[2024-01-11 04:06:40.942] [DEBUG] - Get Deploy by name
[2024-01-11 04:06:40.942] [DEBUG] - mkdir /root/.obd/cluster/
[2024-01-11 04:06:40.943] [DEBUG] - mkdir /root/.obd/config_parser/
[2024-01-11 04:06:40.943] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_obcluster
[2024-01-11 04:06:40.944] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obcluster, count 1
[2024-01-11 04:06:40.953] [DEBUG] - Deploy status judge
[2024-01-11 04:06:40.954] [INFO] Get local repositories
[2024-01-11 04:06:40.954] [DEBUG] - mkdir /root/.obd/repository
[2024-01-11 04:06:40.954] [DEBUG] - Get local repository oceanbase-ce-4.2.0.0-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:40.955] [DEBUG] - Search repository oceanbase-ce version: 4.2.0.0, tag: e77b40f1634befc1d8cb411beafef876e33e31f1, release: None, package_hash: None
[2024-01-11 04:06:40.955] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2024-01-11 04:06:40.955] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 1
[2024-01-11 04:06:40.955] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce
[2024-01-11 04:06:40.959] [DEBUG] - Found repository oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:40.959] [DEBUG] - Get local repository obagent-4.2.0-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:40.959] [DEBUG] - Search repository obagent version: 4.2.0, tag: 5401a2aba62669742bda2c8309fab96520aa528b, release: None, package_hash: None
[2024-01-11 04:06:40.960] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 2
[2024-01-11 04:06:40.960] [DEBUG] - mkdir /root/.obd/repository/obagent
[2024-01-11 04:06:40.963] [DEBUG] - Found repository obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:40.964] [DEBUG] - Get deploy config
[2024-01-11 04:06:41.005] [INFO] Search plugins
[2024-01-11 04:06:41.005] [DEBUG] - Searching start_check plugin for components …
[2024-01-11 04:06:41.005] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:41.006] [DEBUG] - mkdir /root/.obd/plugins
[2024-01-11 04:06:41.007] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.0.0
[2024-01-11 04:06:41.007] [DEBUG] - Searching start_check plugin for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:41.008] [DEBUG] - Found for obagent-py_script_start_check-1.3.0 for obagent-4.2.0
[2024-01-11 04:06:41.008] [DEBUG] - Searching create_tenant plugin for components …
[2024-01-11 04:06:41.008] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:41.009] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.0.0.0 for oceanbase-ce-4.2.0.0
[2024-01-11 04:06:41.009] [DEBUG] - Searching create_tenant plugin for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:41.009] [DEBUG] - No such create_tenant plugin for obagent-4.2.0
[2024-01-11 04:06:41.010] [DEBUG] - Searching start plugin for components …
[2024-01-11 04:06:41.010] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:41.010] [DEBUG] - Found for oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.2.0.0
[2024-01-11 04:06:41.011] [DEBUG] - Searching start plugin for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:41.011] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-4.2.0
[2024-01-11 04:06:41.011] [DEBUG] - Searching connect plugin for components …
[2024-01-11 04:06:41.011] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:41.012] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.0.0
[2024-01-11 04:06:41.012] [DEBUG] - Searching connect plugin for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:41.013] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-4.2.0
[2024-01-11 04:06:41.013] [DEBUG] - Searching bootstrap plugin for components …
[2024-01-11 04:06:41.013] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:41.014] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.2.0.0
[2024-01-11 04:06:41.014] [DEBUG] - Searching bootstrap plugin for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:41.014] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-4.2.0
[2024-01-11 04:06:41.015] [DEBUG] - Searching display plugin for components …
[2024-01-11 04:06:41.015] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:41.015] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.0.0
[2024-01-11 04:06:41.016] [DEBUG] - Searching display plugin for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:41.016] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-4.2.0
[2024-01-11 04:06:41.017] [INFO] Open ssh connection
[2024-01-11 04:06:41.017] [INFO] Load cluster param plugin
[2024-01-11 04:06:41.018] [DEBUG] - Searching param plugin for components …
[2024-01-11 04:06:41.018] [DEBUG] - Search param plugin for oceanbase-ce
[2024-01-11 04:06:41.018] [DEBUG] - Found for oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.0.0
[2024-01-11 04:06:41.019] [DEBUG] - Applying oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:41.884] [DEBUG] - Search param plugin for obagent
[2024-01-11 04:06:41.884] [DEBUG] - Found for obagent-param-1.3.0 for obagent-4.2.0
[2024-01-11 04:06:41.884] [DEBUG] - Applying obagent-param-1.3.0 for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:41.979] [INFO] Cluster status check
[2024-01-11 04:06:41.979] [DEBUG] - Searching status plugin for components …
[2024-01-11 04:06:41.979] [DEBUG] - Searching status plugin for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:41.979] [DEBUG] - Found for oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.0.0
[2024-01-11 04:06:41.980] [DEBUG] - Searching status plugin for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:41.980] [DEBUG] - Found for obagent-py_script_status-1.3.0 for obagent-4.2.0
[2024-01-11 04:06:41.980] [DEBUG] - Call oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:41.980] [DEBUG] - import status
[2024-01-11 04:06:41.981] [DEBUG] - add status ref count to 1
[2024-01-11 04:06:41.981] [DEBUG] – local execute: cat /root/ob/run/observer.pid
[2024-01-11 04:06:41.991] [DEBUG] – exited code 1, error output:
[2024-01-11 04:06:41.991] [DEBUG] cat: /root/ob/run/observer.pid: No such file or directory
[2024-01-11 04:06:41.991] [DEBUG]
[2024-01-11 04:06:41.991] [DEBUG] - sub status ref count to 0
[2024-01-11 04:06:41.991] [DEBUG] - export status
[2024-01-11 04:06:41.991] [DEBUG] - Call obagent-py_script_status-1.3.0 for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:41.991] [DEBUG] - import status
[2024-01-11 04:06:41.992] [DEBUG] - add status ref count to 1
[2024-01-11 04:06:41.993] [DEBUG] – local execute: cat /root/obagent/run/ob_agentd.pid
[2024-01-11 04:06:42.001] [DEBUG] – exited code 0
[2024-01-11 04:06:42.001] [DEBUG] – local execute: ls /proc/704
[2024-01-11 04:06:42.010] [DEBUG] – exited code 2, error output:
[2024-01-11 04:06:42.010] [DEBUG] ls: cannot access ‘/proc/704’: No such file or directory
[2024-01-11 04:06:42.010] [DEBUG]
[2024-01-11 04:06:42.011] [DEBUG] - sub status ref count to 0
[2024-01-11 04:06:42.011] [DEBUG] - export status
[2024-01-11 04:06:42.011] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:42.011] [DEBUG] - import start_check
[2024-01-11 04:06:42.020] [DEBUG] - add start_check ref count to 1
[2024-01-11 04:06:42.021] [INFO] Check before start observer
[2024-01-11 04:06:42.022] [DEBUG] – local execute: ls /root/ob/store/clog/tenant_1/
[2024-01-11 04:06:42.031] [DEBUG] – exited code 0
[2024-01-11 04:06:42.031] [DEBUG] – local execute: cat /root/ob/run/observer.pid
[2024-01-11 04:06:42.039] [DEBUG] – exited code 1, error output:
[2024-01-11 04:06:42.040] [DEBUG] cat: /root/ob/run/observer.pid: No such file or directory
[2024-01-11 04:06:42.040] [DEBUG]
[2024-01-11 04:06:42.040] [DEBUG] – 127.0.0.1 port check
[2024-01-11 04:06:42.040] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2024-01-11 04:06:42.065] [DEBUG] – exited code 0
[2024-01-11 04:06:42.065] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2024-01-11 04:06:42.090] [DEBUG] – exited code 0
[2024-01-11 04:06:42.091] [DEBUG] – local execute: ls /root/ob/store/sstable/block_file
[2024-01-11 04:06:42.100] [DEBUG] – exited code 0
[2024-01-11 04:06:42.100] [DEBUG] – local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2024-01-11 04:06:42.109] [DEBUG] – exited code 0
[2024-01-11 04:06:42.110] [DEBUG] – local execute: ulimit -a
[2024-01-11 04:06:42.115] [DEBUG] – exited code 0
[2024-01-11 04:06:42.116] [DEBUG] – local execute: cat /proc/meminfo
[2024-01-11 04:06:42.125] [DEBUG] – exited code 0
[2024-01-11 04:06:42.126] [DEBUG] – local execute: df --block-size=1024
[2024-01-11 04:06:42.135] [DEBUG] – exited code 0
[2024-01-11 04:06:42.136] [DEBUG] – get disk info for path /, total: 75125227520 avail: 4389531648
[2024-01-11 04:06:42.136] [DEBUG] – get disk info for path /dev, total: 67108864 avail: 67108864
[2024-01-11 04:06:42.136] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 33619075072 avail: 33619075072
[2024-01-11 04:06:42.136] [DEBUG] – get disk info for path /dev/shm, total: 67108864 avail: 67108864
[2024-01-11 04:06:42.136] [DEBUG] – get disk info for path /etc/hosts, total: 75125227520 avail: 4389531648
[2024-01-11 04:06:42.137] [DEBUG] – get disk info for path /proc/acpi, total: 33619075072 avail: 33619075072
[2024-01-11 04:06:42.137] [DEBUG] – get disk info for path /proc/scsi, total: 33619075072 avail: 33619075072
[2024-01-11 04:06:42.137] [DEBUG] – get disk info for path /sys/firmware, total: 33619075072 avail: 33619075072
[2024-01-11 04:06:42.137] [DEBUG] – disk: {’/’: {‘total’: 75125227520, ‘avail’: 4389531648, ‘need’: 0}, ‘/dev’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 33619075072, ‘avail’: 33619075072, ‘need’: 0}, ‘/dev/shm’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/etc/hosts’: {‘total’: 75125227520, ‘avail’: 4389531648, ‘need’: 0}, ‘/proc/acpi’: {‘total’: 33619075072, ‘avail’: 33619075072, ‘need’: 0}, ‘/proc/scsi’: {‘total’: 33619075072, ‘avail’: 33619075072, ‘need’: 0}, ‘/sys/firmware’: {‘total’: 33619075072, ‘avail’: 33619075072, ‘need’: 0}}
[2024-01-11 04:06:42.137] [DEBUG] – local execute: date +%s%N
[2024-01-11 04:06:42.146] [DEBUG] – exited code 0
[2024-01-11 04:06:42.146] [DEBUG] – 127.0.0.1 time delta -1.88037109375
[2024-01-11 04:06:42.147] [DEBUG] - sub start_check ref count to 0
[2024-01-11 04:06:42.147] [DEBUG] - export start_check
[2024-01-11 04:06:42.148] [DEBUG] - Call obagent-py_script_start_check-1.3.0 for obagent-4.2.0-100000062023080210.el7-5401a2aba62669742bda2c8309fab96520aa528b
[2024-01-11 04:06:42.148] [DEBUG] - import start_check
[2024-01-11 04:06:42.151] [DEBUG] - add start_check ref count to 1
[2024-01-11 04:06:42.151] [INFO] Check before start obagent
[2024-01-11 04:06:42.154] [DEBUG] – local execute: cat /root/obagent/run/ob_agentd.pid
[2024-01-11 04:06:42.163] [DEBUG] – exited code 0
[2024-01-11 04:06:42.163] [DEBUG] – local execute: ls /proc/704
[2024-01-11 04:06:42.172] [DEBUG] – exited code 2, error output:
[2024-01-11 04:06:42.172] [DEBUG] ls: cannot access ‘/proc/704’: No such file or directory
[2024-01-11 04:06:42.172] [DEBUG]
[2024-01-11 04:06:42.172] [DEBUG] – 127.0.0.1 port check
[2024-01-11 04:06:42.173] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:1F99’ | awk -F’ ’ ‘{print $2}’ | uniq
[2024-01-11 04:06:42.198] [DEBUG] – exited code 0
[2024-01-11 04:06:42.198] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:1F98’ | awk -F’ ’ ‘{print $2}’ | uniq
[2024-01-11 04:06:42.224] [DEBUG] – exited code 0
[2024-01-11 04:06:42.224] [DEBUG] - sub start_check ref count to 0
[2024-01-11 04:06:42.225] [DEBUG] - export start_check
[2024-01-11 04:06:42.225] [DEBUG] - Call oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.2.0.0-100010032023083021.el7-e77b40f1634befc1d8cb411beafef876e33e31f1
[2024-01-11 04:06:42.225] [DEBUG] - import start
[2024-01-11 04:06:42.228] [DEBUG] - add start ref count to 1
[2024-01-11 04:06:42.229] [INFO] Start observer
[2024-01-11 04:06:42.229] [DEBUG] – local execute: ls /root/ob/store/clog/tenant_1/
[2024-01-11 04:06:42.238] [DEBUG] – exited code 0
[2024-01-11 04:06:42.238] [DEBUG] – local execute: cat /root/ob/run/observer.pid
[2024-01-11 04:06:42.247] [DEBUG] – exited code 1, error output:
[2024-01-11 04:06:42.247] [DEBUG] cat: /root/ob/run/observer.pid: No such file or directory
[2024-01-11 04:06:42.247] [DEBUG]
[2024-01-11 04:06:42.247] [DEBUG] – 127.0.0.1 start command construction
[2024-01-11 04:06:42.248] [DEBUG] – starting 127.0.0.1 observer
[2024-01-11 04:06:42.249] [DEBUG] – root@127.0.0.1 set env LD_LIBRARY_PATH to ‘/root/ob/lib:’
[2024-01-11 04:06:42.249] [DEBUG] – local execute: cd /root/ob; /root/ob/bin/observer -r ‘127.0.0.1:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -n ‘obcluster’ -c 1 -d ‘/root/ob/store’ -i ‘lo’ -l ‘INFO’ -o __min_full_resource_pool_memory=2147483648,memory_limit=‘6G’,system_memory=‘1G’,datafile_size=‘5G’,log_disk_size=‘5G’,cpu_count=16,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4
[2024-01-11 04:06:42.370] [DEBUG] – exited code 0
[2024-01-11 04:06:42.371] [DEBUG] – root@127.0.0.1 delete env LD_LIBRARY_PATH
[2024-01-11 04:06:42.372] [INFO] observer program health check
[2024-01-11 04:06:45.375] [DEBUG] – 127.0.0.1 program health check
[2024-01-11 04:06:45.376] [DEBUG] – local execute: cat /root/ob/run/observer.pid
[2024-01-11 04:06:45.386] [DEBUG] – exited code 1, error output:
[2024-01-11 04:06:45.386] [DEBUG] cat: /root/ob/run/observer.pid: No such file or directory
[2024-01-11 04:06:45.386] [DEBUG]
[2024-01-11 04:06:45.387] [WARNING] OBD-2002: Failed to start 127.0.0.1 observer
[2024-01-11 04:06:45.388] [DEBUG] - sub start ref count to 0
[2024-01-11 04:06:45.388] [DEBUG] - export start
[2024-01-11 04:06:45.388] [ERROR] oceanbase-ce start failed
[2024-01-11 04:06:45.393] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2024-01-11 04:06:45.394] [INFO] Trace ID: d2f9e170-b036-11ee-8015-0242ac110003
[2024-01-11 04:06:45.394] [INFO] If you want to view detailed obd logs, please run: obd display-trace d2f9e170-b036-11ee-8015-0242ac110003
[2024-01-11 04:06:45.395] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[2024-01-11 04:06:45.395] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2024-01-11 04:06:45.395] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2024-01-11 04:06:45.396] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obcluster release, count 0
[2024-01-11 04:06:45.396] [DEBUG] - unlock /root/.obd/lock/deploy_obcluster
[2024-01-11 04:06:45.396] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2024-01-11 04:06:45.396] [DEBUG] - unlock /root/.obd/lock/global
df -h看看呢
根据日志中的 Trace ID 捞下日志看看