【 使用环境 】测试环境
【 OB or 其他组件 】
【 使用版本 】3.1.4
【问题描述】obd cluster start xx启动失败
快速体验 OceanBase-OceanBase 数据库-OceanBase文档中心-分布式数据库使用文档
参考上述文档部署一个新环境,但是一直无法启动,报错信息:
$ obd cluster start ob1
Get local repositories ok
Search plugins ok
Open ssh connection ok
Load cluster param plugin ok
Check before start observer ok
[WARN] (127.0.0.1) clog and data use the same disk (/)
Start observer ok
observer program health check x
[WARN] OBD-2002: Failed to start server1(127.0.0.1) observer
[ERROR] oceanbase-ce start failed
See OceanBase 社区 .
详细日志:
[2022-11-01 20:46:07.531] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - mkdir /root/.obd/lock/
[2022-11-01 20:46:07.532] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - try to get share lock /root/.obd/lock/global
[2022-11-01 20:46:07.532] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - share lock /root/.obd/lock/global
, count 1
[2022-11-01 20:46:07.532] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - cmd: [‘ob1’]
[2022-11-01 20:46:07.532] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2022-11-01 20:46:07.532] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Get Deploy by name
[2022-11-01 20:46:07.532] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - mkdir /root/.obd/cluster/
[2022-11-01 20:46:07.533] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - mkdir /root/.obd/config_parser/
[2022-11-01 20:46:07.533] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_ob1
[2022-11-01 20:46:07.533] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - exclusive lock /root/.obd/lock/deploy_ob1
, count 1
[2022-11-01 20:46:07.542] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Deploy status judge
[2022-11-01 20:46:07.543] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO] Get local repositories
[2022-11-01 20:46:07.544] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - mkdir /root/.obd/repository
[2022-11-01 20:46:07.544] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Get local repository oceanbase-ce-3.1.4-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:07.545] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Search repository oceanbase-ce version: 3.1.4, tag: c5cd94f4f190317b6a883c58a26460a506205ce6, release: None, package_hash: None
[2022-11-01 20:46:07.545] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2022-11-01 20:46:07.545] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo
, count 1
[2022-11-01 20:46:07.546] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce
[2022-11-01 20:46:07.551] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Found repository oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:07.674] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Get deploy config
[2022-11-01 20:46:07.725] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO] Search plugins
[2022-11-01 20:46:07.726] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching start_check plugin for components …
[2022-11-01 20:46:07.727] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:07.727] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - mkdir /root/.obd/plugins
[2022-11-01 20:46:07.729] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.4
[2022-11-01 20:46:07.729] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching create_tenant plugin for components …
[2022-11-01 20:46:07.729] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:07.730] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-3.1.0 for oceanbase-ce-3.1.4
[2022-11-01 20:46:07.730] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching start plugin for components …
[2022-11-01 20:46:07.730] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:07.731] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.4
[2022-11-01 20:46:07.732] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching connect plugin for components …
[2022-11-01 20:46:07.732] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:07.733] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.4
[2022-11-01 20:46:07.733] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching bootstrap plugin for components …
[2022-11-01 20:46:07.733] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:07.734] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.4
[2022-11-01 20:46:07.734] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching display plugin for components …
[2022-11-01 20:46:07.734] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:07.735] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.4
[2022-11-01 20:46:07.859] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO] Open ssh connection
[2022-11-01 20:46:07.992] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO] Load cluster param plugin
[2022-11-01 20:46:07.993] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Searching param plugin for components …
[2022-11-01 20:46:07.993] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Search param plugin for oceanbase-ce
[2022-11-01 20:46:07.994] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.4
[2022-11-01 20:46:07.994] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:08.887] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:08.887] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - import start_check
[2022-11-01 20:46:08.895] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - add start_check ref count to 1
[2022-11-01 20:46:08.896] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO] Check before start observer
[2022-11-01 20:46:08.897] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: cat /data/oceanbase/run/observer.pid
[2022-11-01 20:46:08.908] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 1, error output:
[2022-11-01 20:46:08.908] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] cat: /data/oceanbase/run/observer.pid: No such file or directory
[2022-11-01 20:46:08.908] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG]
[2022-11-01 20:46:08.909] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – server1(127.0.0.1) port check
[2022-11-01 20:46:08.909] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp,udp}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2022-11-01 20:46:08.922] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:08.923] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp,udp}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2022-11-01 20:46:08.934] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:08.935] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: ls /data/oceanbase/store/sstable/block_file
[2022-11-01 20:46:08.945] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 2, error output:
[2022-11-01 20:46:08.945] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] ls: cannot access /data/oceanbase/store/sstable/block_file: No such file or directory
[2022-11-01 20:46:08.945] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG]
[2022-11-01 20:46:08.945] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: grep -e ‘^ *lo:’ /proc/net/dev
[2022-11-01 20:46:08.953] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:08.954] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2022-11-01 20:46:08.960] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:08.961] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: ulimit -a
[2022-11-01 20:46:08.966] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:08.967] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: cat /proc/meminfo
[2022-11-01 20:46:08.974] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:08.976] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: df --block-size=1024
[2022-11-01 20:46:08.986] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:08.987] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: df --block-size=1024 /data/oceanbase/store
[2022-11-01 20:46:08.994] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:08.994] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: df --block-size=1024 /data/oceanbase/store/clog
[2022-11-01 20:46:09.000] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:09.001] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – disk: {’/’: {‘total’: 316631339008, ‘avail’: 233093627904, ‘need’: 0, ‘threshold’: 2}, ‘/dev’: {‘total’: 8311627776, ‘avail’: 8311627776, ‘need’: 0, ‘threshold’: 2}, ‘/dev/shm’: {‘total’: 8328122368, ‘avail’: 8328122368, ‘need’: 0, ‘threshold’: 2}, ‘/run’: {‘total’: 8328122368, ‘avail’: 8234708992, ‘need’: 0, ‘threshold’: 2}, ‘/sys/fs/cgroup’: {‘total’: 8328122368, ‘avail’: 8328122368, ‘need’: 0, ‘threshold’: 2}, ‘/boot’: {‘total’: 311070720, ‘avail’: 103374848, ‘need’: 0, ‘threshold’: 2}, ‘/run/user/0’: {‘total’: 1665626112, ‘avail’: 1665626112, ‘need’: 0, ‘threshold’: 2}, ‘/var/lib/docker/overlay2/4ab1bb670c7ebca2b45bd252aa64557e0b48ce48f6e49263e89da6ac10eb0778/merged’: {‘total’: 316631339008, ‘avail’: 233093627904, ‘need’: 0, ‘threshold’: 2}, ‘/var/lib/docker/containers/331a9ccd6ea76761792b58265dfc0f601ed0d60a48ad883eae48d03abe61e708/shm’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0, ‘threshold’: 2}}
[2022-11-01 20:46:09.001] [279b7366-59e3-11ed-bd4f-00163e6a68de] [WARNING] [WARN] (127.0.0.1) clog and data use the same disk (/)
[2022-11-01 20:46:09.002] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: date +%s%N
[2022-11-01 20:46:09.009] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:09.009] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – 127.0.0.1 time delta -1.362060546875
[2022-11-01 20:46:09.029] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO] [WARN] (127.0.0.1) clog and data use the same disk (/)
[2022-11-01 20:46:09.029] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO]
[2022-11-01 20:46:09.029] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - sub start_check ref count to 0
[2022-11-01 20:46:09.029] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - export start_check
[2022-11-01 20:46:09.029] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.4-10000092022071511.el7-c5cd94f4f190317b6a883c58a26460a506205ce6
[2022-11-01 20:46:09.030] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - import start
[2022-11-01 20:46:09.033] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - add start ref count to 1
[2022-11-01 20:46:09.034] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO] Start observer
[2022-11-01 20:46:09.035] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: ls /data/oceanbase/store/ilog/
[2022-11-01 20:46:09.043] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:09.044] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: cat /data/oceanbase/run/observer.pid
[2022-11-01 20:46:09.049] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 1, error output:
[2022-11-01 20:46:09.050] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] cat: /data/oceanbase/run/observer.pid: No such file or directory
[2022-11-01 20:46:09.050] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG]
[2022-11-01 20:46:09.050] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – server1(127.0.0.1) start command construction
[2022-11-01 20:46:09.050] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – starting server1(127.0.0.1) observer
[2022-11-01 20:46:09.052] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – root@127.0.0.1 set env LD_LIBRARY_PATH to ‘/data/oceanbase/lib:’
[2022-11-01 20:46:09.052] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: cd /data/oceanbase; /data/oceanbase/bin/observer -r ‘127.0.0.1:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -n ‘ob1’ -c 1 -d ‘/data/oceanbase/store’ -i ‘lo’ -l ‘INFO’ -o __min_full_resource_pool_memory=268435456,memory_limit=‘2G’,system_memory=‘4G’,stack_size=‘512K’,cpu_count=16,cache_wash_threshold=‘256M’,workers_per_cpu_quota=10,schema_history_expire_time=‘1d’,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=‘5G’,enable_syslog_wf=True,enable_syslog_recycle=True,max_syslog_file_count=4
[2022-11-01 20:46:09.403] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 0
[2022-11-01 20:46:09.403] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – root@127.0.0.1 set env LD_LIBRARY_PATH to ‘/tmp/_MEIKqL03u::/root/.obd/repository/oceanbase-ce-libs/3.1.4/6d5437b0cad486b55963f89b8ef3769af7995350/:/data/oceanbase/lib/’
[2022-11-01 20:46:09.428] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO] observer program health check
[2022-11-01 20:46:12.432] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – server1(127.0.0.1) program health check
[2022-11-01 20:46:12.433] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – local execute: cat /data/oceanbase/run/observer.pid
[2022-11-01 20:46:12.447] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] – exited code 1, error output:
[2022-11-01 20:46:12.448] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] cat: /data/oceanbase/run/observer.pid: No such file or directory
[2022-11-01 20:46:12.448] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG]
[2022-11-01 20:46:12.565] [279b7366-59e3-11ed-bd4f-00163e6a68de] [WARNING] [WARN] OBD-2002: Failed to start server1(127.0.0.1) observer
[2022-11-01 20:46:12.565] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - sub start ref count to 0
[2022-11-01 20:46:12.566] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - export start
[2022-11-01 20:46:12.566] [279b7366-59e3-11ed-bd4f-00163e6a68de] [ERROR] [ERROR] oceanbase-ce start failed
[2022-11-01 20:46:12.566] [279b7366-59e3-11ed-bd4f-00163e6a68de] [INFO] See OceanBase 社区 .
[2022-11-01 20:46:12.566] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2022-11-01 20:46:12.567] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2022-11-01 20:46:12.567] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - exclusive lock /root/.obd/lock/deploy_ob1 release, count 0
[2022-11-01 20:46:12.567] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - unlock /root/.obd/lock/deploy_ob1
[2022-11-01 20:46:12.567] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2022-11-01 20:46:12.567] [279b7366-59e3-11ed-bd4f-00163e6a68de] [DEBUG] - unlock /root/.obd/lock/global
另外,报错信息输出中的链接打开后总是跳转到文档首页
See OceanBase 社区 .