jks@jks2:~$ sudo sysctl fs.aio-max-nr=1048576
[sudo] jks 的密码:
fs.aio-max-nr = 1048576
jks@jks2:~$ obd cluster start obytt100
Get local repositories ok
Search plugins ok
Load cluster param plugin ok
Open ssh connection ok
Check before start observer ok
[WARN] (127.0.0.1) failed to get open files
[WARN] (127.0.0.1) failed to get max user processes
[WARN] (127.0.0.1) failed to get core file size
[WARN] (127.0.0.1) failed to get stack size
Start observer ok
[ERROR] 127.0.0.1 obshell failed
[ERROR] oceanbase-ce start failed
See
Trace ID: bb9980f2-c1fc-11ef-8a6a-f707f2a9521f
If you want to view detailed obd logs, please run: obd display-trace bb9980f2-c1fc-11ef-8a6a-f707f2a9521f
jks@jks2:~$ obd display-trace bb9980f2-c1fc-11ef-8a6a-f707f2a9521f
[2024-12-24 13:41:09.531] [DEBUG] - cmd: [‘obytt100’]
[2024-12-24 13:41:09.547] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2024-12-24 13:41:09.547] [DEBUG] - mkdir /home/jks/.obd/lock/
[2024-12-24 13:41:09.547] [DEBUG] - unknown lock mode
[2024-12-24 13:41:09.548] [DEBUG] - try to get share lock /home/jks/.obd/lock/global
[2024-12-24 13:41:09.548] [DEBUG] - share lock /home/jks/.obd/lock/global
, count 1
[2024-12-24 13:41:09.548] [DEBUG] - Get Deploy by name
[2024-12-24 13:41:09.548] [DEBUG] - mkdir /home/jks/.obd/cluster/
[2024-12-24 13:41:09.548] [DEBUG] - mkdir /home/jks/.obd/config_parser/
[2024-12-24 13:41:09.548] [DEBUG] - try to get exclusive lock /home/jks/.obd/lock/deploy_obytt100
[2024-12-24 13:41:09.549] [DEBUG] - exclusive lock /home/jks/.obd/lock/deploy_obytt100
, count 1
[2024-12-24 13:41:09.565] [DEBUG] - Deploy status judge
[2024-12-24 13:41:09.590] [INFO] Get local repositories
[2024-12-24 13:41:09.590] [DEBUG] - mkdir /home/jks/.obd/repository
[2024-12-24 13:41:09.591] [DEBUG] - Get local repository oceanbase-ce-4.2.1.10-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.591] [DEBUG] - Search repository oceanbase-ce version: 4.2.1.10, tag: 19e05fb07f899bc2640bbcc4edb3f125b642810a, release: None, package_hash: None
[2024-12-24 13:41:09.591] [DEBUG] - try to get share lock /home/jks/.obd/lock/mirror_and_repo
[2024-12-24 13:41:09.591] [DEBUG] - share lock /home/jks/.obd/lock/mirror_and_repo
, count 1
[2024-12-24 13:41:09.591] [DEBUG] - mkdir /home/jks/.obd/repository/oceanbase-ce
[2024-12-24 13:41:09.594] [DEBUG] - Found repository oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.721] [DEBUG] - Get deploy config
[2024-12-24 13:41:09.745] [INFO] Search plugins
[2024-12-24 13:41:09.746] [DEBUG] - Searching start_check plugin for components …
[2024-12-24 13:41:09.746] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.746] [DEBUG] - mkdir /home/jks/.obd/plugins
[2024-12-24 13:41:09.764] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.1.4 for oceanbase-ce-4.2.1.10
[2024-12-24 13:41:09.764] [DEBUG] - Searching create_tenant plugin for components …
[2024-12-24 13:41:09.764] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.764] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.1.10
[2024-12-24 13:41:09.765] [DEBUG] - Searching tenant_optimize plugin for components …
[2024-12-24 13:41:09.765] [DEBUG] - Searching tenant_optimize plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.765] [DEBUG] - No such tenant_optimize plugin for oceanbase-ce-4.2.1.10
[2024-12-24 13:41:09.765] [DEBUG] - Searching start plugin for components …
[2024-12-24 13:41:09.765] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.766] [DEBUG] - Found for oceanbase-ce-py_script_start-4.2.1.4 for oceanbase-ce-4.2.1.10
[2024-12-24 13:41:09.766] [DEBUG] - Searching connect plugin for components …
[2024-12-24 13:41:09.766] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.766] [DEBUG] - Found for oceanbase-ce-py_script_connect-4.2.1.4 for oceanbase-ce-4.2.1.10
[2024-12-24 13:41:09.766] [DEBUG] - Searching bootstrap plugin for components …
[2024-12-24 13:41:09.766] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.767] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.1.4 for oceanbase-ce-4.2.1.10
[2024-12-24 13:41:09.767] [DEBUG] - Searching display plugin for components …
[2024-12-24 13:41:09.767] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.767] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.1.10
[2024-12-24 13:41:09.876] [INFO] Load cluster param plugin
[2024-12-24 13:41:09.877] [DEBUG] - Get local repository oceanbase-ce-4.2.1.10-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:09.877] [DEBUG] - Searching param plugin for components …
[2024-12-24 13:41:09.877] [DEBUG] - Search param plugin for oceanbase-ce
[2024-12-24 13:41:09.877] [DEBUG] - Found for oceanbase-ce-param-4.2.1.4 for oceanbase-ce-4.2.1.10
[2024-12-24 13:41:09.877] [DEBUG] - Applying oceanbase-ce-param-4.2.1.4 for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:10.432] [INFO] Open ssh connection
[2024-12-24 13:41:10.563] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.1.4 for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:10.563] [DEBUG] - import start_check
[2024-12-24 13:41:10.570] [DEBUG] - add start_check ref count to 1
[2024-12-24 13:41:10.570] [INFO] Check before start observer
[2024-12-24 13:41:10.572] [DEBUG] – local execute: ls /home/jks/observer/store/clog/tenant_1/
[2024-12-24 13:41:10.576] [DEBUG] – exited code 0
[2024-12-24 13:41:10.577] [DEBUG] – local execute: cat /home/jks/observer/run/observer.pid
[2024-12-24 13:41:10.597] [DEBUG] – exited code 0
[2024-12-24 13:41:10.598] [DEBUG] – local execute: ls /proc/6496
[2024-12-24 13:41:10.602] [DEBUG] – exited code 2, error output:
[2024-12-24 13:41:10.602] [DEBUG] ls: 无法访问 ‘/proc/6496’: 没有那个文件或目录
[2024-12-24 13:41:10.602] [DEBUG]
[2024-12-24 13:41:10.602] [DEBUG] – 127.0.0.1 port check
[2024-12-24 13:41:10.602] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B41’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-12-24 13:41:10.615] [DEBUG] – exited code 0
[2024-12-24 13:41:10.615] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B42’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-12-24 13:41:10.622] [DEBUG] – exited code 0
[2024-12-24 13:41:10.623] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B46’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-12-24 13:41:10.630] [DEBUG] – exited code 0
[2024-12-24 13:41:10.631] [DEBUG] – local execute: ls /home/jks/observer/store/sstable/block_file
[2024-12-24 13:41:10.634] [DEBUG] – exited code 0
[2024-12-24 13:41:10.634] [DEBUG] – local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2024-12-24 13:41:10.637] [DEBUG] – exited code 0
[2024-12-24 13:41:10.638] [DEBUG] – local execute: ulimit -a
[2024-12-24 13:41:10.640] [DEBUG] – exited code 0
[2024-12-24 13:41:10.641] [WARNING] (127.0.0.1) failed to get open files
[2024-12-24 13:41:10.641] [WARNING] (127.0.0.1) failed to get max user processes
[2024-12-24 13:41:10.641] [WARNING] (127.0.0.1) failed to get core file size
[2024-12-24 13:41:10.641] [WARNING] (127.0.0.1) failed to get stack size
[2024-12-24 13:41:10.641] [DEBUG] – local execute: sysctl -a
[2024-12-24 13:41:10.683] [DEBUG] – exited code 0
[2024-12-24 13:41:10.686] [DEBUG] – local execute: cat /proc/meminfo
[2024-12-24 13:41:10.689] [DEBUG] – exited code 0
[2024-12-24 13:41:10.689] [DEBUG] – local execute: df --block-size=1024
[2024-12-24 13:41:10.693] [DEBUG] – exited code 0
[2024-12-24 13:41:10.693] [DEBUG] – get disk info for path /run, total: 1668837376 avail: 1664049152
[2024-12-24 13:41:10.693] [DEBUG] – get disk info for path /, total: 491104411648 avail: 434735550464
[2024-12-24 13:41:10.693] [DEBUG] – get disk info for path /dev/shm, total: 8344186880 avail: 8344182784
[2024-12-24 13:41:10.693] [DEBUG] – get disk info for path /run/lock, total: 5242880 avail: 5234688
[2024-12-24 13:41:10.694] [DEBUG] – get disk info for path /run/user/1000, total: 1668837376 avail: 1668677632
[2024-12-24 13:41:10.694] [DEBUG] – disk: {’/run’: {‘total’: 1668837376, ‘avail’: 1664049152, ‘need’: 0}, ‘/’: {‘total’: 491104411648, ‘avail’: 434735550464, ‘need’: 0}, ‘/dev/shm’: {‘total’: 8344186880, ‘avail’: 8344182784, ‘need’: 0}, ‘/run/lock’: {‘total’: 5242880, ‘avail’: 5234688, ‘need’: 0}, ‘/run/user/1000’: {‘total’: 1668837376, ‘avail’: 1668677632, ‘need’: 0}}
[2024-12-24 13:41:10.694] [DEBUG] – local execute: date +%s%N
[2024-12-24 13:41:10.697] [DEBUG] – exited code 0
[2024-12-24 13:41:10.697] [DEBUG] – 127.0.0.1 time delta 0.930419921875
[2024-12-24 13:41:10.701] [INFO] [WARN] (127.0.0.1) failed to get open files
[2024-12-24 13:41:10.701] [INFO] [WARN] (127.0.0.1) failed to get max user processes
[2024-12-24 13:41:10.701] [INFO] [WARN] (127.0.0.1) failed to get core file size
[2024-12-24 13:41:10.701] [INFO] [WARN] (127.0.0.1) failed to get stack size
[2024-12-24 13:41:10.701] [INFO]
[2024-12-24 13:41:10.701] [DEBUG] - sub start_check ref count to 0
[2024-12-24 13:41:10.702] [DEBUG] - export start_check
[2024-12-24 13:41:10.702] [DEBUG] - plugin oceanbase-ce-py_script_start_check-4.2.1.4 result: True
[2024-12-24 13:41:10.702] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.1.4 for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[2024-12-24 13:41:10.702] [DEBUG] - import start
[2024-12-24 13:41:10.705] [DEBUG] - add start ref count to 1
[2024-12-24 13:41:10.705] [INFO] Start observer
[2024-12-24 13:41:10.705] [DEBUG] – local execute: ls /home/jks/observer/store/clog/tenant_1/
[2024-12-24 13:41:10.709] [DEBUG] – exited code 0
[2024-12-24 13:41:10.709] [DEBUG] – local execute: cat /home/jks/observer/run/observer.pid
[2024-12-24 13:41:10.712] [DEBUG] – exited code 0
[2024-12-24 13:41:10.712] [DEBUG] – local execute: ls /proc/6496
[2024-12-24 13:41:10.716] [DEBUG] – exited code 2, error output:
[2024-12-24 13:41:10.716] [DEBUG] ls: 无法访问 ‘/proc/6496’: 没有那个文件或目录
[2024-12-24 13:41:10.716] [DEBUG]
[2024-12-24 13:41:10.716] [DEBUG] – 127.0.0.1 start command construction
[2024-12-24 13:41:10.717] [DEBUG] – starting 127.0.0.1 observer
[2024-12-24 13:41:10.717] [DEBUG] – jks@127.0.0.1 export LD_LIBRARY_PATH=’/home/jks/observer/lib:’
[2024-12-24 13:41:10.717] [DEBUG] – local execute: cd /home/jks/observer; /home/jks/observer/bin/observer -p 2881 -P 2882 -z ‘zone1’ -n ‘obytt100’ -c 100 -d ‘/home/jks/observer/store’ -I ‘127.0.0.1’ -o __min_full_resource_pool_memory=2147483648,memory_limit=‘4G’,system_memory=‘1G’,datafile_size=‘1G’,datafile_next=‘1G’,datafile_maxsize=‘20G’,log_disk_size=‘10G’,cpu_count=8,enable_syslog_wf=False,max_syslog_file_count=4
[2024-12-24 13:41:13.941] [DEBUG] – exited code 0
[2024-12-24 13:41:13.942] [DEBUG] – jks@127.0.0.1 delete env LD_LIBRARY_PATH
[2024-12-24 13:41:13.964] [DEBUG] – start_obshell: True
[2024-12-24 13:41:13.964] [DEBUG] – local execute: cat /home/jks/observer/run/obshell.pid
[2024-12-24 13:41:13.968] [DEBUG] – exited code 1, error output:
[2024-12-24 13:41:13.968] [DEBUG] cat: /home/jks/observer/run/obshell.pid: 没有那个文件或目录
[2024-12-24 13:41:13.968] [DEBUG]
[2024-12-24 13:41:13.968] [DEBUG] – jks@127.0.0.1 export OB_ROOT_PASSWORD=’******’
[2024-12-24 13:41:13.968] [DEBUG] – start obshell: cd /home/jks/observer; /home/jks/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886
[2024-12-24 13:41:13.968] [DEBUG] – local execute: cd /home/jks/observer; /home/jks/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886
[2024-12-24 13:46:14.070] [DEBUG] – exited code 255, error output:
[2024-12-24 13:46:14.070] [DEBUG] Command ‘cd /home/jks/observer; /home/jks/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886’ timed out after 300 seconds
[2024-12-24 13:46:14.072] [ERROR] Traceback (most recent call last):
[2024-12-24 13:46:14.072] [ERROR] File “core.py”, line 2104, in start_cluster
[2024-12-24 13:46:14.072] [ERROR] File “core.py”, line 2198, in _start_cluster
[2024-12-24 13:46:14.072] [ERROR] File “core.py”, line 198, in call_plugin
[2024-12-24 13:46:14.072] [ERROR] File “_plugin.py”, line 348, in call
[2024-12-24 13:46:14.072] [ERROR] File “_plugin.py”, line 305, in _new_func
[2024-12-24 13:46:14.072] [ERROR] File “/home/jks/.obd/plugins/oceanbase-ce/4.2.1.4/start.py”, line 286, in start
[2024-12-24 13:46:14.072] [ERROR] if not client.execute_command(cmd):
[2024-12-24 13:46:14.072] [ERROR] File “_plugin.py”, line 225, in new_method
[2024-12-24 13:46:14.072] [ERROR] File “_stdio.py”, line 956, in func_wrapper
[2024-12-24 13:46:14.072] [ERROR] File “ssh.py”, line 499, in execute_command
[2024-12-24 13:46:14.072] [ERROR] File “_stdio.py”, line 956, in func_wrapper
[2024-12-24 13:46:14.073] [ERROR] File “ssh.py”, line 210, in execute_command
[2024-12-24 13:46:14.073] [ERROR] File “subprocess.py”, line 1028, in communicate
[2024-12-24 13:46:14.073] [ERROR] File “subprocess.py”, line 1869, in _communicate
[2024-12-24 13:46:14.073] [ERROR] File “subprocess.py”, line 1072, in _check_timeout
[2024-12-24 13:46:14.073] [ERROR] subprocess.TimeoutExpired: Command ‘cd /home/jks/observer; /home/jks/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886’ timed out after 300 seconds
[2024-12-24 13:46:14.073] [ERROR]
[2024-12-24 13:46:14.073] [ERROR] 127.0.0.1 obshell failed
[2024-12-24 13:46:14.073] [DEBUG] - sub start ref count to 0
[2024-12-24 13:46:14.073] [DEBUG] - export start
[2024-12-24 13:46:14.073] [DEBUG] - plugin oceanbase-ce-py_script_start-4.2.1.4 result: None
[2024-12-24 13:46:14.074] [ERROR] oceanbase-ce start failed
[2024-12-24 13:46:14.077] [DEBUG] - share lock /home/jks/.obd/lock/mirror_and_repo release, count 0
[2024-12-24 13:46:14.077] [DEBUG] - unlock /home/jks/.obd/lock/mirror_and_repo
[2024-12-24 13:46:14.077] [DEBUG] - exclusive lock /home/jks/.obd/lock/deploy_obytt100 release, count 0
[2024-12-24 13:46:14.077] [DEBUG] - unlock /home/jks/.obd/lock/deploy_obytt100
[2024-12-24 13:46:14.077] [DEBUG] - share lock /home/jks/.obd/lock/global release, count 0
[2024-12-24 13:46:14.077] [DEBUG] - unlock /home/jks/.obd/lock/global
[2024-12-24 13:46:14.078] [INFO] See
[2024-12-24 13:46:14.078] [INFO] Trace ID: bb9980f2-c1fc-11ef-8a6a-f707f2a9521f
[2024-12-24 13:46:14.078] [INFO] If you want to view detailed obd logs, please run: obd display-trace bb9980f2-c1fc-11ef-8a6a-f707f2a9521f
jks@jks2:~$