测试环境 ocp
【 使用版本 】 4.2.2
单机部署后正常使用，重启服务器后服务无法启动
系统资源
[admin@ocp ~]$ free -m
total used free shared buff/cache available
Mem: 31993 3008 28222 11 763 25370
Swap: 2047 0 2047
[admin@ocp ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 197G 157G 41G 80% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 12M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 1014M 145M 870M 15% /boot
tmpfs 3.2G 0 3.2G 0% /run/user/1000
【以下为日志】
[admin@ocp ~]$ obd cluster start ocp
Get local repositories ok
Search plugins ok
Load cluster param plugin ok
Open ssh connection ok
Check before start observer ok
[WARN] OBD-1007: (10.1.1.10) The recommended number of open files is 655350 (Current value: 20000)
[WARN] OBD-1007: (10.1.1.10) The recommended number of max user processes is 655350 (Current value: 120000)
[WARN] OBD-1007: (10.1.1.10) The recommended number of core file size is unlimited (Current value: 0)
[WARN] OBD-1007: (10.1.1.10) The recommended number of stack size is unlimited (Current value: 10240)
Check before start obproxy ok
Check before start ocp-server ok
Start observer x
[ERROR] OBD-2002: Failed to start 10.1.1.10 observer:
[ERROR] oceanbase-ce start failed
See https://www.oceanbase.com/product/ob-deployer/error-codes .
Trace ID: 53d7b810-7262-11ef-8329-000c29b5e83e
If you want to view detailed obd logs, please run: obd display-trace 53d7b810-7262-11ef-8329-000c29b5e83e
[admin@ocp ~]$ obd display-trace 53d7b810-7262-11ef-8329-000c29b5e83e
[2024-09-14 14:26:51.078] [DEBUG] - cmd: [‘ocp’]
[2024-09-14 14:26:51.079] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2024-09-14 14:26:51.079] [DEBUG] - mkdir /home/admin/.obd/lock/
[2024-09-14 14:26:51.079] [DEBUG] - unknown lock mode
[2024-09-14 14:26:51.079] [DEBUG] - try to get share lock /home/admin/.obd/lock/global
[2024-09-14 14:26:51.080] [DEBUG] - share lock /home/admin/.obd/lock/global, count 1
[2024-09-14 14:26:51.080] [DEBUG] - Get Deploy by name
[2024-09-14 14:26:51.080] [DEBUG] - mkdir /home/admin/.obd/cluster/
[2024-09-14 14:26:51.080] [DEBUG] - mkdir /home/admin/.obd/config_parser/
[2024-09-14 14:26:51.081] [DEBUG] - try to get exclusive lock /home/admin/.obd/lock/deploy_ocp
[2024-09-14 14:26:51.081] [DEBUG] - exclusive lock /home/admin/.obd/lock/deploy_ocp, count 1
[2024-09-14 14:26:51.090] [DEBUG] - Deploy status judge
[2024-09-14 14:26:51.173] [INFO] Get local repositories
[2024-09-14 14:26:51.174] [DEBUG] - mkdir /home/admin/.obd/repository
[2024-09-14 14:26:51.174] [DEBUG] - Get local repository oceanbase-ce-4.2.1.2-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:51.174] [DEBUG] - Search repository oceanbase-ce version: 4.2.1.2, tag: 1873bbe80cbbe5d00d5f276a4f7302cfca677fb6, release: None, package_hash: None
[2024-09-14 14:26:51.175] [DEBUG] - try to get share lock /home/admin/.obd/lock/mirror_and_repo
[2024-09-14 14:26:51.175] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo, count 1
[2024-09-14 14:26:51.175] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce
[2024-09-14 14:26:51.288] [DEBUG] - Found repository oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:51.289] [DEBUG] - Get local repository obproxy-ce-4.2.1.0-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:51.289] [DEBUG] - Search repository obproxy-ce version: 4.2.1.0, tag: 0aed4b782120e4248b749f67be3d2cc82cdcb70d, release: None, package_hash: None
[2024-09-14 14:26:51.289] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo, count 2
[2024-09-14 14:26:51.289] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy-ce
[2024-09-14 14:26:51.567] [DEBUG] - Found repository obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:51.567] [DEBUG] - Get local repository ocp-server-ce-4.2.2-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:51.567] [DEBUG] - Search repository ocp-server-ce version: 4.2.2, tag: 84a65bf0ec004e73b772938696b2b896a0ef2b2f, release: None, package_hash: None
[2024-09-14 14:26:51.567] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo, count 3
[2024-09-14 14:26:51.567] [DEBUG] - mkdir /home/admin/.obd/repository/ocp-server-ce
[2024-09-14 14:26:51.612] [DEBUG] - Found repository ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:51.698] [DEBUG] - Get deploy config
[2024-09-14 14:26:51.837] [INFO] Search plugins
[2024-09-14 14:26:51.838] [DEBUG] - Searching start_check plugin for components …
[2024-09-14 14:26:51.838] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:51.838] [DEBUG] - mkdir /home/admin/.obd/plugins
[2024-09-14 14:26:52.998] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.1.2
[2024-09-14 14:26:52.998] [DEBUG] - Searching start_check plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:53.029] [DEBUG] - Found for obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.2.1.0
[2024-09-14 14:26:53.029] [DEBUG] - Searching start_check plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:53.306] [DEBUG] - Found for ocp-server-ce-py_script_start_check-4.2.2 for ocp-server-ce-4.2.2
[2024-09-14 14:26:53.306] [DEBUG] - Searching create_tenant plugin for components …
[2024-09-14 14:26:53.306] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:53.307] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.1.2
[2024-09-14 14:26:53.307] [DEBUG] - Searching create_tenant plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:53.308] [DEBUG] - No such create_tenant plugin for obproxy-ce-4.2.1.0
[2024-09-14 14:26:53.308] [DEBUG] - Searching create_tenant plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:53.308] [DEBUG] - No such create_tenant plugin for ocp-server-ce-4.2.2
[2024-09-14 14:26:53.308] [DEBUG] - Searching tenant_optimize plugin for components …
[2024-09-14 14:26:53.308] [DEBUG] - Searching tenant_optimize plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:53.309] [DEBUG] - No such tenant_optimize plugin for oceanbase-ce-4.2.1.2
[2024-09-14 14:26:53.309] [DEBUG] - Searching tenant_optimize plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:53.309] [DEBUG] - No such tenant_optimize plugin for obproxy-ce-4.2.1.0
[2024-09-14 14:26:53.309] [DEBUG] - Searching tenant_optimize plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:53.309] [DEBUG] - No such tenant_optimize plugin for ocp-server-ce-4.2.2
[2024-09-14 14:26:53.310] [DEBUG] - Searching start plugin for components …
[2024-09-14 14:26:53.310] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:53.311] [DEBUG] - Found for oceanbase-ce-py_script_start-4.2.0.0 for oceanbase-ce-4.2.1.2
[2024-09-14 14:26:53.311] [DEBUG] - Searching start plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:53.311] [DEBUG] - Found for obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.2.1.0
[2024-09-14 14:26:53.311] [DEBUG] - Searching start plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:53.311] [DEBUG] - Found for ocp-server-ce-py_script_start-4.2.1 for ocp-server-ce-4.2.2
[2024-09-14 14:26:53.311] [DEBUG] - Searching connect plugin for components …
[2024-09-14 14:26:53.312] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:53.312] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.1.2
[2024-09-14 14:26:53.313] [DEBUG] - Searching connect plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:53.313] [DEBUG] - Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.2.1.0
[2024-09-14 14:26:53.313] [DEBUG] - Searching connect plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:53.313] [DEBUG] - Found for ocp-server-ce-py_script_connect-4.2.1 for ocp-server-ce-4.2.2
[2024-09-14 14:26:53.313] [DEBUG] - Searching bootstrap plugin for components …
[2024-09-14 14:26:53.313] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:53.314] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.1.0 for oceanbase-ce-4.2.1.2
[2024-09-14 14:26:53.314] [DEBUG] - Searching bootstrap plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:53.315] [DEBUG] - Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.2.1.0
[2024-09-14 14:26:53.315] [DEBUG] - Searching bootstrap plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:53.315] [DEBUG] - Found for ocp-server-ce-py_script_bootstrap-4.2.1 for ocp-server-ce-4.2.2
[2024-09-14 14:26:53.315] [DEBUG] - Searching display plugin for components …
[2024-09-14 14:26:53.315] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:53.316] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.1.2
[2024-09-14 14:26:53.316] [DEBUG] - Searching display plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:53.316] [DEBUG] - Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.2.1.0
[2024-09-14 14:26:53.316] [DEBUG] - Searching display plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:53.317] [DEBUG] - Found for ocp-server-ce-py_script_display-4.2.1 for ocp-server-ce-4.2.2
[2024-09-14 14:26:53.403] [INFO] Load cluster param plugin
[2024-09-14 14:26:53.404] [DEBUG] - Get local repository oceanbase-ce-4.2.1.2-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:53.404] [DEBUG] - Get local repository obproxy-ce-4.2.1.0-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:53.405] [DEBUG] - Get local repository ocp-server-ce-4.2.2-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:53.405] [DEBUG] - Searching param plugin for components …
[2024-09-14 14:26:53.405] [DEBUG] - Search param plugin for oceanbase-ce
[2024-09-14 14:26:53.406] [DEBUG] - Found for oceanbase-ce-param-4.2.1.0 for oceanbase-ce-4.2.1.2
[2024-09-14 14:26:53.406] [DEBUG] - Applying oceanbase-ce-param-4.2.1.0 for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:54.579] [DEBUG] - Search param plugin for obproxy-ce
[2024-09-14 14:26:54.580] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0
[2024-09-14 14:26:54.580] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:54.835] [DEBUG] - Search param plugin for ocp-server-ce
[2024-09-14 14:26:54.836] [DEBUG] - Found for ocp-server-ce-param-4.2.1 for ocp-server-ce-4.2.2
[2024-09-14 14:26:54.836] [DEBUG] - Applying ocp-server-ce-param-4.2.1 for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:55.077] [INFO] Open ssh connection
[2024-09-14 14:26:55.078] [DEBUG] - host: 10.1.1.10, port: 22, user: admin, password: admin
[2024-09-14 14:26:55.209] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:26:55.209] [DEBUG] - import start_check
[2024-09-14 14:26:55.233] [DEBUG] - add start_check ref count to 1
[2024-09-14 14:26:55.234] [INFO] Check before start observer
[2024-09-14 14:26:55.236] [DEBUG] – admin@10.1.1.10 execute: ls /data/1/clog/tenant_1/
[2024-09-14 14:26:55.612] [DEBUG] – exited code 0
[2024-09-14 14:26:55.612] [DEBUG] – admin@10.1.1.10 execute: cat /home/admin/oceanbase/run/observer.pid
[2024-09-14 14:26:55.959] [DEBUG] – exited code 0
[2024-09-14 14:26:55.959] [DEBUG] – admin@10.1.1.10 execute: ls /proc/32157
[2024-09-14 14:26:56.019] [DEBUG] – exited code 2, error output:
[2024-09-14 14:26:56.019] [DEBUG] ls: cannot access /proc/32157: No such file or directory
[2024-09-14 14:26:56.019] [DEBUG]
[2024-09-14 14:26:56.020] [DEBUG] – 10.1.1.10 port check
[2024-09-14 14:26:56.020] [DEBUG] – admin@10.1.1.10 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B41’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-09-14 14:26:56.082] [DEBUG] – exited code 0
[2024-09-14 14:26:56.082] [DEBUG] – admin@10.1.1.10 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B42’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-09-14 14:26:56.143] [DEBUG] – exited code 0
[2024-09-14 14:26:56.144] [DEBUG] – admin@10.1.1.10 execute: ls /data/1/sstable/block_file
[2024-09-14 14:26:56.496] [DEBUG] – exited code 0
[2024-09-14 14:26:56.496] [DEBUG] – admin@10.1.1.10 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2024-09-14 14:26:56.552] [DEBUG] – exited code 0
[2024-09-14 14:26:56.553] [DEBUG] – admin@10.1.1.10 execute: ulimit -a
[2024-09-14 14:26:56.608] [DEBUG] – exited code 0
[2024-09-14 14:26:56.609] [WARNING] OBD-1007: (10.1.1.10) The recommended number of open files is 655350 (Current value: 20000)
[2024-09-14 14:26:56.609] [WARNING] OBD-1007: (10.1.1.10) The recommended number of max user processes is 655350 (Current value: 120000)
[2024-09-14 14:26:56.610] [WARNING] OBD-1007: (10.1.1.10) The recommended number of core file size is unlimited (Current value: 0)
[2024-09-14 14:26:56.610] [WARNING] OBD-1007: (10.1.1.10) The recommended number of stack size is unlimited (Current value: 10240)
[2024-09-14 14:26:56.610] [DEBUG] – admin@10.1.1.10 execute: sysctl -a
[2024-09-14 14:26:56.790] [DEBUG] – exited code 0
[2024-09-14 14:26:56.793] [DEBUG] – admin@10.1.1.10 execute: cat /proc/meminfo
[2024-09-14 14:26:56.852] [DEBUG] – exited code 0
[2024-09-14 14:26:56.854] [DEBUG] – admin@10.1.1.10 execute: df --block-size=1024
[2024-09-14 14:26:56.913] [DEBUG] – exited code 0
[2024-09-14 14:26:56.914] [DEBUG] – get disk info for path /, total: 211415470080 avail: 43662729216
[2024-09-14 14:26:56.914] [DEBUG] – get disk info for path /dev, total: 16761651200 avail: 16761651200
[2024-09-14 14:26:56.914] [DEBUG] – get disk info for path /dev/shm, total: 16774012928 avail: 16774012928
[2024-09-14 14:26:56.914] [DEBUG] – get disk info for path /run, total: 16774012928 avail: 16761864192
[2024-09-14 14:26:56.914] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 16774012928 avail: 16774012928
[2024-09-14 14:26:56.914] [DEBUG] – get disk info for path /boot, total: 1063256064 avail: 911314944
[2024-09-14 14:26:56.915] [DEBUG] – get disk info for path /run/user/1000, total: 3354804224 avail: 3354804224
[2024-09-14 14:26:56.915] [DEBUG] – disk: {’/’: {‘total’: 211415470080, ‘avail’: 43662729216, ‘need’: 0}, ‘/dev’: {‘total’: 16761651200, ‘avail’: 16761651200, ‘need’: 0}, ‘/dev/shm’: {‘total’: 16774012928, ‘avail’: 16774012928, ‘need’: 0}, ‘/run’: {‘total’: 16774012928, ‘avail’: 16761864192, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 16774012928, ‘avail’: 16774012928, ‘need’: 0}, ‘/boot’: {‘total’: 1063256064, ‘avail’: 911314944, ‘need’: 0}, ‘/run/user/1000’: {‘total’: 3354804224, ‘avail’: 3354804224, ‘need’: 0}}
[2024-09-14 14:26:56.915] [DEBUG] – admin@10.1.1.10 execute: date +%s%N
[2024-09-14 14:26:56.970] [DEBUG] – exited code 0
[2024-09-14 14:26:56.970] [DEBUG] – 10.1.1.10 time delta -1.529052734375
[2024-09-14 14:26:57.063] [INFO] [WARN] OBD-1007: (10.1.1.10) The recommended number of open files is 655350 (Current value: 20000)
[2024-09-14 14:26:57.063] [INFO] [WARN] OBD-1007: (10.1.1.10) The recommended number of max user processes is 655350 (Current value: 120000)
[2024-09-14 14:26:57.064] [INFO] [WARN] OBD-1007: (10.1.1.10) The recommended number of core file size is unlimited (Current value: 0)
[2024-09-14 14:26:57.064] [INFO] [WARN] OBD-1007: (10.1.1.10) The recommended number of stack size is unlimited (Current value: 10240)
[2024-09-14 14:26:57.064] [INFO]
[2024-09-14 14:26:57.064] [DEBUG] - sub start_check ref count to 0
[2024-09-14 14:26:57.064] [DEBUG] - export start_check
[2024-09-14 14:26:57.064] [DEBUG] - Call obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[2024-09-14 14:26:57.064] [DEBUG] - import start_check
[2024-09-14 14:26:57.075] [DEBUG] - add start_check ref count to 1
[2024-09-14 14:26:57.076] [INFO] Check before start obproxy
[2024-09-14 14:26:57.077] [DEBUG] – admin@10.1.1.10 execute: cat /home/admin/obproxy/run/obproxy-10.1.1.10-2883.pid
[2024-09-14 14:26:57.309] [DEBUG] – exited code 0
[2024-09-14 14:26:57.309] [DEBUG] – admin@10.1.1.10 execute: ls /proc/37221/fd
[2024-09-14 14:26:57.366] [DEBUG] – exited code 2, error output:
[2024-09-14 14:26:57.366] [DEBUG] ls: cannot access /proc/37221/fd: No such file or directory
[2024-09-14 14:26:57.366] [DEBUG]
[2024-09-14 14:26:57.367] [DEBUG] – 10.1.1.10 port check
[2024-09-14 14:26:57.367] [DEBUG] – admin@10.1.1.10 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B43’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-09-14 14:26:57.426] [DEBUG] – exited code 0
[2024-09-14 14:26:57.427] [DEBUG] – admin@10.1.1.10 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B44’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-09-14 14:26:57.487] [DEBUG] – exited code 0
[2024-09-14 14:26:57.598] [DEBUG] - sub start_check ref count to 0
[2024-09-14 14:26:57.599] [DEBUG] - export start_check
[2024-09-14 14:26:57.599] [DEBUG] - Call ocp-server-ce-py_script_start_check-4.2.2 for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f
[2024-09-14 14:26:57.599] [DEBUG] - import start_check
[2024-09-14 14:26:57.833] [DEBUG] - add start_check ref count to 1
[2024-09-14 14:26:57.834] [INFO] Check before start ocp-server
[2024-09-14 14:26:57.839] [DEBUG] – oceanbase version check
[2024-09-14 14:26:57.840] [DEBUG] – admin@10.1.1.10 execute: sudo -n true
[2024-09-14 14:26:58.850] [DEBUG] – exited code 0
[2024-09-14 14:26:58.851] [DEBUG] – admin@10.1.1.10 execute: cat /home/admin/ocp/run/ocp-server.pid
[2024-09-14 14:26:59.250] [DEBUG] – exited code 0
[2024-09-14 14:26:59.250] [DEBUG] – admin@10.1.1.10 execute: ls /proc/40069
[2024-09-14 14:26:59.309] [DEBUG] – exited code 2, error output:
[2024-09-14 14:26:59.309] [DEBUG] ls: cannot access /proc/40069: No such file or directory
[2024-09-14 14:26:59.310] [DEBUG]
[2024-09-14 14:26:59.310] [DEBUG] – user check
[2024-09-14 14:26:59.310] [DEBUG] – port check
[2024-09-14 14:26:59.310] [DEBUG] – admin@10.1.1.10 execute: bash -c ‘cat /proc/net/{udp*,tcp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:1F90’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-09-14 14:26:59.371] [DEBUG] – exited code 0
[2024-09-14 14:26:59.371] [DEBUG] – java check
[2024-09-14 14:26:59.372] [DEBUG] – admin@10.1.1.10 append ‘/home/admin/ocp/jre/bin:’ to PATH
[2024-09-14 14:26:59.372] [DEBUG] – admin@10.1.1.10 execute: java -version
[2024-09-14 14:27:12.754] [DEBUG] – exited code 0
[2024-09-14 14:27:12.755] [DEBUG] – java version <ssh.SshReturn object at 0x7f86ba69eb20>
[2024-09-14 14:27:12.755] [DEBUG] – java_major_version 1.8.0
[2024-09-14 14:27:12.756] [DEBUG] – java_update_version 332
[2024-09-14 14:27:12.756] [DEBUG] – clockdiff check
[2024-09-14 14:27:12.756] [DEBUG] – admin@10.1.1.10 execute: clockdiff -o 127.0.0.1
[2024-09-14 14:27:12.845] [DEBUG] – exited code 0
[2024-09-14 14:27:12.845] [DEBUG] – memory check
[2024-09-14 14:27:12.845] [DEBUG] – admin@10.1.1.10 execute: cat /proc/meminfo
[2024-09-14 14:27:12.900] [DEBUG] – exited code 0
[2024-09-14 14:27:12.900] [DEBUG] – disk check
[2024-09-14 14:27:12.901] [DEBUG] – admin@10.1.1.10 execute: df --block-size=1024
[2024-09-14 14:27:12.958] [DEBUG] – exited code 0
[2024-09-14 14:27:12.958] [DEBUG] – get disk info for path /, total: 211415470080 avail: 43661705216
[2024-09-14 14:27:12.959] [DEBUG] – get disk info for path /dev, total: 16761651200 avail: 16761651200
[2024-09-14 14:27:12.959] [DEBUG] – get disk info for path /dev/shm, total: 16774012928 avail: 16774012928
[2024-09-14 14:27:12.959] [DEBUG] – get disk info for path /run, total: 16774012928 avail: 16761864192
[2024-09-14 14:27:12.959] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 16774012928 avail: 16774012928
[2024-09-14 14:27:12.959] [DEBUG] – get disk info for path /boot, total: 1063256064 avail: 911314944
[2024-09-14 14:27:12.959] [DEBUG] – get disk info for path /run/user/1000, total: 3354804224 avail: 3354804224
[2024-09-14 14:27:12.960] [DEBUG] – admin@10.1.1.10 execute: df --block-size=1024 /home/admin/logs
[2024-09-14 14:27:13.014] [DEBUG] – exited code 0
[2024-09-14 14:27:13.015] [DEBUG] – get disk info for path /, total: 211415470080 avail: 43661705216
[2024-09-14 14:27:13.091] [DEBUG] - sub start_check ref count to 0
[2024-09-14 14:27:13.091] [DEBUG] - export start_check
[2024-09-14 14:27:13.091] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.0.0 for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[2024-09-14 14:27:13.091] [DEBUG] - import start
[2024-09-14 14:27:13.668] [DEBUG] - add start ref count to 1
[2024-09-14 14:27:13.668] [INFO] Start observer
[2024-09-14 14:27:13.669] [DEBUG] – admin@10.1.1.10 execute: ls /data/1/clog/tenant_1/
[2024-09-14 14:27:13.687] [DEBUG] – exited code 0
[2024-09-14 14:27:13.688] [DEBUG] – admin@10.1.1.10 execute: cat /home/admin/oceanbase/run/observer.pid
[2024-09-14 14:27:13.743] [DEBUG] – exited code 0
[2024-09-14 14:27:13.744] [DEBUG] – admin@10.1.1.10 execute: ls /proc/32157
[2024-09-14 14:27:13.799] [DEBUG] – exited code 2, error output:
[2024-09-14 14:27:13.799] [DEBUG] ls: cannot access /proc/32157: No such file or directory
[2024-09-14 14:27:13.799] [DEBUG]
[2024-09-14 14:27:13.800] [DEBUG] – 10.1.1.10 start command construction
[2024-09-14 14:27:13.800] [DEBUG] – starting 10.1.1.10 observer
[2024-09-14 14:27:13.800] [DEBUG] – admin@10.1.1.10 set env LD_LIBRARY_PATH to ‘/home/admin/oceanbase/lib:’
[2024-09-14 14:27:13.801] [DEBUG] – admin@10.1.1.10 execute: cd /home/admin/oceanbase; /home/admin/oceanbase/bin/observer -r ‘10.1.1.10:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -n ‘ocp’ -c 1725847271 -d ‘/data/1’ -I ‘10.1.1.10’ -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit=‘24290M’,datafile_size=‘72869M’,system_memory=‘5120M’,log_disk_size=‘62629M’,cpu_count=16,datafile_maxsize=‘116G’,datafile_next=‘11922M’
[2024-09-14 14:27:43.844] [ERROR] Traceback (most recent call last):
[2024-09-14 14:27:43.844] [ERROR] File “paramiko/channel.py”, line 699, in recv
[2024-09-14 14:27:43.844] [ERROR] File “paramiko/buffered_pipe.py”, line 164, in read
[2024-09-14 14:27:43.844] [ERROR] paramiko.buffered_pipe.PipeTimeout
[2024-09-14 14:27:43.844] [ERROR]
[2024-09-14 14:27:43.844] [ERROR] During handling of the above exception, another exception occurred:
[2024-09-14 14:27:43.844] [ERROR]
[2024-09-14 14:27:43.844] [ERROR] Traceback (most recent call last):
[2024-09-14 14:27:43.844] [ERROR] File “core.py”, line 2058, in start_cluster
[2024-09-14 14:27:43.844] [ERROR] File “core.py”, line 2152, in _start_cluster
[2024-09-14 14:27:43.844] [ERROR] File “core.py”, line 197, in call_plugin
[2024-09-14 14:27:43.845] [ERROR] File “_plugin.py”, line 347, in call
[2024-09-14 14:27:43.845] [ERROR] File “_plugin.py”, line 305, in _new_func
[2024-09-14 14:27:43.845] [ERROR] File “/home/admin/.obd/plugins/oceanbase-ce/4.2.0.0/start.py”, line 212, in start
[2024-09-14 14:27:43.845] [ERROR] ret = client.execute_command(clusters_cmd[server])
[2024-09-14 14:27:43.845] [ERROR] File “_plugin.py”, line 225, in new_method
[2024-09-14 14:27:43.845] [ERROR] File “_stdio.py”, line 895, in func_wrapper
[2024-09-14 14:27:43.845] [ERROR] File “ssh.py”, line 503, in execute_command
[2024-09-14 14:27:43.845] [ERROR] File “_stdio.py”, line 895, in func_wrapper
[2024-09-14 14:27:43.845] [ERROR] File “ssh.py”, line 464, in _execute_command
[2024-09-14 14:27:43.845] [ERROR] File “paramiko/file.py”, line 200, in read
[2024-09-14 14:27:43.845] [ERROR] File “paramiko/channel.py”, line 1361, in _read
[2024-09-14 14:27:43.845] [ERROR] File “paramiko/channel.py”, line 701, in recv
[2024-09-14 14:27:43.845] [ERROR] socket.timeout
[2024-09-14 14:27:43.845] [ERROR]
[2024-09-14 14:27:43.846] [DEBUG] – admin@10.1.1.10 delete env LD_LIBRARY_PATH
[2024-09-14 14:27:43.920] [ERROR] OBD-2002: Failed to start 10.1.1.10 observer:
[2024-09-14 14:27:43.920] [DEBUG] - sub start ref count to 0
[2024-09-14 14:27:43.920] [DEBUG] - export start
[2024-09-14 14:27:43.920] [ERROR] oceanbase-ce start failed
[2024-09-14 14:27:43.923] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2024-09-14 14:27:43.923] [INFO] Trace ID: 53d7b810-7262-11ef-8329-000c29b5e83e
[2024-09-14 14:27:43.923] [INFO] If you want to view detailed obd logs, please run: obd display-trace 53d7b810-7262-11ef-8329-000c29b5e83e
[2024-09-14 14:27:43.924] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 2
[2024-09-14 14:27:43.924] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 1
[2024-09-14 14:27:43.924] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 0
[2024-09-14 14:27:43.924] [DEBUG] - unlock /home/admin/.obd/lock/mirror_and_repo
[2024-09-14 14:27:43.924] [DEBUG] - exclusive lock /home/admin/.obd/lock/deploy_ocp release, count 0
[2024-09-14 14:27:43.924] [DEBUG] - unlock /home/admin/.obd/lock/deploy_ocp
[2024-09-14 14:27:43.924] [DEBUG] - share lock /home/admin/.obd/lock/global release, count 0
[2024-09-14 14:27:43.925] [DEBUG] - unlock /home/admin/.obd/lock/global
[admin@ocp ~]$