[admin@sh1-kurento log]$ obd display-trace e8dc9108-f7b3-11ee-8254-fa163e7437b3 [2024-04-11 11:30:57.304] [DEBUG] - cmd: ['myob'] [2024-04-11 11:30:57.304] [DEBUG] - opts: {'servers': None, 'components': 'ocp-server-ce', 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-11 11:30:57.304] [DEBUG] - mkdir /home/admin/.obd/lock/ [2024-04-11 11:30:57.304] [DEBUG] - unknown lock mode [2024-04-11 11:30:57.305] [DEBUG] - try to get share lock /home/admin/.obd/lock/global [2024-04-11 11:30:57.305] [DEBUG] - share lock `/home/admin/.obd/lock/global`, count 1 [2024-04-11 11:30:57.305] [DEBUG] - Get Deploy by name [2024-04-11 11:30:57.305] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-11 11:30:57.305] [DEBUG] - mkdir /home/admin/.obd/config_parser/ [2024-04-11 11:30:57.305] [DEBUG] - try to get exclusive lock /home/admin/.obd/lock/deploy_myob [2024-04-11 11:30:57.305] [DEBUG] - exclusive lock `/home/admin/.obd/lock/deploy_myob`, count 1 [2024-04-11 11:30:57.311] [DEBUG] - Deploy status judge [2024-04-11 11:30:57.312] [INFO] Get local repositories [2024-04-11 11:30:57.312] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-11 11:30:57.312] [DEBUG] - Get local repository oceanbase-ce-4.3.0.1-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.313] [DEBUG] - Search repository oceanbase-ce version: 4.3.0.1, tag: c4a03c83614f50c99ddb1c37dda858fa5d9b14b7, release: None, package_hash: None [2024-04-11 11:30:57.313] [DEBUG] - try to get share lock /home/admin/.obd/lock/mirror_and_repo [2024-04-11 11:30:57.313] [DEBUG] - share lock `/home/admin/.obd/lock/mirror_and_repo`, count 1 [2024-04-11 11:30:57.313] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-11 11:30:57.315] [DEBUG] - Found repository oceanbase-ce-4.3.0.1-100000242024032211.el7-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.315] [DEBUG] - Get local repository obproxy-ce-4.2.3.0-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.315] [DEBUG] - Search repository obproxy-ce version: 4.2.3.0, tag: 0490ebc04220def8d25cb9cac9ac61a4efa6d639, release: None, package_hash: None [2024-04-11 11:30:57.315] [DEBUG] - share lock `/home/admin/.obd/lock/mirror_and_repo`, count 2 [2024-04-11 11:30:57.315] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy-ce [2024-04-11 11:30:57.317] [DEBUG] - Found repository obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.317] [DEBUG] - Get local repository ocp-server-ce-4.2.2-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.317] [DEBUG] - Search repository ocp-server-ce version: 4.2.2, tag: 84a65bf0ec004e73b772938696b2b896a0ef2b2f, release: None, package_hash: None [2024-04-11 11:30:57.317] [DEBUG] - share lock `/home/admin/.obd/lock/mirror_and_repo`, count 3 [2024-04-11 11:30:57.317] [DEBUG] - mkdir /home/admin/.obd/repository/ocp-server-ce [2024-04-11 11:30:57.319] [DEBUG] - Found repository ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.443] [DEBUG] - Get deploy config [2024-04-11 11:30:57.467] [INFO] Search plugins [2024-04-11 11:30:57.467] [DEBUG] - Searching start_check plugin for components ... [2024-04-11 11:30:57.467] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.3.0.1-100000242024032211.el7-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.467] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-11 11:30:57.468] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.3.0.1 [2024-04-11 11:30:57.468] [DEBUG] - Searching start_check plugin for obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.468] [DEBUG] - Found for obproxy-ce-py_script_start_check-4.2.3 for obproxy-ce-4.2.3.0 [2024-04-11 11:30:57.469] [DEBUG] - Searching start_check plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.469] [DEBUG] - Found for ocp-server-ce-py_script_start_check-4.2.2 for ocp-server-ce-4.2.2 [2024-04-11 11:30:57.469] [DEBUG] - Searching create_tenant plugin for components ... [2024-04-11 11:30:57.469] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.3.0.1-100000242024032211.el7-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.469] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.3.0.0 for oceanbase-ce-4.3.0.1 [2024-04-11 11:30:57.469] [DEBUG] - Searching create_tenant plugin for obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.469] [DEBUG] - No such create_tenant plugin for obproxy-ce-4.2.3.0 [2024-04-11 11:30:57.469] [DEBUG] - Searching create_tenant plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.470] [DEBUG] - No such create_tenant plugin for ocp-server-ce-4.2.2 [2024-04-11 11:30:57.470] [DEBUG] - Searching tenant_optimize plugin for components ... [2024-04-11 11:30:57.470] [DEBUG] - Searching tenant_optimize plugin for oceanbase-ce-4.3.0.1-100000242024032211.el7-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.470] [DEBUG] - Found for oceanbase-ce-py_script_tenant_optimize-4.3.0.0 for oceanbase-ce-4.3.0.1 [2024-04-11 11:30:57.470] [DEBUG] - Searching tenant_optimize plugin for obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.470] [DEBUG] - No such tenant_optimize plugin for obproxy-ce-4.2.3.0 [2024-04-11 11:30:57.470] [DEBUG] - Searching tenant_optimize plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.470] [DEBUG] - No such tenant_optimize plugin for ocp-server-ce-4.2.2 [2024-04-11 11:30:57.470] [DEBUG] - Searching start plugin for components ... [2024-04-11 11:30:57.470] [DEBUG] - Searching start plugin for oceanbase-ce-4.3.0.1-100000242024032211.el7-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.471] [DEBUG] - Found for oceanbase-ce-py_script_start-4.3.0.0 for oceanbase-ce-4.3.0.1 [2024-04-11 11:30:57.471] [DEBUG] - Searching start plugin for obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.471] [DEBUG] - Found for obproxy-ce-py_script_start-4.2.3 for obproxy-ce-4.2.3.0 [2024-04-11 11:30:57.471] [DEBUG] - Searching start plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.471] [DEBUG] - Found for ocp-server-ce-py_script_start-4.2.1 for ocp-server-ce-4.2.2 [2024-04-11 11:30:57.471] [DEBUG] - Searching connect plugin for components ... [2024-04-11 11:30:57.471] [DEBUG] - Searching connect plugin for oceanbase-ce-4.3.0.1-100000242024032211.el7-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.472] [DEBUG] - Found for oceanbase-ce-py_script_connect-4.2.2.0 for oceanbase-ce-4.3.0.1 [2024-04-11 11:30:57.472] [DEBUG] - Searching connect plugin for obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.472] [DEBUG] - Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.2.3.0 [2024-04-11 11:30:57.472] [DEBUG] - Searching connect plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.472] [DEBUG] - Found for ocp-server-ce-py_script_connect-4.2.1 for ocp-server-ce-4.2.2 [2024-04-11 11:30:57.472] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-11 11:30:57.472] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.3.0.1-100000242024032211.el7-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.473] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.2.0 for oceanbase-ce-4.3.0.1 [2024-04-11 11:30:57.473] [DEBUG] - Searching bootstrap plugin for obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.473] [DEBUG] - Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.2.3.0 [2024-04-11 11:30:57.473] [DEBUG] - Searching bootstrap plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.473] [DEBUG] - Found for ocp-server-ce-py_script_bootstrap-4.2.1 for ocp-server-ce-4.2.2 [2024-04-11 11:30:57.473] [DEBUG] - Searching display plugin for components ... [2024-04-11 11:30:57.473] [DEBUG] - Searching display plugin for oceanbase-ce-4.3.0.1-100000242024032211.el7-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.474] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.3.0.1 [2024-04-11 11:30:57.474] [DEBUG] - Searching display plugin for obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.474] [DEBUG] - Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.2.3.0 [2024-04-11 11:30:57.474] [DEBUG] - Searching display plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.474] [DEBUG] - Found for ocp-server-ce-py_script_display-4.2.1 for ocp-server-ce-4.2.2 [2024-04-11 11:30:57.598] [INFO] Load cluster param plugin [2024-04-11 11:30:57.599] [DEBUG] - Get local repository oceanbase-ce-4.3.0.1-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:57.599] [DEBUG] - Get local repository obproxy-ce-4.2.3.0-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:57.599] [DEBUG] - Get local repository ocp-server-ce-4.2.2-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:57.599] [DEBUG] - Searching param plugin for components ... [2024-04-11 11:30:57.599] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-11 11:30:57.599] [DEBUG] - Found for oceanbase-ce-param-4.3.0.0 for oceanbase-ce-4.3.0.1 [2024-04-11 11:30:57.599] [DEBUG] - Applying oceanbase-ce-param-4.3.0.0 for oceanbase-ce-4.3.0.1-100000242024032211.el7-c4a03c83614f50c99ddb1c37dda858fa5d9b14b7 [2024-04-11 11:30:58.070] [DEBUG] - Search param plugin for obproxy-ce [2024-04-11 11:30:58.070] [DEBUG] - Found for obproxy-ce-param-4.2.3 for obproxy-ce-4.2.3.0 [2024-04-11 11:30:58.070] [DEBUG] - Applying obproxy-ce-param-4.2.3 for obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 [2024-04-11 11:30:58.179] [DEBUG] - Search param plugin for ocp-server-ce [2024-04-11 11:30:58.179] [DEBUG] - Found for ocp-server-ce-param-4.2.1 for ocp-server-ce-4.2.2 [2024-04-11 11:30:58.179] [DEBUG] - Applying ocp-server-ce-param-4.2.1 for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:58.281] [INFO] Open ssh connection [2024-04-11 11:30:58.282] [DEBUG] - host: xx.xx.xx.75, port: 22, user: admin, password: MyPasswd__2023 [2024-04-11 11:30:58.345] [DEBUG] - host: xx.xx.xx.15, port: 22, user: admin, password: MyPasswd__2023 [2024-04-11 11:30:58.408] [DEBUG] - host: xx.xx.xx.11, port: 22, user: admin, password: MyPasswd__2023 [2024-04-11 11:30:58.542] [DEBUG] - Call ocp-server-ce-py_script_start_check-4.2.2 for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:58.542] [DEBUG] - import start_check [2024-04-11 11:30:58.547] [DEBUG] - add start_check ref count to 1 [2024-04-11 11:30:58.547] [INFO] Check before start ocp-server [2024-04-11 11:30:58.551] [DEBUG] -- oceanbase version check [2024-04-11 11:30:58.551] [DEBUG] -- admin@xx.xx.xx.75 execute: sudo -n true [2024-04-11 11:30:58.585] [DEBUG] -- exited code 0 [2024-04-11 11:30:58.585] [DEBUG] -- admin@xx.xx.xx.75 execute: cat /home/admin/ocp/run/ocp-server.pid [2024-04-11 11:30:58.651] [DEBUG] -- exited code 0 [2024-04-11 11:30:58.652] [DEBUG] -- admin@xx.xx.xx.75 execute: ls /proc/11940 [2024-04-11 11:30:58.720] [DEBUG] -- exited code 2, error output: [2024-04-11 11:30:58.720] [DEBUG] ls: cannot access /proc/11940: No such file or directory [2024-04-11 11:30:58.720] [DEBUG] [2024-04-11 11:30:58.720] [DEBUG] -- user check [2024-04-11 11:30:58.720] [DEBUG] -- port check [2024-04-11 11:30:58.720] [DEBUG] -- admin@xx.xx.xx.75 execute: bash -c 'cat /proc/net/{udp*,tcp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1F90' | awk -F' ' '{print $3}' | uniq [2024-04-11 11:30:58.790] [DEBUG] -- exited code 0 [2024-04-11 11:30:58.790] [DEBUG] -- java check [2024-04-11 11:30:58.791] [DEBUG] -- admin@xx.xx.xx.75 append '/home/admin/ocp/jre/bin:' to PATH [2024-04-11 11:30:58.791] [DEBUG] -- admin@xx.xx.xx.75 execute: java -version [2024-04-11 11:30:58.908] [DEBUG] -- exited code 0 [2024-04-11 11:30:58.909] [DEBUG] -- java version [2024-04-11 11:30:58.909] [DEBUG] -- java_major_version 1.8.0 [2024-04-11 11:30:58.909] [DEBUG] -- java_update_version 402 [2024-04-11 11:30:58.909] [DEBUG] -- clockdiff check [2024-04-11 11:30:58.909] [DEBUG] -- admin@xx.xx.xx.75 execute: clockdiff -o 127.0.0.1 [2024-04-11 11:30:58.975] [DEBUG] -- exited code 0 [2024-04-11 11:30:58.975] [DEBUG] -- memory check [2024-04-11 11:30:58.975] [DEBUG] -- admin@xx.xx.xx.75 execute: cat /proc/meminfo [2024-04-11 11:30:59.042] [DEBUG] -- exited code 0 [2024-04-11 11:30:59.043] [DEBUG] -- disk check [2024-04-11 11:30:59.043] [DEBUG] -- admin@xx.xx.xx.75 execute: df --block-size=1024 [2024-04-11 11:30:59.110] [DEBUG] -- exited code 0 [2024-04-11 11:30:59.110] [DEBUG] -- get disk info for path /dev, total: 33681616896 avail: 33681616896 [2024-04-11 11:30:59.111] [DEBUG] -- get disk info for path /dev/shm, total: 33692610560 avail: 33692610560 [2024-04-11 11:30:59.111] [DEBUG] -- get disk info for path /run, total: 33692610560 avail: 33633005568 [2024-04-11 11:30:59.111] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 33692610560 avail: 33692610560 [2024-04-11 11:30:59.111] [DEBUG] -- get disk info for path /, total: 38634766336 avail: 6149804032 [2024-04-11 11:30:59.111] [DEBUG] -- get disk info for path /webapps, total: 214691745792 avail: 65430011904 [2024-04-11 11:30:59.111] [DEBUG] -- get disk info for path /run/user/2000, total: 6738526208 avail: 6738526208 [2024-04-11 11:30:59.111] [DEBUG] -- get disk info for path /run/user/33006, total: 6738526208 avail: 6738526208 [2024-04-11 11:30:59.111] [DEBUG] -- admin@xx.xx.xx.75 execute: df --block-size=1024 /webapps/obdata/logs [2024-04-11 11:30:59.178] [DEBUG] -- exited code 0 [2024-04-11 11:30:59.178] [DEBUG] -- get disk info for path /webapps, total: 214691745792 avail: 65430011904 [2024-04-11 11:30:59.199] [DEBUG] - sub start_check ref count to 0 [2024-04-11 11:30:59.199] [DEBUG] - export start_check [2024-04-11 11:30:59.199] [DEBUG] - Call ocp-server-ce-py_script_start-4.2.1 for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-04-11 11:30:59.200] [DEBUG] - import start [2024-04-11 11:30:59.204] [DEBUG] - add start ref count to 1 [2024-04-11 11:30:59.204] [DEBUG] -- metadb connect check [2024-04-11 11:30:59.205] [DEBUG] -- connect xx.xx.xx.75 -P2883 -uroot@ocp_meta -pMyPasswd__2023 [2024-04-11 11:30:59.206] [ERROR] ocp-server-ce-py_script_start-4.2.1 RuntimeError: (2003, "Can't connect to MySQL server on 'xx.xx.xx.75' ([Errno 111] Connection refused)") [2024-04-11 11:30:59.206] [ERROR] Traceback (most recent call last): [2024-04-11 11:30:59.206] [ERROR] File "pymysql/connections.py", line 613, in connect [2024-04-11 11:30:59.206] [ERROR] File "socket.py", line 808, in create_connection [2024-04-11 11:30:59.206] [ERROR] File "socket.py", line 796, in create_connection [2024-04-11 11:30:59.206] [ERROR] ConnectionRefusedError: [Errno 111] Connection refused [2024-04-11 11:30:59.206] [ERROR] [2024-04-11 11:30:59.206] [ERROR] During handling of the above exception, another exception occurred: [2024-04-11 11:30:59.206] [ERROR] [2024-04-11 11:30:59.206] [ERROR] Traceback (most recent call last): [2024-04-11 11:30:59.206] [ERROR] File "core.py", line 2058, in start_cluster [2024-04-11 11:30:59.206] [ERROR] File "core.py", line 2152, in _start_cluster [2024-04-11 11:30:59.206] [ERROR] File "core.py", line 197, in call_plugin [2024-04-11 11:30:59.206] [ERROR] File "_plugin.py", line 347, in __call__ [2024-04-11 11:30:59.206] [ERROR] File "_plugin.py", line 305, in _new_func [2024-04-11 11:30:59.206] [ERROR] File "/home/admin/.obd/plugins/ocp-server-ce/4.2.1/start.py", line 457, in start [2024-04-11 11:30:59.206] [ERROR] if not start_cluster(): [2024-04-11 11:30:59.206] [ERROR] File "/home/admin/.obd/plugins/ocp-server-ce/4.2.1/start.py", line 242, in start_cluster [2024-04-11 11:30:59.206] [ERROR] meta_cursor = Cursor(jdbc_host, jdbc_port, meta_user, meta_tenant, meta_password, stdio) [2024-04-11 11:30:59.206] [ERROR] File "_stdio.py", line 908, in wrapper [2024-04-11 11:30:59.206] [ERROR] File "tool.py", line 766, in __init__ [2024-04-11 11:30:59.206] [ERROR] File "tool.py", line 796, in _connect [2024-04-11 11:30:59.206] [ERROR] File "pymysql/connections.py", line 353, in __init__ [2024-04-11 11:30:59.206] [ERROR] File "pymysql/connections.py", line 664, in connect [2024-04-11 11:30:59.207] [ERROR] pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'xx.xx.xx.75' ([Errno 111] Connection refused)") [2024-04-11 11:30:59.207] [ERROR] [2024-04-11 11:30:59.207] [DEBUG] - sub start ref count to 0 [2024-04-11 11:30:59.207] [DEBUG] - export start [2024-04-11 11:30:59.207] [ERROR] ocp-server-ce start failed [2024-04-11 11:30:59.209] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2024-04-11 11:30:59.210] [INFO] Trace ID: e8dc9108-f7b3-11ee-8254-fa163e7437b3 [2024-04-11 11:30:59.210] [INFO] If you want to view detailed obd logs, please run: obd display-trace e8dc9108-f7b3-11ee-8254-fa163e7437b3 [2024-04-11 11:30:59.210] [DEBUG] - unlock /home/admin/.obd/lock/global [2024-04-11 11:30:59.210] [DEBUG] - unlock /home/admin/.obd/lock/deploy_myob [2024-04-11 11:30:59.210] [DEBUG] - unlock /home/admin/.obd/lock/mirror_and_repo