admin@ubuntu:~$ obd display-trace 174682d4-249a-11ef-94ab-c62cc4bad044 [2024-06-07 14:49:30.584] [DEBUG] - cmd: ['ocp-only'] [2024-06-07 14:49:30.584] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-06-07 14:49:30.584] [DEBUG] - mkdir /home/admin/.obd/lock/ [2024-06-07 14:49:30.585] [DEBUG] - unknown lock mode [2024-06-07 14:49:30.585] [DEBUG] - try to get share lock /home/admin/.obd/lock/global [2024-06-07 14:49:30.585] [DEBUG] - share lock `/home/admin/.obd/lock/global`, count 1 [2024-06-07 14:49:30.585] [DEBUG] - Get Deploy by name [2024-06-07 14:49:30.586] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-06-07 14:49:30.586] [DEBUG] - mkdir /home/admin/.obd/config_parser/ [2024-06-07 14:49:30.587] [DEBUG] - try to get exclusive lock /home/admin/.obd/lock/deploy_ocp-only [2024-06-07 14:49:30.587] [DEBUG] - exclusive lock `/home/admin/.obd/lock/deploy_ocp-only`, count 1 [2024-06-07 14:49:30.600] [DEBUG] - Deploy status judge [2024-06-07 14:49:30.601] [INFO] Get local repositories [2024-06-07 14:49:30.602] [DEBUG] - mkdir /home/admin/.obd/repository [2024-06-07 14:49:30.602] [DEBUG] - Get local repository ocp-server-ce-4.2.2-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.603] [DEBUG] - Search repository ocp-server-ce version: 4.2.2, tag: 84a65bf0ec004e73b772938696b2b896a0ef2b2f, release: None, package_hash: None [2024-06-07 14:49:30.603] [DEBUG] - try to get share lock /home/admin/.obd/lock/mirror_and_repo [2024-06-07 14:49:30.603] [DEBUG] - share lock `/home/admin/.obd/lock/mirror_and_repo`, count 1 [2024-06-07 14:49:30.603] [DEBUG] - mkdir /home/admin/.obd/repository/ocp-server-ce [2024-06-07 14:49:30.610] [DEBUG] - Found repository ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.733] [DEBUG] - Get deploy config [2024-06-07 14:49:30.770] [INFO] Search plugins [2024-06-07 14:49:30.771] [DEBUG] - Searching start_check plugin for components ... [2024-06-07 14:49:30.771] [DEBUG] - Searching start_check plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.771] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-06-07 14:49:30.773] [DEBUG] - Found for ocp-server-ce-py_script_start_check-4.2.2 for ocp-server-ce-4.2.2 [2024-06-07 14:49:30.773] [DEBUG] - Searching create_tenant plugin for components ... [2024-06-07 14:49:30.773] [DEBUG] - Searching create_tenant plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.774] [DEBUG] - No such create_tenant plugin for ocp-server-ce-4.2.2 [2024-06-07 14:49:30.774] [DEBUG] - Searching tenant_optimize plugin for components ... [2024-06-07 14:49:30.774] [DEBUG] - Searching tenant_optimize plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.775] [DEBUG] - No such tenant_optimize plugin for ocp-server-ce-4.2.2 [2024-06-07 14:49:30.775] [DEBUG] - Searching start plugin for components ... [2024-06-07 14:49:30.775] [DEBUG] - Searching start plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.776] [DEBUG] - Found for ocp-server-ce-py_script_start-4.2.1 for ocp-server-ce-4.2.2 [2024-06-07 14:49:30.776] [DEBUG] - Searching connect plugin for components ... [2024-06-07 14:49:30.776] [DEBUG] - Searching connect plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.777] [DEBUG] - Found for ocp-server-ce-py_script_connect-4.2.1 for ocp-server-ce-4.2.2 [2024-06-07 14:49:30.777] [DEBUG] - Searching bootstrap plugin for components ... [2024-06-07 14:49:30.777] [DEBUG] - Searching bootstrap plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.778] [DEBUG] - Found for ocp-server-ce-py_script_bootstrap-4.2.1 for ocp-server-ce-4.2.2 [2024-06-07 14:49:30.778] [DEBUG] - Searching display plugin for components ... [2024-06-07 14:49:30.778] [DEBUG] - Searching display plugin for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.779] [DEBUG] - Found for ocp-server-ce-py_script_display-4.2.1 for ocp-server-ce-4.2.2 [2024-06-07 14:49:30.902] [INFO] Load cluster param plugin [2024-06-07 14:49:30.903] [DEBUG] - Get local repository ocp-server-ce-4.2.2-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:30.903] [DEBUG] - Searching param plugin for components ... [2024-06-07 14:49:30.904] [DEBUG] - Search param plugin for ocp-server-ce [2024-06-07 14:49:30.904] [DEBUG] - Found for ocp-server-ce-param-4.2.1 for ocp-server-ce-4.2.2 [2024-06-07 14:49:30.904] [DEBUG] - Applying ocp-server-ce-param-4.2.1 for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:31.336] [INFO] Open ssh connection [2024-06-07 14:49:31.337] [DEBUG] - host: 10.xx.xx.126, port: 22, user: admin, password: None [2024-06-07 14:49:31.666] [DEBUG] - Call ocp-server-ce-py_script_start_check-4.2.2 for ocp-server-ce-4.2.2-20240315150922.el7-84a65bf0ec004e73b772938696b2b896a0ef2b2f [2024-06-07 14:49:31.667] [DEBUG] - import start_check [2024-06-07 14:49:31.678] [DEBUG] - add start_check ref count to 1 [2024-06-07 14:49:31.679] [INFO] Check before start ocp-server [2024-06-07 14:49:31.690] [DEBUG] -- oceanbase version check [2024-06-07 14:49:31.691] [DEBUG] -- admin@10.xx.xx.126 execute: sudo -n true [2024-06-07 14:49:31.708] [DEBUG] -- exited code 0 [2024-06-07 14:49:31.708] [DEBUG] -- admin@10.xx.xx.126 execute: cat /data/admin/ocp/run/ocp-server.pid [2024-06-07 14:49:31.759] [DEBUG] -- exited code 1, error output: [2024-06-07 14:49:31.759] [DEBUG] cat: /data/admin/ocp/run/ocp-server.pid: No such file or directory [2024-06-07 14:49:31.759] [DEBUG] [2024-06-07 14:49:31.760] [DEBUG] -- metadb connect check [2024-06-07 14:49:31.760] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:31.763] [DEBUG] -- check cursor failed [2024-06-07 14:49:32.764] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:32.767] [DEBUG] -- check cursor failed [2024-06-07 14:49:33.769] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:33.771] [DEBUG] -- check cursor failed [2024-06-07 14:49:34.772] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:34.775] [DEBUG] -- check cursor failed [2024-06-07 14:49:35.776] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:35.779] [DEBUG] -- check cursor failed [2024-06-07 14:49:36.780] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:36.783] [DEBUG] -- check cursor failed [2024-06-07 14:49:37.784] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:37.787] [DEBUG] -- check cursor failed [2024-06-07 14:49:38.788] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:38.791] [DEBUG] -- check cursor failed [2024-06-07 14:49:39.792] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:39.795] [DEBUG] -- check cursor failed [2024-06-07 14:49:40.796] [DEBUG] -- connect 10.xx.xx.126 -P2883 -umeta@ocp_meta -poceanbase [2024-06-07 14:49:40.799] [DEBUG] -- check cursor failed [2024-06-07 14:49:41.800] [ERROR] failed to connect meta db [2024-06-07 14:49:41.801] [DEBUG] -- time check [2024-06-07 14:49:41.801] [DEBUG] -- admin@10.xx.xx.126 execute: date +"%Y-%m-%d %H:%M:%S" [2024-06-07 14:49:41.808] [DEBUG] -- exited code 0 [2024-06-07 14:49:41.813] [DEBUG] -- now: 2024-06-07 14:49:41 [2024-06-07 14:49:41.813] [DEBUG] -- cursor: [2024-06-07 14:49:41.813] [DEBUG] -- user check [2024-06-07 14:49:41.813] [DEBUG] -- port check [2024-06-07 14:49:41.814] [DEBUG] -- admin@10.xx.xx.126 execute: bash -c 'cat /proc/net/{udp*,tcp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':6DB0' | awk -F' ' '{print $3}' | uniq [2024-06-07 14:49:41.869] [DEBUG] -- exited code 0 [2024-06-07 14:49:41.869] [DEBUG] -- java check [2024-06-07 14:49:41.870] [DEBUG] -- admin@10.xx.xx.126 append '/data/admin/ocp/jre/bin:' to PATH [2024-06-07 14:49:41.870] [DEBUG] -- admin@10.xx.xx.126 execute: java -version [2024-06-07 14:49:42.002] [DEBUG] -- exited code 0 [2024-06-07 14:49:42.003] [DEBUG] -- java version [2024-06-07 14:49:42.004] [DEBUG] -- java_major_version 1.8.0 [2024-06-07 14:49:42.004] [DEBUG] -- java_update_version 162 [2024-06-07 14:49:42.004] [DEBUG] -- clockdiff check [2024-06-07 14:49:42.004] [DEBUG] -- admin@10.xx.xx.126 execute: clockdiff -o 127.0.0.1 [2024-06-07 14:49:42.051] [DEBUG] -- exited code 1, error output: [2024-06-07 14:49:42.051] [DEBUG] clockdiff: nice: Operation not permitted [2024-06-07 14:49:42.051] [DEBUG] [2024-06-07 14:49:42.052] [DEBUG] -- admin@10.xx.xx.126 execute: sudo -n true [2024-06-07 14:49:42.107] [DEBUG] -- exited code 0 [2024-06-07 14:49:42.108] [DEBUG] -- admin@10.xx.xx.126 execute: sudo clockdiff -o 127.0.0.1 [2024-06-07 14:49:42.164] [DEBUG] -- exited code 0 [2024-06-07 14:49:42.164] [DEBUG] -- admin@10.xx.xx.126 execute: type -P clockdiff [2024-06-07 14:49:42.218] [DEBUG] -- exited code 0 [2024-06-07 14:49:42.218] [DEBUG] -- admin@10.xx.xx.126 execute: sudo chmod u+s /usr/bin/clockdiff [2024-06-07 14:49:42.218] [DEBUG] [2024-06-07 14:49:42.275] [DEBUG] -- exited code 0 [2024-06-07 14:49:42.276] [DEBUG] -- admin@10.xx.xx.126 execute: sudo setcap 'cap_net_raw+ep' /usr/bin/clockdiff [2024-06-07 14:49:42.276] [DEBUG] [2024-06-07 14:49:42.335] [DEBUG] -- exited code 0 [2024-06-07 14:49:42.336] [DEBUG] -- memory check [2024-06-07 14:49:42.336] [DEBUG] -- admin@10.xx.xx.126 execute: cat /proc/meminfo [2024-06-07 14:49:42.382] [DEBUG] -- exited code 0 [2024-06-07 14:49:42.383] [DEBUG] -- disk check [2024-06-07 14:49:42.384] [DEBUG] -- admin@10.xx.xx.126 execute: df --block-size=1024 [2024-06-07 14:49:42.430] [DEBUG] -- exited code 0 [2024-06-07 14:49:42.432] [DEBUG] -- get disk info for path /run, total: 6645743616 avail: 6643662848 [2024-06-07 14:49:42.432] [DEBUG] -- get disk info for path /, total: 29347631104 avail: 10520719360 [2024-06-07 14:49:42.432] [DEBUG] -- get disk info for path /dev/shm, total: 33228701696 avail: 33225408512 [2024-06-07 14:49:42.432] [DEBUG] -- get disk info for path /run/lock, total: 5242880 avail: 5242880 [2024-06-07 14:49:42.432] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 4194304 avail: 4194304 [2024-06-07 14:49:42.433] [DEBUG] -- get disk info for path /data, total: 434423177216 avail: 132594311168 [2024-06-07 14:49:42.433] [DEBUG] -- get disk info for path /run/user/0, total: 6645739520 avail: 6645739520 [2024-06-07 14:49:42.433] [DEBUG] -- get disk info for path /run/user/7038, total: 6645739520 avail: 6645739520 [2024-06-07 14:49:42.433] [DEBUG] -- get disk info for path /run/user/1002, total: 6645739520 avail: 6645739520 [2024-06-07 14:49:42.433] [DEBUG] -- admin@10.xx.xx.126 execute: df --block-size=1024 /data/admin/ocp/logs [2024-06-07 14:49:42.478] [DEBUG] -- exited code 0 [2024-06-07 14:49:42.479] [DEBUG] -- get disk info for path /data, total: 434423177216 avail: 132594311168 [2024-06-07 14:49:42.479] [DEBUG] -- admin@10.xx.xx.126 execute: ls /data/admin/ocp/.bootstrapped [2024-06-07 14:49:42.530] [DEBUG] -- exited code 2, error output: [2024-06-07 14:49:42.530] [DEBUG] ls: cannot access '/data/admin/ocp/.bootstrapped': No such file or directory [2024-06-07 14:49:42.530] [DEBUG] [2024-06-07 14:49:42.634] [INFO] [ERROR] failed to connect meta db [2024-06-07 14:49:42.634] [INFO] [2024-06-07 14:49:42.634] [DEBUG] - sub start_check ref count to 0 [2024-06-07 14:49:42.634] [DEBUG] - export start_check [2024-06-07 14:49:42.634] [DEBUG] - ocp-server-ce starting check failed. [2024-06-07 14:49:42.638] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2024-06-07 14:49:42.638] [INFO] Trace ID: 174682d4-249a-11ef-94ab-c62cc4bad044 [2024-06-07 14:49:42.639] [INFO] If you want to view detailed obd logs, please run: obd display-trace 174682d4-249a-11ef-94ab-c62cc4bad044 [2024-06-07 14:49:42.639] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 0 [2024-06-07 14:49:42.639] [DEBUG] - unlock /home/admin/.obd/lock/mirror_and_repo [2024-06-07 14:49:42.639] [DEBUG] - exclusive lock /home/admin/.obd/lock/deploy_ocp-only release, count 0 [2024-06-07 14:49:42.639] [DEBUG] - unlock /home/admin/.obd/lock/deploy_ocp-only [2024-06-07 14:49:42.640] [DEBUG] - share lock /home/admin/.obd/lock/global release, count 0 [2024-06-07 14:49:42.640] [DEBUG] - unlock /home/admin/.obd/lock/global