[root@localhost ~]$ obd display-trace 48d1de34-da29-11f0-8a44-6c92bf076f4b [2025-12-16 10:45:29.871] [DEBUG] - cmd: ['zhangchong_ob_test'] [2025-12-16 10:45:29.872] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2025-12-16 10:45:29.872] [DEBUG] - mkdir /root/.obd/lock/ [2025-12-16 10:45:29.872] [DEBUG] - unknown lock mode [2025-12-16 10:45:29.872] [DEBUG] - try to get share lock /root/.obd/lock/global [2025-12-16 10:45:29.872] [DEBUG] - share lock `/root/.obd/lock/global`, count 1 [2025-12-16 10:45:29.872] [DEBUG] - Get Deploy by name [2025-12-16 10:45:29.873] [DEBUG] - mkdir /root/.obd/cluster/ [2025-12-16 10:45:29.874] [DEBUG] - mkdir /root/.obd/config_parser/ [2025-12-16 10:45:29.874] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_zhangchong_ob_test [2025-12-16 10:45:29.874] [DEBUG] - exclusive lock `/root/.obd/lock/deploy_zhangchong_ob_test`, count 1 [2025-12-16 10:45:29.881] [DEBUG] - Deploy status judge [2025-12-16 10:45:29.881] [INFO] Get local repositories [2025-12-16 10:45:29.882] [DEBUG] - mkdir /root/.obd/repository [2025-12-16 10:45:29.882] [DEBUG] - Get local repository oceanbase-ce-4.3.5.4-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:29.882] [DEBUG] - Search repository oceanbase-ce version: 4.3.5.4, tag: a30a15dd7a80f6acd0f113993bb70b2e56d40f80, release: None, package_hash: None [2025-12-16 10:45:29.882] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo [2025-12-16 10:45:29.882] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 1 [2025-12-16 10:45:29.883] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce [2025-12-16 10:45:29.884] [DEBUG] - Found repository oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:29.884] [DEBUG] - Get local repository obproxy-ce-4.3.5.0-f17b277b681adb1c86bfc3cfda369ad88896da9d [2025-12-16 10:45:29.884] [DEBUG] - Search repository obproxy-ce version: 4.3.5.0, tag: f17b277b681adb1c86bfc3cfda369ad88896da9d, release: None, package_hash: None [2025-12-16 10:45:29.885] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 2 [2025-12-16 10:45:29.885] [DEBUG] - mkdir /root/.obd/repository/obproxy-ce [2025-12-16 10:45:29.886] [DEBUG] - Found repository obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d [2025-12-16 10:45:29.886] [DEBUG] - Get local repository ob-configserver-1.0.1-324f9c93ded9c562896719234748062dfdd93cd8 [2025-12-16 10:45:29.886] [DEBUG] - Search repository ob-configserver version: 1.0.1, tag: 324f9c93ded9c562896719234748062dfdd93cd8, release: None, package_hash: None [2025-12-16 10:45:29.886] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 3 [2025-12-16 10:45:29.887] [DEBUG] - mkdir /root/.obd/repository/ob-configserver [2025-12-16 10:45:29.888] [DEBUG] - Found repository ob-configserver-1.0.1-1.0.1.el7-324f9c93ded9c562896719234748062dfdd93cd8 [2025-12-16 10:45:29.888] [DEBUG] - Get local repository obagent-4.2.4-bae1154d745991c109af6cb5b151fb2831594df4 [2025-12-16 10:45:29.888] [DEBUG] - Search repository obagent version: 4.2.4, tag: bae1154d745991c109af6cb5b151fb2831594df4, release: None, package_hash: None [2025-12-16 10:45:29.888] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 4 [2025-12-16 10:45:29.888] [DEBUG] - mkdir /root/.obd/repository/obagent [2025-12-16 10:45:29.890] [DEBUG] - Found repository obagent-4.2.4-200000022025090416.el7-bae1154d745991c109af6cb5b151fb2831594df4 [2025-12-16 10:45:29.890] [DEBUG] - Get local repository prometheus-2.37.1-d5fe6d40b6ccd6de9de036fd294966d044a3c328 [2025-12-16 10:45:29.890] [DEBUG] - Search repository prometheus version: 2.37.1, tag: d5fe6d40b6ccd6de9de036fd294966d044a3c328, release: None, package_hash: None [2025-12-16 10:45:29.890] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 5 [2025-12-16 10:45:29.890] [DEBUG] - mkdir /root/.obd/repository/prometheus [2025-12-16 10:45:29.892] [DEBUG] - Found repository prometheus-2.37.1-10000272025073110.el7-d5fe6d40b6ccd6de9de036fd294966d044a3c328 [2025-12-16 10:45:29.892] [DEBUG] - Get local repository grafana-7.5.17-1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6 [2025-12-16 10:45:29.892] [DEBUG] - Search repository grafana version: 7.5.17, tag: 1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6, release: None, package_hash: None [2025-12-16 10:45:29.892] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 6 [2025-12-16 10:45:29.892] [DEBUG] - mkdir /root/.obd/repository/grafana [2025-12-16 10:45:29.894] [DEBUG] - Found repository grafana-7.5.17-1-1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6 [2025-12-16 10:45:30.012] [DEBUG] - Get deploy config [2025-12-16 10:45:30.013] [INFO] Load cluster param plugin [2025-12-16 10:45:30.013] [DEBUG] - Get local repository oceanbase-ce-4.3.5.4-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:30.014] [DEBUG] - Get local repository obproxy-ce-4.3.5.0-f17b277b681adb1c86bfc3cfda369ad88896da9d [2025-12-16 10:45:30.014] [DEBUG] - Get local repository ob-configserver-1.0.1-324f9c93ded9c562896719234748062dfdd93cd8 [2025-12-16 10:45:30.014] [DEBUG] - Get local repository obagent-4.2.4-bae1154d745991c109af6cb5b151fb2831594df4 [2025-12-16 10:45:30.014] [DEBUG] - Get local repository prometheus-2.37.1-d5fe6d40b6ccd6de9de036fd294966d044a3c328 [2025-12-16 10:45:30.014] [DEBUG] - Get local repository grafana-7.5.17-1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6 [2025-12-16 10:45:30.014] [DEBUG] - Searching param plugin for components ... [2025-12-16 10:45:30.014] [DEBUG] - Search param plugin for oceanbase-ce [2025-12-16 10:45:30.014] [DEBUG] - mkdir /root/.obd/plugins [2025-12-16 10:45:30.016] [DEBUG] - Found for oceanbase-ce-param-4.3.3.0 for oceanbase-ce-4.3.5.4 [2025-12-16 10:45:30.016] [DEBUG] - Applying oceanbase-ce-param-4.3.3.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:30.416] [DEBUG] - Search param plugin for obproxy-ce [2025-12-16 10:45:30.416] [DEBUG] - Found for obproxy-ce-param-4.3.0 for obproxy-ce-4.3.5.0 [2025-12-16 10:45:30.416] [DEBUG] - Applying obproxy-ce-param-4.3.0 for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d [2025-12-16 10:45:30.516] [DEBUG] - Search param plugin for ob-configserver [2025-12-16 10:45:30.516] [DEBUG] - Found for ob-configserver-param-1.0.0 for ob-configserver-1.0.1 [2025-12-16 10:45:30.517] [DEBUG] - Applying ob-configserver-param-1.0.0 for ob-configserver-1.0.1-1.0.1.el7-324f9c93ded9c562896719234748062dfdd93cd8 [2025-12-16 10:45:30.536] [DEBUG] - Search param plugin for obagent [2025-12-16 10:45:30.536] [DEBUG] - Found for obagent-param-4.2.2 for obagent-4.2.4 [2025-12-16 10:45:30.536] [DEBUG] - Applying obagent-param-4.2.2 for obagent-4.2.4-200000022025090416.el7-bae1154d745991c109af6cb5b151fb2831594df4 [2025-12-16 10:45:30.584] [DEBUG] - Search param plugin for prometheus [2025-12-16 10:45:30.584] [DEBUG] - Found for prometheus-param-2.37.1 for prometheus-2.37.1 [2025-12-16 10:45:30.584] [DEBUG] - Applying prometheus-param-2.37.1 for prometheus-2.37.1-10000272025073110.el7-d5fe6d40b6ccd6de9de036fd294966d044a3c328 [2025-12-16 10:45:30.597] [DEBUG] - Search param plugin for grafana [2025-12-16 10:45:30.597] [DEBUG] - Found for grafana-param-7.5.17 for grafana-7.5.17 [2025-12-16 10:45:30.597] [DEBUG] - Applying grafana-param-7.5.17 for grafana-7.5.17-1-1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6 [2025-12-16 10:45:30.749] [INFO] Open ssh connection [2025-12-16 10:45:30.758] [DEBUG] - host: 10.1.224.54, port: 22, user: root, password: ****** [2025-12-16 10:45:30.863] [DEBUG] - host: 10.1.224.54, port: 22, user: root, password: ****** [2025-12-16 10:45:30.960] [DEBUG] - host: 10.1.224.55, port: 22, user: root, password: ****** [2025-12-16 10:45:31.058] [DEBUG] - host: 10.1.224.53, port: 22, user: root, password: ****** [2025-12-16 10:45:31.184] [DEBUG] - host: 10.1.224.55, port: 22, user: root, password: ****** [2025-12-16 10:45:31.276] [DEBUG] - host: 10.1.224.52, port: 22, user: root, password: ****** [2025-12-16 10:45:31.371] [DEBUG] - host: 10.1.224.53, port: 22, user: root, password: ****** [2025-12-16 10:45:31.541] [DEBUG] - Searching start_check template for components ... [2025-12-16 10:45:31.541] [DEBUG] - mkdir /root/.obd/workflows [2025-12-16 10:45:31.544] [DEBUG] - Call workflow oceanbase-ce-py_script_workflow_start_check-4.2.5.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:31.544] [DEBUG] - mkdir /root/.obd/mirror [2025-12-16 10:45:31.545] [DEBUG] - mkdir /root/.obd/mirror/remote [2025-12-16 10:45:31.545] [DEBUG] - mkdir /root/.obd/mirror/local [2025-12-16 10:45:31.545] [DEBUG] - mkdir /root/.obd/optimize/ [2025-12-16 10:45:31.545] [DEBUG] - mkdir /root/.obd/tool/ [2025-12-16 10:45:31.546] [DEBUG] - import start_check [2025-12-16 10:45:31.547] [DEBUG] - add start_check ref count to 1 [2025-12-16 10:45:31.547] [DEBUG] - sub start_check ref count to 0 [2025-12-16 10:45:31.547] [DEBUG] - export start_check [2025-12-16 10:45:31.547] [DEBUG] - plugin oceanbase-ce-py_script_workflow_start_check-4.2.5.0 result: True [2025-12-16 10:45:31.548] [DEBUG] - Found for oceanbase-ce-py_script_workflow_start_check-4.2.5.0 for oceanbase-ce-4.2.5.0 [2025-12-16 10:45:31.548] [DEBUG] - Searching start_check template for components ... [2025-12-16 10:45:31.548] [DEBUG] - Call workflow obproxy-ce-py_script_workflow_start_check-4.2.3 for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d [2025-12-16 10:45:31.548] [DEBUG] - import start_check [2025-12-16 10:45:31.549] [DEBUG] - add start_check ref count to 1 [2025-12-16 10:45:31.549] [DEBUG] - sub start_check ref count to 0 [2025-12-16 10:45:31.549] [DEBUG] - export start_check [2025-12-16 10:45:31.550] [DEBUG] - plugin obproxy-ce-py_script_workflow_start_check-4.2.3 result: True [2025-12-16 10:45:31.550] [DEBUG] - Found for obproxy-ce-py_script_workflow_start_check-4.2.3 for obproxy-ce-4.2.3 [2025-12-16 10:45:31.550] [DEBUG] - Searching start_check template for components ... [2025-12-16 10:45:31.550] [DEBUG] - Call workflow ob-configserver-py_script_workflow_start_check-0.1 for ob-configserver-1.0.1-1.0.1.el7-324f9c93ded9c562896719234748062dfdd93cd8 [2025-12-16 10:45:31.550] [DEBUG] - import start_check [2025-12-16 10:45:31.551] [DEBUG] - add start_check ref count to 1 [2025-12-16 10:45:31.551] [DEBUG] - sub start_check ref count to 0 [2025-12-16 10:45:31.551] [DEBUG] - export start_check [2025-12-16 10:45:31.552] [DEBUG] - plugin ob-configserver-py_script_workflow_start_check-0.1 result: True [2025-12-16 10:45:31.552] [DEBUG] - Found for ob-configserver-py_script_workflow_start_check-0.1 for ob-configserver-0.1 [2025-12-16 10:45:31.552] [DEBUG] - Searching start_check template for components ... [2025-12-16 10:45:31.552] [DEBUG] - Call workflow obagent-py_script_workflow_start_check-1.3.0 for obagent-4.2.4-200000022025090416.el7-bae1154d745991c109af6cb5b151fb2831594df4 [2025-12-16 10:45:31.552] [DEBUG] - import start_check [2025-12-16 10:45:31.553] [DEBUG] - add start_check ref count to 1 [2025-12-16 10:45:31.553] [DEBUG] - sub start_check ref count to 0 [2025-12-16 10:45:31.553] [DEBUG] - export start_check [2025-12-16 10:45:31.553] [DEBUG] - plugin obagent-py_script_workflow_start_check-1.3.0 result: True [2025-12-16 10:45:31.553] [DEBUG] - Found for obagent-py_script_workflow_start_check-1.3.0 for obagent-1.3.0 [2025-12-16 10:45:31.554] [DEBUG] - Searching start_check template for components ... [2025-12-16 10:45:31.554] [DEBUG] - Call workflow prometheus-py_script_workflow_start_check-2.37.1 for prometheus-2.37.1-10000272025073110.el7-d5fe6d40b6ccd6de9de036fd294966d044a3c328 [2025-12-16 10:45:31.554] [DEBUG] - import start_check [2025-12-16 10:45:31.555] [DEBUG] - add start_check ref count to 1 [2025-12-16 10:45:31.555] [DEBUG] - sub start_check ref count to 0 [2025-12-16 10:45:31.555] [DEBUG] - export start_check [2025-12-16 10:45:31.555] [DEBUG] - plugin prometheus-py_script_workflow_start_check-2.37.1 result: True [2025-12-16 10:45:31.555] [DEBUG] - Found for prometheus-py_script_workflow_start_check-2.37.1 for prometheus-2.37.1 [2025-12-16 10:45:31.555] [DEBUG] - Searching start_check template for components ... [2025-12-16 10:45:31.556] [DEBUG] - Call workflow grafana-py_script_workflow_start_check-7.5.17 for grafana-7.5.17-1-1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6 [2025-12-16 10:45:31.556] [DEBUG] - import start_check [2025-12-16 10:45:31.556] [DEBUG] - add start_check ref count to 1 [2025-12-16 10:45:31.557] [DEBUG] - sub start_check ref count to 0 [2025-12-16 10:45:31.557] [DEBUG] - export start_check [2025-12-16 10:45:31.557] [DEBUG] - plugin grafana-py_script_workflow_start_check-7.5.17 result: True [2025-12-16 10:45:31.557] [DEBUG] - Found for grafana-py_script_workflow_start_check-7.5.17 for grafana-7.5.17 [2025-12-16 10:45:31.557] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 7 [2025-12-16 10:45:31.560] [DEBUG] - Searching start_check plugin for components ... [2025-12-16 10:45:31.560] [DEBUG] - Searching start_check plugin for ob-configserver-1.0.1-1.0.1.el7-324f9c93ded9c562896719234748062dfdd93cd8 [2025-12-16 10:45:31.561] [DEBUG] - Found for ob-configserver-py_script_start_check-1.0.0 for ob-configserver-1.0.1 [2025-12-16 10:45:31.561] [DEBUG] - Call plugin ob-configserver-py_script_start_check-1.0.0 for ob-configserver-1.0.1-1.0.1.el7-324f9c93ded9c562896719234748062dfdd93cd8 [2025-12-16 10:45:31.561] [DEBUG] - import start_check [2025-12-16 10:45:31.566] [DEBUG] - add start_check ref count to 1 [2025-12-16 10:45:31.567] [INFO] Check before start ob-configserver [2025-12-16 10:45:31.569] [DEBUG] -- root@10.1.224.53 execute: cat /root/dba_zc/ob_soft/zhangchong_ob_test/obconfigserver/run/ob-configserver.pid [2025-12-16 10:45:31.659] [DEBUG] -- exited code 0 [2025-12-16 10:45:31.660] [DEBUG] -- root@10.1.224.53 execute: ls /proc/23796 [2025-12-16 10:45:31.718] [DEBUG] -- exited code 2, error output: [2025-12-16 10:45:31.718] [DEBUG] ls: cannot access /proc/23796: No such file or directory [2025-12-16 10:45:31.718] [DEBUG] [2025-12-16 10:45:31.719] [DEBUG] -- 10.1.224.53 port check [2025-12-16 10:45:31.719] [DEBUG] -- root@10.1.224.53 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1F90' | awk -F' ' '{print $3}' | uniq [2025-12-16 10:45:31.791] [DEBUG] -- exited code 0 [2025-12-16 10:45:31.792] [DEBUG] -- 10.1.224.53 parameter check [2025-12-16 10:45:31.792] [DEBUG] -- 10.1.224.53 database check [2025-12-16 10:45:31.831] [DEBUG] - sub start_check ref count to 0 [2025-12-16 10:45:31.831] [DEBUG] - export start_check [2025-12-16 10:45:31.831] [DEBUG] - plugin ob-configserver-py_script_start_check-1.0.0 result: True [2025-12-16 10:45:31.832] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 8 [2025-12-16 10:45:31.836] [DEBUG] - Searching start_check_pre plugin for components ... [2025-12-16 10:45:31.836] [DEBUG] - Searching start_check_pre plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:31.838] [DEBUG] - Found for oceanbase-ce-py_script_start_check_pre-4.3.0.0 for oceanbase-ce-4.3.5.4 [2025-12-16 10:45:31.838] [DEBUG] - Call plugin oceanbase-ce-py_script_start_check_pre-4.3.0.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:31.839] [DEBUG] - import start_check_pre [2025-12-16 10:45:31.843] [DEBUG] - add start_check_pre ref count to 1 [2025-12-16 10:45:31.848] [DEBUG] - sub start_check_pre ref count to 0 [2025-12-16 10:45:31.848] [DEBUG] - export start_check_pre [2025-12-16 10:45:31.848] [DEBUG] - plugin oceanbase-ce-py_script_start_check_pre-4.3.0.0 result: True [2025-12-16 10:45:31.848] [DEBUG] - Searching status_check plugin for components ... [2025-12-16 10:45:31.848] [DEBUG] - Searching status_check plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:31.849] [DEBUG] - Found for oceanbase-ce-py_script_status_check-4.2.1.4 for oceanbase-ce-4.3.5.4 [2025-12-16 10:45:31.850] [DEBUG] - Call plugin oceanbase-ce-py_script_status_check-4.2.1.4 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:31.850] [DEBUG] - import status_check [2025-12-16 10:45:31.851] [DEBUG] - add status_check ref count to 1 [2025-12-16 10:45:31.852] [DEBUG] -- root@10.1.224.53 execute: ls /root/dba_zc/ob_data/1/clog/tenant_1/ [2025-12-16 10:45:31.870] [DEBUG] -- exited code 0 [2025-12-16 10:45:31.870] [DEBUG] -- root@10.1.224.53 execute: cat /root/dba_zc/ob_soft/zhangchong_ob_test/oceanbase/run/observer.pid [2025-12-16 10:45:31.928] [DEBUG] -- exited code 1, error output: [2025-12-16 10:45:31.928] [DEBUG] cat: /root/dba_zc/ob_soft/zhangchong_ob_test/oceanbase/run/observer.pid: No such file or directory [2025-12-16 10:45:31.929] [DEBUG] [2025-12-16 10:45:31.930] [DEBUG] -- root@10.1.224.54 execute: ls /root/dba_zc/ob_data/1/clog/tenant_1/ [2025-12-16 10:45:31.970] [DEBUG] -- exited code 0 [2025-12-16 10:45:31.971] [DEBUG] -- root@10.1.224.54 execute: cat /root/dba_zc/ob_soft/zhangchong_ob_test/oceanbase/run/observer.pid [2025-12-16 10:45:32.027] [DEBUG] -- exited code 1, error output: [2025-12-16 10:45:32.027] [DEBUG] cat: /root/dba_zc/ob_soft/zhangchong_ob_test/oceanbase/run/observer.pid: No such file or directory [2025-12-16 10:45:32.027] [DEBUG] [2025-12-16 10:45:32.029] [DEBUG] -- root@10.1.224.55 execute: ls /root/dba_zc/ob_data/1/clog/tenant_1/ [2025-12-16 10:45:32.067] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.068] [DEBUG] -- root@10.1.224.55 execute: cat /root/dba_zc/ob_soft/zhangchong_ob_test/oceanbase/run/observer.pid [2025-12-16 10:45:32.129] [DEBUG] -- exited code 1, error output: [2025-12-16 10:45:32.130] [DEBUG] cat: /root/dba_zc/ob_soft/zhangchong_ob_test/oceanbase/run/observer.pid: No such file or directory [2025-12-16 10:45:32.130] [DEBUG] [2025-12-16 10:45:32.130] [DEBUG] - sub status_check ref count to 0 [2025-12-16 10:45:32.130] [DEBUG] - export status_check [2025-12-16 10:45:32.130] [DEBUG] - plugin oceanbase-ce-py_script_status_check-4.2.1.4 result: True [2025-12-16 10:45:32.131] [DEBUG] - Searching parameter_check plugin for components ... [2025-12-16 10:45:32.131] [DEBUG] - Searching parameter_check plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:32.132] [DEBUG] - Found for oceanbase-ce-py_script_parameter_check-4.0.0.0 for oceanbase-ce-4.3.5.4 [2025-12-16 10:45:32.132] [DEBUG] - Call plugin oceanbase-ce-py_script_parameter_check-4.0.0.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:32.132] [DEBUG] - import parameter_check [2025-12-16 10:45:32.134] [DEBUG] - add parameter_check ref count to 1 [2025-12-16 10:45:32.135] [DEBUG] -- root@10.1.224.53 execute: ls /root/dba_zc/ob_data/1/sstable/block_file [2025-12-16 10:45:32.158] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.159] [DEBUG] -- root@10.1.224.54 execute: ls /root/dba_zc/ob_data/1/sstable/block_file [2025-12-16 10:45:32.179] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.180] [DEBUG] -- root@10.1.224.55 execute: ls /root/dba_zc/ob_data/1/sstable/block_file [2025-12-16 10:45:32.198] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.198] [DEBUG] - sub parameter_check ref count to 0 [2025-12-16 10:45:32.198] [DEBUG] - export parameter_check [2025-12-16 10:45:32.198] [DEBUG] - plugin oceanbase-ce-py_script_parameter_check-4.0.0.0 result: True [2025-12-16 10:45:32.199] [DEBUG] - Searching system_limits_check plugin for components ... [2025-12-16 10:45:32.199] [DEBUG] - Searching system_limits_check plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:32.200] [DEBUG] - Found for oceanbase-ce-py_script_system_limits_check-3.1.0 for oceanbase-ce-4.3.5.4 [2025-12-16 10:45:32.200] [DEBUG] - Call plugin oceanbase-ce-py_script_system_limits_check-3.1.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:32.200] [DEBUG] - import system_limits_check [2025-12-16 10:45:32.204] [DEBUG] - add system_limits_check ref count to 1 [2025-12-16 10:45:32.205] [DEBUG] -- root@10.1.224.53 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2025-12-16 10:45:32.225] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.225] [WARNING] [WARN] OBD-1011: (10.1.224.53) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536) [2025-12-16 10:45:32.226] [DEBUG] -- root@10.1.224.53 execute: bash -c "ulimit -a" [2025-12-16 10:45:32.287] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.288] [WARNING] [WARN] OBD-1007: (10.1.224.53) The recommended number of open files is 655350 (Current value: 102400) [2025-12-16 10:45:32.288] [WARNING] [WARN] OBD-1007: (10.1.224.53) The recommended number of core file size is unlimited (Current value: 0) [2025-12-16 10:45:32.289] [WARNING] [WARN] OBD-1007: (10.1.224.53) The recommended number of stack size is unlimited (Current value: 8192) [2025-12-16 10:45:32.289] [DEBUG] -- root@10.1.224.53 execute: sysctl -a [2025-12-16 10:45:32.410] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.415] [WARNING] [WARN] OBD-1017: (10.1.224.53) The value of the "vm.max_map_count" must be within [327600, 1310720] (Current value: 65530, Recommended value: 655360) [2025-12-16 10:45:32.416] [WARNING] [WARN] OBD-1017: (10.1.224.53) The value of the "vm.overcommit_memory" must be 0 (Current value: 1, Recommended value: 0) [2025-12-16 10:45:32.416] [WARNING] [WARN] OBD-1017: (10.1.224.53) The value of the "fs.file-max" must be greater than 6573688 (Current value: 6301056, Recommended value: 6573688) [2025-12-16 10:45:32.416] [DEBUG] -- root@10.1.224.54 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2025-12-16 10:45:32.445] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.446] [WARNING] [WARN] OBD-1011: (10.1.224.54) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536) [2025-12-16 10:45:32.446] [DEBUG] -- root@10.1.224.54 execute: bash -c "ulimit -a" [2025-12-16 10:45:32.516] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.517] [WARNING] [WARN] OBD-1007: (10.1.224.54) The recommended number of open files is 655350 (Current value: 102400) [2025-12-16 10:45:32.517] [WARNING] [WARN] OBD-1007: (10.1.224.54) The recommended number of core file size is unlimited (Current value: 0) [2025-12-16 10:45:32.517] [WARNING] [WARN] OBD-1007: (10.1.224.54) The recommended number of stack size is unlimited (Current value: 8192) [2025-12-16 10:45:32.518] [DEBUG] -- root@10.1.224.54 execute: sysctl -a [2025-12-16 10:45:32.686] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.694] [WARNING] [WARN] OBD-1017: (10.1.224.54) The value of the "vm.max_map_count" must be within [327600, 1310720] (Current value: 65530, Recommended value: 655360) [2025-12-16 10:45:32.695] [WARNING] [WARN] OBD-1017: (10.1.224.54) The value of the "vm.overcommit_memory" must be 0 (Current value: 1, Recommended value: 0) [2025-12-16 10:45:32.695] [WARNING] [WARN] OBD-1017: (10.1.224.54) The value of the "fs.file-max" must be greater than 6573688 (Current value: 6301026, Recommended value: 6573688) [2025-12-16 10:45:32.695] [DEBUG] -- root@10.1.224.55 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2025-12-16 10:45:32.727] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.728] [WARNING] [WARN] OBD-1011: (10.1.224.55) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536) [2025-12-16 10:45:32.728] [DEBUG] -- root@10.1.224.55 execute: bash -c "ulimit -a" [2025-12-16 10:45:32.794] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.795] [WARNING] [WARN] OBD-1007: (10.1.224.55) The recommended number of open files is 655350 (Current value: 102400) [2025-12-16 10:45:32.795] [WARNING] [WARN] OBD-1007: (10.1.224.55) The recommended number of core file size is unlimited (Current value: 0) [2025-12-16 10:45:32.795] [WARNING] [WARN] OBD-1007: (10.1.224.55) The recommended number of stack size is unlimited (Current value: 8192) [2025-12-16 10:45:32.796] [DEBUG] -- root@10.1.224.55 execute: sysctl -a [2025-12-16 10:45:32.944] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.951] [WARNING] [WARN] OBD-1017: (10.1.224.55) The value of the "vm.max_map_count" must be within [327600, 1310720] (Current value: 65530, Recommended value: 655360) [2025-12-16 10:45:32.951] [WARNING] [WARN] OBD-1017: (10.1.224.55) The value of the "vm.overcommit_memory" must be 0 (Current value: 1, Recommended value: 0) [2025-12-16 10:45:32.951] [WARNING] [WARN] OBD-1017: (10.1.224.55) The value of the "fs.file-max" must be greater than 6573688 (Current value: 6301042, Recommended value: 6573688) [2025-12-16 10:45:32.952] [DEBUG] - sub system_limits_check ref count to 0 [2025-12-16 10:45:32.952] [DEBUG] - export system_limits_check [2025-12-16 10:45:32.952] [DEBUG] - plugin oceanbase-ce-py_script_system_limits_check-3.1.0 result: True [2025-12-16 10:45:32.952] [DEBUG] - Searching resource_check plugin for components ... [2025-12-16 10:45:32.952] [DEBUG] - Searching resource_check plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:32.953] [DEBUG] - Found for oceanbase-ce-py_script_resource_check-4.0.0.0 for oceanbase-ce-4.3.5.4 [2025-12-16 10:45:32.953] [DEBUG] - Call plugin oceanbase-ce-py_script_resource_check-4.0.0.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80 [2025-12-16 10:45:32.954] [DEBUG] - import resource_check [2025-12-16 10:45:32.957] [DEBUG] - add resource_check ref count to 1 [2025-12-16 10:45:32.958] [DEBUG] -- root@10.1.224.53 execute: cat /proc/meminfo [2025-12-16 10:45:32.972] [DEBUG] -- exited code 0 [2025-12-16 10:45:32.972] [DEBUG] -- root@10.1.224.53 execute: df --block-size=1024 [2025-12-16 10:45:33.028] [DEBUG] -- exited code 0 [2025-12-16 10:45:33.029] [DEBUG] -- get disk info for path /dev, total: 33664856064 avail: 33664856064 [2025-12-16 10:45:33.029] [DEBUG] -- get disk info for path /dev/shm, total: 33675821056 avail: 33675821056 [2025-12-16 10:45:33.029] [DEBUG] -- get disk info for path /run, total: 33675821056 avail: 33665314816 [2025-12-16 10:45:33.029] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 33675821056 avail: 33675821056 [2025-12-16 10:45:33.029] [DEBUG] -- get disk info for path /, total: 281114873856 avail: 119465185280 [2025-12-16 10:45:33.029] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 914710528 [2025-12-16 10:45:33.029] [DEBUG] -- get disk info for path /run/user/1000, total: 6735167488 avail: 6735167488 [2025-12-16 10:45:33.029] [DEBUG] -- get disk info for path /run/user/0, total: 6735167488 avail: 6735167488 [2025-12-16 10:45:33.029] [DEBUG] -- disk: {'/dev': {'total': 33664856064, 'avail': 33664856064, 'need': 0}, '/dev/shm': {'total': 33675821056, 'avail': 33675821056, 'need': 0}, '/run': {'total': 33675821056, 'avail': 33665314816, 'need': 0}, '/sys/fs/cgroup': {'total': 33675821056, 'avail': 33675821056, 'need': 0}, '/': {'total': 281114873856, 'avail': 119465185280, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 914710528, 'need': 0}, '/run/user/1000': {'total': 6735167488, 'avail': 6735167488, 'need': 0}, '/run/user/0': {'total': 6735167488, 'avail': 6735167488, 'need': 0}} [2025-12-16 10:45:33.030] [DEBUG] -- root@10.1.224.54 execute: cat /proc/meminfo [2025-12-16 10:45:33.047] [DEBUG] -- exited code 0 [2025-12-16 10:45:33.048] [DEBUG] -- root@10.1.224.54 execute: df --block-size=1024 [2025-12-16 10:45:33.113] [DEBUG] -- exited code 0 [2025-12-16 10:45:33.114] [DEBUG] -- get disk info for path /dev, total: 33664774144 avail: 33664774144 [2025-12-16 10:45:33.114] [DEBUG] -- get disk info for path /dev/shm, total: 33675739136 avail: 33675739136 [2025-12-16 10:45:33.114] [DEBUG] -- get disk info for path /run, total: 33675739136 avail: 33665216512 [2025-12-16 10:45:33.115] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 33675739136 avail: 33675739136 [2025-12-16 10:45:33.115] [DEBUG] -- get disk info for path /, total: 281114873856 avail: 170514780160 [2025-12-16 10:45:33.115] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 914616320 [2025-12-16 10:45:33.115] [DEBUG] -- get disk info for path /run/user/1000, total: 6735151104 avail: 6735151104 [2025-12-16 10:45:33.115] [DEBUG] -- get disk info for path /run/user/0, total: 6735151104 avail: 6735151104 [2025-12-16 10:45:33.115] [DEBUG] -- disk: {'/dev': {'total': 33664774144, 'avail': 33664774144, 'need': 0}, '/dev/shm': {'total': 33675739136, 'avail': 33675739136, 'need': 0}, '/run': {'total': 33675739136, 'avail': 33665216512, 'need': 0}, '/sys/fs/cgroup': {'total': 33675739136, 'avail': 33675739136, 'need': 0}, '/': {'total': 281114873856, 'avail': 170514780160, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 914616320, 'need': 0}, '/run/user/1000': {'total': 6735151104, 'avail': 6735151104, 'need': 0}, '/run/user/0': {'total': 6735151104, 'avail': 6735151104, 'need': 0}} [2025-12-16 10:45:33.115] [DEBUG] -- root@10.1.224.55 execute: cat /proc/meminfo [2025-12-16 10:45:33.140] [DEBUG] -- exited code 0 [2025-12-16 10:45:33.141] [DEBUG] -- root@10.1.224.55 execute: df --block-size=1024 [2025-12-16 10:45:33.208] [DEBUG] -- exited code 0 [2025-12-16 10:45:33.209] [DEBUG] -- get disk info for path /dev, total: 33664856064 avail: 33664856064 [2025-12-16 10:45:33.209] [DEBUG] -- get disk info for path /dev/shm, total: 33675821056 avail: 33675821056 [2025-12-16 10:45:33.209] [DEBUG] -- get disk info for path /run, total: 33675821056 avail: 33665363968 [2025-12-16 10:45:33.209] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 33675821056 avail: 33675821056 [2025-12-16 10:45:33.210] [DEBUG] -- get disk info for path /, total: 281114873856 avail: 175665844224 [2025-12-16 10:45:33.210] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 914710528 [2025-12-16 10:45:33.210] [DEBUG] -- get disk info for path /run/user/0, total: 6735167488 avail: 6735167488 [2025-12-16 10:45:33.210] [DEBUG] -- get disk info for path /run/user/1000, total: 6735167488 avail: 6735167488 [2025-12-16 10:45:33.210] [DEBUG] -- disk: {'/dev': {'total': 33664856064, 'avail': 33664856064, 'need': 0}, '/dev/shm': {'total': 33675821056, 'avail': 33675821056, 'need': 0}, '/run': {'total': 33675821056, 'avail': 33665363968, 'need': 0}, '/sys/fs/cgroup': {'total': 33675821056, 'avail': 33675821056, 'need': 0}, '/': {'total': 281114873856, 'avail': 175665844224, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 914710528, 'need': 0}, '/run/user/0': {'total': 6735167488, 'avail': 6735167488, 'need': 0}, '/run/user/1000': {'total': 6735167488, 'avail': 6735167488, 'need': 0}} [2025-12-16 10:45:33.210] [DEBUG] -- root@10.1.224.53 execute: date +%s%N [2025-12-16 10:45:33.230] [DEBUG] -- exited code 0 [2025-12-16 10:45:33.230] [DEBUG] -- 10.1.224.53 time delta 731.6767578125 [2025-12-16 10:45:33.231] [DEBUG] -- root@10.1.224.54 execute: date +%s%N [2025-12-16 10:45:33.257] [DEBUG] -- exited code 0 [2025-12-16 10:45:33.257] [DEBUG] -- 10.1.224.54 time delta -168.306396484375 [2025-12-16 10:45:33.257] [DEBUG] -- root@10.1.224.55 execute: date +%s%N [2025-12-16 10:45:33.284] [DEBUG] -- exited code 0 [2025-12-16 10:45:33.285] [DEBUG] -- 10.1.224.55 time delta -215.521240234375 [2025-12-16 10:45:33.287] [ERROR] [ERROR] oceanbase-ce-py_script_resource_check-4.0.0.0 RuntimeError: cannot access local variable 'server' where it is not associated with a value [2025-12-16 10:45:33.287] [ERROR] Traceback (most recent call last): [2025-12-16 10:45:33.287] [ERROR] File "core.py", line 2376, in start_cluster [2025-12-16 10:45:33.287] [ERROR] File "core.py", line 2429, in _start_cluster [2025-12-16 10:45:33.287] [ERROR] File "core.py", line 263, in run_workflow [2025-12-16 10:45:33.287] [ERROR] File "core.py", line 305, in run_plugin_template [2025-12-16 10:45:33.287] [ERROR] File "core.py", line 354, in call_plugin [2025-12-16 10:45:33.287] [ERROR] File "_plugin.py", line 348, in __call__ [2025-12-16 10:45:33.287] [ERROR] File "_plugin.py", line 304, in _new_func [2025-12-16 10:45:33.288] [ERROR] File "/root/.obd/plugins/oceanbase-ce/4.0.0.0/resource_check.py", line 252, in resource_check [2025-12-16 10:45:33.288] [ERROR] critical(server, 'ntp', err.EC_OBSERVER_TIME_OUT_OF_SYNC.format(), [err.SUG_OBSERVER_TIME_OUT_OF_SYNC.format()]) [2025-12-16 10:45:33.288] [ERROR] ^^^^^^ [2025-12-16 10:45:33.288] [ERROR] UnboundLocalError: cannot access local variable 'server' where it is not associated with a value [2025-12-16 10:45:33.288] [ERROR] [2025-12-16 10:45:33.288] [DEBUG] - sub resource_check ref count to 0 [2025-12-16 10:45:33.288] [DEBUG] - export resource_check [2025-12-16 10:45:33.288] [DEBUG] - plugin oceanbase-ce-py_script_resource_check-4.0.0.0 result: False [2025-12-16 10:45:33.291] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2025-12-16 10:45:33.291] [INFO] Trace ID: 48d1de34-da29-11f0-8a44-6c92bf076f4b [2025-12-16 10:45:33.291] [INFO] If you want to view detailed obd logs, please run: obd display-trace 48d1de34-da29-11f0-8a44-6c92bf076f4b [2025-12-16 10:45:33.292] [DEBUG] - unlock /root/.obd/lock/global [2025-12-16 10:45:33.292] [DEBUG] - unlock /root/.obd/lock/deploy_zhangchong_ob_test [2025-12-16 10:45:33.292] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo