[sudoroot@localhost ~]$ obd cluster start dh_ob -c oceanbase-ce -s 10.1.4.180 Get local repositories ok Search plugins ok Load cluster param plugin ok Cluster status check ok Check before start observer x [WARN] OBD-1007: (10.1.4.180) The recommended number of max user processes is 655350 (Current value: 131072) [WARN] OBD-1007: (10.1.4.180) The recommended number of stack size is unlimited (Current value: 8192) [ERROR] oceanbase-ce-py_script_start_check-4.2.2.0 RuntimeError: <10.1.4.180> See https://www.oceanbase.com/product/ob-deployer/error-codes . Trace ID: e1e13d28-bdf0-11ef-963a-566f351b0096 If you want to view detailed obd logs, please run: obd display-trace e1e13d28-bdf0-11ef-963a-566f351b0096 [sudoroot@localhost ~]$ [sudoroot@localhost ~]$ [sudoroot@localhost ~]$ obd display-trace e1e13d28-bdf0-11ef-963a-566f351b0096 [2024-12-19 18:06:15.121] [DEBUG] - cmd: ['dh_ob'] [2024-12-19 18:06:15.121] [DEBUG] - opts: {'servers': '10.1.4.180', 'components': 'oceanbase-ce', 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-12-19 18:06:15.121] [DEBUG] - mkdir /home/sudoroot/.obd/lock/ [2024-12-19 18:06:15.122] [DEBUG] - unknown lock mode [2024-12-19 18:06:15.122] [DEBUG] - try to get share lock /home/sudoroot/.obd/lock/global [2024-12-19 18:06:15.122] [DEBUG] - share lock `/home/sudoroot/.obd/lock/global`, count 1 [2024-12-19 18:06:15.122] [DEBUG] - Get Deploy by name [2024-12-19 18:06:15.122] [DEBUG] - mkdir /home/sudoroot/.obd/cluster/ [2024-12-19 18:06:15.123] [DEBUG] - mkdir /home/sudoroot/.obd/config_parser/ [2024-12-19 18:06:15.123] [DEBUG] - try to get exclusive lock /home/sudoroot/.obd/lock/deploy_dh_ob [2024-12-19 18:06:15.123] [DEBUG] - exclusive lock `/home/sudoroot/.obd/lock/deploy_dh_ob`, count 1 [2024-12-19 18:06:15.131] [DEBUG] - Deploy status judge [2024-12-19 18:06:15.131] [INFO] Get local repositories [2024-12-19 18:06:15.132] [DEBUG] - mkdir /home/sudoroot/.obd/repository [2024-12-19 18:06:15.132] [DEBUG] - Get local repository oceanbase-ce-4.2.2.0-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:15.132] [DEBUG] - Search repository oceanbase-ce version: 4.2.2.0, tag: 5d2a012023c63e651a979e1cbcbf096d9340b7d3, release: None, package_hash: None [2024-12-19 18:06:15.133] [DEBUG] - try to get share lock /home/sudoroot/.obd/lock/mirror_and_repo [2024-12-19 18:06:15.133] [DEBUG] - share lock `/home/sudoroot/.obd/lock/mirror_and_repo`, count 1 [2024-12-19 18:06:15.133] [DEBUG] - mkdir /home/sudoroot/.obd/repository/oceanbase-ce [2024-12-19 18:06:15.135] [DEBUG] - Found repository oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:15.136] [DEBUG] - Get local repository obproxy-ce-4.2.1.0-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:15.136] [DEBUG] - Search repository obproxy-ce version: 4.2.1.0, tag: 6b1c8edcc1606189c94090acd2131e73899d1a61, release: None, package_hash: None [2024-12-19 18:06:15.136] [DEBUG] - share lock `/home/sudoroot/.obd/lock/mirror_and_repo`, count 2 [2024-12-19 18:06:15.136] [DEBUG] - mkdir /home/sudoroot/.obd/repository/obproxy-ce [2024-12-19 18:06:15.138] [DEBUG] - Found repository obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:15.138] [DEBUG] - Get local repository obagent-4.2.2-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:15.138] [DEBUG] - Search repository obagent version: 4.2.2, tag: bf152b880953c2043ddaf80d6180cf22bb8c8ac2, release: None, package_hash: None [2024-12-19 18:06:15.138] [DEBUG] - share lock `/home/sudoroot/.obd/lock/mirror_and_repo`, count 3 [2024-12-19 18:06:15.138] [DEBUG] - mkdir /home/sudoroot/.obd/repository/obagent [2024-12-19 18:06:15.140] [DEBUG] - Found repository obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:15.140] [DEBUG] - Get local repository ocp-express-4.2.2-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:15.141] [DEBUG] - Search repository ocp-express version: 4.2.2, tag: e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75, release: None, package_hash: None [2024-12-19 18:06:15.141] [DEBUG] - share lock `/home/sudoroot/.obd/lock/mirror_and_repo`, count 4 [2024-12-19 18:06:15.141] [DEBUG] - mkdir /home/sudoroot/.obd/repository/ocp-express [2024-12-19 18:06:15.143] [DEBUG] - Found repository ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:15.263] [DEBUG] - Get deploy config [2024-12-19 18:06:15.298] [INFO] Search plugins [2024-12-19 18:06:15.298] [DEBUG] - Searching start_check plugin for components ... [2024-12-19 18:06:15.299] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:15.299] [DEBUG] - mkdir /home/sudoroot/.obd/plugins [2024-12-19 18:06:15.300] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-12-19 18:06:15.300] [DEBUG] - Searching start_check plugin for obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:15.300] [DEBUG] - Found for obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.2.1.0 [2024-12-19 18:06:15.300] [DEBUG] - Searching start_check plugin for obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:15.301] [DEBUG] - Found for obagent-py_script_start_check-1.3.0 for obagent-4.2.2 [2024-12-19 18:06:15.301] [DEBUG] - Searching start_check plugin for ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:15.301] [DEBUG] - Found for ocp-express-py_script_start_check-4.2.1 for ocp-express-4.2.2 [2024-12-19 18:06:15.301] [DEBUG] - Searching create_tenant plugin for components ... [2024-12-19 18:06:15.301] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:15.302] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.2.0 [2024-12-19 18:06:15.302] [DEBUG] - Searching create_tenant plugin for obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:15.302] [DEBUG] - No such create_tenant plugin for obproxy-ce-4.2.1.0 [2024-12-19 18:06:15.302] [DEBUG] - Searching create_tenant plugin for obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:15.302] [DEBUG] - No such create_tenant plugin for obagent-4.2.2 [2024-12-19 18:06:15.303] [DEBUG] - Searching create_tenant plugin for ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:15.303] [DEBUG] - No such create_tenant plugin for ocp-express-4.2.2 [2024-12-19 18:06:15.303] [DEBUG] - Searching start plugin for components ... [2024-12-19 18:06:15.303] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:15.303] [DEBUG] - Found for oceanbase-ce-py_script_start-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-12-19 18:06:15.303] [DEBUG] - Searching start plugin for obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:15.304] [DEBUG] - Found for obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.2.1.0 [2024-12-19 18:06:15.304] [DEBUG] - Searching start plugin for obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:15.304] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-4.2.2 [2024-12-19 18:06:15.304] [DEBUG] - Searching start plugin for ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:15.304] [DEBUG] - Found for ocp-express-py_script_start-4.2.1 for ocp-express-4.2.2 [2024-12-19 18:06:15.304] [DEBUG] - Searching connect plugin for components ... [2024-12-19 18:06:15.305] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:15.305] [DEBUG] - Found for oceanbase-ce-py_script_connect-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-12-19 18:06:15.305] [DEBUG] - Searching connect plugin for obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:15.305] [DEBUG] - Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.2.1.0 [2024-12-19 18:06:15.306] [DEBUG] - Searching connect plugin for obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:15.306] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-4.2.2 [2024-12-19 18:06:15.306] [DEBUG] - Searching connect plugin for ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:15.306] [DEBUG] - Found for ocp-express-py_script_connect-1.0.1 for ocp-express-4.2.2 [2024-12-19 18:06:15.306] [DEBUG] - Searching bootstrap plugin for components ... [2024-12-19 18:06:15.306] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:15.307] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-12-19 18:06:15.307] [DEBUG] - Searching bootstrap plugin for obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:15.307] [DEBUG] - Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.2.1.0 [2024-12-19 18:06:15.307] [DEBUG] - Searching bootstrap plugin for obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:15.307] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-4.2.2 [2024-12-19 18:06:15.308] [DEBUG] - Searching bootstrap plugin for ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:15.308] [DEBUG] - Found for ocp-express-py_script_bootstrap-1.0.1 for ocp-express-4.2.2 [2024-12-19 18:06:15.308] [DEBUG] - Searching display plugin for components ... [2024-12-19 18:06:15.308] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:15.308] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.2.0 [2024-12-19 18:06:15.309] [DEBUG] - Searching display plugin for obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:15.309] [DEBUG] - Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.2.1.0 [2024-12-19 18:06:15.309] [DEBUG] - Searching display plugin for obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:15.309] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-4.2.2 [2024-12-19 18:06:15.309] [DEBUG] - Searching display plugin for ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:15.309] [DEBUG] - Found for ocp-express-py_script_display-1.0.1 for ocp-express-4.2.2 [2024-12-19 18:06:15.430] [INFO] Load cluster param plugin [2024-12-19 18:06:15.430] [DEBUG] - Get local repository oceanbase-ce-4.2.2.0-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:15.430] [DEBUG] - Get local repository obproxy-ce-4.2.1.0-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:15.430] [DEBUG] - Get local repository obagent-4.2.2-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:15.430] [DEBUG] - Get local repository ocp-express-4.2.2-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:15.431] [DEBUG] - Searching param plugin for components ... [2024-12-19 18:06:15.431] [DEBUG] - Search param plugin for oceanbase-ce [2024-12-19 18:06:15.431] [DEBUG] - Found for oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-12-19 18:06:15.431] [DEBUG] - Applying oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:16.056] [DEBUG] - Search param plugin for obproxy-ce [2024-12-19 18:06:16.057] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0 [2024-12-19 18:06:16.057] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:16.203] [DEBUG] - Search param plugin for obagent [2024-12-19 18:06:16.203] [DEBUG] - Found for obagent-param-1.3.0 for obagent-4.2.2 [2024-12-19 18:06:16.203] [DEBUG] - Applying obagent-param-1.3.0 for obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:16.276] [DEBUG] - Search param plugin for ocp-express [2024-12-19 18:06:16.276] [DEBUG] - Found for ocp-express-param-4.1.0 for ocp-express-4.2.2 [2024-12-19 18:06:16.277] [DEBUG] - Applying ocp-express-param-4.1.0 for ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:16.450] [INFO] Cluster status check [2024-12-19 18:06:16.450] [DEBUG] - Searching status plugin for components ... [2024-12-19 18:06:16.450] [DEBUG] - Searching status plugin for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:16.451] [DEBUG] - Found for oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.2.0 [2024-12-19 18:06:16.451] [DEBUG] - Searching status plugin for obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:16.451] [DEBUG] - Found for obproxy-ce-py_script_status-3.1.0 for obproxy-ce-4.2.1.0 [2024-12-19 18:06:16.452] [DEBUG] - Searching status plugin for obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:16.452] [DEBUG] - Found for obagent-py_script_status-1.3.0 for obagent-4.2.2 [2024-12-19 18:06:16.452] [DEBUG] - Searching status plugin for ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:16.452] [DEBUG] - Found for ocp-express-py_script_status-1.0 for ocp-express-4.2.2 [2024-12-19 18:06:16.453] [DEBUG] - host: 10.1.4.177, port: 22, user: sudoroot, password: RB>]GB)hS"BUf//h,1q3 [2024-12-19 18:06:16.542] [DEBUG] - host: 10.1.4.179, port: 22, user: sudoroot, password: RB>]GB)hS"BUf//h,1q3 [2024-12-19 18:06:16.635] [DEBUG] - host: 10.1.4.180, port: 22, user: sudoroot, password: RB>]GB)hS"BUf//h,1q3 [2024-12-19 18:06:16.724] [DEBUG] - host: 10.1.4.178, port: 22, user: sudoroot, password: RB>]GB)hS"BUf//h,1q3 [2024-12-19 18:06:16.842] [DEBUG] - Call oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:16.842] [DEBUG] - import status [2024-12-19 18:06:16.843] [DEBUG] - add status ref count to 1 [2024-12-19 18:06:16.843] [DEBUG] -- sudoroot@10.1.4.178 execute: cat /opt/cluster/dh_ob/oceanbase/run/observer.pid [2024-12-19 18:06:16.905] [DEBUG] -- exited code 0 [2024-12-19 18:06:16.905] [DEBUG] -- sudoroot@10.1.4.178 execute: ls /proc/494719 [2024-12-19 18:06:16.965] [DEBUG] -- exited code 0 [2024-12-19 18:06:16.966] [DEBUG] -- sudoroot@10.1.4.179 execute: cat /opt/cluster/dh_ob/oceanbase/run/observer.pid [2024-12-19 18:06:16.989] [DEBUG] -- exited code 0 [2024-12-19 18:06:16.990] [DEBUG] -- sudoroot@10.1.4.179 execute: ls /proc/489314 [2024-12-19 18:06:17.052] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.053] [DEBUG] -- sudoroot@10.1.4.180 execute: cat /opt/cluster/dh_ob/oceanbase/run/observer.pid [2024-12-19 18:06:17.075] [DEBUG] -- exited code 1, error output: [2024-12-19 18:06:17.076] [DEBUG] cat: /opt/cluster/dh_ob/oceanbase/run/observer.pid: 没有那个文件或目录 [2024-12-19 18:06:17.076] [DEBUG] [2024-12-19 18:06:17.076] [DEBUG] - sub status ref count to 0 [2024-12-19 18:06:17.076] [DEBUG] - export status [2024-12-19 18:06:17.076] [DEBUG] - oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 cluster status is inconsistent [2024-12-19 18:06:17.076] [DEBUG] - Call obproxy-ce-py_script_status-3.1.0 for obproxy-ce-4.2.1.0-11.el8-6b1c8edcc1606189c94090acd2131e73899d1a61 [2024-12-19 18:06:17.077] [DEBUG] - import status [2024-12-19 18:06:17.077] [DEBUG] - add status ref count to 1 [2024-12-19 18:06:17.078] [DEBUG] -- sudoroot@10.1.4.177 execute: cat /opt/cluster/dh_ob/obproxy/run/obproxy-10.1.4.177-2883.pid [2024-12-19 18:06:17.099] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.099] [DEBUG] -- sudoroot@10.1.4.177 execute: ls /proc/494177 [2024-12-19 18:06:17.159] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.159] [DEBUG] - sub status ref count to 0 [2024-12-19 18:06:17.159] [DEBUG] - export status [2024-12-19 18:06:17.159] [DEBUG] - Call obagent-py_script_status-1.3.0 for obagent-4.2.2-100000042024011120.el8-bf152b880953c2043ddaf80d6180cf22bb8c8ac2 [2024-12-19 18:06:17.159] [DEBUG] - import status [2024-12-19 18:06:17.160] [DEBUG] - add status ref count to 1 [2024-12-19 18:06:17.160] [DEBUG] -- sudoroot@10.1.4.178 execute: cat /opt/cluster/dh_ob/obagent/run/ob_agentd.pid [2024-12-19 18:06:17.181] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.181] [DEBUG] -- sudoroot@10.1.4.178 execute: ls /proc/496620 [2024-12-19 18:06:17.241] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.242] [DEBUG] -- sudoroot@10.1.4.179 execute: cat /opt/cluster/dh_ob/obagent/run/ob_agentd.pid [2024-12-19 18:06:17.263] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.263] [DEBUG] -- sudoroot@10.1.4.179 execute: ls /proc/491222 [2024-12-19 18:06:17.326] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.326] [DEBUG] -- sudoroot@10.1.4.180 execute: cat /opt/cluster/dh_ob/obagent/run/ob_agentd.pid [2024-12-19 18:06:17.348] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.348] [DEBUG] -- sudoroot@10.1.4.180 execute: ls /proc/496320 [2024-12-19 18:06:17.410] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.410] [DEBUG] - sub status ref count to 0 [2024-12-19 18:06:17.410] [DEBUG] - export status [2024-12-19 18:06:17.411] [DEBUG] - Call ocp-express-py_script_status-1.0 for ocp-express-4.2.2-100000022024011120.el8-e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 [2024-12-19 18:06:17.411] [DEBUG] - import status [2024-12-19 18:06:17.411] [DEBUG] - add status ref count to 1 [2024-12-19 18:06:17.412] [DEBUG] -- sudoroot@10.1.4.177 execute: cat /opt/cluster/dh_ob/ocpexpress/run/ocp-express.pid [2024-12-19 18:06:17.441] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.441] [DEBUG] -- sudoroot@10.1.4.177 execute: ls /proc/901237 [2024-12-19 18:06:17.500] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.501] [DEBUG] - sub status ref count to 0 [2024-12-19 18:06:17.501] [DEBUG] - export status [2024-12-19 18:06:17.501] [DEBUG] - Deploy status inconsistent [2024-12-19 18:06:17.501] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.2.0-100010012024022719.el8-5d2a012023c63e651a979e1cbcbf096d9340b7d3 [2024-12-19 18:06:17.501] [DEBUG] - import start_check [2024-12-19 18:06:17.510] [DEBUG] - add start_check ref count to 1 [2024-12-19 18:06:17.510] [DEBUG] - plugin oceanbase-ce-py_script_start_check-4.2.2.0 target_servers: [<10.1.4.180>] [2024-12-19 18:06:17.511] [INFO] Check before start observer [2024-12-19 18:06:17.512] [DEBUG] -- sudoroot@10.1.4.180 execute: ls /opt/cluster/dh_ob/oceanbase/store/clog/tenant_1/ [2024-12-19 18:06:17.535] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.535] [DEBUG] -- sudoroot@10.1.4.180 execute: cat /opt/cluster/dh_ob/oceanbase/run/observer.pid [2024-12-19 18:06:17.597] [DEBUG] -- exited code 1, error output: [2024-12-19 18:06:17.597] [DEBUG] cat: /opt/cluster/dh_ob/oceanbase/run/observer.pid: 没有那个文件或目录 [2024-12-19 18:06:17.597] [DEBUG] [2024-12-19 18:06:17.597] [DEBUG] -- 10.1.4.180 port check [2024-12-19 18:06:17.597] [DEBUG] -- sudoroot@10.1.4.180 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-12-19 18:06:17.683] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.683] [DEBUG] -- sudoroot@10.1.4.180 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-12-19 18:06:17.770] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.770] [DEBUG] -- sudoroot@10.1.4.180 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep 'B404010A:0B46' | awk -F' ' '{print $2}' | uniq [2024-12-19 18:06:17.856] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.856] [DEBUG] -- sudoroot@10.1.4.180 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B46' | awk -F' ' '{print $2}' | uniq [2024-12-19 18:06:17.941] [DEBUG] -- exited code 0 [2024-12-19 18:06:17.941] [DEBUG] -- sudoroot@10.1.4.180 execute: ls /opt/cluster/dh_ob/oceanbase/store/sstable/block_file [2024-12-19 18:06:18.003] [DEBUG] -- exited code 0 [2024-12-19 18:06:18.003] [DEBUG] -- sudoroot@10.1.4.180 execute: [ -w /tmp/ ] || [ -w /tmp/obshell ] [2024-12-19 18:06:18.063] [DEBUG] -- exited code 0 [2024-12-19 18:06:18.063] [DEBUG] -- sudoroot@10.1.4.180 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-12-19 18:06:18.123] [DEBUG] -- exited code 0 [2024-12-19 18:06:18.123] [DEBUG] -- sudoroot@10.1.4.180 execute: ulimit -a [2024-12-19 18:06:18.183] [DEBUG] -- exited code 0 [2024-12-19 18:06:18.184] [WARNING] OBD-1007: (10.1.4.180) The recommended number of max user processes is 655350 (Current value: 131072) [2024-12-19 18:06:18.184] [WARNING] OBD-1007: (10.1.4.180) The recommended number of stack size is unlimited (Current value: 8192) [2024-12-19 18:06:18.184] [DEBUG] -- sudoroot@10.1.4.180 execute: sysctl -a [2024-12-19 18:06:18.260] [DEBUG] -- exited code 0 [2024-12-19 18:06:18.264] [DEBUG] -- sudoroot@10.1.4.180 execute: cat /proc/meminfo [2024-12-19 18:06:18.322] [DEBUG] -- exited code 0 [2024-12-19 18:06:18.323] [DEBUG] -- sudoroot@10.1.4.180 execute: df --block-size=1024 [2024-12-19 18:06:18.383] [DEBUG] -- exited code 0 [2024-12-19 18:06:18.384] [DEBUG] -- get disk info for path /dev, total: 33472049152 avail: 33472049152 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /dev/shm, total: 33489457152 avail: 33489457152 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /run, total: 33489457152 avail: 30224388096 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 33489457152 avail: 33489457152 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /, total: 46670925824 avail: 40094486528 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /tmp, total: 33489457152 avail: 33489424384 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /boot, total: 1020702720 avail: 808050688 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /boot/efi, total: 627900416 avail: 619917312 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /opt, total: 482943700992 avail: 7224631296 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /run/user/1003, total: 6697889792 avail: 6697889792 [2024-12-19 18:06:18.385] [DEBUG] -- get disk info for path /run/user/1000, total: 6697889792 avail: 6697889792 [2024-12-19 18:06:18.385] [DEBUG] -- disk: {'/dev': {'total': 33472049152, 'avail': 33472049152, 'need': 0}, '/dev/shm': {'total': 33489457152, 'avail': 33489457152, 'need': 0}, '/run': {'total': 33489457152, 'avail': 30224388096, 'need': 0}, '/sys/fs/cgroup': {'total': 33489457152, 'avail': 33489457152, 'need': 0}, '/': {'total': 46670925824, 'avail': 40094486528, 'need': 0}, '/tmp': {'total': 33489457152, 'avail': 33489424384, 'need': 0}, '/boot': {'total': 1020702720, 'avail': 808050688, 'need': 0}, '/boot/efi': {'total': 627900416, 'avail': 619917312, 'need': 0}, '/opt': {'total': 482943700992, 'avail': 7224631296, 'need': 0}, '/run/user/1003': {'total': 6697889792, 'avail': 6697889792, 'need': 0}, '/run/user/1000': {'total': 6697889792, 'avail': 6697889792, 'need': 0}} [2024-12-19 18:06:18.388] [ERROR] oceanbase-ce-py_script_start_check-4.2.2.0 RuntimeError: <10.1.4.180> [2024-12-19 18:06:18.388] [ERROR] Traceback (most recent call last): [2024-12-19 18:06:18.388] [ERROR] File "core.py", line 2020, in start_cluster [2024-12-19 18:06:18.388] [ERROR] File "core.py", line 2098, in _start_cluster [2024-12-19 18:06:18.388] [ERROR] File "core.py", line 188, in call_plugin [2024-12-19 18:06:18.388] [ERROR] File "_plugin.py", line 347, in __call__ [2024-12-19 18:06:18.388] [ERROR] File "_plugin.py", line 305, in _new_func [2024-12-19 18:06:18.388] [ERROR] File "/home/sudoroot/.obd/plugins/oceanbase-ce/4.2.2.0/start_check.py", line 744, in start_check [2024-12-19 18:06:18.388] [ERROR] log_disk_size = servers_log_disk_size[server] [2024-12-19 18:06:18.388] [ERROR] KeyError: <10.1.4.180> [2024-12-19 18:06:18.388] [ERROR] [2024-12-19 18:06:18.388] [DEBUG] - plugin oceanbase-ce-py_script_start_check-4.2.2.0 restore servers: [<10.1.4.178>, <10.1.4.179>, <10.1.4.180>] [2024-12-19 18:06:18.388] [DEBUG] - sub start_check ref count to 0 [2024-12-19 18:06:18.388] [DEBUG] - export start_check [2024-12-19 18:06:18.388] [DEBUG] - oceanbase-ce starting check failed. [2024-12-19 18:06:18.391] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2024-12-19 18:06:18.391] [INFO] Trace ID: e1e13d28-bdf0-11ef-963a-566f351b0096 [2024-12-19 18:06:18.392] [INFO] If you want to view detailed obd logs, please run: obd display-trace e1e13d28-bdf0-11ef-963a-566f351b0096 [2024-12-19 18:06:18.424] [INFO] [WARN] OBD-1007: (10.1.4.180) The recommended number of max user processes is 655350 (Current value: 131072) [2024-12-19 18:06:18.425] [INFO] [WARN] OBD-1007: (10.1.4.180) The recommended number of stack size is unlimited (Current value: 8192) [2024-12-19 18:06:18.425] [INFO] [ERROR] oceanbase-ce-py_script_start_check-4.2.2.0 RuntimeError: <10.1.4.180> [2024-12-19 18:06:18.425] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2024-12-19 18:06:18.425] [INFO] Trace ID: e1e13d28-bdf0-11ef-963a-566f351b0096 [2024-12-19 18:06:18.425] [INFO] If you want to view detailed obd logs, please run: obd display-trace e1e13d28-bdf0-11ef-963a-566f351b0096 [2024-12-19 18:06:18.425] [INFO] [2024-12-19 18:06:18.425] [DEBUG] - unlock /home/sudoroot/.obd/lock/global [2024-12-19 18:06:18.425] [DEBUG] - unlock /home/sudoroot/.obd/lock/deploy_dh_ob [2024-12-19 18:06:18.426] [DEBUG] - unlock /home/sudoroot/.obd/lock/mirror_and_repo