[2024-06-17 16:06:02.350] [DEBUG] - cmd: ['zcobcluster'] [2024-06-17 16:06:02.350] [DEBUG] - opts: {'config': 'zcobcluster_deploy.yaml', 'force': False, 'clean': False, 'unuselibrepo': None, 'auto_create_tenant': None} [2024-06-17 16:06:02.350] [DEBUG] - mkdir /home/admin/.obd/lock/ [2024-06-17 16:06:02.351] [DEBUG] - unknown lock mode [2024-06-17 16:06:02.351] [DEBUG] - try to get share lock /home/admin/.obd/lock/global [2024-06-17 16:06:02.351] [DEBUG] - share lock `/home/admin/.obd/lock/global`, count 1 [2024-06-17 16:06:02.351] [DEBUG] - Get Deploy by name [2024-06-17 16:06:02.351] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-06-17 16:06:02.351] [DEBUG] - mkdir /home/admin/.obd/config_parser/ [2024-06-17 16:06:02.351] [DEBUG] - try to get exclusive lock /home/admin/.obd/lock/deploy_zcobcluster [2024-06-17 16:06:02.351] [DEBUG] - exclusive lock `/home/admin/.obd/lock/deploy_zcobcluster`, count 1 [2024-06-17 16:06:02.351] [DEBUG] - Get deploy info [2024-06-17 16:06:02.351] [DEBUG] - judge deploy status [2024-06-17 16:06:02.351] [DEBUG] - config path is None or not [2024-06-17 16:06:02.351] [DEBUG] - Create deploy by configuration path [2024-06-17 16:06:02.351] [DEBUG] - exclusive lock `/home/admin/.obd/lock/deploy_zcobcluster`, count 2 [2024-06-17 16:06:02.351] [DEBUG] - mkdir /home/admin/.obd/cluster/zcobcluster [2024-06-17 16:06:02.351] [DEBUG] - copy zcobcluster_deploy.yaml /home/admin/.obd/cluster/zcobcluster/config.yaml [2024-06-17 16:06:02.398] [DEBUG] - Get deploy configuration [2024-06-17 16:06:02.414] [DEBUG] - Search best suitable repository [2024-06-17 16:06:02.414] [DEBUG] - Search package for components... [2024-06-17 16:06:02.414] [DEBUG] - Get oceanbase-ce repository [2024-06-17 16:06:02.414] [DEBUG] - mkdir /home/admin/.obd/repository [2024-06-17 16:06:02.414] [DEBUG] - Search repository oceanbase-ce version: None, tag: None, release: None, package_hash: None [2024-06-17 16:06:02.414] [DEBUG] - try to get share lock /home/admin/.obd/lock/mirror_and_repo [2024-06-17 16:06:02.415] [DEBUG] - share lock `/home/admin/.obd/lock/mirror_and_repo`, count 1 [2024-06-17 16:06:02.415] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-06-17 16:06:02.416] [DEBUG] - Found repository oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:02.416] [DEBUG] - share lock `/home/admin/.obd/lock/mirror_and_repo`, count 2 [2024-06-17 16:06:02.417] [DEBUG] - Search oceanbase-ce package from mirror [2024-06-17 16:06:02.417] [DEBUG] - mkdir /home/admin/.obd/mirror [2024-06-17 16:06:02.417] [DEBUG] - mkdir /home/admin/.obd/mirror/remote [2024-06-17 16:06:02.417] [DEBUG] - mkdir /home/admin/.obd/mirror/local [2024-06-17 16:06:02.417] [DEBUG] - try to get exclusive lock /home/admin/.obd/lock/mirror_and_repo [2024-06-17 16:06:02.417] [DEBUG] - exclusive lock `/home/admin/.obd/lock/mirror_and_repo`, count 1 [2024-06-17 16:06:02.417] [DEBUG] - exclusive lock `/home/admin/.obd/lock/mirror_and_repo`, count 2 [2024-06-17 16:06:02.418] [DEBUG] - load repo config: /home/admin/.obd/mirror/remote/OceanBase.repo [2024-06-17 16:06:02.418] [DEBUG] - open /home/admin/.obd/mirror/remote/OceanBase.repo for r [2024-06-17 16:06:02.418] [DEBUG] - exclusive lock `/home/admin/.obd/lock/mirror_and_repo`, count 3 [2024-06-17 16:06:02.418] [DEBUG] - md5 is None [2024-06-17 16:06:02.418] [DEBUG] - name is oceanbase-ce [2024-06-17 16:06:02.418] [DEBUG] - arch is ['ia32e', 'x86_64', 'athlon', 'i686', 'i586', 'i486', 'i386', 'noarch'] [2024-06-17 16:06:02.418] [DEBUG] - release is None [2024-06-17 16:06:02.418] [DEBUG] - version is None [2024-06-17 16:06:02.418] [DEBUG] - MirrorRepositoryType.LOCAL mirror local found pkg: name: oceanbase-ce [2024-06-17 16:06:02.418] [DEBUG] version: 4.2.1.1 [2024-06-17 16:06:02.418] [DEBUG] release:101000062023110109.el7 [2024-06-17 16:06:02.418] [DEBUG] arch: x86_64 [2024-06-17 16:06:02.419] [DEBUG] md5: ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:02.419] [DEBUG] - get RPM package by name: oceanbase-ce [2024-06-17 16:06:02.419] [DEBUG] version: 4.2.1.1 [2024-06-17 16:06:02.419] [DEBUG] release:101000062023110109.el7 [2024-06-17 16:06:02.419] [DEBUG] arch: x86_64 [2024-06-17 16:06:02.419] [DEBUG] md5: ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:02.419] [DEBUG] - Found Package oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:02.419] [DEBUG] - name: oceanbase-ce [2024-06-17 16:06:02.419] [DEBUG] version: 4.2.1.1 [2024-06-17 16:06:02.419] [DEBUG] release:101000062023110109.el7 [2024-06-17 16:06:02.419] [DEBUG] arch: x86_64 [2024-06-17 16:06:02.419] [DEBUG] md5: ee47f7d89ad323bd634a5976f6e89f04911f81a4 as same as oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4, Use package name: oceanbase-ce [2024-06-17 16:06:02.419] [DEBUG] version: 4.2.1.1 [2024-06-17 16:06:02.419] [DEBUG] release:101000062023110109.el7 [2024-06-17 16:06:02.419] [DEBUG] arch: x86_64 [2024-06-17 16:06:02.419] [DEBUG] md5: ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:02.419] [DEBUG] - Get obproxy-ce repository [2024-06-17 16:06:02.419] [DEBUG] - Search repository obproxy-ce version: None, tag: None, release: None, package_hash: None [2024-06-17 16:06:02.419] [DEBUG] - share lock `/home/admin/.obd/lock/mirror_and_repo`, count 3 [2024-06-17 16:06:02.419] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy-ce [2024-06-17 16:06:02.421] [DEBUG] - Found repository obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:06:02.421] [DEBUG] - share lock `/home/admin/.obd/lock/mirror_and_repo`, count 4 [2024-06-17 16:06:02.422] [DEBUG] - Search obproxy-ce package from mirror [2024-06-17 16:06:02.422] [DEBUG] - exclusive lock `/home/admin/.obd/lock/mirror_and_repo`, count 4 [2024-06-17 16:06:02.422] [DEBUG] - exclusive lock `/home/admin/.obd/lock/mirror_and_repo`, count 5 [2024-06-17 16:06:02.422] [DEBUG] - load repo config: /home/admin/.obd/mirror/remote/OceanBase.repo [2024-06-17 16:06:02.422] [DEBUG] - exclusive lock `/home/admin/.obd/lock/mirror_and_repo`, count 6 [2024-06-17 16:06:02.422] [DEBUG] - md5 is None [2024-06-17 16:06:02.422] [DEBUG] - name is obproxy-ce [2024-06-17 16:06:02.422] [DEBUG] - arch is ['ia32e', 'x86_64', 'athlon', 'i686', 'i586', 'i486', 'i386', 'noarch'] [2024-06-17 16:06:02.422] [DEBUG] - release is None [2024-06-17 16:06:02.422] [DEBUG] - version is None [2024-06-17 16:06:02.422] [DEBUG] - MirrorRepositoryType.LOCAL mirror local found pkg: name: obproxy-ce [2024-06-17 16:06:02.422] [DEBUG] version: 4.2.1.0 [2024-06-17 16:06:02.422] [DEBUG] release:11.el7 [2024-06-17 16:06:02.422] [DEBUG] arch: x86_64 [2024-06-17 16:06:02.422] [DEBUG] md5: 0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:06:02.422] [DEBUG] - get RPM package by name: obproxy-ce [2024-06-17 16:06:02.422] [DEBUG] version: 4.2.1.0 [2024-06-17 16:06:02.422] [DEBUG] release:11.el7 [2024-06-17 16:06:02.423] [DEBUG] arch: x86_64 [2024-06-17 16:06:02.423] [DEBUG] md5: 0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:06:02.423] [DEBUG] - Found Package obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:06:02.423] [DEBUG] - name: obproxy-ce [2024-06-17 16:06:02.423] [DEBUG] version: 4.2.1.0 [2024-06-17 16:06:02.423] [DEBUG] release:11.el7 [2024-06-17 16:06:02.423] [DEBUG] arch: x86_64 [2024-06-17 16:06:02.423] [DEBUG] md5: 0aed4b782120e4248b749f67be3d2cc82cdcb70d as same as obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d, Use package name: obproxy-ce [2024-06-17 16:06:02.423] [DEBUG] version: 4.2.1.0 [2024-06-17 16:06:02.423] [DEBUG] release:11.el7 [2024-06-17 16:06:02.423] [DEBUG] arch: x86_64 [2024-06-17 16:06:02.423] [DEBUG] md5: 0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:06:02.423] [DEBUG] - Searching install plugin for components ... [2024-06-17 16:06:02.423] [DEBUG] - Searching install plugin for components ... [2024-06-17 16:06:02.423] [DEBUG] - Search install plugin for oceanbase-ce [2024-06-17 16:06:02.423] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-06-17 16:06:02.424] [DEBUG] - Found for oceanbase-ce-install-4.0.0.0 for oceanbase-ce-4.2.1.1 [2024-06-17 16:06:02.424] [DEBUG] - Search install plugin for obproxy-ce [2024-06-17 16:06:02.424] [DEBUG] - Found for obproxy-ce-install-3.1.0 for obproxy-ce-4.2.1.0 [2024-06-17 16:06:02.424] [DEBUG] - create instance repository for oceanbase-ce-4.2.1.1 [2024-06-17 16:06:02.427] [DEBUG] - oceanbase-ce-4.2.1.1 is already install [2024-06-17 16:06:02.427] [DEBUG] - create instance repository for obproxy-ce-4.2.1.0 [2024-06-17 16:06:02.428] [DEBUG] - obproxy-ce-4.2.1.0 is already install [2024-06-17 16:06:02.428] [DEBUG] - Searching install plugin for components ... [2024-06-17 16:06:02.428] [DEBUG] - Search install plugin for oceanbase-ce [2024-06-17 16:06:02.429] [DEBUG] - Found for oceanbase-ce-install-4.0.0.0 for oceanbase-ce-4.2.1.1 [2024-06-17 16:06:02.429] [DEBUG] - Search install plugin for obproxy-ce [2024-06-17 16:06:02.429] [DEBUG] - Found for obproxy-ce-install-3.1.0 for obproxy-ce-4.2.1.0 [2024-06-17 16:06:02.429] [INFO] +--------------------------------------------------------------------------------------------+ [2024-06-17 16:06:02.429] [INFO] | Packages | [2024-06-17 16:06:02.429] [INFO] +--------------+---------+------------------------+------------------------------------------+ [2024-06-17 16:06:02.429] [INFO] | Repository | Version | Release | Md5 | [2024-06-17 16:06:02.429] [INFO] +--------------+---------+------------------------+------------------------------------------+ [2024-06-17 16:06:02.429] [INFO] | oceanbase-ce | 4.2.1.1 | 101000062023110109.el7 | ee47f7d89ad323bd634a5976f6e89f04911f81a4 | [2024-06-17 16:06:02.429] [INFO] | obproxy-ce | 4.2.1.0 | 11.el7 | 0aed4b782120e4248b749f67be3d2cc82cdcb70d | [2024-06-17 16:06:02.429] [INFO] +--------------+---------+------------------------+------------------------------------------+ [2024-06-17 16:06:02.430] [INFO] Repository integrity check [2024-06-17 16:06:02.606] [INFO] Parameter check [2024-06-17 16:06:02.631] [DEBUG] - Searching param plugin for components ... [2024-06-17 16:06:02.631] [DEBUG] - Search param plugin for oceanbase-ce [2024-06-17 16:06:02.631] [DEBUG] - Found for oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.1.1 [2024-06-17 16:06:02.631] [DEBUG] - Applying oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:03.291] [DEBUG] - Search param plugin for obproxy-ce [2024-06-17 16:06:03.291] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0 [2024-06-17 16:06:03.291] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:06:03.395] [DEBUG] - Searching generate_config plugin for components ... [2024-06-17 16:06:03.395] [DEBUG] - Searching generate_config plugin for oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:03.395] [DEBUG] - Found for oceanbase-ce-py_script_generate_config-4.2.0.0 for oceanbase-ce-4.2.1.1 [2024-06-17 16:06:03.395] [DEBUG] - Searching generate_config plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:06:03.395] [DEBUG] - Found for obproxy-ce-py_script_generate_config-3.1.0 for obproxy-ce-4.2.1.0 [2024-06-17 16:06:53.112] [DEBUG] - Call oceanbase-ce-py_script_generate_config-4.2.0.0 for oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:53.112] [DEBUG] - import generate_config [2024-06-17 16:06:53.116] [DEBUG] - add generate_config ref count to 1 [2024-06-17 16:06:53.118] [DEBUG] - open /home/admin/.obd/cluster/zcobcluster/inner_config.yaml for w [2024-06-17 16:06:53.118] [DEBUG] - mkdir /home/admin/.obd/cluster/zcobcluster [2024-06-17 16:06:53.124] [DEBUG] - sub generate_config ref count to 0 [2024-06-17 16:06:53.124] [DEBUG] - export generate_config [2024-06-17 16:06:53.124] [DEBUG] - Call obproxy-ce-py_script_generate_config-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:06:53.124] [DEBUG] - import generate_config [2024-06-17 16:06:53.125] [DEBUG] - add generate_config ref count to 1 [2024-06-17 16:06:53.126] [DEBUG] - open /home/admin/.obd/cluster/zcobcluster/inner_config.yaml for w [2024-06-17 16:06:53.131] [DEBUG] - sub generate_config ref count to 0 [2024-06-17 16:06:53.131] [DEBUG] - export generate_config [2024-06-17 16:06:53.131] [DEBUG] - Cluster param configuration check [2024-06-17 16:06:53.132] [DEBUG] - server1(192.168.166.181) oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 param check [2024-06-17 16:06:53.132] [DEBUG] - server2(192.168.166.182) oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 param check [2024-06-17 16:06:53.132] [DEBUG] - server3(192.168.166.183) oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 param check [2024-06-17 16:06:53.132] [DEBUG] - 192.168.166.182 obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d param check [2024-06-17 16:06:53.132] [INFO] Cluster status check [2024-06-17 16:06:53.132] [DEBUG] - Searching status plugin for components ... [2024-06-17 16:06:53.133] [DEBUG] - Searching status plugin for oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:53.133] [DEBUG] - Found for oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.1.1 [2024-06-17 16:06:53.133] [DEBUG] - Searching status plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:06:53.133] [DEBUG] - Found for obproxy-ce-py_script_status-3.1.0 for obproxy-ce-4.2.1.0 [2024-06-17 16:06:53.133] [DEBUG] - Call oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:06:53.133] [DEBUG] - import status [2024-06-17 16:06:53.134] [DEBUG] - add status ref count to 1 [2024-06-17 16:06:53.134] [DEBUG] -- admin@192.168.166.181 execute: cat /home/admin/zcobserver/oceanbase/run/observer.pid [2024-06-17 16:06:53.226] [DEBUG] -- exited code 1, error output: [2024-06-17 16:06:53.227] [DEBUG] cat: /home/admin/zcobserver/oceanbase/run/observer.pid: No such file or directory [2024-06-17 16:06:53.227] [DEBUG] [2024-06-17 16:06:53.227] [DEBUG] -- admin@192.168.166.182 execute: cat /home/admin/zcobserver/oceanbase/run/observer.pid [2024-06-17 16:06:53.245] [DEBUG] -- exited code 1, error output: [2024-06-17 16:06:53.245] [DEBUG] cat: /home/admin/zcobserver/oceanbase/run/observer.pid: No such file or directory [2024-06-17 16:06:53.245] [DEBUG] [2024-06-17 16:06:53.245] [DEBUG] -- admin@192.168.166.183 execute: cat /home/admin/zcobserver/oceanbase/run/observer.pid [2024-06-17 16:07:23.247] [ERROR] Traceback (most recent call last): [2024-06-17 16:07:23.248] [ERROR] File "ssh.py", line 411, in _execute_command [2024-06-17 16:07:23.248] [ERROR] File "paramiko/client.py", line 510, in exec_command [2024-06-17 16:07:23.248] [ERROR] File "paramiko/transport.py", line 919, in open_session [2024-06-17 16:07:23.248] [ERROR] File "paramiko/transport.py", line 1054, in open_channel [2024-06-17 16:07:23.248] [ERROR] paramiko.ssh_exception.SSHException: Timeout opening channel. [2024-06-17 16:07:23.248] [ERROR] [2024-06-17 16:07:23.248] [ERROR] During handling of the above exception, another exception occurred: [2024-06-17 16:07:23.248] [ERROR] [2024-06-17 16:07:23.248] [ERROR] Traceback (most recent call last): [2024-06-17 16:07:23.248] [ERROR] File "core.py", line 1398, in deploy_cluster [2024-06-17 16:07:23.248] [ERROR] File "core.py", line 1450, in _deploy_cluster [2024-06-17 16:07:23.248] [ERROR] File "core.py", line 950, in cluster_status_check [2024-06-17 16:07:23.248] [ERROR] File "core.py", line 184, in call_plugin [2024-06-17 16:07:23.248] [ERROR] File "_plugin.py", line 324, in __call__ [2024-06-17 16:07:23.248] [ERROR] File "_plugin.py", line 286, in _new_func [2024-06-17 16:07:23.248] [ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/status.py", line 37, in status [2024-06-17 16:07:23.248] [ERROR] remote_pid = client.execute_command('cat %s' % remote_pid_path).stdout.strip() [2024-06-17 16:07:23.248] [ERROR] File "_plugin.py", line 223, in new_method [2024-06-17 16:07:23.248] [ERROR] File "_stdio.py", line 888, in func_wrapper [2024-06-17 16:07:23.248] [ERROR] File "ssh.py", line 450, in execute_command [2024-06-17 16:07:23.248] [ERROR] File "_stdio.py", line 888, in func_wrapper [2024-06-17 16:07:23.248] [ERROR] File "ssh.py", line 427, in _execute_command [2024-06-17 16:07:23.248] [ERROR] File "_stdio.py", line 888, in func_wrapper [2024-06-17 16:07:23.248] [ERROR] File "ssh.py", line 411, in _execute_command [2024-06-17 16:07:23.248] [ERROR] File "paramiko/client.py", line 510, in exec_command [2024-06-17 16:07:23.248] [ERROR] AttributeError: 'NoneType' object has no attribute 'open_session' [2024-06-17 16:07:23.248] [ERROR] [2024-06-17 16:07:23.248] [DEBUG] - sub status ref count to 0 [2024-06-17 16:07:23.248] [DEBUG] - export status [2024-06-17 16:07:23.248] [DEBUG] - Call obproxy-ce-py_script_status-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:07:23.248] [DEBUG] - import status [2024-06-17 16:07:23.249] [DEBUG] - add status ref count to 1 [2024-06-17 16:07:23.249] [DEBUG] -- admin@192.168.166.182 execute: cat /home/admin/obproxy/run/obproxy-192.168.166.182-2883.pid [2024-06-17 16:07:23.264] [DEBUG] -- exited code 1, error output: [2024-06-17 16:07:23.264] [DEBUG] cat: /home/admin/obproxy/run/obproxy-192.168.166.182-2883.pid: No such file or directory [2024-06-17 16:07:23.264] [DEBUG] [2024-06-17 16:07:23.264] [DEBUG] - sub status ref count to 0 [2024-06-17 16:07:23.264] [DEBUG] - export status [2024-06-17 16:07:23.373] [DEBUG] - Search init plugin [2024-06-17 16:07:23.374] [DEBUG] - Searching init plugin for components ... [2024-06-17 16:07:23.374] [DEBUG] - Searching init plugin for oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:07:23.374] [DEBUG] - Found for oceanbase-ce-py_script_init-4.0.0.0 for oceanbase-ce-4.2.1.1 [2024-06-17 16:07:23.374] [DEBUG] - Searching init plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:07:23.374] [DEBUG] - Found for obproxy-ce-py_script_init-3.1.0 for obproxy-ce-4.2.1.0 [2024-06-17 16:07:23.374] [DEBUG] - Exec oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 init plugin [2024-06-17 16:07:23.374] [DEBUG] - Apply oceanbase-ce-py_script_init-4.0.0.0 for oceanbase-ce-4.2.1.1 [2024-06-17 16:07:23.374] [DEBUG] - Call oceanbase-ce-py_script_init-4.0.0.0 for oceanbase-ce-4.2.1.1-101000062023110109.el7-ee47f7d89ad323bd634a5976f6e89f04911f81a4 [2024-06-17 16:07:23.374] [DEBUG] - import init [2024-06-17 16:07:23.376] [DEBUG] - add init ref count to 1 [2024-06-17 16:07:23.376] [DEBUG] -- option `force` is False [2024-06-17 16:07:23.376] [INFO] Initializes observer work home [2024-06-17 16:07:23.377] [DEBUG] -- server1(192.168.166.181) initializes observer work home [2024-06-17 16:07:23.377] [DEBUG] -- admin@192.168.166.181 execute: mkdir -p /home/admin/zcobserver/oceanbase [2024-06-17 16:07:23.392] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.392] [DEBUG] -- admin@192.168.166.181 execute: ls /home/admin/zcobserver/oceanbase [2024-06-17 16:07:23.447] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.447] [DEBUG] -- admin@192.168.166.181 execute: bash -c "mkdir -p /home/admin/zcobserver/oceanbase/{etc,admin,.conf,log,bin,lib}" [2024-06-17 16:07:23.503] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.504] [DEBUG] -- admin@192.168.166.181 execute: mkdir -p /data [2024-06-17 16:07:23.558] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.559] [DEBUG] -- admin@192.168.166.181 execute: ls /data [2024-06-17 16:07:23.613] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.613] [DEBUG] -- admin@192.168.166.181 execute: bash -c "mkdir -p /data/sstable" [2024-06-17 16:07:23.669] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.669] [DEBUG] -- admin@192.168.166.181 execute: if [ ! '/data' -ef '/home/admin/zcobserver/oceanbase/store' ]; then ln -sf /data /home/admin/zcobserver/oceanbase/store; fi [2024-06-17 16:07:23.725] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.725] [DEBUG] -- admin@192.168.166.181 execute: mkdir -p /redo/clog [2024-06-17 16:07:23.777] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.778] [DEBUG] -- admin@192.168.166.181 execute: ls /redo/clog [2024-06-17 16:07:23.832] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.832] [DEBUG] -- admin@192.168.166.181 execute: mkdir -p /redo/clog [2024-06-17 16:07:23.885] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.886] [DEBUG] -- admin@192.168.166.181 execute: if [ ! '/redo/clog' -ef '/data/clog' ]; then ln -sf /redo/clog /data/clog; fi [2024-06-17 16:07:23.939] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.940] [DEBUG] -- admin@192.168.166.181 execute: mkdir -p /data/slog [2024-06-17 16:07:23.994] [DEBUG] -- exited code 0 [2024-06-17 16:07:23.994] [DEBUG] -- admin@192.168.166.181 execute: ls /data/slog [2024-06-17 16:07:24.047] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.048] [DEBUG] -- admin@192.168.166.181 execute: mkdir -p /data/slog [2024-06-17 16:07:24.104] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.105] [DEBUG] -- admin@192.168.166.181 execute: if [ ! '/data/slog' -ef '/data/slog' ]; then ln -sf /data/slog /data/slog; fi [2024-06-17 16:07:24.159] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.159] [DEBUG] -- check slog dir in the same disk with data dir [2024-06-17 16:07:24.159] [DEBUG] -- admin@192.168.166.181 execute: df --block-size=1024 /data/slog | awk 'NR == 2 { print $1 }' [2024-06-17 16:07:24.213] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.213] [DEBUG] -- slog disk is /dev/sda3 [2024-06-17 16:07:24.213] [DEBUG] -- admin@192.168.166.181 execute: df --block-size=1024 /data | awk 'NR == 2 { print $1 }' [2024-06-17 16:07:24.270] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.270] [DEBUG] -- data disk is /dev/sda3 [2024-06-17 16:07:24.270] [DEBUG] -- server2(192.168.166.182) initializes observer work home [2024-06-17 16:07:24.270] [DEBUG] -- admin@192.168.166.182 execute: mkdir -p /home/admin/zcobserver/oceanbase [2024-06-17 16:07:24.283] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.283] [DEBUG] -- admin@192.168.166.182 execute: ls /home/admin/zcobserver/oceanbase [2024-06-17 16:07:24.338] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.338] [DEBUG] -- admin@192.168.166.182 execute: bash -c "mkdir -p /home/admin/zcobserver/oceanbase/{etc,admin,.conf,log,bin,lib}" [2024-06-17 16:07:24.393] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.393] [DEBUG] -- admin@192.168.166.182 execute: mkdir -p /data [2024-06-17 16:07:24.449] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.449] [DEBUG] -- admin@192.168.166.182 execute: ls /data [2024-06-17 16:07:24.503] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.504] [DEBUG] -- admin@192.168.166.182 execute: bash -c "mkdir -p /data/sstable" [2024-06-17 16:07:24.559] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.560] [DEBUG] -- admin@192.168.166.182 execute: if [ ! '/data' -ef '/home/admin/zcobserver/oceanbase/store' ]; then ln -sf /data /home/admin/zcobserver/oceanbase/store; fi [2024-06-17 16:07:24.648] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.649] [DEBUG] -- admin@192.168.166.182 execute: mkdir -p /redo/clog [2024-06-17 16:07:24.704] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.704] [DEBUG] -- admin@192.168.166.182 execute: ls /redo/clog [2024-06-17 16:07:24.759] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.759] [DEBUG] -- admin@192.168.166.182 execute: mkdir -p /redo/clog [2024-06-17 16:07:24.814] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.814] [DEBUG] -- admin@192.168.166.182 execute: if [ ! '/redo/clog' -ef '/data/clog' ]; then ln -sf /redo/clog /data/clog; fi [2024-06-17 16:07:24.869] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.869] [DEBUG] -- admin@192.168.166.182 execute: mkdir -p /data/slog [2024-06-17 16:07:24.925] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.925] [DEBUG] -- admin@192.168.166.182 execute: ls /data/slog [2024-06-17 16:07:24.979] [DEBUG] -- exited code 0 [2024-06-17 16:07:24.980] [DEBUG] -- admin@192.168.166.182 execute: mkdir -p /data/slog [2024-06-17 16:07:25.032] [DEBUG] -- exited code 0 [2024-06-17 16:07:25.033] [DEBUG] -- admin@192.168.166.182 execute: if [ ! '/data/slog' -ef '/data/slog' ]; then ln -sf /data/slog /data/slog; fi [2024-06-17 16:07:25.085] [DEBUG] -- exited code 0 [2024-06-17 16:07:25.086] [DEBUG] -- check slog dir in the same disk with data dir [2024-06-17 16:07:25.086] [DEBUG] -- admin@192.168.166.182 execute: df --block-size=1024 /data/slog | awk 'NR == 2 { print $1 }' [2024-06-17 16:07:25.188] [DEBUG] -- exited code 0 [2024-06-17 16:07:25.189] [DEBUG] -- slog disk is /dev/sda3 [2024-06-17 16:07:25.189] [DEBUG] -- admin@192.168.166.182 execute: df --block-size=1024 /data | awk 'NR == 2 { print $1 }' [2024-06-17 16:07:25.244] [DEBUG] -- exited code 0 [2024-06-17 16:07:25.245] [DEBUG] -- data disk is /dev/sda3 [2024-06-17 16:07:25.245] [DEBUG] -- server3(192.168.166.183) initializes observer work home [2024-06-17 16:07:25.245] [DEBUG] -- admin@192.168.166.183 execute: mkdir -p /home/admin/zcobserver/oceanbase [2024-06-17 16:07:25.245] [ERROR] Traceback (most recent call last): [2024-06-17 16:07:25.245] [ERROR] File "core.py", line 1398, in deploy_cluster [2024-06-17 16:07:25.245] [ERROR] File "core.py", line 1468, in _deploy_cluster [2024-06-17 16:07:25.246] [ERROR] File "core.py", line 184, in call_plugin [2024-06-17 16:07:25.246] [ERROR] File "_plugin.py", line 324, in __call__ [2024-06-17 16:07:25.246] [ERROR] File "_plugin.py", line 286, in _new_func [2024-06-17 16:07:25.246] [ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/4.0.0.0/init.py", line 122, in init [2024-06-17 16:07:25.246] [ERROR] if client.execute_command('mkdir -p %s' % home_path): [2024-06-17 16:07:25.246] [ERROR] File "_plugin.py", line 223, in new_method [2024-06-17 16:07:25.246] [ERROR] File "_stdio.py", line 888, in func_wrapper [2024-06-17 16:07:25.246] [ERROR] File "ssh.py", line 450, in execute_command [2024-06-17 16:07:25.246] [ERROR] File "_stdio.py", line 888, in func_wrapper [2024-06-17 16:07:25.246] [ERROR] File "ssh.py", line 411, in _execute_command [2024-06-17 16:07:25.246] [ERROR] File "paramiko/client.py", line 510, in exec_command [2024-06-17 16:07:25.246] [ERROR] AttributeError: 'NoneType' object has no attribute 'open_session' [2024-06-17 16:07:25.246] [ERROR] [2024-06-17 16:07:25.246] [ERROR] OBD-1002: Fail to init server3(192.168.166.183) home path: create /home/admin/zcobserver/oceanbase failed. [2024-06-17 16:07:25.246] [DEBUG] -- admin@192.168.166.183 execute: bash -c "mkdir -p /home/admin/zcobserver/oceanbase/{etc,admin,.conf,log,bin,lib}" [2024-06-17 16:07:25.246] [ERROR] Traceback (most recent call last): [2024-06-17 16:07:25.246] [ERROR] File "core.py", line 1398, in deploy_cluster [2024-06-17 16:07:25.246] [ERROR] File "core.py", line 1468, in _deploy_cluster [2024-06-17 16:07:25.246] [ERROR] File "core.py", line 184, in call_plugin [2024-06-17 16:07:25.246] [ERROR] File "_plugin.py", line 324, in __call__ [2024-06-17 16:07:25.246] [ERROR] File "_plugin.py", line 286, in _new_func [2024-06-17 16:07:25.246] [ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/4.0.0.0/init.py", line 129, in init [2024-06-17 16:07:25.246] [ERROR] ret = client.execute_command('bash -c "mkdir -p %s/{etc,admin,.conf,log,bin,lib}"' % home_path) [2024-06-17 16:07:25.246] [ERROR] File "_plugin.py", line 223, in new_method [2024-06-17 16:07:25.246] [ERROR] File "_stdio.py", line 888, in func_wrapper [2024-06-17 16:07:25.246] [ERROR] File "ssh.py", line 450, in execute_command [2024-06-17 16:07:25.246] [ERROR] File "_stdio.py", line 888, in func_wrapper [2024-06-17 16:07:25.246] [ERROR] File "ssh.py", line 411, in _execute_command [2024-06-17 16:07:25.246] [ERROR] File "paramiko/client.py", line 510, in exec_command [2024-06-17 16:07:25.246] [ERROR] AttributeError: 'NoneType' object has no attribute 'open_session' [2024-06-17 16:07:25.247] [ERROR] [2024-06-17 16:07:25.247] [ERROR] OBD-1002: Fail to init server3(192.168.166.183) home path: /home/admin/zcobserver/oceanbase permission denied . [2024-06-17 16:07:25.359] [INFO] [ERROR] OBD-1002: Fail to init server3(192.168.166.183) home path: create /home/admin/zcobserver/oceanbase failed. [2024-06-17 16:07:25.359] [INFO] [ERROR] OBD-1002: Fail to init server3(192.168.166.183) home path: /home/admin/zcobserver/oceanbase permission denied . [2024-06-17 16:07:25.359] [INFO] [2024-06-17 16:07:25.359] [DEBUG] - sub init ref count to 0 [2024-06-17 16:07:25.359] [DEBUG] - export init [2024-06-17 16:07:25.359] [DEBUG] - Exec obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d init plugin [2024-06-17 16:07:25.359] [DEBUG] - Apply obproxy-ce-py_script_init-3.1.0 for obproxy-ce-4.2.1.0 [2024-06-17 16:07:25.359] [DEBUG] - Call obproxy-ce-py_script_init-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-06-17 16:07:25.359] [DEBUG] - import init [2024-06-17 16:07:25.360] [DEBUG] - add init ref count to 1 [2024-06-17 16:07:25.361] [INFO] Initializes obproxy work home [2024-06-17 16:07:25.361] [DEBUG] -- 192.168.166.182 init cluster work home [2024-06-17 16:07:25.361] [DEBUG] -- admin@192.168.166.182 execute: bash -c 'mkdir -p /home/admin/obproxy/{run,bin,lib}' [2024-06-17 16:07:25.375] [DEBUG] -- exited code 0 [2024-06-17 16:07:25.494] [DEBUG] - sub init ref count to 0 [2024-06-17 16:07:25.494] [DEBUG] - export init [2024-06-17 16:07:25.589] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2024-06-17 16:07:25.589] [INFO] Trace ID: 70500664-2c80-11ef-ac5b-000c29732329 [2024-06-17 16:07:25.589] [INFO] If you want to view detailed obd logs, please run: obd display-trace 70500664-2c80-11ef-ac5b-000c29732329 [2024-06-17 16:07:25.589] [DEBUG] - exclusive lock /home/admin/.obd/lock/mirror_and_repo release, count 5 [2024-06-17 16:07:25.589] [DEBUG] - exclusive lock /home/admin/.obd/lock/mirror_and_repo release, count 4 [2024-06-17 16:07:25.589] [DEBUG] - exclusive lock /home/admin/.obd/lock/mirror_and_repo release, count 3 [2024-06-17 16:07:25.589] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 3 [2024-06-17 16:07:25.589] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 2 [2024-06-17 16:07:25.589] [DEBUG] - exclusive lock /home/admin/.obd/lock/mirror_and_repo release, count 2 [2024-06-17 16:07:25.589] [DEBUG] - exclusive lock /home/admin/.obd/lock/mirror_and_repo release, count 1 [2024-06-17 16:07:25.589] [DEBUG] - exclusive lock /home/admin/.obd/lock/mirror_and_repo release, count 0 [2024-06-17 16:07:25.589] [DEBUG] - try to get share lock /home/admin/.obd/lock/mirror_and_repo [2024-06-17 16:07:25.590] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 1 [2024-06-17 16:07:25.590] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 0 [2024-06-17 16:07:25.590] [DEBUG] - unlock /home/admin/.obd/lock/mirror_and_repo [2024-06-17 16:07:25.590] [DEBUG] - exclusive lock /home/admin/.obd/lock/deploy_zcobcluster release, count 1 [2024-06-17 16:07:25.590] [DEBUG] - exclusive lock /home/admin/.obd/lock/deploy_zcobcluster release, count 0 [2024-06-17 16:07:25.590] [DEBUG] - unlock /home/admin/.obd/lock/deploy_zcobcluster [2024-06-17 16:07:25.590] [DEBUG] - share lock /home/admin/.obd/lock/global release, count 0 [2024-06-17 16:07:25.590] [DEBUG] - unlock /home/admin/.obd/lock/global