[2023-05-03 16:12:58.082] [DEBUG] - mkdir /root/.obd/lock/ [2023-05-03 16:12:58.082] [DEBUG] - unknown lock mode [2023-05-03 16:12:58.082] [DEBUG] - try to get share lock /root/.obd/lock/global [2023-05-03 16:12:58.083] [DEBUG] - share lock `/root/.obd/lock/global`, count 1 [2023-05-03 16:12:58.083] [DEBUG] - cmd: ['myoceanbase'] [2023-05-03 16:12:58.083] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2023-05-03 16:12:58.083] [DEBUG] - Get Deploy by name [2023-05-03 16:12:58.083] [DEBUG] - mkdir /root/.obd/cluster/ [2023-05-03 16:12:58.083] [DEBUG] - mkdir /root/.obd/config_parser/ [2023-05-03 16:12:58.083] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_myoceanbase [2023-05-03 16:12:58.083] [DEBUG] - exclusive lock `/root/.obd/lock/deploy_myoceanbase`, count 1 [2023-05-03 16:12:58.088] [DEBUG] - Deploy status judge [2023-05-03 16:12:58.088] [INFO] Get local repositories [2023-05-03 16:12:58.088] [DEBUG] - mkdir /root/.obd/repository [2023-05-03 16:12:58.089] [DEBUG] - Get local repository oceanbase-ce-4.1.0.0-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:58.089] [DEBUG] - Search repository oceanbase-ce version: 4.1.0.0, tag: 8439ecf8db5e0649bd49671b41ea9e8c85756b63, release: None, package_hash: None [2023-05-03 16:12:58.089] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo [2023-05-03 16:12:58.089] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 1 [2023-05-03 16:12:58.089] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce [2023-05-03 16:12:58.091] [DEBUG] - Found repository oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:58.092] [DEBUG] - Get local repository obproxy-ce-4.1.0.0-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:12:58.092] [DEBUG] - Search repository obproxy-ce version: 4.1.0.0, tag: 2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe, release: None, package_hash: None [2023-05-03 16:12:58.092] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 2 [2023-05-03 16:12:58.092] [DEBUG] - mkdir /root/.obd/repository/obproxy-ce [2023-05-03 16:12:58.093] [DEBUG] - Found repository obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:12:58.093] [DEBUG] - Get local repository obagent-1.3.0-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:12:58.093] [DEBUG] - Search repository obagent version: 1.3.0, tag: d57fbb4962b2fbecb6282358c59295fdfba4d6ac, release: None, package_hash: None [2023-05-03 16:12:58.093] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 3 [2023-05-03 16:12:58.094] [DEBUG] - mkdir /root/.obd/repository/obagent [2023-05-03 16:12:58.095] [DEBUG] - Found repository obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:12:58.222] [DEBUG] - Get deploy config [2023-05-03 16:12:58.279] [INFO] Search plugins [2023-05-03 16:12:58.280] [DEBUG] - Searching start_check plugin for components ... [2023-05-03 16:12:58.280] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:58.280] [DEBUG] - mkdir /root/.obd/plugins [2023-05-03 16:12:58.281] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.0.0.0 for oceanbase-ce-4.1.0.0 [2023-05-03 16:12:58.281] [DEBUG] - Searching start_check plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:12:58.281] [DEBUG] - Found for obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.1.0.0 [2023-05-03 16:12:58.281] [DEBUG] - Searching start_check plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:12:58.281] [DEBUG] - Found for obagent-py_script_start_check-1.3.0 for obagent-1.3.0 [2023-05-03 16:12:58.281] [DEBUG] - Searching create_tenant plugin for components ... [2023-05-03 16:12:58.281] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:58.282] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.0.0.0 for oceanbase-ce-4.1.0.0 [2023-05-03 16:12:58.282] [DEBUG] - Searching create_tenant plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:12:58.282] [DEBUG] - No such create_tenant plugin for obproxy-ce-4.1.0.0 [2023-05-03 16:12:58.282] [DEBUG] - Searching create_tenant plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:12:58.282] [DEBUG] - No such create_tenant plugin for obagent-1.3.0 [2023-05-03 16:12:58.282] [DEBUG] - Searching start plugin for components ... [2023-05-03 16:12:58.282] [DEBUG] - Searching start plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:58.282] [DEBUG] - Found for oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.1.0.0 [2023-05-03 16:12:58.283] [DEBUG] - Searching start plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:12:58.283] [DEBUG] - Found for obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.1.0.0 [2023-05-03 16:12:58.283] [DEBUG] - Searching start plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:12:58.283] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-1.3.0 [2023-05-03 16:12:58.283] [DEBUG] - Searching connect plugin for components ... [2023-05-03 16:12:58.283] [DEBUG] - Searching connect plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:58.284] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.1.0.0 [2023-05-03 16:12:58.284] [DEBUG] - Searching connect plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:12:58.284] [DEBUG] - Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.1.0.0 [2023-05-03 16:12:58.284] [DEBUG] - Searching connect plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:12:58.284] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-1.3.0 [2023-05-03 16:12:58.284] [DEBUG] - Searching bootstrap plugin for components ... [2023-05-03 16:12:58.284] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:58.285] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.1.0.0 [2023-05-03 16:12:58.285] [DEBUG] - Searching bootstrap plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:12:58.285] [DEBUG] - Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.1.0.0 [2023-05-03 16:12:58.285] [DEBUG] - Searching bootstrap plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:12:58.285] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-1.3.0 [2023-05-03 16:12:58.285] [DEBUG] - Searching display plugin for components ... [2023-05-03 16:12:58.285] [DEBUG] - Searching display plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:58.286] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.1.0.0 [2023-05-03 16:12:58.286] [DEBUG] - Searching display plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:12:58.286] [DEBUG] - Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.1.0.0 [2023-05-03 16:12:58.286] [DEBUG] - Searching display plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:12:58.286] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-1.3.0 [2023-05-03 16:12:58.412] [INFO] Open ssh connection [2023-05-03 16:12:59.074] [INFO] Load cluster param plugin [2023-05-03 16:12:59.078] [DEBUG] - Searching param plugin for components ... [2023-05-03 16:12:59.079] [DEBUG] - Search param plugin for oceanbase-ce [2023-05-03 16:12:59.082] [DEBUG] - Found for oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.1.0.0 [2023-05-03 16:12:59.083] [DEBUG] - Applying oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:59.532] [DEBUG] - Search param plugin for obproxy-ce [2023-05-03 16:12:59.532] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.1.0.0 [2023-05-03 16:12:59.532] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:12:59.630] [DEBUG] - Search param plugin for obagent [2023-05-03 16:12:59.631] [DEBUG] - Found for obagent-param-1.3.0 for obagent-1.3.0 [2023-05-03 16:12:59.631] [DEBUG] - Applying obagent-param-1.3.0 for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:12:59.765] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:12:59.765] [DEBUG] - import start_check [2023-05-03 16:12:59.791] [DEBUG] - add start_check ref count to 1 [2023-05-03 16:12:59.792] [INFO] Check before start observer [2023-05-03 16:12:59.794] [DEBUG] -- root@192.168.1.228 execute: ls /root/observer/store/clog/tenant_1/ [2023-05-03 16:12:59.811] [DEBUG] -- exited code 0 [2023-05-03 16:12:59.811] [DEBUG] -- root@192.168.1.228 execute: cat /root/observer/run/observer.pid [2023-05-03 16:12:59.865] [DEBUG] -- exited code 1, error output: [2023-05-03 16:12:59.866] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory [2023-05-03 16:12:59.866] [DEBUG] [2023-05-03 16:12:59.867] [DEBUG] -- server1(192.168.1.228) port check [2023-05-03 16:12:59.868] [DEBUG] -- root@192.168.1.228 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:12:59.925] [DEBUG] -- exited code 0 [2023-05-03 16:12:59.928] [DEBUG] -- root@192.168.1.228 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:12:59.986] [DEBUG] -- exited code 0 [2023-05-03 16:12:59.988] [DEBUG] -- root@192.168.1.228 execute: ls /root/observer/store/sstable/block_file [2023-05-03 16:13:00.041] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.046] [DEBUG] -- root@192.168.1.235 execute: ls /root/observer/store/clog/tenant_1/ [2023-05-03 16:13:00.089] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.090] [DEBUG] -- root@192.168.1.235 execute: cat /root/observer/run/observer.pid [2023-05-03 16:13:00.168] [DEBUG] -- exited code 1, error output: [2023-05-03 16:13:00.168] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory [2023-05-03 16:13:00.168] [DEBUG] [2023-05-03 16:13:00.169] [DEBUG] -- server2(192.168.1.235) port check [2023-05-03 16:13:00.169] [DEBUG] -- root@192.168.1.235 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:00.262] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.265] [DEBUG] -- root@192.168.1.235 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:00.359] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.360] [DEBUG] -- root@192.168.1.235 execute: ls /root/observer/store/sstable/block_file [2023-05-03 16:13:00.520] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.521] [DEBUG] -- root@192.168.1.239 execute: ls /root/observer/store/clog/tenant_1/ [2023-05-03 16:13:00.578] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.578] [DEBUG] -- root@192.168.1.239 execute: cat /root/observer/run/observer.pid [2023-05-03 16:13:00.663] [DEBUG] -- exited code 1, error output: [2023-05-03 16:13:00.663] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory [2023-05-03 16:13:00.663] [DEBUG] [2023-05-03 16:13:00.663] [DEBUG] -- server3(192.168.1.239) port check [2023-05-03 16:13:00.663] [DEBUG] -- root@192.168.1.239 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:00.754] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.755] [DEBUG] -- root@192.168.1.239 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:00.847] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.848] [DEBUG] -- root@192.168.1.239 execute: ls /root/observer/store/sstable/block_file [2023-05-03 16:13:00.934] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.937] [DEBUG] -- root@192.168.1.228 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2023-05-03 16:13:00.953] [DEBUG] -- exited code 0 [2023-05-03 16:13:00.955] [DEBUG] -- root@192.168.1.228 execute: ulimit -a [2023-05-03 16:13:01.010] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.013] [DEBUG] -- root@192.168.1.228 execute: cat /proc/meminfo [2023-05-03 16:13:01.064] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.065] [DEBUG] -- root@192.168.1.228 execute: df --block-size=1024 [2023-05-03 16:13:01.125] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.128] [DEBUG] -- get disk info for path /dev, total: 12728172544 avail: 12728172544 [2023-05-03 16:13:01.129] [DEBUG] -- get disk info for path /dev/shm, total: 12744282112 avail: 12744282112 [2023-05-03 16:13:01.129] [DEBUG] -- get disk info for path /run, total: 12744282112 avail: 12731199488 [2023-05-03 16:13:01.130] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 12744282112 avail: 12744282112 [2023-05-03 16:13:01.130] [DEBUG] -- get disk info for path /, total: 77089300480 avail: 15604674560 [2023-05-03 16:13:01.130] [DEBUG] -- get disk info for path /boot, total: 311070720 avail: 140349440 [2023-05-03 16:13:01.130] [DEBUG] -- get disk info for path /run/user/42, total: 2548858880 avail: 2548846592 [2023-05-03 16:13:01.131] [DEBUG] -- get disk info for path /run/user/0, total: 2548858880 avail: 2548858880 [2023-05-03 16:13:01.131] [DEBUG] -- disk: {'/dev': {'total': 12728172544, 'avail': 12728172544, 'need': 0}, '/dev/shm': {'total': 12744282112, 'avail': 12744282112, 'need': 0}, '/run': {'total': 12744282112, 'avail': 12731199488, 'need': 0}, '/sys/fs/cgroup': {'total': 12744282112, 'avail': 12744282112, 'need': 0}, '/': {'total': 77089300480, 'avail': 15604674560, 'need': 0}, '/boot': {'total': 311070720, 'avail': 140349440, 'need': 0}, '/run/user/42': {'total': 2548858880, 'avail': 2548846592, 'need': 0}, '/run/user/0': {'total': 2548858880, 'avail': 2548858880, 'need': 0}} [2023-05-03 16:13:01.131] [DEBUG] -- root@192.168.1.235 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2023-05-03 16:13:01.180] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.181] [DEBUG] -- root@192.168.1.235 execute: ulimit -a [2023-05-03 16:13:01.241] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.242] [DEBUG] -- root@192.168.1.235 execute: cat /proc/meminfo [2023-05-03 16:13:01.322] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.323] [DEBUG] -- root@192.168.1.235 execute: df --block-size=1024 [2023-05-03 16:13:01.398] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.400] [DEBUG] -- get disk info for path /dev, total: 12728156160 avail: 12728156160 [2023-05-03 16:13:01.400] [DEBUG] -- get disk info for path /dev/shm, total: 12744282112 avail: 12744282112 [2023-05-03 16:13:01.401] [DEBUG] -- get disk info for path /run, total: 12744282112 avail: 12706131968 [2023-05-03 16:13:01.402] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 12744282112 avail: 12744282112 [2023-05-03 16:13:01.402] [DEBUG] -- get disk info for path /, total: 77089300480 avail: 14396502016 [2023-05-03 16:13:01.402] [DEBUG] -- get disk info for path /boot, total: 311070720 avail: 140279808 [2023-05-03 16:13:01.403] [DEBUG] -- get disk info for path /run/user/42, total: 2548858880 avail: 2548846592 [2023-05-03 16:13:01.403] [DEBUG] -- get disk info for path /run/user/0, total: 2548858880 avail: 2548858880 [2023-05-03 16:13:01.403] [DEBUG] -- disk: {'/dev': {'total': 12728156160, 'avail': 12728156160, 'need': 0}, '/dev/shm': {'total': 12744282112, 'avail': 12744282112, 'need': 0}, '/run': {'total': 12744282112, 'avail': 12706131968, 'need': 0}, '/sys/fs/cgroup': {'total': 12744282112, 'avail': 12744282112, 'need': 0}, '/': {'total': 77089300480, 'avail': 14396502016, 'need': 0}, '/boot': {'total': 311070720, 'avail': 140279808, 'need': 0}, '/run/user/42': {'total': 2548858880, 'avail': 2548846592, 'need': 0}, '/run/user/0': {'total': 2548858880, 'avail': 2548858880, 'need': 0}} [2023-05-03 16:13:01.404] [DEBUG] -- root@192.168.1.239 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2023-05-03 16:13:01.442] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.444] [DEBUG] -- root@192.168.1.239 execute: ulimit -a [2023-05-03 16:13:01.526] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.528] [DEBUG] -- root@192.168.1.239 execute: cat /proc/meminfo [2023-05-03 16:13:01.616] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.617] [DEBUG] -- root@192.168.1.239 execute: df --block-size=1024 [2023-05-03 16:13:01.704] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.705] [DEBUG] -- get disk info for path /dev, total: 12728156160 avail: 12728156160 [2023-05-03 16:13:01.705] [DEBUG] -- get disk info for path /dev/shm, total: 12744282112 avail: 12744282112 [2023-05-03 16:13:01.705] [DEBUG] -- get disk info for path /run, total: 12744282112 avail: 12722851840 [2023-05-03 16:13:01.705] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 12744282112 avail: 12744282112 [2023-05-03 16:13:01.705] [DEBUG] -- get disk info for path /, total: 77089300480 avail: 14861488128 [2023-05-03 16:13:01.705] [DEBUG] -- get disk info for path /boot, total: 311070720 avail: 140275712 [2023-05-03 16:13:01.705] [DEBUG] -- get disk info for path /run/user/42, total: 2548858880 avail: 2548846592 [2023-05-03 16:13:01.705] [DEBUG] -- get disk info for path /run/user/0, total: 2548858880 avail: 2548858880 [2023-05-03 16:13:01.705] [DEBUG] -- disk: {'/dev': {'total': 12728156160, 'avail': 12728156160, 'need': 0}, '/dev/shm': {'total': 12744282112, 'avail': 12744282112, 'need': 0}, '/run': {'total': 12744282112, 'avail': 12722851840, 'need': 0}, '/sys/fs/cgroup': {'total': 12744282112, 'avail': 12744282112, 'need': 0}, '/': {'total': 77089300480, 'avail': 14861488128, 'need': 0}, '/boot': {'total': 311070720, 'avail': 140275712, 'need': 0}, '/run/user/42': {'total': 2548858880, 'avail': 2548846592, 'need': 0}, '/run/user/0': {'total': 2548858880, 'avail': 2548858880, 'need': 0}} [2023-05-03 16:13:01.705] [DEBUG] -- root@192.168.1.228 execute: date +%s%N [2023-05-03 16:13:01.717] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.718] [DEBUG] -- 192.168.1.228 time delta 3.988037109375 [2023-05-03 16:13:01.718] [DEBUG] -- root@192.168.1.235 execute: date +%s%N [2023-05-03 16:13:01.758] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.759] [DEBUG] -- 192.168.1.235 time delta -5.39892578125 [2023-05-03 16:13:01.759] [DEBUG] -- root@192.168.1.239 execute: date +%s%N [2023-05-03 16:13:01.801] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.802] [DEBUG] -- 192.168.1.239 time delta -4.58251953125 [2023-05-03 16:13:01.815] [DEBUG] - sub start_check ref count to 0 [2023-05-03 16:13:01.815] [DEBUG] - export start_check [2023-05-03 16:13:01.816] [DEBUG] - Call obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe [2023-05-03 16:13:01.816] [DEBUG] - import start_check [2023-05-03 16:13:01.818] [DEBUG] - add start_check ref count to 1 [2023-05-03 16:13:01.819] [INFO] Check before start obproxy [2023-05-03 16:13:01.822] [DEBUG] -- root@192.168.1.231 execute: cat /root/obproxy/run/obproxy-192.168.1.231-2883.pid [2023-05-03 16:13:01.865] [DEBUG] -- exited code 1, error output: [2023-05-03 16:13:01.866] [DEBUG] cat: /root/obproxy/run/obproxy-192.168.1.231-2883.pid: 没有那个文件或目录 [2023-05-03 16:13:01.866] [DEBUG] [2023-05-03 16:13:01.866] [DEBUG] -- 192.168.1.231 port check [2023-05-03 16:13:01.866] [DEBUG] -- root@192.168.1.231 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:01.963] [DEBUG] -- exited code 0 [2023-05-03 16:13:01.965] [DEBUG] -- root@192.168.1.231 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:02.061] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.087] [DEBUG] - sub start_check ref count to 0 [2023-05-03 16:13:02.088] [DEBUG] - export start_check [2023-05-03 16:13:02.088] [DEBUG] - Call obagent-py_script_start_check-1.3.0 for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac [2023-05-03 16:13:02.088] [DEBUG] - import start_check [2023-05-03 16:13:02.096] [DEBUG] - add start_check ref count to 1 [2023-05-03 16:13:02.097] [INFO] Check before start obagent [2023-05-03 16:13:02.113] [DEBUG] -- root@192.168.1.228 execute: cat /root/obagent/run/ob_agentd.pid [2023-05-03 16:13:02.127] [DEBUG] -- exited code 1, error output: [2023-05-03 16:13:02.127] [DEBUG] cat: /root/obagent/run/ob_agentd.pid: No such file or directory [2023-05-03 16:13:02.127] [DEBUG] [2023-05-03 16:13:02.128] [DEBUG] -- server1(192.168.1.228) port check [2023-05-03 16:13:02.128] [DEBUG] -- root@192.168.1.228 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:1F99' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:02.180] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.181] [DEBUG] -- root@192.168.1.228 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:1F98' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:02.233] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.234] [DEBUG] -- root@192.168.1.235 execute: cat /root/obagent/run/ob_agentd.pid [2023-05-03 16:13:02.272] [DEBUG] -- exited code 1, error output: [2023-05-03 16:13:02.273] [DEBUG] cat: /root/obagent/run/ob_agentd.pid: No such file or directory [2023-05-03 16:13:02.273] [DEBUG] [2023-05-03 16:13:02.273] [DEBUG] -- server2(192.168.1.235) port check [2023-05-03 16:13:02.273] [DEBUG] -- root@192.168.1.235 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:1F99' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:02.358] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.358] [DEBUG] -- root@192.168.1.235 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:1F98' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:02.445] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.445] [DEBUG] -- root@192.168.1.239 execute: cat /root/obagent/run/ob_agentd.pid [2023-05-03 16:13:02.477] [DEBUG] -- exited code 1, error output: [2023-05-03 16:13:02.477] [DEBUG] cat: /root/obagent/run/ob_agentd.pid: No such file or directory [2023-05-03 16:13:02.477] [DEBUG] [2023-05-03 16:13:02.477] [DEBUG] -- server3(192.168.1.239) port check [2023-05-03 16:13:02.477] [DEBUG] -- root@192.168.1.239 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:1F99' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:02.566] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.566] [DEBUG] -- root@192.168.1.239 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:1F98' | awk -F' ' '{print $2}' | uniq [2023-05-03 16:13:02.655] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.766] [DEBUG] - sub start_check ref count to 0 [2023-05-03 16:13:02.766] [DEBUG] - export start_check [2023-05-03 16:13:02.767] [DEBUG] - Call oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:13:02.767] [DEBUG] - import start [2023-05-03 16:13:02.768] [DEBUG] - add start ref count to 1 [2023-05-03 16:13:02.768] [INFO] Start observer [2023-05-03 16:13:02.769] [DEBUG] -- root@192.168.1.228 execute: ls /root/observer/store/clog/tenant_1/ [2023-05-03 16:13:02.779] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.780] [DEBUG] -- root@192.168.1.228 execute: cat /root/observer/run/observer.pid [2023-05-03 16:13:02.830] [DEBUG] -- exited code 1, error output: [2023-05-03 16:13:02.830] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory [2023-05-03 16:13:02.830] [DEBUG] [2023-05-03 16:13:02.831] [DEBUG] -- server1(192.168.1.228) start command construction [2023-05-03 16:13:02.831] [DEBUG] -- root@192.168.1.235 execute: ls /root/observer/store/clog/tenant_1/ [2023-05-03 16:13:02.865] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.865] [DEBUG] -- root@192.168.1.235 execute: cat /root/observer/run/observer.pid [2023-05-03 16:13:02.939] [DEBUG] -- exited code 1, error output: [2023-05-03 16:13:02.939] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory [2023-05-03 16:13:02.939] [DEBUG] [2023-05-03 16:13:02.939] [DEBUG] -- server2(192.168.1.235) start command construction [2023-05-03 16:13:02.939] [DEBUG] -- root@192.168.1.239 execute: ls /root/observer/store/clog/tenant_1/ [2023-05-03 16:13:02.976] [DEBUG] -- exited code 0 [2023-05-03 16:13:02.976] [DEBUG] -- root@192.168.1.239 execute: cat /root/observer/run/observer.pid [2023-05-03 16:13:03.050] [DEBUG] -- exited code 1, error output: [2023-05-03 16:13:03.050] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory [2023-05-03 16:13:03.050] [DEBUG] [2023-05-03 16:13:03.050] [DEBUG] -- server3(192.168.1.239) start command construction [2023-05-03 16:13:03.050] [DEBUG] -- starting server1(192.168.1.228) observer [2023-05-03 16:13:03.051] [DEBUG] -- root@192.168.1.228 set env LD_LIBRARY_PATH to '/root/observer/lib:' [2023-05-03 16:13:03.051] [DEBUG] -- root@192.168.1.228 execute: cd /root/observer; /root/observer/bin/observer -r '192.168.1.228:2882:2881;192.168.1.235:2882:2881;192.168.1.239:2882:2881' -p 2881 -P 2882 -z 'zone1' -c 1 -d '/root/observer/store' -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,system_memory='2G',cpu_count=16,memory_limit='8G',datafile_size='27G',log_disk_size='24G' [2023-05-03 16:13:03.083] [DEBUG] -- exited code 0 [2023-05-03 16:13:03.083] [DEBUG] -- root@192.168.1.228 delete env LD_LIBRARY_PATH [2023-05-03 16:13:03.083] [DEBUG] -- starting server2(192.168.1.235) observer [2023-05-03 16:13:03.084] [DEBUG] -- root@192.168.1.235 set env LD_LIBRARY_PATH to '/root/observer/lib:' [2023-05-03 16:13:03.084] [DEBUG] -- root@192.168.1.235 execute: cd /root/observer; /root/observer/bin/observer -r '192.168.1.228:2882:2881;192.168.1.235:2882:2881;192.168.1.239:2882:2881' -p 2881 -P 2882 -z 'zone2' -c 1 -d '/root/observer/store' -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,system_memory='2G',cpu_count=16,memory_limit='7G',datafile_size='24G',log_disk_size='21G' [2023-05-03 16:13:03.184] [DEBUG] -- exited code 0 [2023-05-03 16:13:03.184] [DEBUG] -- root@192.168.1.235 delete env LD_LIBRARY_PATH [2023-05-03 16:13:03.184] [DEBUG] -- starting server3(192.168.1.239) observer [2023-05-03 16:13:03.184] [DEBUG] -- root@192.168.1.239 set env LD_LIBRARY_PATH to '/root/observer/lib:' [2023-05-03 16:13:03.184] [DEBUG] -- root@192.168.1.239 execute: cd /root/observer; /root/observer/bin/observer -r '192.168.1.228:2882:2881;192.168.1.235:2882:2881;192.168.1.239:2882:2881' -p 2881 -P 2882 -z 'zone3' -c 1 -d '/root/observer/store' -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,system_memory='2G',cpu_count=16,datafile_size='28G',memory_limit='8G',log_disk_size='24G' [2023-05-03 16:13:03.251] [DEBUG] -- exited code 0 [2023-05-03 16:13:03.251] [DEBUG] -- root@192.168.1.239 delete env LD_LIBRARY_PATH [2023-05-03 16:13:03.296] [INFO] observer program health check [2023-05-03 16:13:06.301] [DEBUG] -- server1(192.168.1.228) program health check [2023-05-03 16:13:06.302] [DEBUG] -- root@192.168.1.228 execute: cat /root/observer/run/observer.pid [2023-05-03 16:13:06.317] [DEBUG] -- exited code 0 [2023-05-03 16:13:06.318] [DEBUG] -- root@192.168.1.228 execute: ls /proc/4599 [2023-05-03 16:13:06.376] [DEBUG] -- exited code 0 [2023-05-03 16:13:06.377] [DEBUG] -- server1(192.168.1.228) observer[pid: 4599] started [2023-05-03 16:13:06.378] [DEBUG] -- server2(192.168.1.235) program health check [2023-05-03 16:13:06.378] [DEBUG] -- root@192.168.1.235 execute: cat /root/observer/run/observer.pid [2023-05-03 16:13:06.420] [DEBUG] -- exited code 0 [2023-05-03 16:13:06.421] [DEBUG] -- root@192.168.1.235 execute: ls /proc/71677 [2023-05-03 16:13:06.516] [DEBUG] -- exited code 0 [2023-05-03 16:13:06.516] [DEBUG] -- server2(192.168.1.235) observer[pid: 71677] started [2023-05-03 16:13:06.517] [DEBUG] -- server3(192.168.1.239) program health check [2023-05-03 16:13:06.517] [DEBUG] -- root@192.168.1.239 execute: cat /root/observer/run/observer.pid [2023-05-03 16:13:06.550] [DEBUG] -- exited code 0 [2023-05-03 16:13:06.551] [DEBUG] -- root@192.168.1.239 execute: ls /proc/100910 [2023-05-03 16:13:06.628] [DEBUG] -- exited code 0 [2023-05-03 16:13:06.631] [DEBUG] -- server3(192.168.1.239) observer[pid: 100910] started [2023-05-03 16:13:06.735] [DEBUG] - sub start ref count to 0 [2023-05-03 16:13:06.735] [DEBUG] - export start [2023-05-03 16:13:06.736] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:13:06.736] [DEBUG] - import connect [2023-05-03 16:13:06.855] [DEBUG] - add connect ref count to 1 [2023-05-03 16:13:06.857] [INFO] Connect to observer [2023-05-03 16:13:06.859] [DEBUG] -- connect 192.168.1.228 -P2881 -uroot -p [2023-05-03 16:13:06.863] [DEBUG] -- connect 192.168.1.235 -P2881 -uroot -p [2023-05-03 16:13:07.256] [DEBUG] - sub connect ref count to 0 [2023-05-03 16:13:07.257] [DEBUG] - export connect [2023-05-03 16:13:07.257] [INFO] Initialize oceanbase-ce [2023-05-03 16:13:07.260] [DEBUG] - Call oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63 [2023-05-03 16:13:07.261] [DEBUG] - import bootstrap [2023-05-03 16:13:07.266] [DEBUG] - add bootstrap ref count to 1 [2023-05-03 16:13:07.267] [DEBUG] -- execute sql: set session ob_query_timeout=1000000000 [2023-05-03 16:13:07.267] [DEBUG] -- execute sql: set session ob_query_timeout=1000000000. args: None [2023-05-03 16:13:07.276] [DEBUG] -- execute sql: alter system bootstrap REGION "sys_region" ZONE "zone1" SERVER "192.168.1.228:2882",REGION "sys_region" ZONE "zone2" SERVER "192.168.1.235:2882",REGION "sys_region" ZONE "zone3" SERVER "192.168.1.239:2882". args: None [2023-05-03 16:29:47.377] [DEBUG] -- OBD-5000: alter system bootstrap REGION "sys_region" ZONE "zone1" SERVER "192.168.1.228:2882",REGION "sys_region" ZONE "zone2" SERVER "192.168.1.235:2882",REGION "sys_region" ZONE "zone3" SERVER "192.168.1.239:2882" execute failed [2023-05-03 16:29:47.381] [ERROR] Traceback (most recent call last): [2023-05-03 16:29:47.381] [ERROR] File "core.py", line 1564, in start_cluster [2023-05-03 16:29:47.381] [ERROR] File "core.py", line 1666, in _start_cluster [2023-05-03 16:29:47.382] [ERROR] File "core.py", line 182, in call_plugin [2023-05-03 16:29:47.382] [ERROR] File "_plugin.py", line 314, in __call__ [2023-05-03 16:29:47.382] [ERROR] File "_plugin.py", line 281, in _new_func [2023-05-03 16:29:47.382] [ERROR] File "/root/.obd/plugins/oceanbase-ce/4.0.0.0/bootstrap.py", line 78, in bootstrap [2023-05-03 16:29:47.382] [ERROR] raise_cursor.execute(sql, exc_level='verbose') [2023-05-03 16:29:47.382] [ERROR] File "_stdio.py", line 864, in func_wrapper [2023-05-03 16:29:47.382] [ERROR] File "/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 91, in execute [2023-05-03 16:29:47.382] [ERROR] self.cursor.execute(sql, args) [2023-05-03 16:29:47.382] [ERROR] File "/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/cursors.py", line 148, in execute [2023-05-03 16:29:47.383] [ERROR] result = self._query(query) [2023-05-03 16:29:47.383] [ERROR] File "/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/cursors.py", line 310, in _query [2023-05-03 16:29:47.383] [ERROR] conn.query(q) [2023-05-03 16:29:47.383] [ERROR] File "/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py", line 548, in query [2023-05-03 16:29:47.383] [ERROR] self._affected_rows = self._read_query_result(unbuffered=unbuffered) [2023-05-03 16:29:47.383] [ERROR] File "/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py", line 775, in _read_query_result [2023-05-03 16:29:47.383] [ERROR] result.read() [2023-05-03 16:29:47.383] [ERROR] File "/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py", line 1156, in read [2023-05-03 16:29:47.384] [ERROR] first_packet = self.connection._read_packet() [2023-05-03 16:29:47.384] [ERROR] File "/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py", line 725, in _read_packet [2023-05-03 16:29:47.384] [ERROR] packet.raise_for_error() [2023-05-03 16:29:47.384] [ERROR] File "/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/protocol.py", line 221, in raise_for_error [2023-05-03 16:29:47.384] [ERROR] err.raise_mysql_exception(self._data) [2023-05-03 16:29:47.384] [ERROR] File "/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/err.py", line 143, in raise_mysql_exception [2023-05-03 16:29:47.385] [ERROR] raise errorclass(errno, errval) [2023-05-03 16:29:47.385] [ERROR] pymysql.err.OperationalError: (4012, 'Timeout') [2023-05-03 16:29:47.385] [ERROR] [2023-05-03 16:29:47.385] [DEBUG] -- execute sql: select column_value from oceanbase.__all_core_table where table_name = '__all_global_stat' and column_name = 'baseline_schema_version'. args: None [2023-05-03 16:46:27.397] [DEBUG] -- OBD-5000: select column_value from oceanbase.__all_core_table where table_name = '__all_global_stat' and column_name = 'baseline_schema_version' execute failed [2023-05-03 16:46:27.397] [DEBUG] - sub bootstrap ref count to 0 [2023-05-03 16:46:27.398] [DEBUG] - export bootstrap [2023-05-03 16:46:27.533] [ERROR] Cluster init failed [2023-05-03 16:46:27.533] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2023-05-03 16:46:27.534] [INFO] Trace ID: 505466d2-e98a-11ed-83bb-000c29bb24bc [2023-05-03 16:46:27.534] [INFO] If you want to view detailed obd logs, please run: obd display-trace 505466d2-e98a-11ed-83bb-000c29bb24bc [2023-05-03 16:46:27.535] [DEBUG] - unlock /root/.obd/lock/global [2023-05-03 16:46:27.535] [DEBUG] - unlock /root/.obd/lock/deploy_myoceanbase [2023-05-03 16:46:27.536] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo