OceanBase 重启失败 Connect to observer x pymysql.err.OperationalError: (5150, 'Tenant not in this server')

赧然一笑 · 2023 年9 月 8 日 11:12

【使用环境】测试环境
【 OB or 其他组件】 OB
【使用版本】V4.2.0_CE_BETA
【问题描述】
1、使用 obd cluster stop obhwxc 停止运行中的集群
2、使用 obd cluster start obhwxc 启动集群失败

Connect to observer x
[ERROR] OBD-1006: Failed to connect to oceanbase-ce
See https://www.oceanbase.com/product/ob-deployer/error-codes .

日志报错
pymysql.err.OperationalError: (5150, ‘Tenant not in this server’)

【复现路径】问题出现前后相关操作
# 查看 OceanBase 状态

[root@08b49246b7a9 /] # obd cluster display obhwxc

Get local repositories and plugins ok

Open ssh connection ok

Cluster status check ok

Connect to observer ok

Wait for observer init ok

+---------------------------------------------+

| observer |

+-----------+---------+------+-------+--------+

| ip | version | port | zone | status |

+-----------+---------+------+-------+--------+

| 127.0.0.1 | 4.2.0.0 | 2881 | zone1 | ACTIVE |

+-----------+---------+------+-------+--------+

obclient -h127.0.0.1 -P2881 -uroot -p 'wDuFDJI92aIGzSnpOvhl' -Doceanbase -A

Trace ID: 3690efaa-4d2a-11ee-b825-0242ac1100c6

If you want to view detailed obd logs, please run: obd display-trace 3690efaa-4d2a-11ee-b825-0242ac1100c6

# 停止运行中的集群
obd cluster stop obhwxc
# 启动 OceanBase 数据库
[root@08b49246b7a9 /]# obd cluster start obhwxc
Get local repositories ok
Search plugins ok
Open ssh connection ok
Load cluster param plugin ok
Check before start observer ok
[WARN] OBD-1011: (127.0.0.1) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)

Start observer ok
observer program health check ok
Connect to observer x
[ERROR] OBD-1006: Failed to connect to oceanbase-ce
See https://www.oceanbase.com/product/ob-deployer/error-codes .
Trace ID: bcde959e-4d2a-11ee-ad88-0242ac110037
If you want to view detailed obd logs, please run: obd display-trace bcde959e-4d2a-11ee-ad88-0242ac110037

【问题现象及影响】
日志报错：
[root@08b49246b7a9 /]# obd display-trace bcde959e-4d2a-11ee-ad88-0242ac110037
[2023-09-07 11:00:44.773] [DEBUG] - cmd: [‘obhwxc’]
[2023-09-07 11:00:44.773] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2023-09-07 11:00:44.774] [DEBUG] - mkdir /root/.obd/lock/
[2023-09-07 11:00:44.774] [DEBUG] - unknown lock mode
[2023-09-07 11:00:44.774] [DEBUG] - try to get share lock /root/.obd/lock/global
[2023-09-07 11:00:44.794] [DEBUG] - share lock /root/.obd/lock/global, count 1
[2023-09-07 11:00:44.794] [DEBUG] - Get Deploy by name
[2023-09-07 11:00:44.795] [DEBUG] - mkdir /root/.obd/cluster/
[2023-09-07 11:00:44.795] [DEBUG] - mkdir /root/.obd/config_parser/
[2023-09-07 11:00:44.795] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_obhwxc
[2023-09-07 11:00:44.795] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obhwxc, count 1
[2023-09-07 11:00:44.801] [DEBUG] - Deploy status judge
[2023-09-07 11:00:44.801] [INFO] Get local repositories
[2023-09-07 11:00:44.802] [DEBUG] - mkdir /root/.obd/repository
[2023-09-07 11:00:44.802] [DEBUG] - Get local repository oceanbase-ce-4.2.0.0-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:44.802] [DEBUG] - Search repository oceanbase-ce version: 4.2.0.0, tag: 763eab84be6e4062201e04d56dde8323c4b059b5, release: None, package_hash: None
[2023-09-07 11:00:44.803] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2023-09-07 11:00:44.803] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 1
[2023-09-07 11:00:44.803] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce
[2023-09-07 11:00:44.805] [DEBUG] - Found repository oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:44.933] [DEBUG] - Get deploy config
[2023-09-07 11:00:44.949] [INFO] Search plugins
[2023-09-07 11:00:44.949] [DEBUG] - Searching start_check plugin for components …
[2023-09-07 11:00:44.950] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:44.950] [DEBUG] - mkdir /root/.obd/plugins
[2023-09-07 11:00:44.951] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.0.0
[2023-09-07 11:00:44.951] [DEBUG] - Searching create_tenant plugin for components …
[2023-09-07 11:00:44.951] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:44.952] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.0.0.0 for oceanbase-ce-4.2.0.0
[2023-09-07 11:00:44.952] [DEBUG] - Searching start plugin for components …
[2023-09-07 11:00:44.952] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:44.952] [DEBUG] - Found for oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.2.0.0
[2023-09-07 11:00:44.953] [DEBUG] - Searching connect plugin for components …
[2023-09-07 11:00:44.953] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:44.953] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.0.0
[2023-09-07 11:00:44.953] [DEBUG] - Searching bootstrap plugin for components …
[2023-09-07 11:00:44.953] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:44.954] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.2.0.0
[2023-09-07 11:00:44.954] [DEBUG] - Searching display plugin for components …
[2023-09-07 11:00:44.954] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:44.955] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.0.0
[2023-09-07 11:00:45.081] [INFO] Open ssh connection
[2023-09-07 11:00:45.212] [INFO] Load cluster param plugin
[2023-09-07 11:00:45.213] [DEBUG] - Searching param plugin for components …
[2023-09-07 11:00:45.213] [DEBUG] - Search param plugin for oceanbase-ce
[2023-09-07 11:00:45.213] [DEBUG] - Found for oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.0.0
[2023-09-07 11:00:45.213] [DEBUG] - Applying oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:45.964] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:45.964] [DEBUG] - import start_check
[2023-09-07 11:00:46.057] [DEBUG] - add start_check ref count to 1
[2023-09-07 11:00:46.057] [INFO] Check before start observer
[2023-09-07 11:00:46.059] [DEBUG] – local execute: ls /root/observer/store/clog/tenant_1/
[2023-09-07 11:00:46.062] [DEBUG] – exited code 0
[2023-09-07 11:00:46.062] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2023-09-07 11:00:46.064] [DEBUG] – exited code 1, error output:
[2023-09-07 11:00:46.065] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2023-09-07 11:00:46.065] [DEBUG]
[2023-09-07 11:00:46.065] [DEBUG] – 127.0.0.1 port check
[2023-09-07 11:00:46.065] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-09-07 11:00:46.082] [DEBUG] – exited code 0
[2023-09-07 11:00:46.083] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-09-07 11:00:46.099] [DEBUG] – exited code 0
[2023-09-07 11:00:46.113] [DEBUG] – local execute: ls /root/observer/store/sstable/block_file
[2023-09-07 11:00:46.116] [DEBUG] – exited code 0
[2023-09-07 11:00:46.116] [DEBUG] – local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2023-09-07 11:00:46.118] [DEBUG] – exited code 0
[2023-09-07 11:00:46.118] [WARNING] OBD-1011: (127.0.0.1) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[2023-09-07 11:00:46.118] [DEBUG] – local execute: ulimit -a
[2023-09-07 11:00:46.120] [DEBUG] – exited code 0
[2023-09-07 11:00:46.121] [DEBUG] – local execute: cat /proc/meminfo
[2023-09-07 11:00:46.123] [DEBUG] – exited code 0
[2023-09-07 11:00:46.124] [DEBUG] – local execute: df --block-size=1024
[2023-09-07 11:00:46.126] [DEBUG] – exited code 0
[2023-09-07 11:00:46.127] [DEBUG] – get disk info for path /, total: 938244935680 avail: 362505523200
[2023-09-07 11:00:46.127] [DEBUG] – get disk info for path /dev, total: 67108864 avail: 67108864
[2023-09-07 11:00:46.127] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 136441954304 avail: 136441954304
[2023-09-07 11:00:46.127] [DEBUG] – get disk info for path /etc/hosts, total: 938244935680 avail: 362505523200
[2023-09-07 11:00:46.127] [DEBUG] – get disk info for path /dev/shm, total: 67108864 avail: 67108864
[2023-09-07 11:00:46.127] [DEBUG] – get disk info for path /proc/acpi, total: 136441954304 avail: 136441954304
[2023-09-07 11:00:46.127] [DEBUG] – get disk info for path /proc/livepatch, total: 136441954304 avail: 136441954304
[2023-09-07 11:00:46.127] [DEBUG] – get disk info for path /proc/scsi, total: 136441954304 avail: 136441954304
[2023-09-07 11:00:46.128] [DEBUG] – get disk info for path /sys/firmware, total: 136441954304 avail: 136441954304
[2023-09-07 11:00:46.128] [DEBUG] – disk: {’/’: {‘total’: 938244935680, ‘avail’: 362505523200, ‘need’: 0}, ‘/dev’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}, ‘/etc/hosts’: {‘total’: 938244935680, ‘avail’: 362505523200, ‘need’: 0}, ‘/dev/shm’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/proc/acpi’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}, ‘/proc/livepatch’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}, ‘/proc/scsi’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}, ‘/sys/firmware’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}}
[2023-09-07 11:00:46.128] [DEBUG] – local execute: date +%s%N
[2023-09-07 11:00:46.130] [DEBUG] – exited code 0
[2023-09-07 11:00:46.130] [DEBUG] – 127.0.0.1 time delta -0.377685546875
[2023-09-07 11:00:46.189] [INFO] [WARN] OBD-1011: (127.0.0.1) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[2023-09-07 11:00:46.189] [INFO]
[2023-09-07 11:00:46.190] [DEBUG] - sub start_check ref count to 0
[2023-09-07 11:00:46.190] [DEBUG] - export start_check
[2023-09-07 11:00:46.190] [DEBUG] - Call oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:46.190] [DEBUG] - import start
[2023-09-07 11:00:49.870] [DEBUG] - add start ref count to 1
[2023-09-07 11:00:49.870] [INFO] Start observer
[2023-09-07 11:00:49.871] [DEBUG] – local execute: ls /root/observer/store/clog/tenant_1/
[2023-09-07 11:00:49.874] [DEBUG] – exited code 0
[2023-09-07 11:00:49.874] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2023-09-07 11:00:49.876] [DEBUG] – exited code 1, error output:
[2023-09-07 11:00:49.877] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2023-09-07 11:00:49.877] [DEBUG]
[2023-09-07 11:00:49.877] [DEBUG] – 127.0.0.1 start command construction
[2023-09-07 11:00:49.877] [DEBUG] – starting 127.0.0.1 observer
[2023-09-07 11:00:49.877] [DEBUG] – root@127.0.0.1 set env LD_LIBRARY_PATH to ‘/root/observer/lib:’
[2023-09-07 11:00:49.878] [DEBUG] – local execute: cd /root/observer; /root/observer/bin/observer -r ‘127.0.0.1:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -c 1 -d ‘/root/observer/store’ -i ‘lo’ -o __min_full_resource_pool_memory=2147483648,memory_limit=‘16G’,system_memory=‘1G’,datafile_size=‘2G’,datafile_next=‘2G’,datafile_maxsize=‘20G’,log_disk_size=‘15G’,cpu_count=16,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4
[2023-09-07 11:00:51.535] [DEBUG] – exited code 0
[2023-09-07 11:00:51.542] [DEBUG] – root@127.0.0.1 delete env LD_LIBRARY_PATH
[2023-09-07 11:00:51.565] [INFO] observer program health check
[2023-09-07 11:00:54.574] [DEBUG] – 127.0.0.1 program health check
[2023-09-07 11:00:54.574] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2023-09-07 11:00:54.577] [DEBUG] – exited code 0
[2023-09-07 11:00:54.577] [DEBUG] – local execute: ls /proc/98
[2023-09-07 11:00:54.580] [DEBUG] – exited code 0
[2023-09-07 11:00:54.580] [DEBUG] – 127.0.0.1 observer[pid: 98] started
[2023-09-07 11:00:54.698] [DEBUG] - sub start ref count to 0
[2023-09-07 11:00:54.699] [DEBUG] - export start
[2023-09-07 11:00:54.699] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.0.0-100010022023081817.el7-763eab84be6e4062201e04d56dde8323c4b059b5
[2023-09-07 11:00:54.699] [DEBUG] - import connect
[2023-09-07 11:00:54.733] [DEBUG] - add connect ref count to 1
[2023-09-07 11:00:54.734] [INFO] Connect to observer
[2023-09-07 11:00:54.734] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:00:57.737] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:00.781] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:03.784] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:06.788] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:09.792] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:12.796] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:15.801] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:18.804] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:21.808] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:24.812] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:27.817] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:30.820] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:33.824] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:36.828] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:39.832] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:42.836] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:45.840] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:48.842] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:51.846] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:01:54.849] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:01:57.853] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:00.856] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:03.860] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:06.864] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:09.868] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:12.872] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:15.876] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:18.880] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:21.884] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:24.887] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:27.890] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:30.894] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:33.898] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:36.901] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:39.905] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:42.909] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:45.913] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:48.916] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:51.920] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:02:54.924] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:02:57.928] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:00.932] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:03.936] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:06.940] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:09.944] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:12.947] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:15.951] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:18.954] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:21.958] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:24.962] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:27.966] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:30.968] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:33.972] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:36.976] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:39.980] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:42.983] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:45.987] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:48.990] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:51.994] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:03:54.998] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:03:58.002] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:01.004] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:04.008] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:07.012] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:10.016] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:13.020] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:16.024] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:19.027] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:22.030] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:25.032] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:28.036] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:31.040] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:34.044] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:37.048] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:40.052] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:43.055] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:46.059] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:49.063] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:52.066] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:04:55.070] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:04:58.073] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:01.091] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:05:04.095] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:07.099] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:05:10.102] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:13.106] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:05:16.109] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:19.112] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:05:22.116] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:25.120] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:05:28.124] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:31.127] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:05:34.131] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:37.135] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:05:40.139] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:43.143] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:05:46.146] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:49.150] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pwDuFDJI92aIGzSnpOvhl
[2023-09-07 11:05:52.154] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-07 11:05:52.156] [ERROR] Traceback (most recent call last):
[2023-09-07 11:05:52.156] [ERROR] File “core.py”, line 1574, in start_cluster
[2023-09-07 11:05:52.156] [ERROR] File “core.py”, line 1666, in _start_cluster
[2023-09-07 11:05:52.156] [ERROR] File “core.py”, line 184, in call_plugin
[2023-09-07 11:05:52.156] [ERROR] File “_plugin.py”, line 324, in call
[2023-09-07 11:05:52.156] [ERROR] File “_plugin.py”, line 286, in _new_func
[2023-09-07 11:05:52.156] [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 136, in connect
[2023-09-07 11:05:52.156] [ERROR] cursor = Cursor(ip=server.ip, port=server_config[‘mysql_port’], tenant=’’, password=password if password is not None else ‘’, stdio=stdio)
[2023-09-07 11:05:52.156] [ERROR] File “_stdio.py”, line 901, in wrapper
[2023-09-07 11:05:52.156] [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 45, in init
[2023-09-07 11:05:52.156] [ERROR] self._connect()
[2023-09-07 11:05:52.156] [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 75, in _connect
[2023-09-07 11:05:52.156] [ERROR] self.db = mysql.connect(host=self.ip, user=self.user, port=int(self.port), password=str(self.password),
[2023-09-07 11:05:52.157] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 353, in init
[2023-09-07 11:05:52.157] [ERROR] self.connect()
[2023-09-07 11:05:52.157] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 633, in connect
[2023-09-07 11:05:52.157] [ERROR] self._request_authentication()
[2023-09-07 11:05:52.157] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 907, in _request_authentication
[2023-09-07 11:05:52.157] [ERROR] auth_packet = self._read_packet()
[2023-09-07 11:05:52.157] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 725, in _read_packet
[2023-09-07 11:05:52.157] [ERROR] packet.raise_for_error()
[2023-09-07 11:05:52.157] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/protocol.py”, line 221, in raise_for_error
[2023-09-07 11:05:52.157] [ERROR] err.raise_mysql_exception(self._data)
[2023-09-07 11:05:52.157] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/err.py”, line 143, in raise_mysql_exception
[2023-09-07 11:05:52.157] [ERROR] raise errorclass(errno, errval)
[2023-09-07 11:05:52.157] [ERROR] pymysql.err.OperationalError: (5150, ‘Tenant not in this server’)
[2023-09-07 11:05:52.157] [ERROR]
[2023-09-07 11:05:55.164] [ERROR] OBD-1006: Failed to connect to oceanbase-ce
[2023-09-07 11:05:55.164] [DEBUG] - sub connect ref count to 0
[2023-09-07 11:05:55.165] [DEBUG] - export connect
[2023-09-07 11:05:55.166] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2023-09-07 11:05:55.166] [INFO] Trace ID: bcde959e-4d2a-11ee-ad88-0242ac110037
[2023-09-07 11:05:55.166] [INFO] If you want to view detailed obd logs, please run: obd display-trace bcde959e-4d2a-11ee-ad88-0242ac110037
[2023-09-07 11:05:55.167] [DEBUG] - unlock /root/.obd/lock/global
[2023-09-07 11:05:55.167] [DEBUG] - unlock /root/.obd/lock/deploy_obhwxc
[2023-09-07 11:05:55.167] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo

【附件】
构建步骤与配置文件
wget https: //obbusiness-private.oss-cn-shanghai.aliyuncs.com/download-center/opensource/oceanbase-all-in-one/7/aarch64/oceanbase-all-in-one-4.2.0.0-100120230804133615.el7.aarch64.tar.gz

tar -xzf oceanbase-all-in-one-*.tar.gz

cd oceanbase-all-in-one/bin/

./install.sh

source ~/.oceanbase-all-in-one/bin/env.sh

步骤二：配置 OBD

cd rpms/

1 、禁用远程仓库

obd mirror disable remote

2 、将安装包添加至本地镜像

obd mirror clone *.rpm

3 、查看本地镜像中安装包列表

obd mirror list local

5 、修改配置文件

cd /root/.oceanbase-all-in-one/conf

vim mini-local-example.yaml

[root @aa897ad8c82b conf]# obd cluster deploy obhwxc -c mini-local-example.yaml

Package oceanbase-ce- 4.2 . 0.0 - 100010022023081817 .el7 is available.

install oceanbase-ce- 4.2 . 0.0 for local ok

+--------------------------------------------------------------------------------------------+

| Packages |

+--------------+---------+------------------------+------------------------------------------+

| Repository | Version | Release | Md5 |

+--------------+---------+------------------------+------------------------------------------+

| oceanbase-ce | 4.2 . 0.0 | 100010022023081817 .el7 | 763eab84be6e4062201e04d56dde8323c4b059b5 |

+--------------+---------+------------------------+------------------------------------------+

Repository integrity check ok

Parameter check ok

Cluster status check ok

Initializes observer work home ok

Remote oceanbase-ce- 4.2 . 0.0 - 100010022023081817 .el7-763eab84be6e4062201e04d56dde8323c4b059b5 repository install ok

Remote oceanbase-ce- 4.2 . 0.0 - 100010022023081817 .el7-763eab84be6e4062201e04d56dde8323c4b059b5 repository lib check !!

Try to get lib-repository

Package oceanbase-ce-libs- 4.2 . 0.0 - 100010022023081817 .el7 is available.

install oceanbase-ce-libs- 4.2 . 0.0 for local ok

Remote oceanbase-ce-libs- 4.2 . 0.0 - 100010022023081817 .el7-612fa788247038ec4b2bfc1928161bce083b1b53 repository install ok

Remote oceanbase-ce- 4.2 . 0.0 - 100010022023081817 .el7-763eab84be6e4062201e04d56dde8323c4b059b5 repository lib check ok

obhwxc deployed

Trace ID: 4be930b8-4ac5-11ee-9a52-0242ac1100c2

If you want to view detailed obd logs, please run: obd display-trace 4be930b8-4ac5-11ee-9a52-0242ac1100c2

# 启动 OceanBase 数据库

[root @aa897ad8c82b conf]# obd cluster start obhwxc

Get local repositories ok

Search plugins ok

Open ssh connection ok

Load cluster param plugin ok

Check before start observer ok

[WARN] OBD- 1011 : ( 127.0 . 0.1 ) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536 )

[WARN] OBD- 2000 : ( 127.0 . 0.1 ) not enough memory. (Free: 7 .5G, Need: 16 .0G)

[WARN] OBD- 1012 : ( 127.0 . 0.1 ) clog and data use the same disk (/)

Start observer ok

observer program health check ok

Connect to observer ok

Initialize oceanbase-ce ok

Wait for observer init ok

+---------------------------------------------+

| observer |

+-----------+---------+------+-------+--------+

| ip | version | port | zone | status |

+-----------+---------+------+-------+--------+

| 127.0 . 0.1 | 4.2 . 0.0 | 2881 | zone1 | ACTIVE |

+-----------+---------+------+-------+--------+

obclient -h127. 0.0 . 1 -P2881 -uroot -p 'tB9LiPxpx0OOwYN0rw9b' -Doceanbase -A

obhwxc running

Trace ID: 6d3bf192-4ac5-11ee-93bd-0242ac1100c2

If you want to view detailed obd logs, please run: obd display-trace 6d3bf192-4ac5-11ee-93bd-0242ac1100c2

# 查看 OceanBase 集群状态

[root @aa897ad8c82b conf]# obd cluster list

+------------------------------------------------------+

| Cluster List |

+--------+---------------------------+-----------------+

| Name | Configuration Path | Status (Cached) |

+--------+---------------------------+-----------------+

| obhwxc | /root/.obd/cluster/obhwxc | running |

+--------+---------------------------+-----------------+

Trace ID: f440c294-4ac5-11ee-be79-0242ac1100c2

If you want to view detailed obd logs, please run: obd display-trace f440c294-4ac5-11ee-be79-0242ac1100c2

[root @aa897ad8c82b conf]# obd cluster display obhwxc

Get local repositories and plugins ok

Open ssh connection ok

Cluster status check ok

Connect to observer ok

Wait for observer init ok

+---------------------------------------------+

| observer |

+-----------+---------+------+-------+--------+

| ip | version | port | zone | status |

+-----------+---------+------+-------+--------+

| 127.0 . 0.1 | 4.2 . 0.0 | 2881 | zone1 | ACTIVE |

+-----------+---------+------+-------+--------+

obclient -h127. 0.0 . 1 -P2881 -uroot -p 'tB9LiPxpx0OOwYN0rw9b' -Doceanbase -A

Trace ID: fb964aa0-4ac5-11ee-a992-0242ac1100c2

If you want to view detailed obd logs, please run: obd display-trace fb964aa0-4ac5-11ee-a992-0242ac1100c2

# 连接 OceanBase 集群

[root @aa897ad8c82b conf]# obclient -h172. 17.0 . 194 -uroot @sys -P2881 -p 'tB9LiPxpx0OOwYN0rw9b'

Welcome to the OceanBase. Commands end with ; or \g.

Your OceanBase connection id is 3221487620

Server version: OceanBase_CE 4.2 . 0.0 (r100010022023081817-0bdf1c0c5674e88c5ae9a8d0ae4f8077465d7fae) (Built Aug 18 2023 17 : 33 : 16 )

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

obclient [(none)]> show databases;

+--------------------+

| Database |

+--------------------+

| information_schema |

| LBACSYS |

| mysql |

| oceanbase |

| ORAAUDITOR |

| SYS |

| test |

+--------------------+

7 rows in set ( 0.004 sec)

赧然一笑 · 2023 年9 月 8 日 11:46

1、部署
[root@25c7ec1ad904 conf]# obd cluster deploy obhwxc -c mini-local-example.yaml
Package oceanbase-ce-4.2.0.0-100000152023080109.el7 is available.
install oceanbase-ce-4.2.0.0 for local ok
±-------------------------------------------------------------------------------------------+
| Packages |
±-------------±--------±-----------------------±-----------------------------------------+
| Repository | Version | Release | Md5 |
±-------------±--------±-----------------------±-----------------------------------------+
| oceanbase-ce | 4.2.0.0 | 100000152023080109.el7 | 3445a0b686b11e11b01ad42f0b17b423fcc88ab0 |
±-------------±--------±-----------------------±-----------------------------------------+
Repository integrity check ok
Parameter check ok
Cluster status check ok
Initializes observer work home ok
Remote oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0 repository install ok
Remote oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0 repository lib check !!
Try to get lib-repository
Package oceanbase-ce-libs-4.2.0.0-100000152023080109.el7 is available.
install oceanbase-ce-libs-4.2.0.0 for local ok
Remote oceanbase-ce-libs-4.2.0.0-100000152023080109.el7-85ca997a6dc2c760e4b6b47d367807d2f130ba93 repository install ok
Remote oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0 repository lib check ok
obhwxc deployed
Trace ID: 0e6a74e2-4df7-11ee-955d-0242ac110073
If you want to view detailed obd logs, please run: obd display-trace 0e6a74e2-4df7-11ee-955d-0242ac110073

2、启动
[root@25c7ec1ad904 conf]# obd cluster start obhwxc
Get local repositories ok
Search plugins ok
Open ssh connection ok
Load cluster param plugin ok
Check before start observer ok
[WARN] OBD-1012: (127.0.0.1) clog and data use the same disk (/)

obhwxc running
Trace ID: 1eeac858-4df7-11ee-b1aa-0242ac110073
If you want to view detailed obd logs, please run: obd display-trace 1eeac858-4df7-11ee-b1aa-0242ac110073

3、连接测试
[root@25c7ec1ad904 conf]# obclient -h127.0.0.1 -P2881 -uroot -p’Ewitkh1skvSIA2ETSH2G’ -Doceanbase -A
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 3221487617
Server version: OceanBase_CE 4.2.0.0 (r100000152023080109-8024d8ff45c45cf7c62a548752b985648a5795c3) (Built Aug 1 2023 09:54:54)

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

obclient [oceanbase]> exit;
Bye

Trace ID: 4518e11c-4df8-11ee-8cc5-0242ac110073
If you want to view detailed obd logs, please run: obd display-trace 4518e11c-4df8-11ee-8cc5-0242ac110073

5、关闭
[root@25c7ec1ad904 conf]# obd cluster stop obhwxc
Get local repositories ok
Search plugins ok
Open ssh connection ok
Stop observer ok
obhwxc stopped
Trace ID: 4a8f3074-4df8-11ee-8bd5-0242ac110073
If you want to view detailed obd logs, please run: obd display-trace 4a8f3074-4df8-11ee-8bd5-0242ac110073

6、启动
[root@25c7ec1ad904 conf]# obd cluster start obhwxc
Get local repositories ok
Search plugins ok
Open ssh connection ok
Load cluster param plugin ok
Check before start observer ok
Start observer ok
observer program health check ok
Connect to observer x
[ERROR] OBD-1006: Failed to connect to oceanbase-ce
See https://www.oceanbase.com/product/ob-deployer/error-codes .
Trace ID: 0ab26b5a-4df9-11ee-967b-0242ac110073
If you want to view detailed obd logs, please run: obd display-trace 0ab26b5a-4df9-11ee-967b-0242ac110073

赧然一笑 · 2023 年9 月 8 日 11:46

7、obd日志
[root@25c7ec1ad904 conf]# obd display-trace 0ab26b5a-4df9-11ee-967b-0242ac110073
[2023-09-08 11:37:31.672] [DEBUG] - cmd: [‘obhwxc’]
[2023-09-08 11:37:31.673] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2023-09-08 11:37:31.673] [DEBUG] - mkdir /root/.obd/lock/
[2023-09-08 11:37:31.673] [DEBUG] - unknown lock mode
[2023-09-08 11:37:31.673] [DEBUG] - try to get share lock /root/.obd/lock/global
[2023-09-08 11:37:31.673] [DEBUG] - share lock /root/.obd/lock/global, count 1
[2023-09-08 11:37:31.674] [DEBUG] - Get Deploy by name
[2023-09-08 11:37:31.674] [DEBUG] - mkdir /root/.obd/cluster/
[2023-09-08 11:37:31.674] [DEBUG] - mkdir /root/.obd/config_parser/
[2023-09-08 11:37:31.674] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_obhwxc
[2023-09-08 11:37:31.674] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obhwxc, count 1
[2023-09-08 11:37:31.680] [DEBUG] - Deploy status judge
[2023-09-08 11:37:31.680] [INFO] Get local repositories
[2023-09-08 11:37:31.681] [DEBUG] - mkdir /root/.obd/repository
[2023-09-08 11:37:31.681] [DEBUG] - Get local repository oceanbase-ce-4.2.0.0-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:31.681] [DEBUG] - Search repository oceanbase-ce version: 4.2.0.0, tag: 3445a0b686b11e11b01ad42f0b17b423fcc88ab0, release: None, package_hash: None
[2023-09-08 11:37:31.682] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2023-09-08 11:37:31.682] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 1
[2023-09-08 11:37:31.682] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce
[2023-09-08 11:37:31.684] [DEBUG] - Found repository oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:31.812] [DEBUG] - Get deploy config
[2023-09-08 11:37:31.827] [INFO] Search plugins
[2023-09-08 11:37:31.832] [DEBUG] - Searching start_check plugin for components …
[2023-09-08 11:37:31.833] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:31.833] [DEBUG] - mkdir /root/.obd/plugins
[2023-09-08 11:37:31.854] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.0.0
[2023-09-08 11:37:31.854] [DEBUG] - Searching create_tenant plugin for components …
[2023-09-08 11:37:31.854] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:31.855] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.0.0.0 for oceanbase-ce-4.2.0.0
[2023-09-08 11:37:31.855] [DEBUG] - Searching start plugin for components …
[2023-09-08 11:37:31.855] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:31.855] [DEBUG] - Found for oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.2.0.0
[2023-09-08 11:37:31.856] [DEBUG] - Searching connect plugin for components …
[2023-09-08 11:37:31.856] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:31.856] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.0.0
[2023-09-08 11:37:31.856] [DEBUG] - Searching bootstrap plugin for components …
[2023-09-08 11:37:31.856] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:31.857] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.2.0.0
[2023-09-08 11:37:31.857] [DEBUG] - Searching display plugin for components …
[2023-09-08 11:37:31.857] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:31.858] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.0.0
[2023-09-08 11:37:31.964] [INFO] Open ssh connection
[2023-09-08 11:37:32.095] [INFO] Load cluster param plugin
[2023-09-08 11:37:32.096] [DEBUG] - Searching param plugin for components …
[2023-09-08 11:37:32.096] [DEBUG] - Search param plugin for oceanbase-ce
[2023-09-08 11:37:32.097] [DEBUG] - Found for oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.0.0
[2023-09-08 11:37:32.097] [DEBUG] - Applying oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:32.840] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:32.840] [DEBUG] - import start_check
[2023-09-08 11:37:32.847] [DEBUG] - add start_check ref count to 1
[2023-09-08 11:37:32.848] [INFO] Check before start observer
[2023-09-08 11:37:32.849] [DEBUG] – local execute: ls /root/observer/store/clog/tenant_1/
[2023-09-08 11:37:32.852] [DEBUG] – exited code 0
[2023-09-08 11:37:32.852] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2023-09-08 11:37:32.855] [DEBUG] – exited code 1, error output:
[2023-09-08 11:37:32.855] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2023-09-08 11:37:32.855] [DEBUG]
[2023-09-08 11:37:32.855] [DEBUG] – 127.0.0.1 port check
[2023-09-08 11:37:32.855] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-09-08 11:37:32.873] [DEBUG] – exited code 0
[2023-09-08 11:37:32.873] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-09-08 11:37:32.890] [DEBUG] – exited code 0
[2023-09-08 11:37:32.891] [DEBUG] – local execute: ls /root/observer/store/sstable/block_file
[2023-09-08 11:37:32.893] [DEBUG] – exited code 0
[2023-09-08 11:37:32.894] [DEBUG] – local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2023-09-08 11:37:32.896] [DEBUG] – exited code 0
[2023-09-08 11:37:32.896] [DEBUG] – local execute: ulimit -a
[2023-09-08 11:37:32.898] [DEBUG] – exited code 0
[2023-09-08 11:37:32.899] [DEBUG] – local execute: cat /proc/meminfo
[2023-09-08 11:37:32.901] [DEBUG] – exited code 0
[2023-09-08 11:37:32.901] [DEBUG] – local execute: df --block-size=1024
[2023-09-08 11:37:32.904] [DEBUG] – exited code 0
[2023-09-08 11:37:32.904] [DEBUG] – get disk info for path /, total: 938244935680 avail: 394510049280
[2023-09-08 11:37:32.905] [DEBUG] – get disk info for path /dev, total: 67108864 avail: 67108864
[2023-09-08 11:37:32.905] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 136441954304 avail: 136441954304
[2023-09-08 11:37:32.905] [DEBUG] – get disk info for path /etc/hosts, total: 938244935680 avail: 394510049280
[2023-09-08 11:37:32.905] [DEBUG] – get disk info for path /dev/shm, total: 67108864 avail: 67108864
[2023-09-08 11:37:32.905] [DEBUG] – get disk info for path /proc/acpi, total: 136441954304 avail: 136441954304
[2023-09-08 11:37:32.905] [DEBUG] – get disk info for path /proc/livepatch, total: 136441954304 avail: 136441954304
[2023-09-08 11:37:32.905] [DEBUG] – get disk info for path /proc/scsi, total: 136441954304 avail: 136441954304
[2023-09-08 11:37:32.905] [DEBUG] – get disk info for path /sys/firmware, total: 136441954304 avail: 136441954304
[2023-09-08 11:37:32.905] [DEBUG] – disk: {’/’: {‘total’: 938244935680, ‘avail’: 394510049280, ‘need’: 0}, ‘/dev’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}, ‘/etc/hosts’: {‘total’: 938244935680, ‘avail’: 394510049280, ‘need’: 0}, ‘/dev/shm’: {‘total’: 67108864, ‘avail’: 67108864, ‘need’: 0}, ‘/proc/acpi’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}, ‘/proc/livepatch’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}, ‘/proc/scsi’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}, ‘/sys/firmware’: {‘total’: 136441954304, ‘avail’: 136441954304, ‘need’: 0}}
[2023-09-08 11:37:32.905] [DEBUG] – local execute: date +%s%N
[2023-09-08 11:37:32.908] [DEBUG] – exited code 0
[2023-09-08 11:37:32.908] [DEBUG] – 127.0.0.1 time delta -0.367431640625
[2023-09-08 11:37:32.980] [DEBUG] - sub start_check ref count to 0
[2023-09-08 11:37:32.980] [DEBUG] - export start_check
[2023-09-08 11:37:32.980] [DEBUG] - Call oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:32.980] [DEBUG] - import start
[2023-09-08 11:37:32.982] [DEBUG] - add start ref count to 1
[2023-09-08 11:37:32.983] [INFO] Start observer
[2023-09-08 11:37:32.983] [DEBUG] – local execute: ls /root/observer/store/clog/tenant_1/
[2023-09-08 11:37:32.986] [DEBUG] – exited code 0
[2023-09-08 11:37:32.986] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2023-09-08 11:37:32.989] [DEBUG] – exited code 1, error output:
[2023-09-08 11:37:32.989] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2023-09-08 11:37:32.989] [DEBUG]
[2023-09-08 11:37:32.989] [DEBUG] – 127.0.0.1 start command construction
[2023-09-08 11:37:32.989] [DEBUG] – starting 127.0.0.1 observer
[2023-09-08 11:37:32.990] [DEBUG] – root@127.0.0.1 set env LD_LIBRARY_PATH to ‘/root/observer/lib:’
[2023-09-08 11:37:32.990] [DEBUG] – local execute: cd /root/observer; /root/observer/bin/observer -r ‘127.0.0.1:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -c 1 -d ‘/root/observer/store’ -i ‘lo’ -o __min_full_resource_pool_memory=2147483648,memory_limit=‘16G’,system_memory=‘1G’,datafile_size=‘2G’,datafile_next=‘2G’,datafile_maxsize=‘20G’,log_disk_size=‘20G’,cpu_count=16,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4
[2023-09-08 11:37:33.033] [DEBUG] – exited code 0
[2023-09-08 11:37:33.034] [DEBUG] – root@127.0.0.1 delete env LD_LIBRARY_PATH
[2023-09-08 11:37:33.114] [INFO] observer program health check
[2023-09-08 11:37:36.118] [DEBUG] – 127.0.0.1 program health check
[2023-09-08 11:37:36.118] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2023-09-08 11:37:36.121] [DEBUG] – exited code 0
[2023-09-08 11:37:36.121] [DEBUG] – local execute: ls /proc/1683
[2023-09-08 11:37:36.124] [DEBUG] – exited code 0
[2023-09-08 11:37:36.124] [DEBUG] – 127.0.0.1 observer[pid: 1683] started
[2023-09-08 11:37:36.242] [DEBUG] - sub start ref count to 0
[2023-09-08 11:37:36.242] [DEBUG] - export start
[2023-09-08 11:37:36.242] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.0.0-100000152023080109.el7-3445a0b686b11e11b01ad42f0b17b423fcc88ab0
[2023-09-08 11:37:36.242] [DEBUG] - import connect
[2023-09-08 11:37:36.276] [DEBUG] - add connect ref count to 1
[2023-09-08 11:37:36.276] [INFO] Connect to observer
[2023-09-08 11:37:36.277] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:37:39.282] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:37:42.285] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:37:45.288] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:37:48.292] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:37:51.296] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:37:54.300] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:37:57.304] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:00.306] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:03.308] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:06.310] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:09.314] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:12.315] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:15.319] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:18.323] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:21.328] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:24.332] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:27.335] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:30.336] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:33.339] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:36.342] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:39.346] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:42.349] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:45.352] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:48.356] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:51.360] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:38:54.364] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:38:57.368] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:00.371] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:03.373] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:06.377] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:09.380] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:12.385] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:15.389] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:18.391] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:21.393] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:24.397] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:27.400] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:30.404] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:33.408] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:36.412] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:39.414] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:42.416] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:45.420] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:48.422] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:51.425] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:39:54.428] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:39:57.432] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:00.436] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:03.440] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:06.444] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:09.447] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:12.449] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:15.452] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:18.455] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:21.456] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:24.460] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:27.462] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:30.466] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:33.470] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:36.474] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:39.478] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:42.480] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:45.484] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:48.488] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:51.491] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:40:54.495] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:40:57.498] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:00.502] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:03.505] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:06.510] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:09.512] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:12.517] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:15.521] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:18.523] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:21.528] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:24.532] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:27.536] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:30.540] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:33.544] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:36.549] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:39.553] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:42.557] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:45.561] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:48.565] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:51.568] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:41:54.572] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:41:57.576] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:42:00.580] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:42:03.585] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:42:06.589] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:42:09.592] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:42:12.596] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:42:15.600] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:42:18.604] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:42:21.608] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:42:24.612] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:42:27.616] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:42:30.619] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pEwitkh1skvSIA2ETSH2G
[2023-09-08 11:42:33.623] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2023-09-08 11:42:33.626] [ERROR] Traceback (most recent call last):
[2023-09-08 11:42:33.626] [ERROR] File “core.py”, line 1574, in start_cluster
[2023-09-08 11:42:33.626] [ERROR] File “core.py”, line 1666, in _start_cluster
[2023-09-08 11:42:33.626] [ERROR] File “core.py”, line 184, in call_plugin
[2023-09-08 11:42:33.626] [ERROR] File “_plugin.py”, line 324, in call
[2023-09-08 11:42:33.626] [ERROR] File “_plugin.py”, line 286, in _new_func
[2023-09-08 11:42:33.626] [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 136, in connect
[2023-09-08 11:42:33.626] [ERROR] cursor = Cursor(ip=server.ip, port=server_config[‘mysql_port’], tenant=’’, password=password if password is not None else ‘’, stdio=stdio)
[2023-09-08 11:42:33.626] [ERROR] File “_stdio.py”, line 901, in wrapper
[2023-09-08 11:42:33.626] [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 45, in init
[2023-09-08 11:42:33.626] [ERROR] self._connect()
[2023-09-08 11:42:33.626] [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 75, in _connect
[2023-09-08 11:42:33.626] [ERROR] self.db = mysql.connect(host=self.ip, user=self.user, port=int(self.port), password=str(self.password),
[2023-09-08 11:42:33.627] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 353, in init
[2023-09-08 11:42:33.627] [ERROR] self.connect()
[2023-09-08 11:42:33.627] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 633, in connect
[2023-09-08 11:42:33.627] [ERROR] self._request_authentication()
[2023-09-08 11:42:33.627] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 907, in _request_authentication
[2023-09-08 11:42:33.627] [ERROR] auth_packet = self._read_packet()
[2023-09-08 11:42:33.627] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 725, in _read_packet
[2023-09-08 11:42:33.627] [ERROR] packet.raise_for_error()
[2023-09-08 11:42:33.627] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/protocol.py”, line 221, in raise_for_error
[2023-09-08 11:42:33.627] [ERROR] err.raise_mysql_exception(self._data)
[2023-09-08 11:42:33.627] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/err.py”, line 143, in raise_mysql_exception
[2023-09-08 11:42:33.627] [ERROR] raise errorclass(errno, errval)
[2023-09-08 11:42:33.627] [ERROR] pymysql.err.OperationalError: (5150, ‘Tenant not in this server’)
[2023-09-08 11:42:33.627] [ERROR]
[2023-09-08 11:42:36.671] [ERROR] OBD-1006: Failed to connect to oceanbase-ce
[2023-09-08 11:42:36.671] [DEBUG] - sub connect ref count to 0
[2023-09-08 11:42:36.671] [DEBUG] - export connect
[2023-09-08 11:42:36.673] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2023-09-08 11:42:36.673] [INFO] Trace ID: 0ab26b5a-4df9-11ee-967b-0242ac110073
[2023-09-08 11:42:36.673] [INFO] If you want to view detailed obd logs, please run: obd display-trace 0ab26b5a-4df9-11ee-967b-0242ac110073
[2023-09-08 11:42:36.673] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2023-09-08 11:42:36.673] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2023-09-08 11:42:36.673] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obhwxc release, count 0
[2023-09-08 11:42:36.673] [DEBUG] - unlock /root/.obd/lock/deploy_obhwxc
[2023-09-08 11:42:36.673] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2023-09-08 11:42:36.673] [DEBUG] - unlock /root/.obd/lock/global

赧然一笑 · 2023 年9 月 8 日 11:47

8、observer.log
tail -f observer.log |grep ERROR
[2023-09-08 11:42:15.533169] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:16.533318] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:17.533482] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:18.533686] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:19.533894] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:20.534102] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:21.534327] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:22.534522] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:23.534721] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:24.569920] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:25.570073] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:26.570273] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:27.570470] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:28.570669] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:29.570871] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:30.571083] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:31.571290] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:32.571500] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:33.571691] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:34.607359] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:35.607520] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:36.607712] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:37.607917] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:38.608108] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:39.608312] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:40.608536] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:41.608738] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:42.608951] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:43.609150] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:44.644276] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:45.644473] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:46.644696] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:47.644904] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:48.645077] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:49.645277] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:50.645475] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:51.645679] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:52.645883] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:53.646082] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:54.681191] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:55.681472] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:56.681674] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:57.681868] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:58.312586] ERROR issue_dba_error (ob_log.cpp:1802) [2175][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:42:58.453945] ERROR issue_dba_error (ob_log.cpp:1802) [2172][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:42:58.595343] ERROR issue_dba_error (ob_log.cpp:1802) [2177][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:42:58.682086] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:59.586137] ERROR issue_dba_error (ob_log.cpp:1802) [2173][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:42:59.682289] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:42:59.708275] ERROR issue_dba_error (ob_log.cpp:1802) [2106][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:42:59.708337] ERROR issue_dba_error (ob_log.cpp:1802) [2171][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:42:59.748533] ERROR issue_dba_error (ob_log.cpp:1802) [2176][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:42:59.755791] ERROR issue_dba_error (ob_log.cpp:1802) [2179][T1_ReplaySrv9][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:42:59.868266] ERROR issue_dba_error (ob_log.cpp:1802) [2174][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:42:59.938017] ERROR issue_dba_error (ob_log.cpp:1802) [2178][T1_ReplaySrv8][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:00.214214] ERROR issue_dba_error (ob_log.cpp:1802) [1683][observer][T0][YB427F000001-000604D0B2767401-0-0] [lt=11][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file=“ob_server_config.cpp”, line_no=291, info=“the hold of tenant_500 is over the system_memory”)
[2023-09-08 11:43:00.682458] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:01.682671] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:02.303020] ERROR issue_dba_error (ob_log.cpp:1802) [2177][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:02.384078] ERROR issue_dba_error (ob_log.cpp:1802) [2175][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:02.384389] ERROR issue_dba_error (ob_log.cpp:1802) [2173][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:02.384404] ERROR issue_dba_error (ob_log.cpp:1802) [2174][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:02.384728] ERROR issue_dba_error (ob_log.cpp:1802) [2106][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:02.682826] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:03.621684] ERROR issue_dba_error (ob_log.cpp:1802) [2179][T1_ReplaySrv9][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:03.683026] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:03.891905] ERROR issue_dba_error (ob_log.cpp:1802) [2178][T1_ReplaySrv8][T1][Y0-0000000000000000-0-0] [lt=79][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:04.718053] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:05.353632] ERROR issue_dba_error (ob_log.cpp:1802) [2176][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:05.718218] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:06.718421] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:06.747239] ERROR issue_dba_error (ob_log.cpp:1802) [2179][T1_ReplaySrv9][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:07.066938] ERROR issue_dba_error (ob_log.cpp:1802) [2106][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:07.718625] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:08.716136] ERROR issue_dba_error (ob_log.cpp:1802) [2177][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:08.718806] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:09.719015] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:10.719209] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:11.016409] ERROR issue_dba_error (ob_log.cpp:1802) [2173][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:11.719417] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:11.897817] ERROR issue_dba_error (ob_log.cpp:1802) [2171][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:12.321607] ERROR issue_dba_error (ob_log.cpp:1802) [2178][T1_ReplaySrv8][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:12.719636] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:13.185014] ERROR issue_dba_error (ob_log.cpp:1802) [2176][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:13.264847] ERROR issue_dba_error (ob_log.cpp:1802) [2179][T1_ReplaySrv9][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:13.305143] ERROR issue_dba_error (ob_log.cpp:1802) [2175][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:13.719836] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:14.363179] ERROR issue_dba_error (ob_log.cpp:1802) [2177][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:14.744289] ERROR issue_dba_error (ob_log.cpp:1802) [2173][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:14.754861] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:14.884593] ERROR issue_dba_error (ob_log.cpp:1802) [2172][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:15.755041] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:15.809050] ERROR issue_dba_error (ob_log.cpp:1802) [2106][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:16.052105] ERROR issue_dba_error (ob_log.cpp:1802) [2176][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:16.274111] ERROR issue_dba_error (ob_log.cpp:1802) [2171][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:16.755177] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:17.755340] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:18.505563] ERROR issue_dba_error (ob_log.cpp:1802) [2172][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:18.755543] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:19.259719] ERROR issue_dba_error (ob_log.cpp:1802) [2175][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:19.380563] ERROR issue_dba_error (ob_log.cpp:1802) [2177][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:19.755781] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:20.755992] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:21.727841] ERROR issue_dba_error (ob_log.cpp:1802) [2176][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:21.756156] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:21.929606] ERROR issue_dba_error (ob_log.cpp:1802) [2178][T1_ReplaySrv8][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:22.269507] ERROR issue_dba_error (ob_log.cpp:1802) [2106][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:22.756372] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:23.756602] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:24.559357] ERROR issue_dba_error (ob_log.cpp:1802) [2176][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4389, file=“ob_ls_adapter.cpp”, line_no=106, info=“single replay task cost too much time. replay may be delayed”)
[2023-09-08 11:43:24.792331] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2023-09-08 11:43:25.792479] WDIAG [SERVER] runTimerTask (ob_server.cpp:3091) [1684][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)

序风 · 2023 年9 月 8 日 12:00

3个observer节点都起了的？配置文件能发一下吗？

序风 · 2023 年9 月 8 日 14:33

system_memory的资源给大一点