【 使用环境 】测试环境
【 OB or 其他组件 】OB-CE + OBD
【 使用版本 】OB-CE:4.2.1.2，OBD:2.4.1
【问题描述】OBD从2.4.0在线升级到了2.4.1，OB集群从4.2.1.0在线升级到了4.2.1.2，升级成功后，重新启动集群WARN：OBD-1007:The recommended number of stack size is unlimited (Current value: 20480)；但是安装部署建议中给的是20480，如下：
集群其中一个节点的 ulimit -a如下：
obd部署的ob集群os用户是root吗？
看看obd日志呢，~/.obd/log/obd 这个日志文件
WARN是警告，额 可以忽略。
用的是admin用户，最近一次启动的OBD日志(部分)如下，还有些ERROR在里面
[] cmd: ['distributedob']
[] opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None}
[] mkdir /home/admin/.obd/lock/
[] unknown lock mode
[] try to get share lock /home/admin/.obd/lock/global
[] share lock `/home/admin/.obd/lock/global`, count 1
[] Get Deploy by name
[] mkdir /home/admin/.obd/cluster/
[] mkdir /home/admin/.obd/config_parser/
[] try to get exclusive lock /home/admin/.obd/lock/deploy_distributedob
[] exclusive lock `/home/admin/.obd/lock/deploy_distributedob`, count 1
[] Deploy status judge
[] Get local repositories
[] mkdir /home/admin/.obd/repository
[] Get local repository oceanbase-ce-4.2.1.2-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] Search repository oceanbase-ce version: 4.2.1.2, tag: 1873bbe80cbbe5d00d5f276a4f7302cfca677fb6, release: None, package_hash: None
[] try to get share lock /home/admin/.obd/lock/mirror_and_repo
[] share lock `/home/admin/.obd/lock/mirror_and_repo`, count 1
[] mkdir /home/admin/.obd/repository/oceanbase-ce
[] Found repository oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] Get local repository obproxy-ce-4.2.1.0-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] Search repository obproxy-ce version: 4.2.1.0, tag: 0aed4b782120e4248b749f67be3d2cc82cdcb70d, release: None, package_hash: None
[] share lock `/home/admin/.obd/lock/mirror_and_repo`, count 2
[] mkdir /home/admin/.obd/repository/obproxy-ce
[] Found repository obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] Get deploy config
[] Search plugins
[] Searching start_check plugin for components ...
[] Searching start_check plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] mkdir /home/admin/.obd/plugins
[] Found for oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.1.2
[] Searching start_check plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] Found for obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.2.1.0
[] Searching create_tenant plugin for components ...
[] Searching create_tenant plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.1.2
[] Searching create_tenant plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] No such create_tenant plugin for obproxy-ce-4.2.1.0
[] Searching start plugin for components ...
[] Searching start plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] Found for oceanbase-ce-py_script_start-4.2.0.0 for oceanbase-ce-4.2.1.2
[] Searching start plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] Found for obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.2.1.0
[] Searching connect plugin for components ...
[] Searching connect plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.1.2
[] Searching connect plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.2.1.0
[] Searching bootstrap plugin for components ...
[] Searching bootstrap plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] Found for oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.2.1.2
[] Searching bootstrap plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.2.1.0
[] Searching display plugin for components ...
[] Searching display plugin for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.1.2
[] Searching display plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.2.1.0
[] Open ssh connection
[] host: 192.168.0.224, port: 22, user: admin, password: admin0901
[] host: 192.168.0.220, port: 22, user: admin, password: admin0901
[] host: 192.168.0.221, port: 22, user: admin, password: admin0901
[] host: 192.168.0.223, port: 22, user: admin, password: admin0901
[] Load cluster param plugin
[] Searching param plugin for components ...
[] Search param plugin for oceanbase-ce
[] Found for oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.1.2
[] Applying oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] Search param plugin for obproxy-ce
[] Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0
[] Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] Call oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] import start_check
[] add start_check ref count to 1
[] Check before start observer
[] admin@192.168.0.221 execute: ls /home/admin/distributedob/oceanbase/store/clog/tenant_1/
[] exited code 0
[] admin@192.168.0.221 execute: cat /home/admin/distributedob/oceanbase/run/observer.pid
[] exited code 1, error output:
[] cat: /home/admin/distributedob/oceanbase/run/observer.pid: 没有那个文件或目录
[]
[] 192.168.0.221 port check
[] admin@192.168.0.221 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] admin@192.168.0.221 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] admin@192.168.0.221 execute: ls /home/admin/distributedob/oceanbase/store/sstable/block_file
[] exited code 0
[] admin@192.168.0.223 execute: cat /home/admin/distributedob/oceanbase/run/observer.pid
[] exited code 1, error output:
[] cat: /home/admin/distributedob/oceanbase/run/observer.pid: 没有那个文件或目录
[]
[] 192.168.0.223 port check
[] admin@192.168.0.223 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] admin@192.168.0.223 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] admin@192.168.0.223 execute: ls /home/admin/distributedob/oceanbase/store/sstable/block_file
[] exited code 0
[] admin@192.168.0.224 execute: cat /home/admin/distributedob/oceanbase/run/observer.pid
[] exited code 1, error output:
[] cat: /home/admin/distributedob/oceanbase/run/observer.pid: 没有那个文件或目录
[]
[] 192.168.0.224 port check
[] admin@192.168.0.224 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] admin@192.168.0.224 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] admin@192.168.0.224 execute: ls /home/admin/distributedob/oceanbase/store/sstable/block_file
[] exited code 0
[] admin@192.168.0.221 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[] exited code 0
[] admin@192.168.0.221 execute: ulimit -a
[] exited code 0
[WARNING] OBD-1007: (192.168.0.221) The recommended number of stack size is unlimited (Current value: 20480)
[] admin@192.168.0.221 execute: sysctl -a
[] exited code 0
[] admin@192.168.0.221 execute: cat /proc/meminfo
[] exited code 0
[] admin@192.168.0.221 execute: df --block-size=1024
[] exited code 0
[] get disk info for path /dev, total: 8310308864 avail: 8310308864
[] get disk info for path /dev/shm, total: 8327835648 avail: 8327835648
[] get disk info for path /run, total: 8327835648 avail: 8292655104
[] get disk info for path /sys/fs/cgroup, total: 8327835648 avail: 8327835648
[] get disk info for path /, total: 53660876800 avail: 30074376192
[] get disk info for path /boot, total: 1063256064 avail: 818429952
[] get disk info for path /home, total: 357641359360 avail: 119862554624
[] get disk info for path /ob-backup, total: 366093533184 avail: 324230184960
[] get disk info for path /run/user/42, total: 1665568768 avail: 1665556480
[] get disk info for path /run/user/0, total: 1665568768 avail: 1665568768
[] get disk info for path /run/user/1001, total: 1665568768 avail: 1665568768
[] disk: {'/dev': {'total': 8310308864, 'avail': 8310308864, 'need': 0}, '/dev/shm': {'total': 8327835648, 'avail': 8327835648, 'need': 0}, '/run': {'total': 8327835648, 'avail': 8292655104, 'need': 0}, '/sys/fs/cgroup': {'total': 8327835648, 'avail': 8327835648, 'need': 0}, '/': {'total': 53660876800, 'avail': 30074376192, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 818429952, 'need': 0}, '/home': {'total': 357641359360, 'avail': 119862554624, 'need': 0}, '/ob-backup': {'total': 366093533184, 'avail': 324230184960, 'need': 0}, '/run/user/42': {'total': 1665568768, 'avail': 1665556480, 'need': 0}, '/run/user/0': {'total': 1665568768, 'avail': 1665568768, 'need': 0}, '/run/user/1001': {'total': 1665568768, 'avail': 1665568768, 'need': 0}}
[] admin@192.168.0.223 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[] exited code 0
[] admin@192.168.0.223 execute: ulimit -a
[] exited code 0
[WARNING] OBD-1007: (192.168.0.223) The recommended number of stack size is unlimited (Current value: 20480)
[] admin@192.168.0.223 execute: sysctl -a
[] exited code 0
[] admin@192.168.0.223 execute: cat /proc/meminfo
[] exited code 0
[] admin@192.168.0.223 execute: df --block-size=1024
[] exited code 0
[] get disk info for path /dev, total: 8310317056 avail: 8310317056
[] get disk info for path /dev/shm, total: 8327843840 avail: 8327843840
[] get disk info for path /run, total: 8327843840 avail: 8292769792
[] get disk info for path /sys/fs/cgroup, total: 8327843840 avail: 8327843840
[] get disk info for path /, total: 53660876800 avail: 46795833344
[] get disk info for path /boot, total: 1063256064 avail: 818421760
[] get disk info for path /home, total: 366092947456 avail: 128536231936
[] get disk info for path /ob-backup, total: 366093533184 avail: 324230184960
[] get disk info for path /run/user/42, total: 1665572864 avail: 1665560576
[] get disk info for path /run/user/0, total: 1665572864 avail: 1665572864
[] get disk info for path /run/user/1001, total: 1665572864 avail: 1665572864
[] disk: {'/dev': {'total': 8310317056, 'avail': 8310317056, 'need': 0}, '/dev/shm': {'total': 8327843840, 'avail': 8327843840, 'need': 0}, '/run': {'total': 8327843840, 'avail': 8292769792, 'need': 0}, '/sys/fs/cgroup': {'total': 8327843840, 'avail': 8327843840, 'need': 0}, '/': {'total': 53660876800, 'avail': 46795833344, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 818421760, 'need': 0}, '/home': {'total': 366092947456, 'avail': 128536231936, 'need': 0}, '/ob-backup': {'total': 366093533184, 'avail': 324230184960, 'need': 0}, '/run/user/42': {'total': 1665572864, 'avail': 1665560576, 'need': 0}, '/run/user/0': {'total': 1665572864, 'avail': 1665572864, 'need': 0}, '/run/user/1001': {'total': 1665572864, 'avail': 1665572864, 'need': 0}}
[] admin@192.168.0.224 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[] exited code 0
[] admin@192.168.0.224 execute: ulimit -a
[] exited code 0
[WARNING] OBD-1007: (192.168.0.224) The recommended number of stack size is unlimited (Current value: 20480)
[] admin@192.168.0.224 execute: sysctl -a
[] exited code 0
[] admin@192.168.0.224 execute: cat /proc/meminfo
[] exited code 0
[] admin@192.168.0.224 execute: df --block-size=1024
[] exited code 0
[] get disk info for path /dev, total: 8310308864 avail: 8310308864
[] get disk info for path /dev/shm, total: 8327835648 avail: 8327835648
[] get disk info for path /run, total: 8327835648 avail: 8292761600
[] get disk info for path /sys/fs/cgroup, total: 8327835648 avail: 8327835648
[] get disk info for path /, total: 53660876800 avail: 46805233664
[] get disk info for path /boot, total: 1063256064 avail: 818536448
[] get disk info for path /home, total: 366092947456 avail: 134216077312
[] get disk info for path /ob-backup, total: 366093533184 avail: 324230184960
[] get disk info for path /run/user/42, total: 1665568768 avail: 1665556480
[] get disk info for path /run/user/0, total: 1665568768 avail: 1665568768
[] get disk info for path /run/user/1001, total: 1665568768 avail: 1665568768
[] disk: {'/dev': {'total': 8310308864, 'avail': 8310308864, 'need': 0}, '/dev/shm': {'total': 8327835648, 'avail': 8327835648, 'need': 0}, '/run': {'total': 8327835648, 'avail': 8292761600, 'need': 0}, '/sys/fs/cgroup': {'total': 8327835648, 'avail': 8327835648, 'need': 0}, '/': {'total': 53660876800, 'avail': 46805233664, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 818536448, 'need': 0}, '/home': {'total': 366092947456, 'avail': 134216077312, 'need': 0}, '/ob-backup': {'total': 366093533184, 'avail': 324230184960, 'need': 0}, '/run/user/42': {'total': 1665568768, 'avail': 1665556480, 'need': 0}, '/run/user/0': {'total': 1665568768, 'avail': 1665568768, 'need': 0}, '/run/user/1001': {'total': 1665568768, 'avail': 1665568768, 'need': 0}}
[] admin@192.168.0.221 execute: date +%s%N
[] exited code 0
[] 192.168.0.221 time delta -2.754150390625
[] admin@192.168.0.223 execute: date +%s%N
[] exited code 0
[] 192.168.0.223 time delta -2.9287109375
[] admin@192.168.0.224 execute: date +%s%N
[] exited code 0
[] 192.168.0.224 time delta -2.7734375
[] e[33m[WARN]e[39m OBD-1007: (192.168.0.221) The recommended number of stack size is unlimited (Current value: 20480)
[] e[33m[WARN]e[39m OBD-1007: (192.168.0.223) The recommended number of stack size is unlimited (Current value: 20480)
[] e[33m[WARN]e[39m OBD-1007: (192.168.0.224) The recommended number of stack size is unlimited (Current value: 20480)
[]
[] sub start_check ref count to 0
[] export start_check
[] Call obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] import start_check
[] add start_check ref count to 1
[] Check before start obproxy
[] admin@192.168.0.220 execute: cat /home/admin/distributedob/obproxy/run/obproxy-192.168.0.220-2883.pid
[] exited code 1, error output:
[] cat: /home/admin/distributedob/obproxy/run/obproxy-192.168.0.220-2883.pid: 没有那个文件或目录
[]
[] 192.168.0.220 port check
[] admin@192.168.0.220 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] admin@192.168.0.220 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] sub start_check ref count to 0
[] export start_check
[] Call oceanbase-ce-py_script_start-4.2.0.0 for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] import start
[] add start ref count to 1
[] Start observer
[] admin@192.168.0.221 execute: ls /home/admin/distributedob/oceanbase/store/clog/tenant_1/
[] exited code 0
[] admin@192.168.0.221 execute: cat /home/admin/distributedob/oceanbase/run/observer.pid
[] exited code 1, error output:
[] cat: /home/admin/distributedob/oceanbase/run/observer.pid: 没有那个文件或目录
[]
[] 192.168.0.221 start command construction
[] admin@192.168.0.223 execute: ls /home/admin/distributedob/oceanbase/store/clog/tenant_1/
[] exited code 0
[] admin@192.168.0.223 execute: cat /home/admin/distributedob/oceanbase/run/observer.pid
[] exited code 1, error output:
[] cat: /home/admin/distributedob/oceanbase/run/observer.pid: 没有那个文件或目录
[]
[] 192.168.0.223 start command construction
[] admin@192.168.0.224 execute: ls /home/admin/distributedob/oceanbase/store/clog/tenant_1/
[] exited code 0
[] admin@192.168.0.224 execute: cat /home/admin/distributedob/oceanbase/run/observer.pid
[] exited code 1, error output:
[] cat: /home/admin/distributedob/oceanbase/run/observer.pid: 没有那个文件或目录
[]
[] 192.168.0.224 start command construction
[] starting 192.168.0.221 observer
[] admin@192.168.0.221 set env LD_LIBRARY_PATH to '/home/admin/distributedob/oceanbase/lib:'
[] admin@192.168.0.221 execute: cd /home/admin/distributedob/oceanbase; /home/admin/distributedob/oceanbase/bin/observer -r '192.168.0.221:2882:2881;192.168.0.223:2882:2881;192.168.0.224:2882:2881' -p 2881 -P 2882 -z 'zone1' -n 'distributedob' -c 1697695072 -d '/home/admin/distributedob/oceanbase/store' -I '192.168.0.221' -o __min_full_resource_pool_memory=2147483648,datafile_size='100GB',datafile_maxsize='3000GB',datafile_next='20GB',log_disk_size='100GB',enable_syslog_recycle=True,max_syslog_file_count=20,memory_limit='12GB',system_memory='3GB',cpu_count=16,enable_syslog_wf=False
[] exited code 0
[] admin@192.168.0.221 delete env LD_LIBRARY_PATH
[] starting 192.168.0.223 observer
[] admin@192.168.0.223 set env LD_LIBRARY_PATH to '/home/admin/distributedob/oceanbase/lib:'
[] admin@192.168.0.223 execute: cd /home/admin/distributedob/oceanbase; /home/admin/distributedob/oceanbase/bin/observer -r '192.168.0.221:2882:2881;192.168.0.223:2882:2881;192.168.0.224:2882:2881' -p 2881 -P 2882 -z 'zone2' -n 'distributedob' -c 1697695072 -d '/home/admin/distributedob/oceanbase/store' -I '192.168.0.223' -o __min_full_resource_pool_memory=2147483648,datafile_size='100GB',datafile_maxsize='3000GB',datafile_next='20GB',log_disk_size='100GB',enable_syslog_recycle=True,max_syslog_file_count=20,memory_limit='12GB',system_memory='3GB',cpu_count=16,enable_syslog_wf=False
[] exited code 0
[] admin@192.168.0.223 delete env LD_LIBRARY_PATH
[] starting 192.168.0.224 observer
[] admin@192.168.0.224 set env LD_LIBRARY_PATH to '/home/admin/distributedob/oceanbase/lib:'
[] admin@192.168.0.224 execute: cd /home/admin/distributedob/oceanbase; /home/admin/distributedob/oceanbase/bin/observer -r '192.168.0.221:2882:2881;192.168.0.223:2882:2881;192.168.0.224:2882:2881' -p 2881 -P 2882 -z 'zone3' -n 'distributedob' -c 1697695072 -d '/home/admin/distributedob/oceanbase/store' -I '192.168.0.224' -o __min_full_resource_pool_memory=2147483648,datafile_size='100GB',datafile_maxsize='3000GB',datafile_next='20GB',log_disk_size='100GB',enable_syslog_recycle=True,max_syslog_file_count=20,memory_limit='12GB',system_memory='3GB',cpu_count=16,enable_syslog_wf=False
[] exited code 0
[] admin@192.168.0.224 delete env LD_LIBRARY_PATH
[] observer program health check
[] 192.168.0.221 program health check
[] admin@192.168.0.221 execute: cat /home/admin/distributedob/oceanbase/run/observer.pid
[] exited code 0
[] admin@192.168.0.221 execute: ls /proc/2670
[] exited code 0
[] 192.168.0.221 observer[pid: 2670] started
[] 192.168.0.223 program health check
[] admin@192.168.0.223 execute: cat /home/admin/distributedob/oceanbase/run/observer.pid
[] exited code 0
[] admin@192.168.0.223 execute: ls /proc/30512
[] exited code 0
[] 192.168.0.223 observer[pid: 30512] started
[] 192.168.0.224 program health check
[] admin@192.168.0.224 execute: cat /home/admin/distributedob/oceanbase/run/observer.pid
[] exited code 0
[] admin@192.168.0.224 execute: ls /proc/29446
[] exited code 0
[] 192.168.0.224 observer[pid: 29446] started
[] sub start ref count to 0
[] export start
[] Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] import connect
[] add connect ref count to 1
[] Connect to observer
[] connect 192.168.0.221 -P2881 -uroot -padmin0901
[] connect 192.168.0.223 -P2881 -uroot -padmin0901
[] connect 192.168.0.224 -P2881 -uroot -padmin0901
[] connect 192.168.0.221 -P2881 -uroot -p
[] execute sql: select 1. args: None
[] sub connect ref count to 0
[] export connect
[] Call obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] import start
[] add start ref count to 1
[] Start obproxy
[] admin@192.168.0.220 execute: ls /home/admin/distributedob/obproxy/etc/obproxy_config.bin
[] exited code 0
[] 192.168.0.220 port check
[] admin@192.168.0.220 execute: cat /home/admin/distributedob/obproxy/run/obproxy-192.168.0.220-2883.pid
[] exited code 1, error output:
[] cat: /home/admin/distributedob/obproxy/run/obproxy-192.168.0.220-2883.pid: 没有那个文件或目录
[]
[] starting 192.168.0.220 obproxy
[] admin@192.168.0.220 set env LD_LIBRARY_PATH to '/home/admin/distributedob/obproxy/lib:'
[] admin@192.168.0.220 execute: cd /home/admin/distributedob/obproxy; /home/admin/distributedob/obproxy/bin/obproxy -o obproxy_sys_password='0f954de0974582bedef02f1ce77b3bae20154269',enable_strict_kernel_release=False,proxy_mem_limited='4GB',skip_proxy_sys_private_check=True,enable_cluster_checkout=False --listen_port 2883 --prometheus_listen_port 2884 --rs_list '192.168.0.221:2881;192.168.0.223:2881;192.168.0.224:2881' --cluster_name 'distributedob'
[] exited code 0
[] admin@192.168.0.220 delete env LD_LIBRARY_PATH
[] admin@192.168.0.220 execute: ps -aux | grep -e '/home/admin/distributedob/obproxy/bin/obproxy -o obproxy_sys_password=0f954de0974582bedef02f1ce77b3bae20154269,enable_strict_kernel_release=False,proxy_mem_limited=4GB,skip_proxy_sys_private_check=True,enable_cluster_checkout=False --listen_port 2883 --prometheus_listen_port 2884 --rs_list 192.168.0.221:2881;192.168.0.223:2881;192.168.0.224:2881 --cluster_name distributedob$' | grep -v grep | awk '{print $2}' > /home/admin/distributedob/obproxy/run/obproxy-192.168.0.220-2883.pid
[] exited code 0
[] obproxy program health check
[] 192.168.0.220 program health check
[] admin@192.168.0.220 execute: cat /home/admin/distributedob/obproxy/run/obproxy-192.168.0.220-2883.pid
[] exited code 0
[] admin@192.168.0.220 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] failed to start 192.168.0.220 obproxy, remaining retries: 299
[] 192.168.0.220 program health check
[] admin@192.168.0.220 execute: cat /home/admin/distributedob/obproxy/run/obproxy-192.168.0.220-2883.pid
[] exited code 0
[] admin@192.168.0.220 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] failed to start 192.168.0.220 obproxy, remaining retries: 298
[] 192.168.0.220 program health check
[] admin@192.168.0.220 execute: cat /home/admin/distributedob/obproxy/run/obproxy-192.168.0.220-2883.pid
[] exited code 0
[] admin@192.168.0.220 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq
[] exited code 0
[] 9468964
[]
[] admin@192.168.0.220 execute: ls -l /proc/27108/fd/ |grep -E 'socket:\[(9468964)\]'
[] exited code 0
[] admin@192.168.0.220 execute: pid=`cat /home/admin/distributedob/obproxy/run/obproxyd-192.168.0.220-2883.pid` && ls /proc/$pid
[] exited code 0
[] 192.168.0.220 obproxy[pid: 27108] started
[] sub start ref count to 0
[] export start
[] Call obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] import connect
[] add connect ref count to 1
[] Connect to obproxy
[] connect 192.168.0.220 -P2883 -uroot@proxysys -padmin0901
[] sub connect ref count to 0
[] export connect
[] Initialize obproxy-ce
[] Call obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d
[] import bootstrap
[] add bootstrap ref count to 1
[] execute sql: alter proxyconfig set observer_sys_password = %s. args: ['oqRh1nDplu']
[] sub bootstrap ref count to 0
[] export bootstrap
[] Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.1.2-102010022023121415.el7-1873bbe80cbbe5d00d5f276a4f7302cfca677fb6
[] import display
[] add display ref count to 1
[] Wait for observer init
[] execute sql: select * from oceanbase.__all_server. args: None
[] OBD-5000: select * from oceanbase.__all_server execute failed
[ERROR] Traceback (most recent call last):
[ERROR] File "core.py", line 1673, in start_cluster
[ERROR] File "core.py", line 1794, in _start_cluster
[ERROR] File "core.py", line 184, in call_plugin
[ERROR] File "_plugin.py", line 324, in __call__
[ERROR] File "_plugin.py", line 286, in _new_func
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 37, in display
[ERROR] servers = cursor.fetchall('select * from oceanbase.__all_server', raise_exception=True, exc_level='verbose')
[ERROR] File "_stdio.py", line 888, in func_wrapper
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 115, in fetchall
[ERROR] return self.execute(sql, args=args, execute_func='fetchall', raise_exception=raise_exception, exc_level=exc_level, stdio=stdio)
[ERROR] File "_stdio.py", line 888, in func_wrapper
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 94, in execute
[ERROR] self.cursor.execute(sql, args)
[ERROR] File "pymysql/cursors.py", line 148, in execute
[ERROR] File "pymysql/cursors.py", line 310, in _query
[ERROR] File "pymysql/connections.py", line 548, in query
[ERROR] File "pymysql/connections.py", line 775, in _read_query_result
[ERROR] File "pymysql/connections.py", line 1156, in read
[ERROR] File "pymysql/connections.py", line 725, in _read_packet
[ERROR] File "pymysql/protocol.py", line 221, in raise_for_error
[ERROR] File "pymysql/err.py", line 143, in raise_mysql_exception
[ERROR] pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist")
[ERROR]
[] execute sql: select * from oceanbase.__all_server. args: None
[] OBD-5000: select * from oceanbase.__all_server execute failed
[ERROR] Traceback (most recent call last):
[ERROR] File "core.py", line 1673, in start_cluster
[ERROR] File "core.py", line 1794, in _start_cluster
[ERROR] File "core.py", line 184, in call_plugin
[ERROR] File "_plugin.py", line 324, in __call__
[ERROR] File "_plugin.py", line 286, in _new_func
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 37, in display
[ERROR] servers = cursor.fetchall('select * from oceanbase.__all_server', raise_exception=True, exc_level='verbose')
[ERROR] File "_stdio.py", line 888, in func_wrapper
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 115, in fetchall
[ERROR] return self.execute(sql, args=args, execute_func='fetchall', raise_exception=raise_exception, exc_level=exc_level, stdio=stdio)
[ERROR] File "_stdio.py", line 888, in func_wrapper
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 94, in execute
[ERROR] self.cursor.execute(sql, args)
[ERROR] File "pymysql/cursors.py", line 148, in execute
[ERROR] File "pymysql/cursors.py", line 310, in _query
[ERROR] File "pymysql/connections.py", line 548, in query
[ERROR] File "pymysql/connections.py", line 775, in _read_query_result
[ERROR] File "pymysql/connections.py", line 1156, in read
[ERROR] File "pymysql/connections.py", line 725, in _read_packet
[ERROR] File "pymysql/protocol.py", line 221, in raise_for_error
[ERROR] File "pymysql/err.py", line 143, in raise_mysql_exception
[ERROR] pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist")
[ERROR]
[] execute sql: select * from oceanbase.__all_server. args: None
[] OBD-5000: select * from oceanbase.__all_server execute failed
[ERROR] Traceback (most recent call last):
[ERROR] File "core.py", line 1673, in start_cluster
[ERROR] File "core.py", line 1794, in _start_cluster
[ERROR] File "core.py", line 184, in call_plugin
[ERROR] File "_plugin.py", line 324, in __call__
[ERROR] File "_plugin.py", line 286, in _new_func
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 37, in display
[ERROR] servers = cursor.fetchall('select * from oceanbase.__all_server', raise_exception=True, exc_level='verbose')
[ERROR] File "_stdio.py", line 888, in func_wrapper
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 115, in fetchall
[ERROR] return self.execute(sql, args=args, execute_func='fetchall', raise_exception=raise_exception, exc_level=exc_level, stdio=stdio)
[ERROR] File "_stdio.py", line 888, in func_wrapper
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 94, in execute
[ERROR] self.cursor.execute(sql, args)
[ERROR] File "pymysql/cursors.py", line 148, in execute
[ERROR] File "pymysql/cursors.py", line 310, in _query
[ERROR] File "pymysql/connections.py", line 548, in query
[ERROR] File "pymysql/connections.py", line 775, in _read_query_result
[ERROR] File "pymysql/connections.py", line 1156, in read
[ERROR] File "pymysql/connections.py", line 725, in _read_packet
[ERROR] File "pymysql/protocol.py", line 221, in raise_for_error
[ERROR] File "pymysql/err.py", line 143, in raise_mysql_exception
[ERROR] pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist")
[ERROR]
[] execute sql: select * from oceanbase.__all_server. args: None
[] OBD-5000: select * from oceanbase.__all_server execute failed
[ERROR] Traceback (most recent call last):
[ERROR] File "core.py", line 1673, in start_cluster
[ERROR] File "core.py", line 1794, in _start_cluster
[ERROR] File "core.py", line 184, in call_plugin
[ERROR] File "_plugin.py", line 324, in __call__
[ERROR] File "_plugin.py", line 286, in _new_func
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 37, in display
[ERROR] servers = cursor.fetchall('select * from oceanbase.__all_server', raise_exception=True, exc_level='verbose')
[ERROR] File "_stdio.py", line 888, in func_wrapper
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 115, in fetchall
[ERROR] return self.execute(sql, args=args, execute_func='fetchall', raise_exception=raise_exception, exc_level=exc_level, stdio=stdio)
[ERROR] File "_stdio.py", line 888, in func_wrapper
[ERROR] File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 94, in execute
[ERROR] self.cursor.execute(sql, args)
[ERROR] File "pymysql/cursors.py", line 148, in execute
[ERROR] File "pymysql/cursors.py", line 310, in _query
[ERROR] File "pymysql/connections.py", line 548, in query
[ERROR] File "pymysql/connections.py", line 775, in _read_query_result
[ERROR] File "pymysql/connections.py", line 1156, in read
[ERROR] File "pymysql/connections.py", line 725, in _read_packet
[ERROR] File "pymysql/protocol.py", line 221, in raise_for_error
[ERROR] File "pymysql/err.py", line 143, in raise_mysql_exception
[ERROR] pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist")
[ERROR]
[] execute sql: select * from oceanbase.__all_server. args: None
[] OBD-5000: select * from oceanbase.__all_server execute failed
现在OBServer集群是起来的吗？
查查这个值，admin用户下查询ulimit -a
[] admin@192.168.0.224 execute: ulimit -a
[] exited code 0
[WARNING] OBD-1007: (192.168.0.224) The recommended number of stack size is unlimited (Current value: 20480)
[
是的，可以正常启动
你上面确实限制了stack size，官方推荐这个是设置成unlimited，这是个弱限制，所以是warning
那现在集群正常可以用的吧？有没有什么问题
可以正常用，只是对这个WARN和obd日志中的ERROR有点疑问
error的话，你连进oceanbase集群，oceanbase数据库下查询看看能不能查
[] execute sql: select * from oceanbase.__all_server. args: None
[] OBD-5000: select * from oceanbase.__all_server execute failed
你用obd restart一下，看看还有没有报错？
用 restart 就没有 WARN 和 ERROR 了，如果是先 stop 再 start 就会有WARN和ERROR
restart 也是stop+start的
查看obd日志，restart确实没有问题，先stop再start就会报错和警告，虽然可以正常启动和使用
老版本的：
start命令 = start check脚本 + start 脚本
restart 命令 = stop + start（脚本）
因为：能start 成功代表环境是没问题的，就跳过了重复检查
2.4.0 之后 restart 才加上了check 机制
因为：2.4.0 增加了好几项内核参数检查，担心用户会在running状态 改了内核参数导致restart 起不来
那这个启动时的ERROR是啥原因嘞：
