【 使用环境 】测试环境
【 OB or 其他组件 】OB
【 使用版本 】社区版 3.1.1
【问题描述】安装部署后启动报错
【复现路径】
【问题现象及影响】
看提示错误码2002，中间按启动的参数memory_limit=‘4G’,system_memory=‘1G’，部署的配置文件里是 memory_limit: 10G，system_memory: 2G。
报错日志如下
[root@Cent73dev5-58 ~]# obd display-trace 7be507d8-f3d1-11ed-bdfb-0050569b2460
[2023-05-16 18:07:37.068] [DEBUG] - mkdir /root/.obd/lock/
[2023-05-16 18:07:37.068] [DEBUG] - unknown lock mode
[2023-05-16 18:07:37.068] [DEBUG] - try to get share lock /root/.obd/lock/global
[2023-05-16 18:07:37.068] [DEBUG] - share lock /root/.obd/lock/global, count 1
[2023-05-16 18:07:37.068] [DEBUG] - cmd: [‘ob’]
[2023-05-16 18:07:37.068] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2023-05-16 18:07:37.068] [DEBUG] - Get Deploy by name
[2023-05-16 18:07:37.068] [DEBUG] - mkdir /root/.obd/cluster/
[2023-05-16 18:07:37.069] [DEBUG] - mkdir /root/.obd/config_parser/
[2023-05-16 18:07:37.069] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_ob
[2023-05-16 18:07:37.069] [DEBUG] - exclusive lock /root/.obd/lock/deploy_ob, count 1
[2023-05-16 18:07:37.073] [DEBUG] - Deploy status judge
[2023-05-16 18:07:37.074] [INFO] Get local repositories
[2023-05-16 18:07:37.074] [DEBUG] - mkdir /root/.obd/repository
[2023-05-16 18:07:37.075] [DEBUG] - Get local repository oceanbase-ce-3.1.1-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:37.075] [DEBUG] - Search repository oceanbase-ce version: 3.1.1, tag: f19f8bfb67723712175fb0dfd60579196b3168f1, release: None, package_hash: None
[2023-05-16 18:07:37.075] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2023-05-16 18:07:37.075] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 1
[2023-05-16 18:07:37.075] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce
[2023-05-16 18:07:37.077] [DEBUG] - Found repository oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:37.205] [DEBUG] - Get deploy config
[2023-05-16 18:07:37.219] [INFO] Search plugins
[2023-05-16 18:07:37.219] [DEBUG] - Searching start_check plugin for components …
[2023-05-16 18:07:37.219] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:37.220] [DEBUG] - mkdir /root/.obd/plugins
[2023-05-16 18:07:37.221] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1
[2023-05-16 18:07:37.221] [DEBUG] - Searching create_tenant plugin for components …
[2023-05-16 18:07:37.221] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:37.221] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-3.1.0 for oceanbase-ce-3.1.1
[2023-05-16 18:07:37.222] [DEBUG] - Searching start plugin for components …
[2023-05-16 18:07:37.222] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:37.222] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1
[2023-05-16 18:07:37.222] [DEBUG] - Searching connect plugin for components …
[2023-05-16 18:07:37.222] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:37.223] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1
[2023-05-16 18:07:37.223] [DEBUG] - Searching bootstrap plugin for components …
[2023-05-16 18:07:37.223] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:37.223] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1
[2023-05-16 18:07:37.224] [DEBUG] - Searching display plugin for components …
[2023-05-16 18:07:37.224] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:37.224] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1
[2023-05-16 18:07:37.352] [INFO] Open ssh connection
[2023-05-16 18:07:37.483] [INFO] Load cluster param plugin
[2023-05-16 18:07:37.484] [DEBUG] - Searching param plugin for components …
[2023-05-16 18:07:37.484] [DEBUG] - Search param plugin for oceanbase-ce
[2023-05-16 18:07:37.484] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1
[2023-05-16 18:07:37.484] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:38.166] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:38.166] [DEBUG] - import start_check
[2023-05-16 18:07:38.171] [DEBUG] - add start_check ref count to 1
[2023-05-16 18:07:38.172] [INFO] Check before start observer
[2023-05-16 18:07:38.173] [DEBUG] – local execute: cat /root/oceanbase/run/observer.pid
[2023-05-16 18:07:38.176] [DEBUG] – exited code 1, error output:
[2023-05-16 18:07:38.176] [DEBUG] cat: /root/oceanbase/run/observer.pid: No such file or directory
[2023-05-16 18:07:38.176] [DEBUG]
[2023-05-16 18:07:38.176] [DEBUG] – 127.0.0.1 port check
[2023-05-16 18:07:38.177] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-05-16 18:07:38.183] [DEBUG] – exited code 0
[2023-05-16 18:07:38.183] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-05-16 18:07:38.188] [DEBUG] – exited code 0
[2023-05-16 18:07:38.189] [DEBUG] – local execute: ls /root/oceanbase/store/sstable/block_file
[2023-05-16 18:07:38.194] [DEBUG] – exited code 2, error output:
[2023-05-16 18:07:38.194] [DEBUG] ls: cannot access /root/oceanbase/store/sstable/block_file: No such file or directory
[2023-05-16 18:07:38.194] [DEBUG]
[2023-05-16 18:07:38.194] [DEBUG] – local execute: grep -e ‘^ *lo:’ /proc/net/dev
[2023-05-16 18:07:38.198] [DEBUG] – exited code 0
[2023-05-16 18:07:38.198] [DEBUG] – local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2023-05-16 18:07:38.201] [DEBUG] – exited code 0
[2023-05-16 18:07:38.201] [DEBUG] – local execute: ulimit -a
[2023-05-16 18:07:38.204] [DEBUG] – exited code 0
[2023-05-16 18:07:38.205] [WARNING] OBD-1007: (127.0.0.1) The recommended number of open files is 655350 (Current value: 102400)
[2023-05-16 18:07:38.205] [DEBUG] – local execute: cat /proc/meminfo
[2023-05-16 18:07:38.208] [DEBUG] – exited code 0
[2023-05-16 18:07:38.209] [DEBUG] – local execute: df --block-size=1024
[2023-05-16 18:07:38.213] [DEBUG] – exited code 0
[2023-05-16 18:07:38.214] [DEBUG] – get disk info for path /, total: 30511435776 avail: 15232610304
[2023-05-16 18:07:38.214] [DEBUG] – get disk info for path /dev, total: 8313335808 avail: 8313335808
[2023-05-16 18:07:38.214] [DEBUG] – get disk info for path /dev/shm, total: 8329363456 avail: 8329277440
[2023-05-16 18:07:38.214] [DEBUG] – get disk info for path /run, total: 8329363456 avail: 8320098304
[2023-05-16 18:07:38.214] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 8329363456 avail: 8329363456
[2023-05-16 18:07:38.215] [DEBUG] – get disk info for path /boot, total: 1023303680 avail: 804036608
[2023-05-16 18:07:38.215] [DEBUG] – get disk info for path /mnt/localyum, total: 8280440832 avail: 0
[2023-05-16 18:07:38.215] [DEBUG] – get disk info for path /run/user/42, total: 1665875968 avail: 1665859584
[2023-05-16 18:07:38.215] [DEBUG] – get disk info for path /run/user/0, total: 1665875968 avail: 1665875968
[2023-05-16 18:07:38.215] [DEBUG] – local execute: df --block-size=1024 /root/oceanbase/store
[2023-05-16 18:07:38.218] [DEBUG] – exited code 0
[2023-05-16 18:07:38.219] [DEBUG] – get disk info for path /, total: 30511435776 avail: 15232610304
[2023-05-16 18:07:38.219] [DEBUG] – local execute: df --block-size=1024 /root/oceanbase/store/clog
[2023-05-16 18:07:38.222] [DEBUG] – exited code 0
[2023-05-16 18:07:38.223] [DEBUG] – get disk info for path /, total: 30511435776 avail: 15232610304
[2023-05-16 18:07:38.223] [DEBUG] – disk: {’/’: {‘total’: 30511435776, ‘avail’: 15232610304, ‘need’: 0, ‘threshold’: 2}, ‘/dev’: {‘total’: 8313335808, ‘avail’: 8313335808, ‘need’: 0, ‘threshold’: 2}, ‘/dev/shm’: {‘total’: 8329363456, ‘avail’: 8329277440, ‘need’: 0, ‘threshold’: 2}, ‘/run’: {‘total’: 8329363456, ‘avail’: 8320098304, ‘need’: 0, ‘threshold’: 2}, ‘/sys/fs/cgroup’: {‘total’: 8329363456, ‘avail’: 8329363456, ‘need’: 0, ‘threshold’: 2}, ‘/boot’: {‘total’: 1023303680, ‘avail’: 804036608, ‘need’: 0, ‘threshold’: 2}, ‘/mnt/localyum’: {‘total’: 8280440832, ‘avail’: 0, ‘need’: 0, ‘threshold’: 2}, ‘/run/user/42’: {‘total’: 1665875968, ‘avail’: 1665859584, ‘need’: 0, ‘threshold’: 2}, ‘/run/user/0’: {‘total’: 1665875968, ‘avail’: 1665875968, ‘need’: 0, ‘threshold’: 2}}
[2023-05-16 18:07:38.223] [WARNING] OBD-1012: (127.0.0.1) clog and data use the same disk (/)
[2023-05-16 18:07:38.223] [DEBUG] – local execute: date +%s%N
[2023-05-16 18:07:38.226] [DEBUG] – exited code 0
[2023-05-16 18:07:38.226] [DEBUG] – 127.0.0.1 time delta -0.406494140625
[2023-05-16 18:07:38.303] [INFO] [WARN] OBD-1007: (127.0.0.1) The recommended number of open files is 655350 (Current value: 102400)
[2023-05-16 18:07:38.303] [INFO] [WARN] OBD-1012: (127.0.0.1) clog and data use the same disk (/)
[2023-05-16 18:07:38.303] [INFO]
[2023-05-16 18:07:38.304] [DEBUG] - sub start_check ref count to 0
[2023-05-16 18:07:38.304] [DEBUG] - export start_check
[2023-05-16 18:07:38.304] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-4.el7-f19f8bfb67723712175fb0dfd60579196b3168f1
[2023-05-16 18:07:38.304] [DEBUG] - import start
[2023-05-16 18:07:38.307] [DEBUG] - add start ref count to 1
[2023-05-16 18:07:38.307] [INFO] Start observer
[2023-05-16 18:07:38.308] [DEBUG] – local execute: ls /root/oceanbase/store/ilog/
[2023-05-16 18:07:38.312] [DEBUG] – exited code 0
[2023-05-16 18:07:38.312] [DEBUG] – local execute: cat /root/oceanbase/run/observer.pid
[2023-05-16 18:07:38.315] [DEBUG] – exited code 1, error output:
[2023-05-16 18:07:38.316] [DEBUG] cat: /root/oceanbase/run/observer.pid: No such file or directory
[2023-05-16 18:07:38.316] [DEBUG]
[2023-05-16 18:07:38.316] [DEBUG] – 127.0.0.1 start command construction
[2023-05-16 18:07:38.316] [DEBUG] – starting 127.0.0.1 observer
[2023-05-16 18:07:38.316] [DEBUG] – root@127.0.0.1 set env LD_LIBRARY_PATH to ‘/root/oceanbase/lib:’
[2023-05-16 18:07:38.317] [DEBUG] – local execute: cd /root/oceanbase; /root/oceanbase/bin/observer -r ‘127.0.0.1:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -c 1 -d ‘/root/oceanbase/store’ -i ‘lo’ -o __min_full_resource_pool_memory=268435456,memory_limit=‘4G’,system_memory=‘1G’,datafile_size=‘8G’,log_disk_size=‘2G’,cpu_count=8,production_mode=False,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4
[2023-05-16 18:07:38.397] [DEBUG] – exited code 0
[2023-05-16 18:07:38.398] [DEBUG] – root@127.0.0.1 set env LD_LIBRARY_PATH to ‘/tmp/_MEIdOG86K’
[2023-05-16 18:07:38.439] [INFO] observer program health check
[2023-05-16 18:07:41.442] [DEBUG] – 127.0.0.1 program health check
[2023-05-16 18:07:41.442] [DEBUG] – local execute: cat /root/oceanbase/run/observer.pid
[2023-05-16 18:07:41.446] [DEBUG] – exited code 1, error output:
[2023-05-16 18:07:41.447] [DEBUG] cat: /root/oceanbase/run/observer.pid: No such file or directory
[2023-05-16 18:07:41.447] [DEBUG]
[2023-05-16 18:07:41.568] [WARNING] OBD-2002: Failed to start 127.0.0.1 observer
[2023-05-16 18:07:41.568] [DEBUG] - sub start ref count to 0
[2023-05-16 18:07:41.568] [DEBUG] - export start
[2023-05-16 18:07:41.569] [ERROR] oceanbase-ce start failed
[2023-05-16 18:07:41.569] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2023-05-16 18:07:41.569] [INFO] Trace ID: 7be507d8-f3d1-11ed-bdfb-0050569b2460
[2023-05-16 18:07:41.569] [INFO] If you want to view detailed obd logs, please run: obd display-trace 7be507d8-f3d1-11ed-bdfb-0050569b2460
[2023-05-16 18:07:41.569] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2023-05-16 18:07:41.569] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2023-05-16 18:07:41.569] [DEBUG] - exclusive lock /root/.obd/lock/deploy_ob release, count 0
[2023-05-16 18:07:41.569] [DEBUG] - unlock /root/.obd/lock/deploy_ob
[2023-05-16 18:07:41.570] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2023-05-16 18:07:41.570] [DEBUG] - unlock /root/.obd/lock/global
部署配置如下：
oceanbase-ce:
servers:
# Please don’t use hostname, only IP can be supported
- 192.168.5.58
global:
# The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
home_path: /root/oceanbase
# The directory for data storage. The default value is $home_path/store.
# data_dir: /data
# The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
# redo_dir: /redo
# Please set devname as the network adaptor’s name whose ip is in the setting of severs.
# if set severs as “127.0.0.1”, please set devname as “lo”
# if current ip is 192.168.1.10, and the ip’s network adaptor’s name is “eth0”, please use “eth0”
devname: lo
mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
zone: zone1
cluster_id: 1
# please set memory limit to a suitable value which is matching resource.
memory_limit: 10G # The maximum running memory for an observer
system_memory: 2G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
datafile_size: 8G # Size of the data file.
log_disk_size: 2G # The size of disk space used by the clog files.
cpu_count: 1
production_mode: false
enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
max_syslog_file_count: 4 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
# root_password: # root user password, can be empty
【附件】
