observer启动报错

oceanvoice · 2022 年1 月 26 日 20:21

虚拟机内存16g

【问题描述】2022-01-26 20:01:00.009348] ERROR [SERVER] main (main.cpp:491) [2095][0][Y0-0000000000000000] [lt=25] observer init fail(ret=-4002) BACKTRACE:0x9786d2e 0x96d0bdb 0x22251fd 0x2223ee8 0x222234a 0x2b4e61072555 0x2220d29

正经点 · 2022 年1 月 26 日 20:26

您好，这边有详细的日志吗，根据这个报错排查不出问题

oceanvoice · 2022 年1 月 26 日 20:46

抱歉, 刚描述的不是很清楚！

环境

=========================

配置项配置值备注

CPU 2[逻辑] vmware

内存 16G -

硬盘 40G -

操作系统 CentOS7.9 -

操作步骤

=========================

echo 'fs.aio-max-nr=1048576' >> /etc/sysctl.conf

sysctl -p

vim /etc/security/limits.conf

soft nofile 655360
hard nofile 655360
soft nproc 655360
hard nproc 655360
soft core unlimited
hard core unlimited
soft stack unlimited
hard stack unlimited

ulimit -a

useradd admin

passwd admin #admin

usermod admin -G wheel

wget https://mirrors.aliyun.com/oceanbase/community/stable/el/7/x86_64/oceanbase-ce-libs-3.1.1-4.el7.x86_64.rpm

wget https://mirrors.aliyun.com/oceanbase/community/stable/el/7/x86_64/oceanbase-ce-3.1.1-4.el7.x86_64.rpm

wget https://mirrors.aliyun.com/oceanbase/community/stable/el/7/x86_64/obproxy-3.2.0-1.el7.x86_64.rpm

wget https://mirrors.aliyun.com/oceanbase/community/stable/el/7/x86_64/libobclient-2.0.0-2.el7.x86_64.rpm

wget https://mirrors.aliyun.com/oceanbase/community/stable/el/7/x86_64/obclient-2.0.0-2.el7.x86_64.rpm

yum localinstall *.rpm -y

#安装ok

[root@master01 ~]# ll /home/admin/

drwxr-xr-x 3 admin admin 17 Jan 26 19:40 obproxy-3.2.0

drwxr-xr-x 5 root root 39 Jan 26 19:40 oceanbase

##建立数据目录

sudo su - admin

sudo mkdir -p /ob/obdemo/

sudo chown -R admin.admin /ob/

echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/oceanbase/lib' >> ~/.bash_profile

source ~/.bash_profile

cd /home/admin/oceanbase/

./bin/observer -i ens33 -p 2881 -P 2882 -z zone1 -d /ob/obdemo/ -r '192.168.33.128:2882:2881' -c 4444 -n vividvoice -o "memory_limit=8G,cache_wash_threshold=1G,__min_full_resource_pool_memory=268435456,system_memory=3G,memory_chunk_cache_size=128M,cpu_count=1,net_thread_count=4,datafile_size=6G,stack_size=1536K,config_additional_dir=/ob/obdemo"

#这步操作之后报错

=========================

日志见附件

observer.zip (56328 KB)

oceanvoice · 2022 年1 月 26 日 21:07

/bin/observer 启动的时候哪些参数是必须的, 只要带上必要的就可以了,其余使用默认就可以....

hope2021 · 2022 年1 月 26 日 21:21

第一次启动会带很多参数：

cd /home/admin/oceanbase/

./bin/observer -i ens33 -p 2881 -P 2882 -z zone1 -d /ob/obdemo/ -r '192.168.33.128:2882:2881' -c 4444 -n vividvoice -o "memory_limit=8G,cache_wash_threshold=1G,__min_full_resource_pool_memory=268435456,system_memory=3G,memory_chunk_cache_size=128M,cpu_count=1,net_thread_count=4,datafile_size=6G,stack_size=1536K,config_additional_dir=/ob/obdemo"

hope2021 · 2022 年1 月 26 日 21:22

第二次启动：

cd /home/admin/oceanbase/ && ./bin/observer 即可

oceanvoice · 2022 年1 月 26 日 21:24

not ok!

[2022-01-26 21:24:04.623243] ERROR [SERVER] main (main.cpp:491) [20508][0][Y0-0000000000000000] [lt=3] [dc=0] observer init fail(ret=-4337) BACKTRACE:0x9786d2e 0x96d9aa1 0x2225

a04 0x22254eb 0x2225251 0x2223ee8 0x222234a 0x2ba1bd756555 0x2220d29

[2022-01-26 21:24:04.623307] INFO [COMMON] ob_kv_storecache.cpp:215 [20508][0][Y0-0000000000000000] [lt=63] [dc=0] Begin destroy the ObKVGlobalCache!

[2022-01-26 21:24:04.623322] INFO ob_timer.cpp:234 [20508][0][Y0-0000000000000000] [lt=3] [dc=0] cancel task(this=0x2ba1c33ac510, thread_id=20557, pos=0, wakeup_time_=65334029

40, tasks_num_=0, task=tasktype:N9oceanbase6common15ObKVGlobalCache15KVStoreWashTaskE)

[2022-01-26 21:24:04.623327] INFO ob_timer.cpp:234 [20508][0][Y0-0000000000000000] [lt=3] [dc=0] cancel task(this=0x2ba1c33ac950, thread_id=20558, pos=0, wakeup_time_=65333668

40, tasks_num_=0, task=tasktype:N9oceanbase6common15ObKVGlobalCache16KVMapReplaceTaskE)

[2022-01-26 21:24:04.623331] INFO ob_timer.cpp:107 [20508][0][Y0-0000000000000000] [lt=3] [dc=0] ObTimer destroy(this=0x2ba1c33ac510, thread_id=20557)

[2022-01-26 21:24:04.623343] INFO ob_timer.cpp:366 [20557][96][Y0-0000000000000000] [lt=3] [dc=0] timer thread exit(this=0x2ba1c33ac510, thread_id=20557)

[2022-01-26 21:24:04.623443] INFO ob_page_manager.cpp:43 [20557][0][Y0-0000000000000000] [lt=5] [dc=0] unregister pm finish(&pm=0x2ba1fe372900, pm.get_tid()=20557)

[2022-01-26 21:24:04.623973] INFO ob_timer.cpp:107 [20508][0][Y0-0000000000000000] [lt=2] [dc=0] ObTimer destroy(this=0x2ba1c33ac950, thread_id=20558)

[2022-01-26 21:24:04.624150] INFO ob_timer.cpp:366 [20558][98][Y0-0000000000000000] [lt=5] [dc=0] timer thread exit(this=0x2ba1c33ac950, thread_id=20558)

[2022-01-26 21:24:04.624339] INFO ob_page_manager.cpp:43 [20558][0][Y0-0000000000000000] [lt=2] [dc=0] unregister pm finish(&pm=0x2ba1fe798900, pm.get_tid()=20558)

[2022-01-26 21:24:04.665371] INFO [COMMON] ob_kv_storecache.cpp:233 [20508][0][Y0-0000000000000000] [lt=48] [dc=0] The ObKVGlobalCache has been destroyed!

[2022-01-26 21:24:04.665477] INFO [SERVER] main.cpp:505 [20508][0][Y0-0000000000000000] [lt=57] [dc=0] observer is exit(observer_version="OceanBase CE 3.1.1")

正经点 · 2022 年1 月 26 日 21:28

1.是否使用admin用户起的

2.把-d /ob/obdemo/改成-d /ob/obdemo

3.根据文档进行排查https://www.oceanbase.com/docs/knowledgeBase/observer-startup-failed-error-code-4337

hope2021 · 2022 年1 月 26 日 21:30

从提的日志看，好像第一次启动参数有问题

oceanvoice · 2022 年1 月 26 日 22:00

##建立数据目录

sudo su - admin

sudo mkdir -p /home/admin/oceanbase/store/{sstable,slog,clog,ilog,sort_dir}

sudo chown -R admin:admin /home/admin/oceanbase/store/

sudo mkdir -p /data/vividvoice/{sstable,slog,clog,ilog,sort_dir}

sudo chown -R admin:admin /data/vividvoice

cd /home/admin/oceanbase/store/

ln -s /home/admin/oceanbase/store/clog /data/vividvoice/clog

ln -s /home/admin/oceanbase/store/ilog /data/vividvoice/ilog

ln -s /home/admin/oceanbase/store/slog /data/vividvoice/slog

ln -s /home/admin/oceanbase/store/sort_dir /data/vividvoice/sort_dir

ln -s /home/admin/oceanbase/store/sstable /data/vividvoice/sstable

echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/oceanbase/lib' >> ~/.bash_profile

source ~/.bash_profile

cd /home/admin/oceanbase/

./bin/observer -i ens33 -p 2881 -P 2882 -z zone1 -d /home/admin/oceanbase/store -r '192.168.33.128:2882:2881' -c 4444 -n vividvoice -o "memory_limit=8G,cache_wash_threshold=1G,__min_full_resource_pool_memory=10G,system_memory=3G,memory_chunk_cache_size=128M,cpu_count=1,net_thread_count=4,datafile_size=6G,stack_size=1536K,config_additional_dir=/home/admin/oceanbase/store"

文档中心没有关于4147解析

observer.zip (11469 KB)

正经点 · 2022 年1 月 26 日 22:06

4147无效的配置，应该是参数的问题https://www.oceanbase.com/docs/oceanbase-database/oceanbase-database-2-1/V2.1/tslkmg

-d /home/admin/oceanbase/store我看上边的数据目录不是这个目录吧应该是-d /ob/obdemo

oceanvoice · 2022 年1 月 26 日 22:13

我重建数据目录了

正经点 · 2022 年1 月 26 日 22:15

如果是第一次安装的话可以参考这个文档https://open.oceanbase.com/blog/1100243

oceanvoice · 2022 年1 月 27 日 09:56

已经换了两个文档了

这次报这个错

./bin/observer -i ens33 -p 2881 -P 2882 -z zone1 -d /home/admin/oceanbase/store -r '192.168.33.128:2882:2881' -c 4444 -n vividvoice -o "memory_limit=8G,cache_wash_threshold=1G,__min_full_resource_pool_memory=1073741824,system_memory=3G,memory_chunk_cache_size=128M,cpu_count=2,net_thread_count=4,datafile_size=6G,stack_size=1536K,sys_bkgd_io_low_percentage=10,sys_bkgd_io_high_percentage=50,config_additional_dir=/home/admin/oceanbase/store"

[2022-01-27 09:53:46.089604] WARN [SERVER] destroy (ob_server.cpp:434) [68176][0][Y0-0000000000000000] [lt=3] [dc=0] log compressor destroyed

[2022-01-27 09:53:46.089615] WARN [SERVER] destroy (ob_server.cpp:435) [68176][0][Y0-0000000000000000] [lt=10] [dc=0] destroy observer end

[2022-01-27 09:53:46.089619] ERROR [SERVER] main (main.cpp:491) [68176][0][Y0-0000000000000000] [lt=3] [dc=0] observer init fail(ret=-4002) BACKTRACE:0x9786d2e 0x96d9aa1 0x2225a04 0x22254eb 0x2225251 0x2223ee8 0x222234a 0x2ab905c86555 0

x2220d29

[2022-01-27 09:53:46.089653] INFO [COMMON] ob_kv_storecache.cpp:215 [68176][0][Y0-0000000000000000] [lt=33] [dc=0] Begin destroy the ObKVGlobalCache!

[2022-01-27 09:53:46.090022] INFO ob_timer.cpp:234 [68176][0][Y0-0000000000000000] [lt=197] [dc=0] cancel task(this=0x2ab90b8dc510, thread_id=68225, pos=0, wakeup_time_=51514771292, tasks_num_=0, task=tasktype:N9oceanbase6common15ObKVG

lobalCache15KVStoreWashTaskE)

和顺 · 2022 年1 月 27 日 09:58

可以参考：https://open.oceanbase.com/docs/tutorials/quickstart/V1.0.0/2-11-advanced-how-to-manually-deploy-an-oceanbase-cluster

日志中错误的地方提示如标红的所示：

[2022-01-26 21:53:24.546493] ERROR get (ob_config_helper.cpp:330) [22918][0][Y0-0000000000000000] [lt=3] set int error(str="10G", valid=false) BACKTRACE:0x9786d2e 0x96d0bdb 0x22602e3 0x22a4a16 0x227b97e 0x227b796 0x229ba57 0x229c93c 0x22753a7 0x913c825 0x9138a6f 0x2222045 0x2b008cb00555 0x2220d29

[2022-01-26 21:53:24.547108] ERROR parse (ob_config.cpp:541) [22918][0][Y0-0000000000000000] [lt=609] set int error(name="__min_full_resource_pool_memory", str="10G", valid=false) BACKTRACE:0x9786d2e 0x96d0bdb 0x22622cc 0x22a0838 0x227b840 0x229ba57 0x229c93c 0x22753a7 0x913c825 0x9138a6f 0x2222045 0x2b008cb00555 0x2220d29

[2022-01-26 21:53:24.547140] WARN [SHARE] add_extra_config (ob_common_config.cpp:97) [22918][0][Y0-0000000000000000] [lt=30] Invalid config value(name="__min_full_resource_pool_memory", value="10G", ret=-4147)

[2022-01-26 21:53:24.547161] ERROR [SERVER] init_config (ob_server.cpp:854) [22918][0][Y0-0000000000000000] [lt=7] invalid config from cmdline

oceanvoice · 2022 年1 月 27 日 10:24

去掉了不行

oceanvoice · 2022 年1 月 27 日 11:32

启动OK！

总结如下:

误解:虚拟机多大内存(CPU),observer设置不能等于该内存(CPU)阈值(一直认为需要给系统预留资源)

比如:虚拟机16G内存(完整),CPU4个(logical)

然而执行observer启动命令如下启动ok:(cpu_count=4)

./bin/observer -i ens33 -p 2881 -P 2882 -z zone1 -d /home/admin/oceanbase/store -r '192.168.33.128:2882:2881' -c 4444 -n vividvoice -o

"memory_limit=14G,

cache_wash_threshold=1G,system_memory=1G,memory_chunk_cache_size=128M,

cpu_count=4,

net_thread_count=4,datafile_size=6G,stack_size=1536K,sys_bkgd_io_low_percentage=10,sys_bkgd_io_high_percentage=90,config_additional_dir=/home/admin/oceanbase/store"

具体参数含义后续看情况了解。

oceanvoice · 2022 年1 月 27 日 12:58

admin@master01 oceanbase]$ obclient -h127.0.0.1 -u root -P2881 -p -c -A

Enter password:

Welcome to the OceanBase. Commands end with ; or \g.

Your MySQL connection id is 3221225472

Server version: 5.7.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>

MySQL [(none)]> set session ob_query_timeout=1000000000; alter system bootstrap ZONE 'zone1' SERVER '192.168.33.128:2882';

ERROR 1235 (0A000): unit min memory less than __min_full_resource_pool_memory not supported

MySQL [(none)]> alter user root identified by '123456'; ##这个命令一直卡着不动

/home/admin/oceanbase/etc/observer.config.bin ##参数持久化在这里? 能不能编辑，

和顺 · 2022 年1 月 27 日 16:09

__min_full_resource_pool_memory 默认值是5G，如果我们测试使用的资源紧张，需要在启动的时候调整一下这个内部的参数。

observer.config.bin 不允许直接去修改改，alter system方式或者obd edit-config方式修改后会持久化到这个文件里，可以通过strings命令来查看。