我参照 实战教程第二章2.7：如何使用 OBD 自动化部署多节点集群 部署CE-4.0版本遇到一个问题想请教一下
我的环境是三台observer：8C16G；一台obproxy：8C16G，obproxy做中控机用obd部署
我的yaml配置文件参照的是教程内的，里面有一条
- 配置文件包含__min_full_resource_pool_memory参数
oceabase.yaml ... __min_full_resource_pool_memory: 268435456 ...
在deploy的时候没有报错，成功完成，但是在启动集群的时候，三台observer都启动失败，手动启动查看日志发现是这个参数是无效参数。
--------------observer log --------------
[2022-11-10 21:15:31.582678] ERROR [SHARE] operator() (ob_common_config.cpp:124) [12607][][T0][Y0-0000000000000000-0-0] [lt=0] Invalid config, value out of range(name="__min_full_resource_pool_memory", value="268435456", ret=-4147) BACKTRACE:0xb553efb 0xb5459d6 0x3c3bfda 0x3c3bcf9 0x3c3bb00 0x3c3b952 0x9a21ee4 0x99fa59f 0x99f9d88 0x5bd2d95 0x5bd1386 0x3c173fc 0x2b71c42d8555 0x3c16184
[2022-11-10 21:15:31.583272] ERROR [SERVER] init_config (ob_server.cpp:1275) [12607][][T0][Y0-0000000000000000-0-0] [lt=585] invalid config from cmdline options(opts_.optstr_="__min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=3G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=8,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=20G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10", ret=-4147, ret="OB_INVALID_CONFIG") BACKTRACE:0xb553efb 0xb5459d6 0x3c3bfda 0x3c3bcf9 0x3c3bb00 0x3c3b952 0x5bf1552 0x5bd39cf 0x5bd1386 0x3c173fc 0x2b71c42d8555 0x3c16184
[2022-11-10 21:15:31.583335] ERROR [SERVER] init (ob_server.cpp:178) [12607][][T0][Y0-0000000000000000-0-0] [lt=39] init config failed(ret=-4147, ret="OB_INVALID_CONFIG") BACKTRACE:0xb553efb 0xb5459d6 0x3c42f9b 0x3c42cb4 0x3c42ac9 0x3c2137b 0x5bd437d 0x5bd1ebb 0x3c173fc 0x2b71c42d8555 0x3c16184
[2022-11-10 21:15:31.583669] WARN [SHARE] add_event (ob_event_history_table_operator.h:373) [12607][][T0][Y0-0000000000000000-0-0] [lt=37] not init(ret=-4006)
[2022-11-10 21:15:31.583802] WARN [SERVER] destroy (ob_service.cpp:349) [12607][][T0][Y0-0000000000000000-0-0] [lt=10] ob_service not init(ret=-4006, ret="OB_NOT_INIT", inited=false)
[2022-11-10 21:15:31.583896] WARN [COMMON] wrlock_all (ob_bucket_lock.cpp:220) [12607][][T0][Y0-0000000000000000-0-0] [lt=11] The ObBucketLock has not been inited, (ret=-4006)
[2022-11-10 21:15:31.583910] WARN [COMMON] ObBucketWLockAllGuard (ob_bucket_lock.h:159) [12607][][T0][Y0-0000000000000000-0-0] [lt=13] Fail to try write lock all buckets(ret=-4006)
[2022-11-10 21:15:31.583973] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=10] hashtable not init, backtrace=0xb553efb 0x8f418e6 0x8ef2e47 0x8eecac2 0x5bd0b7f 0x5bd1f16 0x3c173fc 0x2b71c42d8555 0x3c16184
[2022-11-10 21:15:31.584062] WARN foreach_refactored (ob_hashtable.h:1307) [12607][][T0][Y0-0000000000000000-0-0] [lt=9] hashtable not init
[2022-11-10 21:15:31.584076] WARN foreach_refactored (ob_hashtable.h:1307) [12607][][T0][Y0-0000000000000000-0-0] [lt=8] hashtable not init
[2022-11-10 21:15:31.584093] WARN [COMMON] get_instance (memory_dump.cpp:94) [12607][][T0][Y0-0000000000000000-0-0] [lt=5] memory dump not init
[2022-11-10 21:15:31.584144] WARN [CLOG] destroy (ob_server_log_block_mgr.cpp:116) [12607][][T0][Y0-0000000000000000-0-0] [lt=5] ObServerLogBlockMgr destroy(this={dir::"", dir_fd:-1, meta_fd:-1, log_pool_meta:{curr_total_size:0, next_total_size:0, status:0}, min_block_id:0, max_block_id:0, is_inited:false})
[2022-11-10 21:15:31.584192] ERROR [SERVER] init (ob_server.cpp:374) [12607][][T0][Y0-0000000000000000-0-0] [lt=31] [OBSERVER_NOTICE] fail to init observer(ret=-4147, ret="OB_INVALID_CONFIG") BACKTRACE:0xb553efb 0xb5459d6 0x3c42f9b 0x3c42cb4 0x3c42ac9 0x3c2137b 0x5bdd615 0x5bd1f27 0x3c173fc 0x2b71c42d8555 0x3c16184
[2022-11-10 21:15:31.584212] ERROR [SERVER] main (main.cpp:529) [12607][][T0][Y0-0000000000000000-0-0] [lt=20] observer init fail(ret=-4147) BACKTRACE:0xb553efb 0xb5459d6 0x3c1bfe0 0x3c1bd09 0x3c1bb28 0x3c163dc 0x3c1840d 0x3c17645 0x2b71c42d8555 0x3c16184
[2022-11-10 21:15:31.584237] WARN [SERVER] destroy (ob_server.cpp:542) [12607][][T0][Y0-0000000000000000-0-0] [lt=25] [OBSERVER_NOTICE] can not destroy observer(has_destroy=true, has_stopped=true)
[2022-11-10 21:15:31.584916] WARN foreach_refactored (ob_hashtable.h:1307) [12607][][T0][Y0-0000000000000000-0-0] [lt=15] hashtable not init
[2022-11-10 21:15:31.584928] WARN foreach_refactored (ob_hashtable.h:1307) [12607][][T0][Y0-0000000000000000-0-0] [lt=12] hashtable not init
[2022-11-10 21:15:31.584951] WARN [COMMON] wrlock_all (ob_bucket_lock.cpp:220) [12607][][T0][Y0-0000000000000000-0-0] [lt=8] The ObBucketLock has not been inited, (ret=-4006)
[2022-11-10 21:15:31.584961] WARN [COMMON] ObBucketWLockAllGuard (ob_bucket_lock.h:159) [12607][][T0][Y0-0000000000000000-0-0] [lt=9] Fail to try write lock all buckets(ret=-4006)
[2022-11-10 21:15:31.584993] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=6] hashtable not init, backtrace=0xb553efb 0x8f418e6 0x8ef2e47 0x8eecac2 0x8ef2bfd 0x2b71c42efce9 0x2b71c42efd37 0x2b71c42d855c 0x3c16184
[2022-11-10 21:15:31.585097] WARN [SERVER] destroy (ob_server.cpp:542) [12607][][T0][Y0-0000000000000000-0-0] [lt=5] [OBSERVER_NOTICE] can not destroy observer(has_destroy=true, has_stopped=true)
[2022-11-10 21:15:31.585106] WARN [CLOG] destroy (ob_server_log_block_mgr.cpp:116) [12607][][T0][Y0-0000000000000000-0-0] [lt=8] ObServerLogBlockMgr destroy(this={dir::"", dir_fd:-1, meta_fd:-1, log_pool_meta:{curr_total_size:0, next_total_size:0, status:0}, min_block_id:0, max_block_id:0, is_inited:false})
[2022-11-10 21:15:31.585573] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=10] hashtable not init, backtrace=0xb553efb 0x4d4ee40 0x4c6cbdf 0x4d958c7 0x5bcfc72 0x2b71c42efce9 0x2b71c42efd37 0x2b71c42d855c 0x3c16184
[2022-11-10 21:15:31.585588] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=8] hashtable not init, backtrace=0xb553efb 0x4d4ef7c 0x4c6cc8f 0x4d958c7 0x5bcfc72 0x2b71c42efce9 0x2b71c42efd37 0x2b71c42d855c 0x3c16184
[2022-11-10 21:15:31.585599] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=4] hashtable not init, backtrace=0xb553efb 0x4d4f0b8 0x4c6cd3f 0x4d958c7 0x5bcfc72 0x2b71c42efce9 0x2b71c42efd37 0x2b71c42d855c 0x3c16184
[2022-11-10 21:15:31.585607] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=4] hashtable not init, backtrace=0xb553efb 0x4d4ee40 0x4c6cdef 0x4d958c7 0x5bcfc72 0x2b71c42efce9 0x2b71c42efd37 0x2b71c42d855c 0x3c16184
[2022-11-10 21:15:31.585616] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=4] hashtable not init, backtrace=0xb553efb 0x4d4f1f4 0x4c6ce9f 0x4d958c7 0x5bcfc72 0x2b71c42efce9 0x2b71c42efd37 0x2b71c42d855c 0x3c16184
[2022-11-10 21:15:31.586373] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=13] hashtable not init, backtrace=0xb553efb 0xb1ae83e 0xb0ef8f5 0xb0ef60d 0x2b71c42efce9 0x2b71c42efd37 0x2b71c42d855c 0x3c16184
[2022-11-10 21:15:31.586415] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=17] hashtable not init, backtrace=0xb553efb 0x98d63b8 0xb0ef9f5 0xb0ef60d 0x2b71c42efce9 0x2b71c42efd37 0x2b71c42d855c 0x3c16184
[2022-11-10 21:15:31.586425] WARN begin (ob_hashtable.h:915) [12607][][T0][Y0-0000000000000000-0-0] [lt=6] hashtable not init, backtrace=0xb553efb 0x98d63b8 0xb0efaf5 0xb0ef60d 0x2b71c42efce9 0x2b71c42efd37 0x2b71c42d855c 0x3c16184
- 配置文件取消__min_full_resource_pool_memory参数
oceabase.yaml ... # __min_full_resource_pool_memory: 268435456 ...
然后我在集群配置文件内注释掉这个参数再次部署也没有报错，再次启动的时候还是同样observer启动失败。obd日志里发现发送到observer机器的启动命令里还是存在这个参数。
--------------------- obd-log ---------------------
[2022-11-10 21:18:43.905] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- admin@10.51.210.101 execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '10.51.210.101:2882:2881;10.51.210.102:2882:2881;10.51.210.103:2882:2881' -p 2881 -P 2882 -z 'zone1' -n 'obcluster' -c 2 -d '/ob-data' -i 'eth0' -l 'WARN' -o __min_full_resource_pool_memory=268435456,memory_limit='8G',system_memory='3G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=8,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size='20G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10
[2022-11-10 21:18:44.029] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- exited code 0
[2022-11-10 21:18:44.029] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- admin@10.51.210.101 delete env LD_LIBRARY_PATH
[2022-11-10 21:18:44.030] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- starting obce02(10.51.210.102) observer
[2022-11-10 21:18:44.030] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- admin@10.51.210.102 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:'
[2022-11-10 21:18:44.030] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- admin@10.51.210.102 execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '10.51.210.101:2882:2881;10.51.210.102:2882:2881;10.51.210.103:2882:2881' -p 2881 -P 2882 -z 'zone2' -n 'obcluster' -c 2 -d '/ob-data' -i 'eth0' -l 'WARN' -o **__min_full_resource_pool_memory=268435456**,memory_limit='8G',system_memory='3G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=8,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size='20G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10
[2022-11-10 21:18:44.137] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- exited code 0
[2022-11-10 21:18:44.137] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- admin@10.51.210.102 delete env LD_LIBRARY_PATH
[2022-11-10 21:18:44.141] [31496b70-60fa-11ed-8490-00505682e16b] [INFO] observer program health check
[2022-11-10 21:18:47.145] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- obce01(10.51.210.101) program health check
[2022-11-10 21:18:47.145] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- admin@10.51.210.101 execute: cat /home/admin/oceanbase-ce/run/observer.pid
[2022-11-10 21:18:47.161] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- exited code 1, error output:
[2022-11-10 21:18:47.161] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory
[2022-11-10 21:18:47.161] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG]
[2022-11-10 21:18:47.162] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- obce02(10.51.210.102) program health check
[2022-11-10 21:18:47.162] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- admin@10.51.210.102 execute: cat /home/admin/oceanbase-ce/run/observer.pid
[2022-11-10 21:18:47.177] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- exited code 1, error output:
[2022-11-10 21:18:47.178] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory
[2022-11-10 21:18:47.178] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG]
[2022-11-10 21:18:47.178] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- obce03(10.51.210.103) program health check
[2022-11-10 21:18:47.178] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- admin@10.51.210.103 execute: cat /home/admin/oceanbase-ce/run/observer.pid
[2022-11-10 21:18:47.197] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- exited code 0
[2022-11-10 21:18:47.198] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- admin@10.51.210.103 execute: ls /proc/12671
[2022-11-10 21:18:47.256] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- exited code 0
[2022-11-10 21:18:47.256] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] -- obce03(10.51.210.103) observer[pid: 12671] started
[2022-11-10 21:18:47.271] [31496b70-60fa-11ed-8490-00505682e16b] [WARNING] [WARN] OBD-2002: Failed to start obce01(10.51.210.101) observer
[2022-11-10 21:18:47.271] [31496b70-60fa-11ed-8490-00505682e16b] [WARNING] [WARN] OBD-2002: Failed to start obce02(10.51.210.102) observer
[2022-11-10 21:18:47.271] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] - sub start ref count to 0
[2022-11-10 21:18:47.271] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] - export start
[2022-11-10 21:18:47.271] [31496b70-60fa-11ed-8490-00505682e16b] [ERROR] [ERROR] oceanbase-ce start failed
[2022-11-10 21:18:47.271] [31496b70-60fa-11ed-8490-00505682e16b] [INFO] See https://open.oceanbase.com/docs/obd-cn/V1.4.0/10000000000436999 .
[2022-11-10 21:18:47.271] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 1
[2022-11-10 21:18:47.272] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] - share lock /home/admin/.obd/lock/mirror_and_repo release, count 0
[2022-11-10 21:18:47.272] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] - unlock /home/admin/.obd/lock/mirror_and_repo
[2022-11-10 21:18:47.272] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] - exclusive lock /home/admin/.obd/lock/deploy_obcluster release, count 0
[2022-11-10 21:18:47.272] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] - unlock /home/admin/.obd/lock/deploy_obcluster
[2022-11-10 21:18:47.272] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] - share lock /home/admin/.obd/lock/global release, count 0
[2022-11-10 21:18:47.272] [31496b70-60fa-11ed-8490-00505682e16b] [DEBUG] - unlock /home/admin/.obd/lock/global
[2022-11-10 21:19:22.506] [4a6c4afa-60fa-11ed-815a-00505682e16b] [DEBUG] - mkdir /home/admin/.obd/lock/
[2022-11-10 21:19:22.507] [4a6c4afa-60fa-11ed-815a-00505682e16b] [DEBUG] - try to get share lock /home/admin/.obd/lock/global
[2022-11-10 21:19:22.507] [4a6c4afa-60fa-11ed-815a-00505682e16b] [DEBUG] - share lock `/home/admin/.obd/lock/global`, count 1
[2022-11-10 21:19:22.507] [4a6c4afa-60fa-11ed-815a-00505682e16b] [DEBUG] - cmd: ['obcluster']
想问下这是我的哪步操作不对吗？ 我后续是通过去掉这个参数，手动启动每个observer，然后再在中控机上成功启动了集群。谢谢