集群启动失败 observice起不起来，observer 状态不正常

egegredsfew · 2023 年11 月 10 日 10:49

容器日志：
2023-11-10T10:41:31.08526+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 44.549µs, comment

2023-11-10T10:41:32.37072+08:00 INFO [7,] caller=shell/exec.go:97:execute: execute shell command start, command=Command{user=admin, program=sh, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD=’’ /home/admin/oceanbase/bin/observer, timeout=10s}
2023/11/10 10:41:32 observer 466 [running]
2023-11-10T10:41:32.46397+08:00 INFO [7,] caller=shell/exec.go:129:execute: execute shell command end, command=Command{user=admin, program=sh, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD=’’ /home/admin/oceanbase/bin/observer, timeout=10s}
2023-11-10T10:41:33.08653+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 74.01µs, comment
2023-11-10T10:41:35.08587+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 75.605µs, comment
2023-11-10T10:41:37.08581+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 60.532µs, comment
2023-11-10T10:41:39.08584+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 82.137µs, comment
2023-11-10T10:41:41.08591+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 43.911µs, comment

2023-11-10T10:41:42.37022+08:00 ERROR [7,] caller=observer/check.go:58:checkerObserver: observer process not running, try restart…

rootservice.log：
[2023-11-10 10:45:09.038468] INFO [RS] destroy (ob_root_service.cpp:1064) [788][][T0][Y0-0000000000000000-0-0] [lt=4] ObDBMSSchedJobMaster destory
[2023-11-10 10:45:09.038477] INFO [RS] destroy (ob_root_service.cpp:1066) [788][][T0][Y0-0000000000000000-0-0] [lt=5] global ctx timer destroyed
[2023-11-10 10:45:09.038480] INFO [RS] destroy (ob_root_service.cpp:1075) [788][][T0][Y0-0000000000000000-0-0] [lt=3] [ROOTSERVICE_NOTICE] destroy rootservice end(ret=0, ret=“OB_SUCCESS”)
[2023-11-10 10:45:09.040653] INFO [RS] stop (ob_disaster_recovery_task_table_updater.cpp:190) [788][][T0][Y0-0000000000000000-0-0] [lt=3] stop ObDRTaskTableUpdater success
[2023-11-10 10:45:09.040659] INFO [RS] wait (ob_disaster_recovery_task_table_updater.cpp:196) [788][][T0][Y0-0000000000000000-0-0] [lt=7] wait ObDRTaskTableUpdater
[2023-11-10 10:45:29.036691] INFO [RS] destroy (ob_root_service.cpp:974) [808][][T0][Y0-0000000000000000-0-0] [lt=3] [ROOTSERVICE_NOTICE] start to destroy rootservice
[2023-11-10 10:45:29.036706] INFO [RS] destroy (ob_root_service.cpp:982) [808][][T0][Y0-0000000000000000-0-0] [lt=10] start destroy archive_service_
[2023-11-10 10:45:29.036714] INFO [RS] destroy (ob_root_service.cpp:987) [808][][T0][Y0-0000000000000000-0-0] [lt=5] finish destroy archive_service_
[2023-11-10 10:45:29.036722] INFO [RS] destroy (ob_rs_reentrant_thread.cpp:115) [808][][T0][Y0-0000000000000000-0-0] [lt=3] rs_monitor_check : reentrant thread check unregister success(thread_name="", last_run_timestamp=0)
[2023-11-10 10:45:29.036730] INFO [RS] destroy (ob_root_service.cpp:995) [808][][T0][Y0-0000000000000000-0-0] [lt=9] root balance destroy
[2023-11-10 10:45:29.036734] INFO [RS] destroy (ob_root_service.cpp:1002) [808][][T0][Y0-0000000000000000-0-0] [lt=4] empty server checker destroy
[2023-11-10 10:45:29.036739] INFO [RS] destroy (ob_root_service.cpp:1009) [808][][T0][Y0-0000000000000000-0-0] [lt=5] rs_monitor_check : thread checker destroy
[2023-11-10 10:45:29.036744] INFO [RS] destroy (ob_root_service.cpp:1015) [808][][T0][Y0-0000000000000000-0-0] [lt=5] schema history recycler destroy
[2023-11-10 10:45:29.036751] INFO [RS] destroy (ob_root_service.cpp:1019) [808][][T0][Y0-0000000000000000-0-0] [lt=5] inner queue destroy
[2023-11-10 10:45:29.036754] INFO [RS] destroy (ob_root_service.cpp:1021) [808][][T0][Y0-0000000000000000-0-0] [lt=3] inspect queue destroy
[2023-11-10 10:45:29.036759] INFO [RS] destroy (ob_root_service.cpp:1023) [808][][T0][Y0-0000000000000000-0-0] [lt=3] ddl builder destroy
[2023-11-10 10:45:29.036764] INFO [RS] destroy (ob_rs_reentrant_thread.cpp:115) [808][][T0][Y0-0000000000000000-0-0] [lt=5] rs_monitor_check : reentrant thread check unregister success(thread_name="", last_run_timestamp=0)
[2023-11-10 10:45:29.036770] INFO [RS] destroy (ob_root_service.cpp:1028) [808][][T0][Y0-0000000000000000-0-0] [lt=4] heartbeat checker destroy
[2023-11-10 10:45:29.036779] INFO [RS] destroy (ob_root_service.cpp:1032) [808][][T0][Y0-0000000000000000-0-0] [lt=5] event table operator destroy
[2023-11-10 10:45:29.036788] INFO [RS] destroy (ob_rs_reentrant_thread.cpp:115) [808][][T0][Y0-0000000000000000-0-0] [lt=8] rs_monitor_check : reentrant thread check unregister success(thread_name="", last_run_timestamp=0)
[2023-11-10 10:45:29.036791] INFO [RS] destroy (ob_root_service.cpp:1039) [808][][T0][Y0-0000000000000000-0-0] [lt=3] root backup task scheduler destroy
[2023-11-10 10:45:29.036794] INFO [RS] destroy (ob_root_service.cpp:1046) [808][][T0][Y0-0000000000000000-0-0] [lt=3] root backup mgr destroy
[2023-11-10 10:45:29.036799] INFO [RS] destroy (ob_root_service.cpp:1049) [808][][T0][Y0-0000000000000000-0-0] [lt=5] start destroy backup_lease_service_
[2023-11-10 10:45:29.036807] INFO [RS] destroy (ob_backup_lease_service.cpp:171) [808][][T0][Y0-0000000000000000-0-0] [lt=5] start destroy ObBackupLeaseService
[2023-11-10 10:45:29.036827] WDIAG [RS] stop_lease (ob_backup_lease_service.cpp:133) [808][][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4006] not inited(ret=-4006)
[2023-11-10 10:45:29.036847] INFO [RS] wait_backup_scheduler_stop_ (ob_backup_lease_service.cpp:274) [808][][T0][Y0-0000000000000000-0-0] [lt=5] [BACKUP_LEASE] start wait_backup_scheduler_stop_
[2023-11-10 10:45:29.036854] INFO [RS] wait_backup_scheduler_stop_ (ob_backup_lease_service.cpp:286) [808][][T0][Y0-0000000000000000-0-0] [lt=6] [BACKUP_LEASE] finish wait_backup_scheduler_stop_(cost_ts=8)
[2023-11-10 10:45:29.036862] INFO [RS] wait_mgr_stop_ (ob_backup_lease_service.cpp:291) [808][][T0][Y0-0000000000000000-0-0] [lt=8] [BACKUP_LEASE] start waiting backup mgr stop
[2023-11-10 10:45:29.036869] INFO [RS] wait_mgr_stop_ (ob_backup_lease_service.cpp:297) [808][][T0][Y0-0000000000000000-0-0] [lt=6] [BACKUP_LEASE] finish waiting backup mgr stop(cost_ts=0)
[2023-11-10 10:45:29.036877] INFO [RS] destroy (ob_backup_lease_service.cpp:180) [808][][T0][Y0-0000000000000000-0-0] [lt=5] finish destroy ObBackupLeaseService(cost_ts=71)
[2023-11-10 10:45:29.036880] INFO [RS] destroy (ob_root_service.cpp:1051) [808][][T0][Y0-0000000000000000-0-0] [lt=3] finish destroy backup_lease_service_
[2023-11-10 10:45:29.036904] WDIAG [RS] destroy (ob_dbms_job_master.cpp:94) [808][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] scheduler task not inited(ret=-4006, inited_=false)
[2023-11-10 10:45:29.036914] INFO [RS] destroy (ob_root_service.cpp:1054) [808][][T0][Y0-0000000000000000-0-0] [lt=8] ObDBMSJobMaster destory
[2023-11-10 10:45:29.036918] INFO [RS] destroy (ob_rs_reentrant_thread.cpp:115) [808][][T0][Y0-0000000000000000-0-0] [lt=3] rs_monitor_check : reentrant thread check unregister success(thread_name="", last_run_timestamp=0)
[2023-11-10 10:45:29.036921] INFO [RS] destroy (ob_root_service.cpp:1060) [808][][T0][Y0-0000000000000000-0-0] [lt=3] disaster recovery task mgr destroy
[2023-11-10 10:45:29.036934] WDIAG [RS] destroy (ob_dbms_sched_job_master.cpp:95) [808][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] scheduler task not inited(ret=-4006, inited_=false)
[2023-11-10 10:45:29.036938] INFO [RS] destroy (ob_root_service.cpp:1064) [808][][T0][Y0-0000000000000000-0-0] [lt=4] ObDBMSSchedJobMaster destory
[2023-11-10 10:45:29.036947] INFO [RS] destroy (ob_root_service.cpp:1066) [808][][T0][Y0-0000000000000000-0-0] [lt=5] global ctx timer destroyed
[2023-11-10 10:45:29.036950] INFO [RS] destroy (ob_root_service.cpp:1075) [808][][T0][Y0-0000000000000000-0-0] [lt=3] [ROOTSERVICE_NOTICE] destroy rootservice end(ret=0, ret=“OB_SUCCESS”)
[2023-11-10 10:45:29.039324] INFO [RS] stop (ob_disaster_recovery_task_table_updater.cpp:190) [808][][T0][Y0-0000000000000000-0-0] [lt=3] stop ObDRTaskTableUpdater success
[2023-11-10 10:45:29.039331] INFO [RS] wait (ob_disaster_recovery_task_table_updater.cpp:196) [808][][T0][Y0-0000000000000000-0-0] [lt=8] wait ObDRTaskTableUpdater

observice.log:
[2023-11-10 10:46:29.031309] INFO destroy_tg (thread_mgr.cpp:95) [868][][T0][Y0-0000000000000000-0-0] [lt=4] destroy tg(tg_id=115, tg=0x7faefb6ada00, tg->attr_={name:CDCSrv, desc:, scope:1, type:3})
[2023-11-10 10:46:29.031313] INFO destroy_tg (thread_mgr.cpp:95) [868][][T0][Y0-0000000000000000-0-0] [lt=4] destroy tg(tg_id=116, tg=0x7faefb6adad0, tg->attr_={name:LogUpdater, desc:, scope:1, type:4})
[2023-11-10 10:46:29.031317] INFO destroy_tg (thread_mgr.cpp:95) [868][][T0][Y0-0000000000000000-0-0] [lt=4] destroy tg(tg_id=117, tg=0x7faefb6adc30, tg->attr_={name:HeartBeatCheckTask, desc:, scope:1, type:4})
[2023-11-10 10:46:29.031362] INFO [SERVER] destroy (ob_server.cpp:462) [868][][T0][Y0-0000000000000000-0-0] [lt=4] [OBSERVER_NOTICE] destroy observer begin
[2023-11-10 10:46:29.031367] WDIAG [SERVER] destroy (ob_server.cpp:629) [868][][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4000] [OBSERVER_NOTICE] can not destroy observer(has_destroy=true, has_stopped=true)
[2023-11-10 10:46:29.031385] ERROR destroy (ob_server.cpp:630) [868][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4394] observer stop process failure(msg=“observer exits without calling destroy()”, has_destroy=true, has_stopped=true)
[2023-11-10 10:46:29.031393] WDIAG [CLOG] destroy (ob_server_log_block_mgr.cpp:120) [868][][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] ObServerLogBlockMgr destroy(this={dir::"", dir_fd:-1, meta_fd:-1, log_pool_meta:{curr_total_size:0, next_total_size:0, status:0}, min_block_id:0, max_block_id:0, min_log_disk_size_for_all_tenants_:0, is_inited:false})
[2023-11-10 10:46:29.031705] INFO [STORAGE.TRANS] destroy (ob_weak_read_service.cpp:53) [868][][T0][Y0-0000000000000000-0-0] [lt=8] [WRS] weak read service begin destroy
[2023-11-10 10:46:29.031710] INFO [STORAGE.TRANS] destroy (ob_weak_read_service.cpp:59) [868][][T0][Y0-0000000000000000-0-0] [lt=6] [WRS] weak read service destroy succ
[2023-11-10 10:46:29.031835] WDIAG begin (ob_hashtable.h:915) [868][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4006] hashtable not init, backtrace=0xd64e645 0x671d053 0x6f8a30d 0x6e7ad73 0x7d5a47d 0x7faefbd06b0c 0x7faefbd06c40 0x7faefbcf049a 0x3e4ba7e
[2023-11-10 10:46:29.031857] WDIAG begin (ob_hashtable.h:915) [868][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4006] hashtable not init, backtrace=0xd64e645 0x705f82f 0x6f8a3c8 0x6e7ad73 0x7d5a47d 0x7faefbd06b0c 0x7faefbd06c40 0x7faefbcf049a 0x3e4ba7e
[2023-11-10 10:46:29.031870] WDIAG begin (ob_hashtable.h:915) [868][][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4006] hashtable not init, backtrace=0xd64e645 0x705f951 0x6f8a488 0x6e7ad73 0x7d5a47d 0x7faefbd06b0c 0x7faefbd06c40 0x7faefbcf049a 0x3e4ba7e
[2023-11-10 10:46:29.031878] WDIAG begin (ob_hashtable.h:915) [868][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] hashtable not init, backtrace=0xd64e645 0x671d053 0x6f8a55d 0x6e7ad73 0x7d5a47d 0x7faefbd06b0c 0x7faefbd06c40 0x7faefbcf049a 0x3e4ba7e
[2023-11-10 10:46:29.031889] WDIAG begin (ob_hashtable.h:915) [868][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] hashtable not init, backtrace=0xd64e645 0x705fa73 0x6f8a618 0x6e7ad73 0x7d5a47d 0x7faefbd06b0c 0x7faefbd06c40 0x7faefbcf049a 0x3e4ba7e
[2023-11-10 10:46:29.033031] WDIAG begin (ob_hashtable.h:915) [868][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] hashtable not init, backtrace=0xd64e645 0xc850f4b 0xc6df82c 0xc6df576 0x7faefbd06b0c 0x7faefbd06c40 0x7faefbcf049a 0x3e4ba7e
[2023-11-10 10:46:29.033050] WDIAG begin (ob_hashtable.h:915) [868][][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] hashtable not init, backtrace=0xd64e645 0xc850fd3 0xc6df931 0xc6df576 0x7faefbd06b0c 0x7faefbd06c40 0x7faefbcf049a 0x3e4ba7e
[2023-11-10 10:46:29.033058] WDIAG begin (ob_hashtable.h:915) [868][][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4006] hashtable not init, backtrace=0xd64e645 0xc850fd3 0xc6dfa31 0xc6df576 0x7faefbd06b0c 0x7faefbd06c40 0x7faefbcf049a 0x3e4ba7e

egegredsfew · 2023 年11 月 10 日 10:56

配置：
apiVersion: OceanBase Cloud
kind: OBCluster
metadata:
name: ob-test
namespace: oceanbase-system
spec:
imageRepo: oceanbasedev/oceanbase-cn
tag: v4.1.0.0-100000192023032010
imageObagent: oceanbase/obagent:1.2.0
clusterID: 1
topology:
- cluster: cn
zone:
- name: zone1
region: region1
nodeSelector:
ob.zone: zone1
replicas: 1
- name: zone2
region: region1
nodeSelector:
ob.zone: zone2
replicas: 1
parameters:
- name: log_disk_size
value: “1G”
resources:
cpu: 1
memory: 1Gi
storage:
- name: data-file
storageClassName: “local-path”
size: 5Gi
- name: data-log
storageClassName: “local-path”
size: 2Gi
- name: log
storageClassName: “local-path”
size: 2Gi
- name: obagent-conf-file
storageClassName: “local-path”
size: 1Gi
volume:
name: backup
nfs:
server: 172.24.241.108
path: /data/pv/oceanbase/ob
readOnly: false

egegredsfew · 2023 年11 月 10 日 10:59

observer日志：

observer.log (8.6 MB)

王利博 · 2023 年11 月 10 日 11:17

看下内存和磁盘大小 1Gi太小
集群创建-配置文件详细介绍

egegredsfew · 2023 年11 月 10 日 14:50

这个 cpu 和内存不能设小一点么？我现在改为了 2cpu 5G内存。还是不行。
下面是容器日志。附件是 ob的日志。

2023-11-10T14:47:31.94154+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 44.297µs, comment

2023-11-10T14:47:33.66397+08:00 INFO [7,] caller=shell/exec.go:97:execute: execute shell command start, command=Command{user=admin, program=sh, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD=’’ /home/admin/oceanbase/bin/observer, timeout=10s}
2023-11-10T14:47:33.75876+08:00 INFO [7,] caller=shell/exec.go:129:execute: execute shell command end, command=Command{user=admin, program=sh, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD=’’ /home/admin/oceanbase/bin/observer, timeout=10s}
2023/11/10 14:47:33 observer 551 []
2023-11-10T14:47:33.94119+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 43.96µs, comment
2023-11-10T14:47:35.94189+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 59.003µs, comment
2023-11-10T14:47:37.94177+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 87.273µs, comment
2023-11-10T14:47:39.94078+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 97.734µs, comment
2023-11-10T14:47:40.52405+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 50.169µs, comment
2023-11-10T14:47:41.94132+08:00 INFO [7,] caller=server/http.go:101:func1: request: from 10.42.0.1, method GET, path /api/ob/readiness, response: status code 500, latency 46.48µs, comment

2023-11-10T14:47:43.66227+08:00 ERROR [7,] caller=observer/check.go:58:checkerObserver: observer process not running, try restart…

observer.log (1.1 MB)
rootservice.log (55.3 KB)
election.log (167 字节)

chris-sun · 2023 年11 月 10 日 15:33

看配置还是用的 ob-operator 1.2 版本，还是推荐用新的 2.x 版本的 ob-operator 吧
这个配置给的资源太小了，

https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000000221085
对于资源都可以参考这个文档
如果使用2.0 版本，可以参考对应分支上的配置文件