oceanbase 4.1-CE 社区版 K8S环境1.20.0

imade · 2023 年6 月 27 日 16:42

【使用环境】测试环境
【 OB or 其他组件】 ob
【使用版本】v4.1.0.0-100000192023032010
【问题描述】observer 启动失败
【复现路径】问题出现前后相关操作
【问题现象及影响】

【附件】

imade · 2023 年6 月 27 日 16:43

imade · 2023 年6 月 27 日 16:44

cluster 配置

apiVersion: OceanBase Cloud
kind: OBCluster
metadata:
name: ob-test
namespace: obcluster
spec:
imageRepo: oceanbasedev/oceanbase-cn
tag: v4.1.0.0-100000192023032010
imageObagent: oceanbase/obagent:1.2.0
clusterID: 1
topology:
- cluster: cn
zone:
- name: zone1
region: region1
nodeSelector:
ob.zone: zone4
replicas: 1
- name: zone2
region: region1
nodeSelector:
ob.zone: zone5
replicas: 1
- name: zone3
region: region1
nodeSelector:
ob.zone: zone6
replicas: 1
parameters:
- name: log_disk_size
value: “40G”
resources:
cpu: 4
memory: 16Gi
storage:
- name: data-file
storageClassName: “local-path”
size: 1024Gi
- name: data-log
storageClassName: “local-path”
size: 50Gi
- name: log
storageClassName: “local-path”
size: 30Gi
- name: obagent-conf-file
storageClassName: “local-path”
size: 1Gi

imade · 2023 年6 月 27 日 16:45

部分日志
[2023-06-27 15:13:19.093475] INFO [SERVER] init (ob_server.cpp:222) [54][][T0][Y0-0000000000000000-0-0] [lt=10] [OBSERVER_NOTICE] start to init observer
[2023-06-27 15:13:19.781589] INFO [SERVER] init (ob_service.cpp:184) [54][][T0][Y0-0000000000000000-0-0] [lt=47] [OBSERVICE_NOTICE] init ob_service begin
[2023-06-27 15:13:19.812418] INFO [SERVER] init (ob_service.cpp:224) [54][][T0][Y0-0000000000000000-0-0] [lt=5] [OBSERVICE_NOTICE] init ob_service finish(ret=0, ret=“OB_SUCCESS”, inited=true)
[2023-06-27 15:13:20.212366] INFO [SERVER] init (ob_server.cpp:445) [54][][T0][Y0-0000000000000000-0-0] [lt=15] [OBSERVER_NOTICE] success to init observer(cluster_id=1, lib::g_runtime_enabled=true)
[2023-06-27 15:13:20.212402] INFO [SERVER] start (ob_server.cpp:653) [54][][T0][Y0-0000000000000000-0-0] [lt=15] [OBSERVER_NOTICE] start observer begin
[2023-06-27 15:13:20.217751] INFO [SERVER] start (ob_server.cpp:880) [54][][T0][Y0-0000000000000000-0-0] [lt=4] [NOTICE] check if sys srs usable(ret=-4184, stop_=true)
[2023-06-27 15:13:20.217790] INFO [SERVER] set_stop (ob_service.cpp:285) [54][][T0][Y0-0000000000000000-0-0] [lt=5] [OBSERVICE_NOTICE] observice need stop now
[2023-06-27 15:13:20.217793] INFO [SERVER] set_stop (ob_server.cpp:953) [54][][T0][Y0-0000000000000000-0-0] [lt=3] [OBSERVER_NOTICE] observer is setted to stop
[2023-06-27 15:13:20.217800] INFO [SERVER] wait (ob_server.cpp:1187) [54][][T0][Y0-0000000000000000-0-0] [lt=7] [OBSERVER_NOTICE] wait observer begin
[2023-06-27 15:13:20.217809] INFO [SERVER] stop (ob_server.cpp:960) [54][][T0][Y0-0000000000000000-0-0] [lt=2] [OBSERVER_NOTICE] stop observer begin
[2023-06-27 15:13:20.217939] INFO [SERVER] stop (ob_service.cpp:291) [54][][T0][Y0-0000000000000000-0-0] [lt=1] [OBSERVICE_NOTICE] start to stop observice
[2023-06-27 15:13:20.218035] INFO [SERVER] stop (ob_service.cpp:318) [54][][T0][Y0-0000000000000000-0-0] [lt=2] [OBSERVICE_NOTICE] observice finish stop(stopped=true)
[2023-06-27 15:13:20.701566] INFO [SERVER] stop (ob_server.cpp:1175) [54][][T0][Y0-0000000000000000-0-0] [lt=19] [OBSERVER_NOTICE] stop observer end(ret=0, ret=“OB_SUCCESS”)
[2023-06-27 15:13:50.039459] INFO [SERVER] wait (ob_service.cpp:323) [54][][T0][Y0-0000000000000000-0-0] [lt=3] [OBSERVICE_NOTICE] wait ob_service begin
[2023-06-27 15:13:50.040045] INFO [SERVER] wait (ob_service.cpp:343) [54][][T0][Y0-0000000000000000-0-0] [lt=5] [OBSERVICE_NOTICE] wait ob_service end
[2023-06-27 15:13:50.040861] INFO [SERVER] wait (ob_server.cpp:1359) [54][][T0][Y0-0000000000000000-0-0] [lt=16] [OBSERVER_NOTICE] wait observer end(ret=0, ret=“OB_SUCCESS”)
[2023-06-27 15:13:50.041015] INFO [SERVER] destroy (ob_server.cpp:462) [54][][T0][Y0-0000000000000000-0-0] [lt=118] [OBSERVER_NOTICE] destroy observer begin
[2023-06-27 15:13:50.089673] INFO [SERVER] destroy (ob_service.cpp:349) [54][][T0][Y0-0000000000000000-0-0] [lt=3] [OBSERVICE_NOTICE] destroy ob_service begin
[2023-06-27 15:13:50.100686] INFO [SERVER] destroy (ob_service.cpp:375) [54][][T0][Y0-0000000000000000-0-0] [lt=3] [OBSERVICE_NOTICE] destroy ob_service end(ret=0, ret=“OB_SUCCESS”)
[2023-06-27 15:13:52.270750] INFO [SERVER] destroy (ob_server.cpp:627) [54][][T0][Y0-0000000000000000-0-0] [lt=4] [OBSERVER_NOTICE] destroy observer end
[2023-06-27 15:13:52.696206] INFO [SERVER] destroy (ob_server.cpp:462) [54][][T0][Y0-0000000000000000-0-0] [lt=6] [OBSERVER_NOTICE] destroy observer begin
[2023-06-27 15:13:52.696213] WDIAG [SERVER] destroy (ob_server.cpp:629) [54][][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] [OBSERVER_NOTICE] can not destroy observer(has_destroy=true, has_stopped=true)
[2023-06-27 15:14:29.094898] INFO [SERVER] init (ob_server.cpp:222) [273][][T0][Y0-0000000000000000-0-0] [lt=9] [OBSERVER_NOTICE] start to init observer
[2023-06-27 15:14:29.096827] INFO [SERVER] set_stop (ob_service.cpp:285) [273][][T0][Y0-0000000000000000-0-0] [lt=7] [OBSERVICE_NOTICE] observice need stop now
[2023-06-27 15:14:29.096847] INFO [SERVER] set_stop (ob_server.cpp:953) [273][][T0][Y0-0000000000000000-0-0] [lt=15] [OBSERVER_NOTICE] observer is setted to stop
[2023-06-27 15:14:29.096858] INFO [SERVER] destroy (ob_server.cpp:462) [273][][T0][Y0-0000000000000000-0-0] [lt=9] [OBSERVER_NOTICE] destroy observer begin
[2023-06-27 15:14:29.098871] INFO [SERVER] destroy (ob_service.cpp:349) [273][][T0][Y0-0000000000000000-0-0] [lt=6] [OBSERVICE_NOTICE] destroy ob_service begin
[2023-06-27 15:14:29.098885] INFO [SERVER] destroy (ob_service.cpp:375) [273][][T0][Y0-0000000000000000-0-0] [lt=7] [OBSERVICE_NOTICE] destroy ob_service end(ret=-4006, ret=“OB_NOT_INIT”)
[2023-06-27 15:14:29.099875] INFO [SERVER] destroy (ob_server.cpp:627) [273][][T0][Y0-0000000000000000-0-0] [lt=5] [OBSERVER_NOTICE] destroy observer end
[2023-06-27 15:14:29.099883] ERROR issue_dba_error (ob_log.cpp:1786) [273][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4147, file=“ob_server.cpp”, line_no=441, info="[OBSERVER_NOTICE] fail to init observer")
[2023-06-27 15:14:29.099895] EDIAG [SERVER] init (ob_server.cpp:441) [273][][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4147] [OBSERVER_NOTICE] fail to init observer(ret=-4147, ret=“OB_INVALID_CONFIG”) BACKTRACE:0xdc493b9 0x59b3469 0x59b2f71 0x59b2d72 0x5994a82 0x7d6d5f6 0x7d62927 0x5989623 0x7fcc529c5493 0x3e4ba7e
[2023-06-27 15:14:29.100022] INFO [SERVER] destroy (ob_server.cpp:462) [273][][T0][Y0-0000000000000000-0-0] [lt=64] [OBSERVER_NOTICE] destroy observer begin
[2023-06-27 15:14:29.100028] WDIAG [SERVER] destroy (ob_server.cpp:629) [273][][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] [OBSERVER_NOTICE] can not destroy observer(has_destroy=true, has_stopped=true)
[2023-06-27 15:14:29.101408] INFO [SERVER] destroy (ob_server.cpp:462) [273][][T0][Y0-0000000000000000-0-0] [lt=7] [OBSERVER_NOTICE] destroy observer begin
[2023-06-27 15:14:29.101415] WDIAG [SERVER] destroy (ob_server.cpp:629) [273][][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] [OBSERVER_NOTICE] can not destroy observer(has_destroy=true, has_stopped=true)
[2023-06-27 15:14:49.107100] INFO [SERVER] init (ob_server.cpp:222) [299][][T0][Y0-0000000000000000-0-0] [lt=17] [OBSERVER_NOTICE] start to init observer
[2023-06-27 15:14:49.109228] INFO [SERVER] set_stop (ob_service.cpp:285) [299][][T0][Y0-0000000000000000-0-0] [lt=7] [OBSERVICE_NOTICE] observice need stop now
[2023-06-27 15:14:49.109249] INFO [SERVER] set_stop (ob_server.cpp:953) [299][][T0][Y0-0000000000000000-0-0] [lt=17] [OBSERVER_NOTICE] observer is setted to stop
[2023-06-27 15:14:49.109261] INFO [SERVER] destroy (ob_server.cpp:462) [299][][T0][Y0-0000000000000000-0-0] [lt=10] [OBSERVER_NOTICE] destroy observer begin
[2023-06-27 15:14:49.111513] INFO [SERVER] destroy (ob_service.cpp:349) [299][][T0][Y0-0000000000000000-0-0] [lt=5] [OBSERVICE_NOTICE] destroy ob_service begin
[2023-06-27 15:14:49.111527] INFO [SERVER] destroy (ob_service.cpp:375) [299][][T0][Y0-0000000000000000-0-0] [lt=6] [OBSERVICE_NOTICE] destroy ob_service end(ret=-4006, ret=“OB_NOT_INIT”)
[2023-06-27 15:14:49.112649] INFO [SERVER] destroy (ob_server.cpp:627) [299][][T0][Y0-0000000000000000-0-0] [lt=5] [OBSERVER_NOTICE] destroy observer end
[2023-06-27 15:14:49.112663] ERROR issue_dba_error (ob_log.cpp:1786) [299][][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4147, file=“ob_server.cpp”, line_no=441, info="[OBSERVER_NOTICE] fail to init observer")
[2023-06-27 15:14:49.112676] EDIAG [SERVER] init (ob_server.cpp:441) [299][][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4147] [OBSERVER_NOTICE] fail to init observer(ret=-4147, ret=“OB_INVALID_CONFIG”) BACKTRACE:0xdc493b9 0x59b3469 0x59b2f71 0x59b2d72 0x5994a82 0x7d6d5f6 0x7d62927 0x5989623 0x7ff537e8c493 0x3e4ba7e
[2023-06-27 15:14:49.112798] INFO [SERVER] destroy (ob_server.cpp:462) [299][][T0][Y0-0000000000000000-0-0] [lt=54] [OBSERVER_NOTICE] destroy observer begin
[2023-06-27 15:14:49.112804] WDIAG [SERVER] destroy (ob_server.cpp:629) [299][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4000] [OBSERVER_NOTICE] can not destroy observer(has_destroy=true, has_stopped=true)
[2023-06-27 15:14:49.114450] INFO [SERVER] destroy (ob_server.cpp:462) [299][][T0][Y0-0000000000000000-0-0] [lt=6] [OBSERVER_NOTICE] destroy observer begin
[2023-06-27 15:14:49.114459] WDIAG [SERVER] destroy (ob_server.cpp:629) [299][][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] [OBSERVER_NOTICE] can not destroy observer(has_destroy=true, has_stopped=true)