OB 4.2.4 destroy之后重新安装，无法启动

AntTech_NPYXWW · 2024 年9 月 13 日 13:03

3节点OB集集群，4.2.4_CE版本，第一次部署可以正常启动，测试都没有问题。后续destroy 之后，重新部署，能部署成功，但是无法启动，一直卡在start observer.
Check before start obagent ok
Check before start ocp-express ok
Start observer ok
observer program health check ok
Connect to observer 172.16.51.35:2881 ok
Initialize oceanbase-ce
卡在这一步无法继续。
日志反复刷：
[2024-09-13 13:00:32.703915] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [3127][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret=“OB_NEED_WAIT”, tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1726203632503804, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master=“0.0.0.0:0”)
[2024-09-13 13:00:32.735615] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:271) [3144][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1})
[2024-09-13 13:00:32.736551] WDIAG load_file_to_string (utility.h:662) [2756][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] read /sys/class/net/eth0/speed failed, errno 22
[2024-09-13 13:00:32.736573] WDIAG get_ethernet_speed (utility.cpp:580) [2756][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000
[2024-09-13 13:00:32.736580] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2807) [2756][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname=“eth0”)
[2024-09-13 13:00:32.736590] WDIAG [SERVER] runTimerTask (ob_server.cpp:3341) [2756][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret=“OB_ERROR”)
[2024-09-13 13:00:32.744994] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2788][KVCacheRep][T0][Y0-0000000
日志如下：
alert.log (1.5 KB)
observer.log (9.7 MB)
rootservice.log (37.8 KB)

皇甫侯 · 2024 年9 月 13 日 13:18

看一个各个节点的时间同步是否正常

辞霜 · 2024 年9 月 13 日 14:01

8004报错属于部署错误，需要尽快停止部署错误的集群。
重新搭建大概率是yaml配置文件存在问题

AntTech_NPYXWW · 2024 年9 月 13 日 14:17

时间是同步的，最开始三台机器的chrony指向的同一个时间源，启动报错，刚才我调整节点2,3指向节点1。redeploy还是卡在同样的位置：
Check before start ocp-express ok
Start observer ok
observer program health check ok
Connect to observer 172.16.51.35:2881 ok
Initialize oceanbase-ce /

clock generator not inited这个异常同样还有。

我补充一个背景：
第一次部署成功是4.2.4_CE版本，采用OMS迁移mysql数据库报错，后续发现是4.2.4_CE的bug。因此就想部署4.2.1版本，先是destroy 集群，然后下载了4.2.1版本，重新安装，deploy集群是成功的，start就是卡在上述步骤。一直无法成功。继而继续部署4.2.4版本，同样卡在start位置，topoloy.yaml文件是同一个。

辞霜 · 2024 年9 月 13 日 14:20

换个新的yaml文件试试

AntTech_NPYXWW · 2024 年9 月 13 日 14:20

3台机器，topoloy配置没调整，第一次部署，start都是正常的。都做了压测。完全没问题。destroy之后，重新deploy就无法start了。

AntTech_NPYXWW · 2024 年9 月 13 日 14:21

内容不调整？

辞霜 · 2024 年9 月 13 日 14:29

你把yaml配置发出来看一下

AntTech_NPYXWW · 2024 年9 月 13 日 14:35

[root@tidb-1 ob]# obd cluster deploy ob-poc -c ob-poc.yaml
±-------------------------------------------------------------------------------------------+
| Packages |
±-------------±--------±-----------------------±-----------------------------------------+
| Repository | Version | Release | Md5 |
±-------------±--------±-----------------------±-----------------------------------------+
| oceanbase-ce | 4.2.4.0 | 100000082024070810.el7 | 7dc8b049b3283ef4660cdf6e3cfa24f81e9d2a78 |
| obproxy-ce | 4.2.3.0 | 3.el7 | 0490ebc04220def8d25cb9cac9ac61a4efa6d639 |
| obagent | 4.2.2 | 100000042024011120.el7 | 19739a07a12eab736aff86ecf357b1ae660b554e |
| ocp-express | 4.2.2 | 100000022024011120.el7 | 09ffcf156d1df9318a78af52656f499d2315e3f7 |
±-------------±--------±-----------------------±-----------------------------------------+
Repository integrity check ok
Load param plugin ok
Open ssh connection ok
Parameter check ok
Cluster status check ok
Initializes observer work home ok
Initializes obproxy work home ok
Initializes obagent work home ok
Initializes ocp-express work home ok
Remote oceanbase-ce-4.2.4.0-100000082024070810.el7-7dc8b049b3283ef4660cdf6e3cfa24f81e9d2a78 repository install ok
Remote oceanbase-ce-4.2.4.0-100000082024070810.el7-7dc8b049b3283ef4660cdf6e3cfa24f81e9d2a78 repository lib check !!
Remote obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 repository install ok
Remote obproxy-ce-4.2.3.0-3.el7-0490ebc04220def8d25cb9cac9ac61a4efa6d639 repository lib check ok
Remote obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e repository install ok
Remote obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e repository lib check ok
Remote ocp-express-4.2.2-100000022024011120.el7-09ffcf156d1df9318a78af52656f499d2315e3f7 repository install ok
Remote ocp-express-4.2.2-100000022024011120.el7-09ffcf156d1df9318a78af52656f499d2315e3f7 repository lib check !!
Try to get lib-repository
Remote oceanbase-ce-libs-4.2.4.0-100000082024070810.el7-eda524874ce5daff6685f114bd28965b8c1834ef repository install ok
Remote oceanbase-ce-4.2.4.0-100000082024070810.el7-7dc8b049b3283ef4660cdf6e3cfa24f81e9d2a78 repository lib check ok
Remote openjdk-jre-1.8.0_322-b09.el7-051aa69c5abb8697d15c2f0dcb1392b3f815f7ed repository install ok
Remote ocp-express-4.2.2-100000022024011120.el7-09ffcf156d1df9318a78af52656f499d2315e3f7 repository lib check ok
ob-poc deployed
Please execute obd cluster start ob-poc to start
Trace ID: 99ceca22-7198-11ef-89cb-fa163e45e664
If you want to view detailed obd logs, please run: obd display-trace 99ceca22-7198-11ef-89cb-fa163e45e664

部署yaml:
ob-poc.log (4.6 KB)

辞霜 · 2024 年9 月 13 日 14:46

system_memory 设置的太大了，改为2G试试。最下面的ocp-express内存也改为2G
或者建议你用obd web白屏化部署

AntTech_NPYXWW · 2024 年9 月 13 日 15:02

调整参数之后，还是一样，卡在那里不继续。
白屏部署，也是一样：

辞霜 · 2024 年9 月 13 日 15:24

看之前的observer日志也存在log日志报错，建议扩容到3-4倍内存大小
log_disk_size：

AntTech_NPYXWW · 2024 年9 月 13 日 15:54

现在调整4倍，还是不行，部署时所有的warn都解决了，还是不行，换了obd服务器，也还是不行，都是卡在同一个地方。

姜涛 · 2024 年9 月 13 日 16:05

执行下cat ~/.obd/cluster/ob-poc/config.yaml 看看配置文件

辞霜 · 2024 年9 月 13 日 16:06

稍等这边让同事帮忙看一下

AntTech_NPYXWW · 2024 年9 月 13 日 16:07

Only need to configure when remote login is required

user:
username: root
password: xxxx

key_file: your ssh-key file path if need

port: 12722

timeout: ssh connection timeout (second), default 30

oceanbase-ce:
servers:

name: server1
Please don’t use hostname, only IP can be supported
ip: 172.16.51.35
name: server2
ip: 172.16.51.36
name: server3
ip: 172.16.51.37
global:
Starting from observer version 4.2, the network selection for the observer is based on the ‘local_ip’ parameter, and the ‘devname’ parameter is no longer mandatory.

If the ‘local_ip’ parameter is set, the observer will first use this parameter for the configuration, regardless of the ‘devname’ parameter.

If only the ‘devname’ parameter is set, the observer will use the ‘devname’ parameter for the configuration.

If neither the ‘devname’ nor the ‘local_ip’ parameters are set, the ‘local_ip’ parameter will be automatically assigned the IP address configured above.
devname: eth0
if current hardware’s memory capacity is smaller than 50G, please use the setting of “mini-single-example.yaml” and do a small adjustment.
memory_limit: 16G # The maximum running memory for an observer
The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
system_memory: 2G
datafile_size: 64G # Size of the data file.
log_disk_size: 64G # The size of disk space used by the clog files.
enable_syslog_wf: true # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
max_syslog_file_count: 4 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
Cluster name for OceanBase Database. The default value is obcluster. When you deploy OceanBase Database and obproxy, this value must be the same as the cluster_name for obproxy.
appname: ob_poc
root_password: ob@poc.test
proxyro_password: obp@poc.test
observer_sys_password: obp@poc.test
ocp_meta_db: ocp_express # The database name of ocp express meta

ocp_meta_username: meta # The username of ocp express meta

ocp_meta_password: ‘’ # The password of ocp express meta

ocp_agent_monitor_password: ‘’ # The password for obagent monitor user
ocp_meta_tenant: # The config for ocp express meta tenant
tenant_name: ocp
max_cpu: 1
memory_size: 2G
log_disk_size: 7680M # The recommend value is (4608 + (expect node num + expect tenant num) * 512) M.
mysql_port: 2881
rpc_port: 2882
obshell_port: 2886
home_path: /u01/app/observer
data_dir: /data1/oceanbase/data
redo_dir: /u01/oceanbase/redo
cluster_id: 1726214411
ocp_agent_monitor_password: reuq1AdaSj
ocp_root_password: 5ZFR7h6MSH
ocp_meta_password: Qq6HymAFkE
server1:
zone: zone1
server2:
zone: zone2
server3:
zone: zone3
obproxy-ce:
depends:
oceanbase-ce
servers:
172.16.51.35
global:
listen_port: 2883
prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
home_path: /u01/app/obproxy
oceanbase root server list

format: ip:mysql_port;ip:mysql_port. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.

rs_list: 192.168.1.2:2881;192.168.1.3:2881;192.168.1.4:2881
enable_cluster_checkout: false
observer cluster name, consistent with oceanbase-ce’s appname. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.

cluster_name: obcluster
skip_proxy_sys_private_check: true
enable_strict_kernel_release: false
obproxy_sys_password: obproxy_sys_password@poc.test
172.16.51.35:
proxy_id: 6346
client_session_id_version: 2
obagent:
depends:
oceanbase-ce
servers:
name: server1
ip: 172.16.51.35
name: server2
ip: 172.16.51.36
name: server3
ip: 172.16.51.37
global:
home_path: /u01/app/obagent
http_basic_auth_password: t79z6QAVf
ocp-express:
depends:
oceanbase-ce
obproxy-ce
obagent
servers:
172.16.51.35
global:
The working directory for prometheus. prometheus is started under this directory. This is a required field.
home_path: /u01/app/ocp-express
log_dir: /u01/oceanbase/ocp-express/log # The log directory of ocp express server. The default value is {home_path}/log.
memory_size: 2G # The memory size of ocp-express server. The recommend value is 512MB * (expect node num + expect tenant num) * 60MB.
logging_file_total_size_cap: 10G # The total log file size of ocp-express server

logging_file_max_history: 1 # The maximum of retention days the log archive log files to keep. The default value is unlimited
admin_passwd: 2Li3.%wZ
ocp_root_password: 9tN%@9zA

AntTech_NPYXWW · 2024 年9 月 13 日 16:09

操作系统：
CentOS Linux release 7.6.1810 (Core)

现在换4.2.1.8版本，也会卡在同样的地方。

姜涛 · 2024 年9 月 13 日 16:10

我这边的配置文件是这样的

AntTech_NPYXWW · 2024 年9 月 13 日 16:39

处理配置的值不同，其他没多大差别，deploy是正常的，只是start的时候不行

姜涛 · 2024 年9 月 13 日 16:40

方便把ob部分截出来看看吗

OB 4.2.4 destroy之后重新安装，无法启动

Only need to configure when remote login is required

key_file: your ssh-key file path if need

timeout: ssh connection timeout (second), default 30

Please don’t use hostname, only IP can be supported

Starting from observer version 4.2, the network selection for the observer is based on the ‘local_ip’ parameter, and the ‘devname’ parameter is no longer mandatory.

If the ‘local_ip’ parameter is set, the observer will first use this parameter for the configuration, regardless of the ‘devname’ parameter.

If only the ‘devname’ parameter is set, the observer will use the ‘devname’ parameter for the configuration.

If neither the ‘devname’ nor the ‘local_ip’ parameters are set, the ‘local_ip’ parameter will be automatically assigned the IP address configured above.

if current hardware’s memory capacity is smaller than 50G, please use the setting of “mini-single-example.yaml” and do a small adjustment.

The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.

Cluster name for OceanBase Database. The default value is obcluster. When you deploy OceanBase Database and obproxy, this value must be the same as the cluster_name for obproxy.

ocp_meta_db: ocp_express # The database name of ocp express meta

ocp_meta_username: meta # The username of ocp express meta

ocp_meta_password: ‘’ # The password of ocp express meta

ocp_agent_monitor_password: ‘’ # The password for obagent monitor user

oceanbase root server list

format: ip:mysql_port;ip:mysql_port. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.

rs_list: 192.168.1.2:2881;192.168.1.3:2881;192.168.1.4:2881

observer cluster name, consistent with oceanbase-ce’s appname. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.

cluster_name: obcluster

The working directory for prometheus. prometheus is started under this directory. This is a required field.

logging_file_total_size_cap: 10G # The total log file size of ocp-express server

logging_file_max_history: 1 # The maximum of retention days the log archive log files to keep. The default value is unlimited