【 使用环境 】生产环境
【 OB or 其他组件 】
【 使用版本 】
oceanbase-ce:4.3.5.4
obproxy-ce:4.3.5.0
【问题描述】
故障说明
使用obd部署三台服务器集群,三台均有observer与obproxy。
发现obproxy会不间断挂掉,写了一个脚本去监测故障发生时间,发现周五(2026-01-23)晚上23点发出监测通知(每十分钟检测一次),发现三台机器通过observer端口均可连接,通过obproxy的连接性如下:
| 服务(obproxy) | obproxy可连性 |
|---|---|
| xx.xx.xx.11 | 可连接 |
| xx.xx.xx.12 | 进程存在不可连接 |
| xx.xx.xx.13 | 进程不存在,不可连接 |
第二台机器obproxy连接提示如下:
➜ ~ mysql -hxx.xx.xx.12 -P18083 -uroot@thtf -p'xxxxxx'
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 2
配置如下
user:
username: root
port: 22
password:
oceanbase-ce:
version: 4.3.5.4
release: 104000042025090916.el7
package_hash: 502277ef3b10b1bcf3fea757b309addf14ad8edb1a8a98d986883045c539603c
xx.xx.xx.11:
zone: zone1
xx.xx.xx.12:
zone: zone2
xx.xx.xx.13:
zone: zone3
servers:
- xx.xx.xx.11
- xx.xx.xx.12
- xx.xx.xx.13
global:
appname: sessob
root_password: 密码3
mysql_port: 18081
rpc_port: 18082
data_dir: /data
redo_dir: /redo
obshell_port: 18086
home_path: /root/sessob/oceanbase
scenario: htap
datafile_size: 2GB
datafile_maxsize: 900GB
datafile_next: 2GB
log_disk_size: 170GB
max_syslog_file_count: '10'
memory_limit: 50GB
system_memory: 5GB
devname: eth0
enable_auto_start: 'True'
cluster_id: 1761301228
ocp_agent_monitor_password: 密码5
proxyro_password: 密码2
enable_syslog_wf: false
cpu_count: 14
depends:
- ob-configserver
obproxy-ce:
version: 4.3.5.0
package_hash: dff2846bc2fae2852480bdb0a2c5ddeee3309071f82257b921a687b3c9274345
release: 3.el7
servers:
- xx.xx.xx.11
- xx.xx.xx.12
- xx.xx.xx.13
global:
prometheus_listen_port: 18084
listen_port: 18083
rpc_listen_port: 18085
home_path: /root/sessob/obproxy
proxy_mem_limited: 4GB
obproxy_sys_password: 密码1
skip_proxy_sys_private_check: true
enable_strict_kernel_release: false
enable_cluster_checkout: false
rs_list: xx.xx.xx.11:18081;xx.xx.xx.12:18081;xx.xx.xx.13:18081
cluster_name: sessob
observer_root_password: 密码3
xx.xx.xx.11:
proxy_id: 6410
client_session_id_version: 2
xx.xx.xx.12:
proxy_id: 6411
client_session_id_version: 2
xx.xx.xx.13:
proxy_id: 6412
client_session_id_version: 2
depends:
- oceanbase-ce
- ob-configserver
obagent:
version: 4.2.3
package_hash: 5f44d62b09e2fc6fb3107ebce02a2a68185f03e81956f98d8578c55ba3c8238f
release: 200000032025071420.el7
servers:
- xx.xx.xx.11
- xx.xx.xx.12
- xx.xx.xx.13
global:
monagent_http_port: 18088
mgragent_http_port: 8089
home_path: /root/sessob/obagent
http_basic_auth_password: 密码4
ob_monitor_status: active
depends:
- oceanbase-ce
ob-configserver:
version: 1.0.1
release: 1.0.1.el7
servers:
- xx.xx.xx.11
global:
listen_port: 8080
home_path: /root/sessob/obconfigserver
日志说明
13服务器日志最新更改时间固定到了发生故障时(2026-01-23 23:00左右),下面是1000行日志
13-obproxy.log.tail1000.txt (544.1 KB)
13-obproxy_stat.log.tail1000.txt (110.0 KB)
obproxy_error.log.tail1000.txt (172.1 KB)
12服务器日志一直在更新,但是无法连接,下面是截取时的1000行日志
12-obproxy.log.tail1000.txt (441.8 KB)
12-obproxy_diagnosis.log.tail1000.txt (587.5 KB)
12-obproxy_error.log.tail1000.txt (168.9 KB)
猜测原因
结合我在论坛里搜到的相关帖子(搜索reading authorization packet),发现我配置的下面两行密码不同
proxyro_password: 密码2
obproxy_sys_password: 密码1
这两个密码是不是应该改成一样的?
【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接(右键跳转查看):
【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试,在发帖时输入 [@论坛小助手] 即可召唤小助手,欢迎试用!
