Failed to connect to obproxy-ce

AntTech_YGJQAX · 2024 年10 月 21 日 22:09

[admin@node74 ~]$ cat /tmp/soft/mini-distributed-with-obproxy-example.yaml
oceanbase-ce:
servers:
- name: z1
ip: 10.10.1.75
- name: z2
ip: 10.10.1.76
- name: z3
ip: 10.10.1.77
global:
devname: ens160
cluster_id: 1
memory_limit: 8G
system_memory: 4G
stack_size: 512K
production_mode: false
cpu_count: 10
cache_wash_threshold: 1G
__min_full_resource_pool_memory: 2684354560
workers_per_cpu_quota: 8
schema_history_expire_time: 1d
net_thread_count: 4
major_freeze_duty_time: Disable
minor_freeze_times: 10
enable_separate_sys_clog: 0
enable_merge_by_turn: FALSE
datafile_disk_percentage: 20
syslog_level: INFO
enable_syslog_recycle: true
max_syslog_file_count: 4
appname: ob_cluster
root_password:
proxyro_password: zhxf_123
z1:
mysql_port: 2881
rpc_port: 2882
home_path: /home/admin/oceanbase
data_dir: /data
redo_dir: /redo
zone: zone1
z2:
mysql_port: 2881
rpc_port: 2882
home_path: /home/admin/oceanbase
data_dir: /data
redo_dir: /redo
zone: zone2
z3:
mysql_port: 2881
rpc_port: 2882
home_path: /home/admin/oceanbase
data_dir: /data
redo_dir: /redo
zone: zone3
obproxy-ce:
servers:
- name: z1
ip: 10.10.1.75
- name: z2
ip: 10.10.1.76
- name: z3
ip: 10.10.1.77
global:
rs_list: 10.10.1.75:2881;10.10.1.76:2881;10.10.1.77:2881
enable_cluster_checkout: false
enable_strict_kernel_release: false
cluster_name: ob_cluster
obproxy_sys_password:
observer_sys_password: zhxf_123
z1:
listen_port: 2883
prometheus_listen_port: 2884
home_path: /home/admin/obproxy
z2:
listen_port: 2883
prometheus_listen_port: 2884
home_path: /home/admin/obproxy
z3:
listen_port: 2883
prometheus_listen_port: 2884
home_path: /home/admin/obproxy
[admin@node74 ~]$

$ obd cluster deploy ceshi -c /tmp/mini-distributed-with-obproxy-example.yaml
无报错。

$ obd cluster start ceshi
Start observer ok
observer program health check ok
Connect to observer 10.10.1.77:2881 ok
Initialize oceanbase-ce ok
Start obproxy ok
obproxy program health check ok
Connect to obproxy x
[ERROR] OBD-1006: Failed to connect to obproxy-ce
Wait for observer init ok
±---------------------------------------------+
| oceanbase-ce |
±-----------±--------±-----±------±-------+
| ip | version | port | zone | status |
±-----------±--------±-----±------±-------+
| 10.10.1.75 | 4.3.3.0 | 2881 | zone1 | ACTIVE |
| 10.10.1.76 | 4.3.3.0 | 2881 | zone2 | ACTIVE |
| 10.10.1.77 | 4.3.3.0 | 2881 | zone3 | ACTIVE |
±-----------±--------±-----±------±-------+

报错了：
[ERROR] OBD-1006: Failed to connect to obproxy-ce

$ obd cluster start ceshi
Get local repositories ok
Search plugins ok
Load cluster param plugin ok
Open ssh connection ok
Check before start observer ok
Check before start obproxy ok
cluster scenario: htap
Start observer ok
observer program health check ok
obshell program health check ok
Connect to observer 10.10.1.75:2881 ok
Start obproxy ok
obproxy program health check ok
Connect to obproxy x
[ERROR] OBD-1006: Failed to connect to obproxy-ce
Wait for observer init ok
±---------------------------------------------+
| oceanbase-ce |
±-----------±--------±-----±------±-------+
| ip | version | port | zone | status |
±-----------±--------±-----±------±-------+
| 10.10.1.75 | 4.3.3.0 | 2881 | zone1 | ACTIVE |
| 10.10.1.76 | 4.3.3.0 | 2881 | zone2 | ACTIVE |
| 10.10.1.77 | 4.3.3.0 | 2881 | zone3 | ACTIVE |
±-----------±--------±-----±------±-------+
obclient -h10.10.1.75 -P2881 -uroot -Doceanbase -A

cluster unique id: 77479861-86ab-5a23-bbde-e022b4bd7021-192af61982f-00030304

报错了：
[ERROR] OBD-1006: Failed to connect to obproxy-ce

皇甫侯 · 2024 年10 月 22 日 09:40

是obproxy报错啊，这块可以用 obd obdiag rca run --scene= disconnection 来诊断下obproxy的日志。。看看是不是有啥异常

https://www.oceanbase.com/docs/common-obdiag-cn-1000000001491170

辞霜 · 2024 年10 月 22 日 10:26

obproxy_sys_password:
observer_sys_password: zhxf_123
root@proxysys的密码没设置所以连接不上proxy

AntTech_YGJQAX · 2024 年10 月 22 日 14:58

不是这个原因，把密码都设置了，还是还报错。

Connect to observer 10.10.1.75:2881 ok
Initialize oceanbase-ce ok
Start obproxy ok
obproxy program health check ok
Connect to obproxy x
[ERROR] OBD-1006: Failed to connect to obproxy-ce
Wait for observer init ok

root_password: ob_7788
proxyro_password: zhxf_7660

obproxy_sys_password: ob_123456
observer_sys_password: zhxf_7660

AntTech_YGJQAX · 2024 年10 月 22 日 15:00

obproxy节点的进程都在的

[root@observer1 ~]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 2561/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2759/master
tcp 0 0 0.0.0.0:2881 0.0.0.0:* LISTEN 15401/observer
tcp 0 0 0.0.0.0:2882 0.0.0.0:* LISTEN 15401/observer
tcp 0 0 0.0.0.0:2883 0.0.0.0:* LISTEN 16574/obproxy
tcp 0 0 0.0.0.0:2884 0.0.0.0:* LISTEN 16574/obproxy
tcp 0 0 0.0.0.0:2885 0.0.0.0:* LISTEN 16574/obproxy
tcp 0 0 10.10.1.75:2886 0.0.0.0:* LISTEN 16157/obshell
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 3166/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2563/sshd
tcp6 0 0 ::1:631 :::* LISTEN 2561/cupsd
tcp6 0 0 ::1:25 :::* LISTEN 2759/master
tcp6 0 0 :::2882 :::* LISTEN 15401/observer
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::22 :::* LISTEN 2563/sshd

AntTech_YGJQAX · 2024 年10 月 22 日 15:05

[admin@node74 soft]$ obd obdiag rca run ceshi --scene=disconnection
Get local repositories and plugins ok
The tool oceanbase-diagnostic-tool is already installed the latest version 2.4.0
Open ssh connection ok
No time option provided, default processing is based on the last 30 minutes
gather from_time: 2024-10-22 14:34:33, to_time: 2024-10-22 15:05:33
Gather ObProxy Log Summary:
±-----------±-----------------±-------±-------±-----------+
| Node | Status | Size | Time | PackPath |
+============+==================+========+========+============+
| 10.10.1.75 | Error:Empty file | 0.000B | 0 s | |
±-----------±-----------------±-------±-------±-----------+
| 10.10.1.76 | Error:Empty file | 0.000B | 0 s | |
±-----------±-----------------±-------±-------±-----------+
| 10.10.1.77 | Error:Empty file | 0.000B | 0 s | |
±-----------±-----------------±-------±-------±-----------+
[WARN] not found log about disconnection. On node: 10.10.1.75
No time option provided, default processing is based on the last 30 minutes
gather from_time: 2024-10-22 14:34:36, to_time: 2024-10-22 15:05:36
Gather ObProxy Log Summary:
±-----------±-----------------±-------±-------±-----------+
| Node | Status | Size | Time | PackPath |
+============+==================+========+========+============+
| 10.10.1.75 | Error:Empty file | 0.000B | 0 s | |
±-----------±-----------------±-------±-------±-----------+
| 10.10.1.76 | Error:Empty file | 0.000B | 0 s | |
±-----------±-----------------±-------±-------±-----------+
| 10.10.1.77 | Error:Empty file | 0.000B | 0 s | |
±-----------±-----------------±-------±-------±-----------+
[WARN] not found log about disconnection. On node: 10.10.1.76
No time option provided, default processing is based on the last 30 minutes
gather from_time: 2024-10-22 14:34:38, to_time: 2024-10-22 15:05:38
Gather ObProxy Log Summary:
±-----------±-----------------±-------±-------±-----------+
| Node | Status | Size | Time | PackPath |
+============+==================+========+========+============+
| 10.10.1.75 | Error:Empty file | 0.000B | 0 s | |
±-----------±-----------------±-------±-------±-----------+
| 10.10.1.76 | Error:Empty file | 0.000B | 0 s | |
±-----------±-----------------±-------±-------±-----------+
| 10.10.1.77 | Error:Empty file | 0.000B | 0 s | |
±-----------±-----------------±-------±-------±-----------+
[WARN] not found log about disconnection. On node: 10.10.1.77
rca finished. For more details, the result on ‘/tmp/soft/rca/obdiag_disconnection_20241022150433’
You can get the suggest by ‘cat /tmp/soft/rca/obdiag_disconnection_20241022150433/record’
Trace ID: e3636cdc-9043-11ef-a928-000c29c252a7
If you want to view detailed obdiag logs, please run: /home/admin/oceanbase-diagnostic-tool/obdiag display-trace e3636cdc-9043-11ef-a928-000c29c252a7

Trace ID: e0787ae4-9043-11ef-aa15-000c29c252a7
If you want to view detailed obd logs, please run: obd display-trace e0787ae4-9043-11ef-aa15-000c29c252a7
[admin@node74 soft]$

AntTech_YGJQAX · 2024 年10 月 22 日 15:06

好像没发现啥

皇甫侯 · 2024 年10 月 22 日 15:09

我搞错了这个命令是排查断联的，看来还是直接去排查obproxy日志比较好。。看看有啥异常报错？

辞霜 · 2024 年10 月 22 日 15:09

-P2883 -uroot@proxysys#xxxx。使用2883黑屏化登录proxy看看
-P2883 -uroot@sys#xxxx。登录ob看看

AntTech_YGJQAX · 2024 年10 月 24 日 11:19

用模板配置文件就OK了，应该是某个参数的问题。