OB高可用验证，OBProxy高可用容灾异常
环境信息
- observer版本:
obclient -u root -p -h 127.0.0.1 -P 2883
Enter password:
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 1998
Server version: 5.6.25 OceanBase 3.1.4 (r10000092022071511-b4bfa011ceaef428782dcb65ae89190c40b78c2f) (Built Jul 15 2022 11:45:12)
Copyright (c) 2000, 2022, OceanBase and/or its affiliates. All rights reserved.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
obclient [(none)]> select version();
+--------------------+
| version() |
+--------------------+
| 3.1.4-OceanBase CE |
+--------------------+
1 row in set (0.004 sec)
- obproxy版本:
/root/obproxy/bin/obproxy --version
obproxy (OceanBase 3.2.3 2.el8)
REVISION: 6-local-99faebfc7130b70ad0f56330a28cab6a32ec9a33
BUILD_TIME: Mar 30 2022 01:53:20
- 集群架构
1-1-1 三节点+OBProxy
测试步骤
- sysbench持续压测
time sysbench --test=./oltp_read_only.lua --mysql-host=127.0.0.1 --mysql-port=2883 --mysql-db=sysbenchdb --mysql-user="ob_cluster:tenant_2:u_sysbench" --mysql-password=123456 --tables=16 --table_size=5000000 --threads=32 --time=3000 --report-interval=1 --db-driver=mysql --db-ps-mode=disable --skip-trx=on --mysql-ignore-errors=6002,6004,4012,2013,4016,1062 run
观察到三个节点都有流量
2.kill 启动一台OB进程
ps -ef|grep observer
root 783095 1 99 10:21 ? 00:07:28 /mydata/observer1/bin/observer -r 10.140.114.12:2882:2881;10.140.60.14:2882:2881;10.140.118.7:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=14G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_disk_percentage=20,enable_syslog_recycle=True,max_syslog_file_count=4 -z zone2 -p 2881 -P 2882 -n ob_cluster -c 1 -d /mydata/observer1/store -i eth0 -l DEBUG
root 783936 779377 0 10:25 pts/0 00:00:00 grep --color=auto observer
[root@dhy02 ~]# kill -9 783095
3.观察结果
- Leader全部切换到另外两个节点上
obclient [oceanbase]> SELECT tenant.tenant_name, meta.table_id, tab.table_name, partition_id, ZONE, concat(svr_ip, ':', svr_port) observer , CASE WHEN ROLE=1 THEN 'leader' WHEN ROLE=2 THEN 'follower' ELSE NULL END AS ROLE, tab.primary_zone FROM __all_virtual_meta_table meta INNER JOIN __all_tenant tenant ON meta.tenant_id=tenant.tenant_id INNER JOIN __all_virtual_table tab ON meta.tenant_id=tab.tenant_id AND meta.table_id=tab.table_id WHERE tenant.tenant_id='1001' and role = 1 ORDER BY tenant.tenant_name, TABLE_NAME, partition_id, ZONE;
+-------------+------------------+------------+--------------+-------+--------------------+--------+--------------+
| tenant_name | table_id | table_name | partition_id | ZONE | observer | ROLE | primary_zone |
+-------------+------------------+------------+--------------+-------+--------------------+--------+--------------+
| tenant_2 | 1100611139453890 | sbtest1 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453900 | sbtest10 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453915 | sbtest11 | 0 | zone1 | 10.140.114.12:2882 | leader | |
| tenant_2 | 1100611139453904 | sbtest12 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453917 | sbtest13 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453906 | sbtest14 | 0 | zone1 | 10.140.114.12:2882 | leader | |
| tenant_2 | 1100611139453919 | sbtest15 | 0 | zone1 | 10.140.114.12:2882 | leader | |
| tenant_2 | 1100611139453910 | sbtest16 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453889 | sbtest2 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453896 | sbtest3 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453892 | sbtest4 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453902 | sbtest5 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453894 | sbtest6 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453908 | sbtest7 | 0 | zone1 | 10.140.114.12:2882 | leader | |
| tenant_2 | 1100611139453898 | sbtest8 | 0 | zone3 | 10.140.118.7:2882 | leader | |
| tenant_2 | 1100611139453913 | sbtest9 | 0 | zone1 | 10.140.114.12:2882 | leader | |
+-------------+------------------+------------+--------------+-------+--------------------+--------+--------------+
16 rows in set (0.060 sec)
-
三台机器上均没有流量
-
sysbench全部掉为0
从现象上并没有能做到当后端ob节点发生切换后，OBProxy能拉取最新的信息，保持会连连接不中断。
