OceanBase三节点部署&&扩容
环境信息
| IP | OB目录 | 端口 |
|---|---|---|
| 10.140.114.12 | /mydata/observer{1,2} | {2881,2882}, {3881,3882} |
| 10.140.60.14 | /mydata/observer{1,2} | {2881,2882}, {3881,3882} |
| 10.140.118.7 | /mydata/observer{1,2} | {2881,2882}, {3881,3882} |
| IP | OBProxy目录 | 端口 |
|---|---|---|
| 10.140.114.12 | /mydata/observer{1,2,3} | {2883,2884} |
搭建三节点(1-1-1)
同样使用OBD快速部署，配置文件:
cat distributed-with-obproxy.yaml
## Only need to configure when remote login is required
user:
username: root
password: xxxxx
# key_file: your ssh-key file path if need
# port: your ssh port, default 22
# timeout: ssh connection timeout (second), default 30
oceanbase-ce:
servers:
- name: z1
# Please don't use hostname, only IP can be supported
ip: 10.140.114.12
- name: z2
ip: 10.140.60.14
- name: z3
ip: 10.140.118.7
global:
# Please set devname as the network adaptor's name whose ip is in the setting of severs.
# if set severs as "127.0.0.1", please set devname as "lo"
# if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"
devname: eth0
# if current hardware's memory capacity is smaller than 50G, please use the setting of "mini-single-example.yaml" and do a small adjustment.
memory_limit: 14G
system_memory: 4G
stack_size: 512K
cpu_count: 16
cache_wash_threshold: 1G
__min_full_resource_pool_memory: 268435456
workers_per_cpu_quota: 10
schema_history_expire_time: 1d
# The value of net_thread_count had better be same as cpu's core number.
net_thread_count: 4
major_freeze_duty_time: Disable
minor_freeze_times: 10
enable_separate_sys_clog: 0
enable_merge_by_turn: FALSE
datafile_disk_percentage: 20
syslog_level: INFO
enable_syslog_recycle: true
max_syslog_file_count: 4
cluster_id: 1
# observer cluster name, consistent with obproxy's cluster_name
appname: ob_cluster
# root_password: # root user password, can be empty
# proxyro_password: # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty
# In this example , support multiple ob process in single node, so different process use different ports.
# If deploy ob cluster in multiple nodes, the port and path setting can be same.
z1:
mysql_port: 2881
rpc_port: 2882
home_path: /mydata/observer1
zone: zone1
z2:
mysql_port: 2881
rpc_port: 2882
home_path: /mydata/observer1
zone: zone2
z3:
mysql_port: 2881
rpc_port: 2882
home_path: /mydata/observer1
zone: zone3
obproxy:
servers:
- 10.140.114.12
global:
listen_port: 2883
prometheus_listen_port: 2884
home_path: /root/obproxy
# oceanbase root server list
# format: ip:mysql_port,ip:mysql_port
rs_list: 10.140.114.12:2881;10.140.60.14:2881;10.140.118.7:2881
#rs_list: 139.198.32.21:2881;139.198.32.253:2881;139.198.30.3:2881
enable_cluster_checkout: false
# observer cluster name, consistent with oceanbase-ce's appname
cluster_name: ob_cluster
# obproxy_sys_password: # obproxy sys user password, can be empty
# observer_sys_password: # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty
obd 部署:
obd cluster deploy cluster -c distributed-with-obproxy.yaml
Update OceanBase-community-stable-el8 ok
Update OceanBase-development-kit-el8 ok
oceanbase-ce-3.1.0 already installed.
obproxy-3.1.0 already installed.
+-----------------------------------------------------------------------------+
| Packages |
+--------------+---------+---------+------------------------------------------+
| Repository | Version | Release | Md5 |
+--------------+---------+---------+------------------------------------------+
| oceanbase-ce | 3.1.0 | 2.el8 | 3052acd41b981aed85ce858174e90f07d804f85b |
| obproxy | 3.1.0 | 1.el8 | d242ea5fe45222b8f61c3135ba2aaa778c61ea22 |
+--------------+---------+---------+------------------------------------------+
Open ssh connection ok
Remote oceanbase-ce-3.1.0-3052acd41b981aed85ce858174e90f07d804f85b repository install ok
Remote oceanbase-ce-3.1.0-3052acd41b981aed85ce858174e90f07d804f85b repository lib check ok
Remote obproxy-3.1.0-d242ea5fe45222b8f61c3135ba2aaa778c61ea22 repository install ok
Remote obproxy-3.1.0-d242ea5fe45222b8f61c3135ba2aaa778c61ea22 repository lib check ok
Cluster status check ok
z1(10.140.114.12) initializes cluster work home
^[[Bz2(10.140.60.14) initializes cluster work home
z3(10.140.118.7) initializes cluster work home
10.140.114.12 init cluster work home
cluster deployed
启动集群:
obd cluster start cluster
Get local repositories and plugins ok
Open ssh connection ok
Cluster param config check ok
Check before start observer ok
Check before start obproxy ok
Start observer ok
observer program health check ok
Connect to observer ok
Initialize cluster
Cluster bootstrap ok
Wait for observer init ok
+-------------------------------------------------+
| observer |
+---------------+---------+------+-------+--------+
| ip | version | port | zone | status |
+---------------+---------+------+-------+--------+
| 10.140.114.12 | 3.1.0 | 2881 | zone1 | active |
| 10.140.118.7 | 3.1.0 | 2881 | zone3 | active |
| 10.140.60.14 | 3.1.0 | 2881 | zone2 | active |
+---------------+---------+------+-------+--------+
Start obproxy ok
obproxy program health check ok
Connect to obproxy ok
Initialize cluster
+-------------------------------------------------+
| obproxy |
+---------------+------+-----------------+--------+
| ip | port | prometheus_port | status |
+---------------+------+-----------------+--------+
| 10.140.114.12 | 2883 | 2884 | active |
+---------------+------+-----------------+--------+
cluster running
创建资源池和租户
通过proxy 连接到observer， 创建资源及租户
obclient -u root -p -h 127.0.0.1 -P 2883
Enter password:
Welcome to the OceanBase. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.6.25 OceanBase 3.1.0 (r-00672c3c730c3df6eef3b359eae548d8c2db5ea2) (Built Jun 29 2021 02:14:01)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> use oceanbase;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MySQL [oceanbase]>
创建资源池，unit数量为1
MySQL [oceanbase]> create resource unit S2 max_cpu=2, min_cpu=2, max_memory='4G', min_memory='2G', max_iops=10000, min_iops=1000, max_session_num=1000000, max_disk_size='50G';
Query OK, 0 rows affected (0.009 sec)
MySQL [oceanbase]> create resource pool pool_2 unit='S2', unit_num=1;
Query OK, 0 rows affected (0.021 sec)
MySQL [oceanbase]> CREATE TENANT IF NOT EXISTS tenant_2
-> charset='utf8mb4',
-> replica_num=3,
-> zone_list=('zone1','zone2','zone3'),
-> primary_zone='RANDOM',
-> resource_pool_list=('pool_2');
Query OK, 0 rows affected (1.477 sec)
通过自己创建的租户tenant_2登录创建表:
[root@dhy01 ~]# obclient -u root@tenant_2 -p -h 127.0.0.1 -P 2881
Enter password:
Welcome to the OceanBase. Commands end with ; or \g.
Your MySQL connection id is 3221547565
Server version: 5.7.25 OceanBase 3.1.0 (r-00672c3c730c3df6eef3b359eae548d8c2db5ea2) (Built Jun 29 2021 02:14:01)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> use test;
Database changed
MySQL [test]> create tablegroup tg partition by hash partitions 3;
Query OK, 0 rows affected (0.059 sec)
MySQL [test]> create table dhytest1(id int , name char(20));
Query OK, 0 rows affected (0.122 sec)
MySQL [test]> create table dhytest2(id int , name char(20)) tablegroup=tg partition by hash (id) partitions 3 ;
Query OK, 0 rows affected (0.122 sec)
MySQL [test]> create table dhytest3(id int , name char(20)) tablegroup=tg partition by hash(id) partitions 3 ;
Query OK, 0 rows affected (0.137 sec)
MySQL [test]> create table dhytest4(id int , name char(20)) partition by hash (id) partitions 6 ;
Query OK, 0 rows affected (0.153 sec)
查看数据分布
这时所有的表leader都在2882端口上，并且dhytest4有6个分区，每个observer上面分布了两个leader
SELECT t.tenant_id, a.tenant_name, t.table_name, d.database_name, tg.tablegroup_name , t.part_num , t2.partition_id, t2.role , t2.ZONE, t2.svr_ip, concat(t2.svr_ip, ':', t2.svr_port) observer , t2.data_size , a.primary_zone , IF(t.locality = '' OR t.locality IS NULL, a.locality, t.locality) AS locality FROM oceanbase.__all_tenant AS a JOIN oceanbase.__all_virtual_database AS d ON ( a.tenant_id = d.tenant_id ) JOIN oceanbase.__all_virtual_table AS t ON (t.tenant_id = d.tenant_id AND t.database_id = d.database_id) JOIN oceanbase.__all_virtual_meta_table t2 ON (t.tenant_id = t2.tenant_id AND (t.table_id=t2.table_id OR t.tablegroup_id=t2.table_id) AND t2.ROLE IN (1) ) LEFT JOIN oceanbase.__all_virtual_tablegroup AS tg ON (t.tenant_id = tg.tenant_id and t.tablegroup_id = tg.tablegroup_id) WHERE a.tenant_id IN (1001,1002 ) AND t.table_type IN (3) and table_name in ('dhytest1','dhytest2','dhytest3', 'dhytest4') ORDER BY t.tenant_id, tg.tablegroup_name, d.database_name, t.table_name, t2.partition_id;
服务器扩容(1-1-1->2-2-2)
设置环境变量&创建目录(三台机器均执行)
export LD_LIBRARY_PATH=/root/.obd/repository/oceanbase-ce-libs/3.1.0/799bcbe7d5d1dba6eb251b8aca44394bfc5e3c14/
mkdir -p /mydata/observer{2,3}/store/{sort_dir,sstable,clog,ilog,slog}
手工启动observe进程
cd /mydata/observer2
10.140.114.12:
/root/.obd/repository/oceanbase-ce/3.1.0/3052acd41b981aed85ce858174e90f07d804f85b/bin/observer -r "10.140.114.12:2882:2881;10.140.60.14:2882:2881;10.140.118.7:2882:2881" -o __min_full_resource_pool_memory=268435456,memory_limit=14G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_disk_percentage=20,enable_syslog_recycle=True,max_syslog_file_count=4 -z zone1 -p 3881 -P 3882 -n ob_cluster -c 1 -d /mydata/observer2/store -i eth0 -l INFO
10.140.118.7:
/root/.obd/repository/oceanbase-ce/3.1.0/3052acd41b981aed85ce858174e90f07d804f85b/bin/observer -r "10.140.114.12:2882:2881;10.140.60.14:2882:2881;10.140.118.7:2882:2881" -o __min_full_resource_pool_memory=268435456,memory_limit=14G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_disk_percentage=20,enable_syslog_recycle=True,max_syslog_file_count=4 -z zone2 -p 3881 -P 3882 -n ob_cluster -c 1 -d /mydata/observer2/store -i eth0 -l INFO
10.140.60.14:
/root/.obd/repository/oceanbase-ce/3.1.0/3052acd41b981aed85ce858174e90f07d804f85b/bin/observer -r "10.140.114.12:2882:2881;10.140.60.14:2882:2881;10.140.118.7:2882:2881" -o __min_full_resource_pool_memory=268435456,memory_limit=14G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_disk_percentage=20,enable_syslog_recycle=True,max_syslog_file_count=4 -z zone3 -p 3881 -P 3882 -n ob_cluster -c 1 -d /mydata/observer2/store -i eth0 -l INFO
添加server
MySQL [oceanbase]>
MySQL [oceanbase]> alter system add server '10.140.114.12:3882' zone 'zone1';
Query OK, 0 rows affected (0.055 sec)
MySQL [oceanbase]> alter system add server '10.140.60.14:3882' zone 'zone2';
Query OK, 0 rows affected (0.079 sec)
MySQL [oceanbase]> alter system add server '10.140.118.7:3882' zone 'zone3';
Query OK, 0 rows affected (0.009 sec)
查看资源情况，现在每个zone内已经有两个observe了:
select zone,concat(svr_ip,':',svr_port) observer, cpu_capacity,cpu_total,cpu_assigned,cpu_assigned_percent, mem_capacity,mem_total,mem_assigned,mem_assigned_percent, unit_Num,round(`load`,2) `load`, round(cpu_weight,2) cpu_weight, round(memory_weight,2) mem_weight, leader_count from __all_virtual_server_stat order by zone,svr_ip;
租户扩容
升级资源的unit数量
MySQL [oceanbase]> alter resource pool pool_2 unit_num=2;
Query OK, 0 rows affected (0.146 sec)
扩容后数据分布，看到dhytest4这个表中的三个分区的leader已经迁移到3882上了:
SELECT t.tenant_id, a.tenant_name, t.table_name, d.database_name, tg.tablegroup_name , t.part_num , t2.partition_id, t2.role , t2.ZONE, t2.svr_ip, concat(t2.svr_ip, ':', t2.svr_port) observer , t2.data_size , a.primary_zone , IF(t.locality = '' OR t.locality IS NULL, a.locality, t.locality) AS locality FROM oceanbase.__all_tenant AS a JOIN oceanbase.__all_virtual_database AS d ON ( a.tenant_id = d.tenant_id ) JOIN oceanbase.__all_virtual_table AS t ON (t.tenant_id = d.tenant_id AND t.database_id = d.database_id) JOIN oceanbase.__all_virtual_meta_table t2 ON (t.tenant_id = t2.tenant_id AND (t.table_id=t2.table_id OR t.tablegroup_id=t2.table_id) AND t2.ROLE IN (1) ) LEFT JOIN oceanbase.__all_virtual_tablegroup AS tg ON (t.tenant_id = tg.tenant_id and t.tablegroup_id = tg.tablegroup_id) WHERE a.tenant_id IN (1001,1002 ) AND t.table_type IN (3) and table_name in ('dhytest1','dhytest2','dhytest3', 'dhytest4') ORDER BY t.tenant_id, tg.tablegroup_name, d.database_name, t.table_name, t2.partition_id;
租户缩容
MySQL [oceanbase]> alter resource pool pool_2 unit_num=1;
Query OK, 0 rows affected (0.146 sec)
查看__all_rootservice_job可查看进度，等待状态变成SUCCESS
删除server
执行完删除server后，通过__all_rootservice_job, __all_virtual_sys_task_status可查看数据迁移的进度， 这里是因为缩容后数据从2882迁移到了3882，如果删除server后，会在进行一次迁移。这里也有个问题，就是扩容的时候 __all_virtual_sys_task_status表中查看不到信息
