修改unit，报ERROR 4624 (HY000)

OceanBase 技术问题
#1

【产品名称】OceanBase

【产品版本】3.1.1

【问题描述】修改unit，报ERROR 4624 (HY000): machine resource ‘"172.20.0.10:2882"MAX_MEM’ is not enough to hold a new unit。操作如下：

mysql> select name,max_memory/1024/1024/1024 from __all_unit_config; show parameters like ‘system_memory’;±----------------±--------------------------+ | name | max_memory/1024/1024/1024 | ±----------------±--------------------------+ | sys_unit_config | 1.500000000000 | | mini | 2.000000000000 | ±----------------±--------------------------+

root@172.20.0.10 : oceanbase 01:33:31> show parameters like ‘system_memory’; ±------±---------±------------±---------±--------------±----------±------±------------------------------------------------------------------------------------------------------------------------------------------------------------------------±---------±--------±--------±------------------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | ±------±---------±------------±---------±--------------±----------±------±------------------------------------------------------------------------------------------------------------------------------------------------------------------------±---------±--------±--------±------------------+ | zone1 | observer | 172.20.0.10 | 2882 | system_memory | NULL | 3G | the memory reserved for internal use which cannot be allocated to any outer-tenant, and should be determined to guarantee every server functions normally. Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone2 | observer | 172.20.0.11 | 2882 | system_memory | NULL | 3G | the memory reserved for internal use which cannot be allocated to any outer-tenant, and should be determined to guarantee every server functions normally. Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone3 | observer | 172.20.0.12 | 2882 | system_memory | NULL | 3G | the memory reserved for internal use which cannot be allocated to any outer-tenant, and should be determined to guarantee every server functions normally. Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | ±------±---------±------------±---------±--------------±----------±------±------------------------------------------------------------------------------------------------------------------------------------------------------------------------±---------±--------±--------±------------------+ 3 rows in set (0.051 sec)

#2 


mysql> show parameters like 'memory_limit';
+-------+----------+-------------+----------+--------------+-----------+-------+--------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone  | svr_type | svr_ip      | svr_port | name         | data_type | value | info                                                                                                                           | section  | scope   | source  | edit_level        |
+-------+----------+-------------+----------+--------------+-----------+-------+--------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
| zone1 | observer | 172.20.0.10 |     2882 | memory_limit | NULL      | 20G   | the size of the memory reserved for internal use(for testing purpose), 0 means follow memory_limit_percentage. Range: 0, [8G,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone2 | observer | 172.20.0.11 |     2882 | memory_limit | NULL      | 20G   | the size of the memory reserved for internal use(for testing purpose), 0 means follow memory_limit_percentage. Range: 0, [8G,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone3 | observer | 172.20.0.12 |     2882 | memory_limit | NULL      | 20G   | the size of the memory reserved for internal use(for testing purpose), 0 means follow memory_limit_percentage. Range: 0, [8G,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+-------------+----------+--------------+-----------+-------+--------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+


#3 


mysql> alter resource unit mini max_memory='6G' ; 
ERROR 4624 (HY000):  machine resource '"172.20.0.10:2882"MAX_MEM' is not enough to hold a new unit


#4

宿主机24C,128G，在其上运行3个docker容器（centos:7）。


root@172.20.0.10 : oceanbase 01:35:25> system free -h
              total        used        free      shared  buff/cache   available
Mem:           125G         25G        525M        338M        100G         99G
Swap:           63G        1.7G         62G


#5

租户情况：


root@172.20.0.10 : oceanbase 01:40:42> select * from gv$tenant; 
+-----------+-------------+-------------------+-------------------+----------------+---------------+-----------+---------------------------------------------+
| tenant_id | tenant_name | zone_list         | primary_zone      | collation_type | info          | read_only | locality                                    |
+-----------+-------------+-------------------+-------------------+----------------+---------------+-----------+---------------------------------------------+
|         1 | sys         | zone1;zone2;zone3 | zone1;zone2,zone3 |              0 | system tenant |         0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
|      1001 | obcp_t1     | zone1;zone2;zone3 | zone1,zone2,zone3 |              0 |               |         0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
|      1002 | obcp_t2     | zone1;zone2;zone3 | zone1,zone2,zone3 |              0 |               |         0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
|      1004 | obcp_t3     | zone1;zone2;zone3 | zone1;zone2,zone3 |              0 |               |         0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
+-----------+-------------+-------------------+-------------------+----------------+---------------+-----------+---------------------------------------------+
4 rows in set (0.074 sec)


#6

可用资源：


root@172.20.0.10 : oceanbase 01:42:08> select a.zone,concat(a.svr_ip,':',a.svr_port) observer, cpu_total, cpu_assigned, (cpu_total-cpu_assigned) cpu_free, mem_total/1024/1024/1024 mem_total_gb, mem_a
ssigned/1024/1024/1024 mem_assign_gb, (mem_total-mem_assigned)/1024/1024/1024 mem_free_gb 
    -> from __all_virtual_server_stat a join __all_server b on (a.svr_ip=b.svr_ip and a.svr_port=b.svr_port)
    -> order by a.zone, a.svr_ip;
+-------+------------------+-----------+--------------+----------+-----------------+----------------+----------------+
| zone  | observer         | cpu_total | cpu_assigned | cpu_free | mem_total_gb    | mem_assign_gb  | mem_free_gb    |
+-------+------------------+-----------+--------------+----------+-----------------+----------------+----------------+
| zone1 | 172.20.0.10:2882 |        14 |          8.5 |      5.5 | 17.000000000000 | 7.250000000000 | 9.750000000000 |
| zone2 | 172.20.0.11:2882 |        14 |          8.5 |      5.5 | 17.000000000000 | 7.250000000000 | 9.750000000000 |
| zone3 | 172.20.0.12:2882 |        14 |          8.5 |      5.5 | 17.000000000000 | 7.250000000000 | 9.750000000000 |
+-------+------------------+-----------+--------------+----------+-----------------+----------------+----------------+
3 rows in set (0.084 sec)


#7

资源分配情况：


root@172.20.0.10 : oceanbase 01:42:51> select t1.name resource_pool_name, t2.

name
unit_config_name, t2.max_cpu, t2.min_cpu, t2.max_memory/1024/1024/1024 max_mem_gb, t2.min_memory/1024/1024/1024 min

_mem_gb, t3.unit_id, t3.zone, concat(t3.svr_ip,':',t3.

svr_port
) observer,t4.tenant_id, t4.tenant_name

  -> from __all_resource_pool t1 join __all_unit_config t2 on (t1.unit_config_id=t2.unit_config_id)

  ->   join __all_unit t3 on (t1.

resource_pool_id = t3.resource_pool_id
)

  ->   left join __all_tenant t4 on (t1.tenant_id=t4.tenant_id)

  -> order by t1.

resource_pool_id, t2.unit_config_id
, t3.unit_id;

+--------------------+------------------+---------+---------+----------------+----------------+---------+-------+------------------+-----------+-------------+

| resource_pool_name | unit_config_name | max_cpu | min_cpu | max_mem_gb   | min_mem_gb   | unit_id | zone | observer     | tenant_id | tenant_name |

+--------------------+------------------+---------+---------+----------------+----------------+---------+-------+------------------+-----------+-------------+

| sys_pool      | sys_unit_config |    5 |   2.5 | 1.500000000000 | 1.250000000000 |    1 | zone1 | 172.20.0.10:2882 |     1 | sys     |

| sys_pool      | sys_unit_config |    5 |   2.5 | 1.500000000000 | 1.250000000000 |    2 | zone2 | 172.20.0.11:2882 |     1 | sys     |

| sys_pool      | sys_unit_config |    5 |   2.5 | 1.500000000000 | 1.250000000000 |    3 | zone3 | 172.20.0.12:2882 |     1 | sys     |

| mini_pool_t1    | mini       |    2 |    2 | 4.000000000000 | 2.000000000000 |  1010 | zone1 | 172.20.0.10:2882 |   1001 | obcp_t1   |

| mini_pool_t1    | mini       |    2 |    2 | 4.000000000000 | 2.000000000000 |  1011 | zone2 | 172.20.0.11:2882 |   1001 | obcp_t1   |

| mini_pool_t1    | mini       |    2 |    2 | 4.000000000000 | 2.000000000000 |  1012 | zone3 | 172.20.0.12:2882 |   1001 | obcp_t1   |

| mini_pool_t2    | mini       |    2 |    2 | 4.000000000000 | 2.000000000000 |  1013 | zone1 | 172.20.0.10:2882 |   1002 | obcp_t2   |

| mini_pool_t2    | mini       |    2 |    2 | 4.000000000000 | 2.000000000000 |  1014 | zone2 | 172.20.0.11:2882 |   1002 | obcp_t2   |

| mini_pool_t2    | mini       |    2 |    2 | 4.000000000000 | 2.000000000000 |  1015 | zone3 | 172.20.0.12:2882 |   1002 | obcp_t2   |

| mini_pool_t3    | mini       |    2 |    2 | 4.000000000000 | 2.000000000000 |  1016 | zone1 | 172.20.0.10:2882 |   1004 | obcp_t3   |

| mini_pool_t3    | mini       |    2 |    2 | 4.000000000000 | 2.000000000000 |  1017 | zone2 | 172.20.0.11:2882 |   1004 | obcp_t3   |

| mini_pool_t3    | mini       |    2 |    2 | 4.000000000000 | 2.000000000000 |  1018 | zone3 | 172.20.0.12:2882 |   1004 | obcp_t3   |

+--------------------+------------------+---------+---------+----------------+----------------+---------+-------+------------------+-----------+-------------+

#8 
root@172.20.0.10 : oceanbase 01:43:21> SELECT svr_ip,svr_port, cpu_total, cpu_assigned,
    ->      round(mem_total/1024/1024/1024,2) mem_total_GB, round(mem_assigned/1024/1024/1024,2) mem_assigned_G,
    ->     round(disk_total/1024/1024/1024,2)  disk_total_GB,round(disk_assigned/1024/1024/1024,2) disk_assigned_GB,zone
    ->     FROM oceanbase.__all_virtual_server_stat ;
+-------------+----------+-----------+--------------+--------------+----------------+---------------+------------------+-------+
| svr_ip      | svr_port | cpu_total | cpu_assigned | mem_total_GB | mem_assigned_G | disk_total_GB | disk_assigned_GB | zone  |
+-------------+----------+-----------+--------------+--------------+----------------+---------------+------------------+-------+
| 172.20.0.10 |     2882 |        14 |          8.5 |        17.00 |           7.25 |         20.00 |            50.00 | zone1 |
| 172.20.0.11 |     2882 |        14 |          8.5 |        17.00 |           7.25 |         20.00 |            50.00 | zone2 |
| 172.20.0.12 |     2882 |        14 |          8.5 |        17.00 |           7.25 |         20.00 |            50.00 | zone3 |
+-------------+----------+-----------+--------------+--------------+----------------+---------------+------------------+-------+
3 rows in set (0.021 sec)


#9

https://www.oceanbase.com/docs/knowledgeBase/failed-to-create-table-error-code-4624

你好，这里面有排查方法

#10

显示可用资源有17GB，分配失败的原因是什么？

#11

大概找到原因：是因为有3个租户，都使用这个2C2G的mini unit，所以改mini unit的时候，设置的容量需要乘以3倍才可以。 

1 个赞