ERROR 1235 (0A000): requested memory over 90 percent of total available memory not supported

[jieguo]

【 使用环境 】测试环境
【 使用版本 】4.2.5单机企业版邀测
【问题描述】修改unit规格报错
【复现路径】
物理内存64G，memory_limit 设置为48G,租户lnzyob_unit设置为40G,现在想调小为32G，8G,2G都报错，竟然报错，why？？？bug?
内存空闲如下：
[root@ob log]# free -g
total used free shared buff/cache available
Mem: 62 28 9 0 24 33
Swap: 31 0 31

obclient(root@sys)[oceanbase]> SHOW PARAMETERS LIKE ‘%memory%’;
±------±---------±---------------±---------±--------------------------------±----------±-----------±------------------------------------------------------------------------------------------------------------------------------------------------------------------------±-------------±--------±--------±------------------±--------------±----------+
| zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | default_value | isdefault |
±------±---------±---------------±---------±--------------------------------±----------±-----------±------------------------------------------------------------------------------------------------------------------------------------------------------------------------±-------------±--------±--------±------------------±--------------±----------+
| zone1 | observer | 192.168.207.18 | 2882 | __min_full_resource_pool_memory | NULL | 1073741824 | the min memory value which is specified for a full resource pool. | LOAD_BALANCE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 5368709120 | 0 |
| zone1 | observer | 192.168.207.18 | 2882 | memory_chunk_cache_size | NULL | 0M | the maximum size of memory cached by memory chunk cache. Range: [0M,], 0 stands for adaptive | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0M | 1 |
| zone1 | observer | 192.168.207.18 | 2882 | memory_limit_percentage | NULL | 60 | the size of the memory reserved for internal use(for testing purpose). Range: [10, 95] | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 80 | 0 |
| zone1 | observer | 192.168.207.18 | 2882 | system_memory | NULL | 7G | the memory reserved for internal use which cannot be allocated to any outer-tenant, and should be determined to guarantee every server functions normally. Range: [0M,) | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0M | 0 |
| zone1 | observer | 192.168.207.18 | 2882 | memory_limit | NULL | 48G | the size of the memory reserved for internal use(for testing purpose), 0 means follow memory_limit_percentage. Range: 0, [1G,). | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0M | 0 |
| zone1 | observer | 192.168.207.18 | 2882 | memory_reserved | NULL | 500M | the size of the system memory reserved for emergency internal use. Range: [10M, total size of memory] | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 500M | 1 |
| zone1 | observer | 192.168.207.18 | 2882 | query_memory_limit_percentage | NULL | 50 | the percentage of tenant memory that can be used by a single SQL. The default value is 50. Range: [0,100] | OBSERVER | TENANT | DEFAULT | DYNAMIC_EFFECTIVE | 50 | 1 |
| zone1 | observer | 192.168.207.18 | 2882 | rpc_memory_limit_percentage | NULL | 0 | maximum memory for rpc in a tenant, as a percentage of total tenant memory, and 0 means no limit to rpc memory | RPC | TENANT | DEFAULT | DYNAMIC_EFFECTIVE | 0 | 1 |
±------±---------±---------------±---------±--------------------------------±----------±-----------±------------------------------------------------------------------------------------------------------------------------------------------------------------------------±-------------±--------±--------±------------------±--------------±----------+
8 rows in set (0.007 sec)

obclient(root@sys)[oceanbase]> SELECT * FROM DBA_OB_UNIT_CONFIGS;
±---------------±----------------±---------------------------±---------------------------±--------±--------±------------±--------------±--------------------±--------------------±------------+
| UNIT_CONFIG_ID | NAME | CREATE_TIME | MODIFY_TIME | MAX_CPU | MIN_CPU | MEMORY_SIZE | LOG_DISK_SIZE | MAX_IOPS | MIN_IOPS | IOPS_WEIGHT |
±---------------±----------------±---------------------------±---------------------------±--------±--------±------------±--------------±--------------------±--------------------±------------+
| 1 | sys_unit_config | 2025-04-10 12:09:50.118951 | 2025-04-10 12:09:50.118951 | 3 | 3 | 1073741824 | 4294967296 | 9223372036854775807 | 9223372036854775807 | 3 |
| 1001 | lnzyob_unit | 2025-04-10 12:10:43.993237 | 2025-04-10 12:10:43.993237 | 12 | 12 | 42949672960 | 134217728000 | 9223372036854775807 | 9223372036854775807 | 12 |
±---------------±----------------±---------------------------±---------------------------±--------±--------±------------±--------------±--------------------±--------------------±------------+
2 rows in set (0.000 sec)

obclient(root@sys)[oceanbase]>
obclient(root@sys)[oceanbase]> ALTER RESOURCE UNIT lnzyob_unit MEMORY_SIZE = ‘32G’;
ERROR 1235 (0A000): requested memory over 90 percent of total available memory not supported
obclient(root@sys)[oceanbase]>
obclient(root@sys)[oceanbase]> ALTER RESOURCE UNIT lnzyob_unit MEMORY_SIZE = ‘8G’;
ERROR 1235 (0A000): requested memory over 90 percent of total available memory not supported
obclient(root@sys)[oceanbase]> ALTER RESOURCE UNIT lnzyob_unit MEMORY_SIZE = ‘2G’;
ERROR 1235 (0A000): requested memory over 90 percent of total available memory not supported
obclient(root@sys)[oceanbase]>

[Giant]

1. lnzyob_unit 租户是不是大量使用了 ？
2. alter system minor freeze 在进行减小unit规格试下。 或者发起合并 alter system major freeze;

[jieguo]

1. lnzyob_unit 租户是不是大量使用了 ？
   –只有一个租户。
   obclient(root@sys)[oceanbase]> select TENANT_ID,hold/1024/1024/1024,free/1024/1024/1024 from V$OB_TENANT_MEMORY;
   ±----------±--------------------±------------------------+
   | TENANT_ID | hold/1024/1024/1024 | free/1024/1024/1024 |
   ±----------±--------------------±------------------------+
   | 1 | 3.110095977783 | 0.889904022217 |
   | 500 | 0.757526397705 | 8589934591.242473601363 |
   | 508 | 0.033847808838 | 0.966152191162 |
   | 1003 | 3.172561645508 | 0.827438354492 |
   | 1004 | 20.386829376221 | 15.613170623779 |
   ±----------±--------------------±------------------------+
   5 rows in set (0.004 sec)
2. alter system minor freeze 在进行减小unit规格试下。 或者发起合并 alter system major freeze;
   –执行命令，没作用。
   
   

[Giant]

连接到 lnzyob_unit 租户下执行 ， 是触发合并和转储的 命令

这两个命令是异步执行的 。 您需要等很久才能看到执行完毕 select * from dba_ob_major_compaction;

变成idle才行。 触发后不久会变成 compacting.

[jieguo]

obclient(GISTAR@lnzyob)[GISTAR]> select * from dba_ob_major_compaction;
±--------------------±---------------------±--------------------±-----------------±-----------------±-----------±---------±-------------±-----+
| FROZEN_SCN | GLOBAL_BROADCAST_SCN | LAST_SCN | LAST_FINISH_TIME | START_TIME | STATUS | IS_ERROR | IS_SUSPENDED | INFO |
±--------------------±---------------------±--------------------±-----------------±-----------------±-----------±---------±-------------±-----+
| 1744394400408453000 | 1744394400408453000 | 1744308001151183000 | 1744308164060005 | 1744394400571833 | COMPACTING | NO | NO | NULL |
±--------------------±---------------------±--------------------±-----------------±-----------------±-----------±---------±-------------±-----+
1 row in set (0.024 sec)

COMPACTING 这个状态需要一直等吗？有没有加快的办法？

变成idle之后就可以修改规格了是吗？

[jieguo]

过了24小时还是一样的状态，没有变化是咋回事？

[独善其身]

sysctl的配置系统参数看看是不是配置小了

[jieguo]

参数如下：
[root@ob log]# free -g
total used free shared buff/cache available
Mem: 62 26 10 0 25 35
Swap: 31 0 31
[root@ob log]#

[root@ob log]# sysctl -p
fs.aio-max-nr = 1048576
vm.max_map_count = 655360
fs.file-max = 6573688
[root@ob log]#
[root@ob log]#

ob的日志也看不到错误的信息，奇怪：
[root@ob log]# pwd
/root/lnob/oceanbase/log
[root@ob log]#
[root@ob log]#
[root@ob log]# tail -5 observer.log
[2025-04-15 08:58:10.255835] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:3173) [7485][observer][T0][YB42C0A8CF12-000632C6A0ADEFEE-0-0] [lt=34] [server_start 15/18] observer check user tenant schema refreshed finish.
[2025-04-15 08:58:10.255850] INFO [SERVER] check_log_replay_over (ob_server.cpp:3180) [7485][observer][T0][YB42C0A8CF12-000632C6A0ADEFEE-0-0] [lt=5] [server_start 16/18] observer check log replay over begin.
[2025-04-15 09:11:38.397919] INFO [SERVER] check_log_replay_over (ob_server.cpp:3206) [7485][observer][T0][YB42C0A8CF12-000632C6A0ADEFEE-0-0] [lt=27] [server_start 17/18] observer check log replay over finish.
[2025-04-15 09:11:38.463783] INFO [SERVER] start (ob_server.cpp:1236) [7485][observer][T0][YB42C0A8CF12-000632C6A0ADEFEE-0-0] [lt=26] [server_start 18/18] observer start success.
[2025-04-15 09:11:38.463810] INFO [SERVER] wait (ob_server.cpp:1633) [7485][observer][T0][YB42C0A8CF12-000632C6A0ADEFEE-0-0] [lt=8] observer process wait begin.
[root@ob log]#
[root@ob log]#

[Giant]

–正在合并的tablet 表级别或者分区级别
select * from gV$OB_TABLET_COMPACTION_PROGRESS;
select * from GV$OB_TABLET_COMPACTION_HISTORY order by start_time desc;

您查询这几个表看下 ， 能不能定位到 那个 表上面 ？？

[jieguo]

@Giant

select * from GV$OB_TABLET_COMPACTION_HISTORY order by start_time desc;

空的

[Giant]

再看下合并结束了没 ？

select * from dba_ob_major_compaction;

如果是idel 了 ， 执行下 unit 那个 降规格 看下

[jieguo]

恢复正常了。感谢！！！
竟然合并了3天？？？

设置了一下 ```
ALTER SYSTEM SET major_compact_trigger = 10000;

[root@ob log]# obclient -h192.168.207.18 -P2881 -uroot@sys -p'linux25' -Doceanbase -A
Welcome to the OceanBase.  Commands end with ; or \g.
Your OceanBase connection id is 3221685204
Server version: OceanBase 4.2.5.3 (r103000152025033110-f5b88cd987f23383677f2eb53cf76ed38a48979b) (Built Mar 31 2025 11:08:20)

Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

obclient(root@sys)[oceanbase]> select * from dba_ob_major_compaction;
+---------------------+----------------------------+----------------------+---------------------+----------------------------+----------------------------+--------+----------+--------------+------+
| FROZEN_SCN          | FROZEN_TIME                | GLOBAL_BROADCAST_SCN | LAST_SCN            | LAST_FINISH_TIME           | START_TIME                 | STATUS | IS_ERROR | IS_SUSPENDED | INFO |
+---------------------+----------------------------+----------------------+---------------------+----------------------------+----------------------------+--------+----------+--------------+------+
| 1744394403474481000 | 2025-04-12 02:00:03.474481 |  1744394403474481000 | 1744394403474481000 | 2025-04-15 11:03:43.741645 | 2025-04-12 02:00:03.533936 | IDLE   | NO       | NO           |      |
+---------------------+----------------------------+----------------------+---------------------+----------------------------+----------------------------+--------+----------+--------------+------+
1 row in set (0.011 sec)


obclient(root@sys)[oceanbase]> select * from dba_ob_major_compaction;
+---------------------+----------------------------+----------------------+---------------------+----------------------------+----------------------------+--------+----------+--------------+------+
| FROZEN_SCN          | FROZEN_TIME                | GLOBAL_BROADCAST_SCN | LAST_SCN            | LAST_FINISH_TIME           | START_TIME                 | STATUS | IS_ERROR | IS_SUSPENDED | INFO |
+---------------------+----------------------------+----------------------+---------------------+----------------------------+----------------------------+--------+----------+--------------+------+
| 1744394403474481000 | 2025-04-12 02:00:03.474481 |  1744394403474481000 | 1744394403474481000 | 2025-04-15 11:03:43.741645 | 2025-04-12 02:00:03.533936 | IDLE   | NO       | NO           |      |
+---------------------+----------------------------+----------------------+---------------------+----------------------------+----------------------------+--------+----------+--------------+------+
1 row in set (0.002 sec)

obclient(root@sys)[oceanbase]>  select TENANT_ID,hold/1024/1024/1024,free/1024/1024/1024 from V$OB_TENANT_MEMORY;
+-----------+---------------------+-------------------------+
| TENANT_ID | hold/1024/1024/1024 | free/1024/1024/1024     |
+-----------+---------------------+-------------------------+
|         1 |      2.143009185791 |          1.856990814209 |
|       500 |      0.790748596191 | 8589934591.209251402877 |
|       508 |      0.033847808838 |          0.966152191162 |
|      1003 |      1.933799743652 |          2.066200256348 |
|      1004 |      5.255088806152 |         30.744911193848 |
+-----------+---------------------+-------------------------+
5 rows in set (0.021 sec)

obclient(root@sys)[oceanbase]> SELECT      ctx_name,      SUM(hold) / 1024 / 1024/1024 AS hold_gb  FROM oceanbase.GV$OB_MEMORY  WHERE tenant_id = 1004 GROUP BY ctx_name;
+-------------------------------+----------------+
| ctx_name                      | hold_gb        |
+-------------------------------+----------------+
| DEFAULT_CTX_ID                | 1.331318899988 |
| DO_NOT_USE_ME                 | 0.000000000000 |
| MEMSTORE_CTX_ID               | 2.023132324219 |
| EXECUTE_CTX_ID                | 0.000000000000 |
| TRANS_CTX_MGR_ID              | 0.000869750977 |
| PLAN_CACHE_CTX_ID             | 0.118287801742 |
| WORK_AREA                     | 0.000000000000 |
| GLIBC                         | 0.017464235425 |
| CO_STACK                      | 0.151405334473 |
| LIBEASY                       | 0.000000000000 |
| LOGGER_CTX_ID                 | 0.000000000000 |
| KVSTORE_CACHE_ID              | 1.296875000000 |
| META_OBJ_CTX_ID               | 0.011421203613 |
| TX_CALLBACK_CTX_ID            | 0.032512247562 |
| LOB_CTX_ID                    | 0.000007569789 |
| PS_CACHE_CTX_ID               | 0.000081211328 |
| RPC_CTX_ID                    | 0.000000000000 |
| PKT_NIO                       | 0.000000000000 |
| TX_DATA_TABLE                 | 0.025560140609 |
| STORAGE_LONG_TERM_META_CTX_ID | 0.000000000000 |
| MDS_DATA_ID                   | 0.001302003860 |
| MDS_CTX_ID                    | 0.000000000000 |
| SCHEMA_SERVICE                | 0.000000000000 |
| UNEXPECTED_IN_500             | 0.000000000000 |
+-------------------------------+----------------+
24 rows in set (0.007 sec)

obclient(root@sys)[oceanbase]> select * from oceanbase.GV$OB_UNITS where TENANT_ID=1004;
+----------------+----------+---------+-----------+-------+-----------+---------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
| SVR_IP         | SVR_PORT | UNIT_ID | TENANT_ID | ZONE  | ZONE_TYPE | REGION        | MAX_CPU | MIN_CPU | MEMORY_SIZE | MAX_IOPS            | MIN_IOPS            | IOPS_WEIGHT | LOG_DISK_SIZE | LOG_DISK_IN_USE | DATA_DISK_IN_USE | STATUS | CREATE_TIME                |
+----------------+----------+---------+-----------+-------+-----------+---------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
| 192.168.207.18 |     2882 |    1001 |      1004 | zone1 | ReadWrite | deault_region |      12 |      12 | 38654705664 | 9223372036854775807 | 9223372036854775807 |          12 |  120795955200 |     29944199004 |        490151936 | NORMAL | 2025-04-10 12:40:23.491977 |
+----------------+----------+---------+-----------+-------+-----------+---------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
1 row in set (0.022 sec)

obclient(root@sys)[oceanbase]> ALTER RESOURCE UNIT lnzyob_unit MEMORY_SIZE = '32G';
Query OK, 0 rows affected (0.021 sec)

obclient(root@sys)[oceanbase]> select * from oceanbase.GV$OB_UNITS where TENANT_ID=1004;
+----------------+----------+---------+-----------+-------+-----------+---------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
| SVR_IP         | SVR_PORT | UNIT_ID | TENANT_ID | ZONE  | ZONE_TYPE | REGION        | MAX_CPU | MIN_CPU | MEMORY_SIZE | MAX_IOPS            | MIN_IOPS            | IOPS_WEIGHT | LOG_DISK_SIZE | LOG_DISK_IN_USE | DATA_DISK_IN_USE | STATUS | CREATE_TIME                |
+----------------+----------+---------+-----------+-------+-----------+---------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
| 192.168.207.18 |     2882 |    1001 |      1004 | zone1 | ReadWrite | deault_region |      12 |      12 | 30923764532 | 9223372036854775807 | 9223372036854775807 |          12 |  120795955200 |     29963919442 |        490151936 | NORMAL | 2025-04-10 12:40:23.491977 |
+----------------+----------+---------+-----------+-------+-----------+---------------+---------+---------+-------------+---------------------+---------------------+-------------+---------------+-----------------+------------------+--------+----------------------------+
1 row in set (0.000 sec)

obclient(root@sys)[oceanbase]>