ocp告警：the hold memory of tenant_500 is over the reserved memory

OceanBase 技术问题
,
#1

【 使用环境 】生产环境
【 OB or 其他组件 】ocp
【 使用版本 】oceanbase-all-in-one-4.3.1.0-100000032024051615.el7.x86_64.tar
【问题描述】ocp接管oceanbase后，出现告警，如下：

image
image1735×878 47.2 KB

#3

system_memory 这个参数
system_memory-OceanBase 数据库-OceanBase文档中心-分布式数据库使用文档

#4

看一下system_memory设置的多大

#5

30G

#6

select * from gv$ob_memory where tenant_id=500 order by used desc limit 20;
可以看下那个模块占用内存大

#7

clog盘是否满过，手工做过什么操作么，
日志中grep过滤下“the hold memory of tenant_500 is over the reserved memory”
执行select * from __all_virtual_memory_info where tenant_id=500 and svr_ip=‘192.168.1.132’ order by hold desc limit 20;

#8
tenant_id svr_ip svr_port ctx_id label ctx_name mod_type mod_id mod_name zone hold used count
500 192.168.1.132 2882 8 CoStack CO_STACK user 0 CoStack zone1 471711744 470834304 914
500 192.168.1.132 2882 22 SchemaSysCache SCHEMA_SERVICE user 0 SchemaSysCache zone1 98779392 98027481 10426
500 192.168.1.132 2882 17 DEFAULT PKT_NIO user 0 DEFAULT zone1 89227296 88707592 168
500 192.168.1.132 2882 0 Compressor DEFAULT_CTX_ID user 0 Compressor zone1 66584704 66551824 33
500 192.168.1.132 2882 0 TZInfoArray DEFAULT_CTX_ID user 0 TZInfoArray zone1 52268992 50996736 3317
500 192.168.1.132 2882 23 CACHE_MB_HANDLE UNEXPECTED_IN_500 user 0 CACHE_MB_HANDLE zone1 39538688 39519360 1
500 192.168.1.132 2882 22 TenantSchemMgr SCHEMA_SERVICE user 0 TenantSchemMgr zone1 37533824 37514752 28
500 192.168.1.132 2882 0 ash_list DEFAULT_CTX_ID user 0 ash_list zone1 31477760 31458304 1
500 192.168.1.132 2882 23 di_tenant_cache UNEXPECTED_IN_500 user 0 di_tenant_cache zone1 30007296 29990928 33
500 192.168.1.132 2882 0 IoControl DEFAULT_CTX_ID user 0 IoControl zone1 22695936 22633792 8
500 192.168.1.132 2882 23 OccamThreadPool UNEXPECTED_IN_500 user 0 OccamThreadPool zone1 20137568 18405184 398
500 192.168.1.132 2882 0 [T]ObSessionDIB DEFAULT_CTX_ID user 0 [T]ObSessionDIB zone1 19095552 17906224 259
500 192.168.1.132 2882 23 StorageLoggerM UNEXPECTED_IN_500 user 0 StorageLoggerM zone1 17329216 8968960 1093
500 192.168.1.132 2882 0 ModulePageAlloc DEFAULT_CTX_ID user 0 ModulePageAlloc zone1 16881280 16744960 2110
500 192.168.1.132 2882 23 CACHE_MAP_BKT UNEXPECTED_IN_500 user 0 CACHE_MAP_BKT zone1 16818256 16777232 3
500 192.168.1.132 2882 23 FixeSizeBlocAll UNEXPECTED_IN_500 user 0 FixeSizeBlocAll zone1 16806272 16785696 3
500 192.168.1.132 2882 7 glibc_malloc GLIBC user 0 glibc_malloc zone1 16429360 10973083 75761
500 192.168.1.132 2882 23 TenantConfig UNEXPECTED_IN_500 user 0 TenantConfig zone1 14028800 13926320 5
500 192.168.1.132 2882 0 MemDumpContext DEFAULT_CTX_ID user 0 MemDumpContext zone1 12873728 12856352 1
500 192.168.1.132 2882 0 Iterator<BtreeI DEFAULT_CTX_ID user 0 Iterator<BtreeI zone1 10502144 10477568 64
#9
TENANT_ID SVR_IP SVR_PORT CTX_NAME MOD_NAME COUNT HOLD USED
500 192.168.1.27 2882 CO_STACK CoStack 919 474292224 473409984
500 192.168.1.132 2882 CO_STACK CoStack 914 471711744 470834304
500 192.168.1.54 2882 CO_STACK CoStack 914 471711744 470834304
500 192.168.1.27 2882 SCHEMA_SERVICE SchemaSysCache 10430 98785344 98033161
500 192.168.1.132 2882 SCHEMA_SERVICE SchemaSysCache 10426 98779392 98027481
500 192.168.1.54 2882 SCHEMA_SERVICE SchemaSysCache 10426 98779424 98027481
500 192.168.1.27 2882 PKT_NIO DEFAULT 168 89227296 88707592
500 192.168.1.132 2882 PKT_NIO DEFAULT 168 89227296 88707592
500 192.168.1.54 2882 PKT_NIO DEFAULT 168 89227296 88707592
500 192.168.1.27 2882 DEFAULT_CTX_ID Compressor 33 66584704 66551824
500 192.168.1.132 2882 DEFAULT_CTX_ID Compressor 33 66584704 66551824
500 192.168.1.54 2882 DEFAULT_CTX_ID Compressor 33 66584704 66551824
500 192.168.1.27 2882 DEFAULT_CTX_ID TZInfoArray 3317 52268992 50996736
500 192.168.1.132 2882 DEFAULT_CTX_ID TZInfoArray 3317 52268992 50996736
500 192.168.1.54 2882 DEFAULT_CTX_ID TZInfoArray 3317 52268992 50996736
500 192.168.1.27 2882 SCHEMA_SERVICE TenantSchemMgr 31 43776128 43753984
500 192.168.1.54 2882 SCHEMA_SERVICE TenantSchemMgr 30 41695360 41674240
500 192.168.1.132 2882 UNEXPECTED_IN_500 CACHE_MB_HANDLE 1 39538688 39519360
500 192.168.1.27 2882 UNEXPECTED_IN_500 CACHE_MB_HANDLE 1 39538688 39519360
500 192.168.1.54 2882 UNEXPECTED_IN_500 CACHE_MB_HANDLE 1 39538688 39519360
#10

用obdiag 捞点信息回来，obdiag gather scene run --scene=observer.memory

#11

也上传一下observer.log日志呢

#12

observer.7z (3.5 MB)

#13

机器什么配置啊，现在看500租户[MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 1,423,876,096，没用多少内存，有重启吗，有错误时间的日志吗

#14

image
image829×82 2.9 KB

image
image705×507 11.4 KB

image
通过白屏安装完成oceanbase后没有重启过，ocp接管集群之后就一直有这个告警。

#15

image
image1360×748 77.1 KB

image
image817×376 35.7 KB

你看下这里呢

#16

image
image1838×873 111 KB

#17

你是使用obd部署的吗，能发下yaml文件吗，

image
image741×397 18.7 KB

#18

图中3.6gb就达到100%了，你sytem_memory设置的30G肯定是有问题的，按楼上的 发一下yaml文件看下

#19

image
image1920×666 49.9 KB

这个参数我是通过ocp设置的，yaml里我也修改了，但是没有重启过。
tmp9_alle9l.zip (957 字节)

#20

直接登录集群里看下参数的值呢，这是我的测试环境，memory_limit=6G, system_memory=1G, 计算处理的值
image

#21

大佬，怎么登录查看呢？:joy: