ocp页面显示所有租户不可用

OceanBase 技术问题
#1

【 使用环境 】生产环境
【 OB or 其他组件 】OCP
【 使用版本 】4.2.2
【问题描述】ocp页面显示所有租户不可用
【复现路径】无
【附件及日志】
当前租户不可用

租户状态：不可用 (持续时间：10天17小时17分钟53秒)
原因：连接 sys 执行 SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = ‘YES’ THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(SWITCHOVER_STATUS, ’ ', ‘_’) as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN (‘SYS’, ‘USER’) 失败，错误信息: (conn=3222313679) No memory or reach tenant memory limit
解决方案：请联系技术支持

扩容sys租户也未恢复，ocp中重启集群失败，报 No memory or reach tenant memory limit

#3

free -h 和select a.zone, a.SVR_IP,a.SVR_PORT, b.status,cpu_capacity,cpu_assigned_max,cpu_capacity-cpu_assigned_max as cpu_free,round(memory_limit /1024/1024/1024 ,2) as memory_total_gb,round((memory_limit-mem_capacity) /1024/1024/1024 ,2) as system_memory_gb,round(mem_assigned /1024/1024/1024 ,2) as mem_assigned_gb,round((mem_capacity-mem_assigned) /1024/1024/1024 ,2) as memory_free_gb,round(log_disk_capacity /1024/1024/1024 ,2) as log_disk_capacity_gb,round(log_disk_assigned /1024/1024/1024 ,2) as log_disk_assigned_gb,round((log_disk_capacity-log_disk_assigned) /1024/1024/1024 ,2) as log_disk_free_gb,round((data_disk_capacity /1024/1024/1024 ),2) as data_disk_gb,round((data_disk_in_use /1024/1024/1024 ),2) as data_disk_used_gb,round((data_disk_capacity-data_disk_in_use) /1024/1024/1024 ,2) as data_disk_free_gb from gv$ob_servers a join oceanbase.DBA_OB_SERVERS b on a.zone=b.zone\G;
麻烦提供下 看看。

#4

ocp的observer.log也看看。

#5

跟ocp可能没啥关系，我命令行到租户里面去执行语句也是报这个错
MySQL [(none)]> SELECT * FROM oceanbase.CDB_OB_ZONE_MAJOR_COMPACTION\G
ERROR 4013 (HY001): No memory or reach tenant memory limit

#6

admin@EM-1RLBCR3: ~$ free -h
total used free shared buff/cache available
Mem: 1.0T 637G 296G 572M 73G 361G
Swap: 16G 0B 16G

image
image1728×674 38 KB

#7

内存占用

租户内存参数：

show parameters where name in ('memstore_limit_percentage','freeze_trigger_percentage');

租户内存持有：

select TENANT_ID,SVR_IP,SVR_PORT,HOLD/1024/1024/1024,FREE/1024/1024/1024 
from oceanbase.GV$OB_TENANT_MEMORY 
where tenant_id =1002;

租户内存模块占用：

select * from V$OB_MEMORY where tenant_id=1002;

memstore占用：

select * from V$OB_MEMSTORE where tenant_id=1002;

总体实际占用的memory 大小以及大小限制：

select * from oceanbase.GV$OB_SERVERS;

memory_limit字段代表实际的memory_limit大小。

MEM_CAPACITY 是 memory_limit - system_memory

  • 内存资源：包括两个配置，MIN_MEMORY和MAX_MEMORY，他们含义如下：
    • MIN_MEMORY：表示为租户分配的最小内存规格，observer上，所有租户的MIN_MEMORY的总和不能超过物理可用内存大小MEM_CAPACITY
    • MAX_MEMORY：表示为租户分配的最大内存规格，observer上，所有租户的MAX_MEMORY的总和不能超过物理可用内存的超卖值：MEM_CAPACITY * resource_hard_limit

看看租户内存大小

#8

ob-sys-memory.txt (47.3 KB)

#9

内存相关问题

https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000000818714

看一下这个文档 有这个报错的解决办法

#10

没有用，调整sys租户这个参数ob_sql_work_area_percentage为10没有效果

#11

如果是测试环境手动发起一次合并试试

#12

老师你好我也遇到这个问题了,想问一下怎么操作呢
image

#13

昨天就发起过了，想要查合并状态也报错了。
MySQL [(none)]> SELECT * FROM oceanbase.CDB_OB_ZONE_MAJOR_COMPACTION\G
ERROR 4013 (HY001): No memory or reach tenant memory limit

#14

先通过报错4013错误去grep一下observer.log日志，可以获取租户id
再根据租户id进行grep ‘malloc_allocator.*tenant: #tenant_id#’ observer.log -A 可以获取租户的内存元信息
麻烦再将两个信息贴出来

#15

sys租户
[2024-06-25 16:29:58.291841] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [62165][MemDumpTimer][T1002][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant: 1002, limit: 1,073,741,824 hold: 936,050,688 rpc_hold: 0 cache_hold: 123,731,968 cache_used: 123,731,968 cache_item_count: 59

#16

执行下这个试试grep ‘malloc_allocator.*tenant: 1002’ observer.log -A 然后发日志发一下看看

#17

[2024-06-25 16:42:48.490525] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:176) [62165][MemDumpTimer][T1002][Y0-0000000000000000-0-0] [lt=6]
[MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 553,189,376 used= 470,721,456 limit= 9,223,372,036,854,775,807
[MEMORY] idle_size= 0 free_size= 0
[MEMORY] wash_related_chunks= 91 washed_blocks= 767 washed_size= 92,307,456
[MEMORY] hold= 209,817,600 used= 209,715,200 count= 5 avg_used= 41,943,040 block_cnt= 5 chunk_cnt= 5 mod=LogGroupBuffer
[MEMORY] hold= 22,888,448 used= 22,877,184 count= 11 avg_used= 2,079,744 block_cnt= 11 chunk_cnt= 11 mod=CACHE_MAP_NODE
[MEMORY] hold= 18,653,184 used= 16,689,651 count= 253 avg_used= 65,967 block_cnt= 253 chunk_cnt= 64 mod=SqlDtlBuf
[MEMORY] hold= 18,616,320 used= 18,493,440 count= 10 avg_used= 1,849,344 block_cnt= 10 chunk_cnt= 8 mod=LogDIOAligned
[MEMORY] hold= 15,777,792 used= 14,024,704 count= 214 avg_used= 65,536 block_cnt= 214 chunk_cnt= 51 mod=MinorMergeMgr
[MEMORY] hold= 14,082,048 used= 12,573,912 count= 191 avg_used= 65,832 block_cnt= 191 chunk_cnt= 31 mod=[T]ObSessionDIB
[MEMORY] hold= 13,303,808 used= 13,289,408 count= 8 avg_used= 1,661,176 block_cnt= 8 chunk_cnt= 8 mod=MysqlRequesReco
[MEMORY] hold= 10,776,576 used= 10,735,904 count= 11 avg_used= 975,991 block_cnt= 11 chunk_cnt= 7 mod=IoControl
[MEMORY] hold= 8,895,632 used= 8,810,249 count= 79 avg_used= 111,522 block_cnt= 25 chunk_cnt= 5 mod=OmtTenant
[MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue
[MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv
[MEMORY] hold= 8,257,536 used= 8,249,344 count= 8 avg_used= 1,031,168 block_cnt= 8 chunk_cnt= 6 mod=Compressor
[MEMORY] hold= 8,257,536 used= 8,240,064 count= 2 avg_used= 4,120,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream
[MEMORY] hold= 5,529,600 used= 4,915,200 count= 75 avg_used= 65,536 block_cnt= 75 chunk_cnt= 36 mod=MajorMergeMgr
[MEMORY] hold= 5,120,000 used= 5,079,040 count= 5 avg_used= 1,015,808 block_cnt= 5 chunk_cnt= 3 mod=CommitVersions
[MEMORY] hold= 3,763,584 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 10 mod=CkptDgnMemCU
[MEMORY] hold= 3,763,584 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 11 mod=CkptDgnMem
[MEMORY] hold= 3,680,096 used= 3,649,568 count= 471 avg_used= 7,748 block_cnt= 471 chunk_cnt= 79 mod=TLDecoderAlloc
[MEMORY] hold= 3,489,792 used= 3,080,192 count= 50 avg_used= 61,603 block_cnt= 50 chunk_cnt= 22 mod=[T]char
[MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 6 mod=SqlDtlQueue
[MEMORY] hold= 2,756,608 used= 2,720,005 count= 3 avg_used= 906,668 block_cnt= 3 chunk_cnt= 2 mod=SqlDtl1stBuf
[MEMORY] hold= 2,588,672 used= 2,560,768 count= 4 avg_used= 640,192 block_cnt= 4 chunk_cnt= 2 mod=XATimeWheel
[MEMORY] hold= 2,588,672 used= 2,560,768 count= 4 avg_used= 640,192 block_cnt= 4 chunk_cnt= 2 mod=TransTimeWheel
[MEMORY] hold= 2,547,712 used= 2,529,856 count= 1 avg_used= 2,529,856 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo
[MEMORY] hold= 2,523,136 used= 2,505,184 count= 11 avg_used= 227,744 block_cnt= 11 chunk_cnt= 10 mod=SQLSessionInfo
[MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId
[MEMORY] hold= 2,437,120 used= 2,414,336 count= 2 avg_used= 1,207,168 block_cnt= 2 chunk_cnt= 2 mod=SqlPlanMon
[MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap
[MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe
[MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta
[MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb
[MEMORY] hold= 2,080,768 used= 2,079,744 count= 1 avg_used= 2,079,744 block_cnt= 1 chunk_cnt= 1 mod=LogReplayTask
[MEMORY] hold= 1,925,120 used= 1,885,600 count= 5 avg_used= 377,120 block_cnt= 5 chunk_cnt= 2 mod=ClogGe
[MEMORY] hold= 1,784,704 used= 1,746,772 count= 14 avg_used= 124,769 block_cnt= 14 chunk_cnt= 14 mod=CsSstableReader
[MEMORY] hold= 1,777,664 used= 1,770,696 count= 1 avg_used= 1,770,696 block_cnt= 1 chunk_cnt= 1 mod=TableProc
[MEMORY] hold= 1,597,440 used= 1,431,456 count= 39 avg_used= 36,704 block_cnt= 39 chunk_cnt= 18 mod=CommSysVarFac
[MEMORY] hold= 1,507,328 used= 1,504,384 count= 23 avg_used= 65,408 block_cnt= 23 chunk_cnt= 15 mod=MemTblObj
[MEMORY] hold= 1,376,256 used= 1,343,488 count= 4 avg_used= 335,872 block_cnt= 4 chunk_cnt= 1 mod=IOWorkerLQ
[MEMORY] hold= 1,265,328 used= 1,253,464 count= 60 avg_used= 20,891 block_cnt= 60 chunk_cnt= 26 mod=TabletMap
[MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac
[MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap
[MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap
[MEMORY] hold= 1,045,344 used= 710,016 count= 4,602 avg_used= 154 block_cnt= 17 chunk_cnt= 14 mod=TLDecoderCtx
[MEMORY] hold= 955,360 used= 853,350 count= 39 avg_used= 21,880 block_cnt= 35 chunk_cnt= 22 mod=SqlSessiQuerSql
[MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue
[MEMORY] hold= 884,736 used= 461,376 count= 54 avg_used= 8,544 block_cnt= 54 chunk_cnt= 9 mod=[T]ObDSActionAr
[MEMORY] hold= 829,632 used= 804,864 count= 258 avg_used= 3,119 block_cnt= 130 chunk_cnt= 2 mod=LSMap
[MEMORY] hold= 811,008 used= 720,896 count= 11 avg_used= 65,536 block_cnt= 11 chunk_cnt= 7 mod=SuspectInfoMgr
[MEMORY] hold= 802,944 used= 786,432 count= 256 avg_used= 3,072 block_cnt= 128 chunk_cnt= 4 mod=TabletToLS
[MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec
[MEMORY] hold= 786,752 used= 524,352 count= 36 avg_used= 14,565 block_cnt= 34 chunk_cnt= 2 mod=LogAlloc
[MEMORY] hold= 738,592 used= 732,448 count= 94 avg_used= 7,792 block_cnt= 94 chunk_cnt= 12 mod=CkptDgnMemNode
[MEMORY] hold= 672,000 used= 666,624 count= 84 avg_used= 7,936 block_cnt= 84 chunk_cnt= 24 mod=SqlSession
[MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog
[MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue
[MEMORY] hold= 655,360 used= 649,216 count= 2 avg_used= 324,608 block_cnt= 2 chunk_cnt= 1 mod=ServerObjecPool
[MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease
[MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF
[MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR
[MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC
[MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC
[MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger
[MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT
[MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer
[MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer
[MEMORY] hold= 614,400 used= 553,040 count= 10 avg_used= 55,304 block_cnt= 10 chunk_cnt= 4 mod=TabletIDSetBkt
[MEMORY] hold= 606,208 used= 589,872 count= 2 avg_used= 294,936 block_cnt= 2 chunk_cnt= 1 mod=TmpBlockMap
[MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap
[MEMORY] hold= 589,824 used= 569,088 count= 6 avg_used= 94,848 block_cnt= 6 chunk_cnt= 5 mod=ResourceGroup
[MEMORY] hold= 535,872 used= 527,424 count= 118 avg_used= 4,469 block_cnt= 112 chunk_cnt= 44 mod=WorkerMap
[MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService
[MEMORY] hold= 458,752 used= 457,856 count= 7 avg_used= 65,408 block_cnt= 7 chunk_cnt= 6 mod=MemTblMgrObj
[MEMORY] hold= 450,560 used= 422,240 count= 5 avg_used= 84,448 block_cnt= 5 chunk_cnt= 3 mod=LSSvr
[MEMORY] hold= 450,560 used= 440,928 count= 3 avg_used= 146,976 block_cnt= 3 chunk_cnt= 3 mod=bf_queue
[MEMORY] hold= 441,952 used= 424,582 count= 5 avg_used= 84,916 block_cnt= 5 chunk_cnt= 4 mod=SqlMemMgr

#18

1002租户应该是只有1G内存吧先扩一下，扩到5G试试 :smile: