这个问题怎么分析?

业务租户反馈写入延迟从 5ms 飙升到 500ms,OBProxy 报大量 OB_TENANT_OUT_OF_MEMORY 错误。登录 Observer 发现 sys CPU 70%+,MemStore 内存使用率 95%。

5 个赞

通过这些视图查内存使用状态,GV$OB_TENANT_MEMORY,GV$OB_MEMORY,GV$OB_MEMSTORE,GV$OB_KVCACHE,GV$OB_SQL_WORKAREA_MEMORY_INFO,GV$OB_PLAN_CACHE_STAT

4 个赞

1、可以查MemStore内存查使用率
SELECT TENANT_ID, SVR_IP,
ROUND(MEMSTORE_USED/1024/1024,1) AS USED_MB,
ROUND(FREEZE_TRIGGER/1024/1024,1) AS TRIGGER_MB,
ROUND(MEMSTORE_LIMIT/1024/1024,1) AS LIMIT_MB,
ROUND(MEMSTORE_USED/MEMSTORE_LIMIT*100,0) AS USED_PCT
FROM GV$OB_MEMSTORE
WHERE TENANT_ID = 1014 ORDER BY SVR_IP;

2、 查冻结/转储状态
– 查当前冻结信息:是否有活跃转储、上次冻结时间
SELECT * FROM GV$OB_FREEZE_INFO WHERE TENANT_ID = 1014;

– 查合并状态:是否正在进行 Major Compaction
SELECT * FROM DBA_OB_MAJOR_COMPACTION;

3、 查长事务
SELECT * FROM GV$OB_TRANSACTION_PARTICIPANTS
WHERE STATE = ‘ACTIVE’ AND ELAPSED_TIME > 60000000;

如果是转储没触发或太慢,就执行: ALTER SYSTEM MINOR FREEZE TENANT = ‘xxx’;
如果是长事务阻塞冻结,就执行: KILL <session_id>; 杀掉长事务
如果是多版本数据堆积,就执行:ALTER SYSTEM MAJOR FREEZE;
如果是阈值太高,冻结太晚,就执行:ALTER SYSTEM SET freeze_trigger_percentage = 10 TENANT = ‘xxx’;
如果是MemStore 上限太低,就执行:ALTER SYSTEM SET memstore_limit_percentage = 60 TENANT = ‘xxx’;

4 个赞

感谢回答

4 个赞

厉害了

学习

66666

厉害了

属于典型的 租户内存写满导致写入限流甚至拒绝服务 的场景。
救场:
1 加快转储释放 MemStore
ALTER SYSTEM SET memstore_limit_percentage = 70 TENANT = ‘<your_tenant_name>’;
ALTER SYSTEM SET freeze_trigger_percentage = 40 TENANT = ‘<your_tenant_name>’;
2 若当前未启用限速,建议设为 80 以避免突发写入压垮内存:
ALTER SYSTEM SET writing_throttling_trigger_percentage = 80 TENANT = ‘<your_tenant_name>’;