logo社区
官网首页
Search

ocp_monitor库，cpu、内存被打满，想知道怎么去查为什么会被打满

OceanBase 技术问题
#14

5节点

#15

monitor租户现在是2个zone，会不会和这个有关

#16

和是否是偶数副本没关系，还是得看ocp上托管的业务集群、机器数量，另外刚也提到过，业务流量大的话也会导致ocp monitordb的资源消耗过大，应为会采集业务集群的sql执行的数据

#17

我看你上面提到有11个集群，是每个集群都是5节点，总计55个节点？

#18

对，打算扩一下monitor看看有没有改善，现在ocp创建租户变的很慢，一个任务要执行很久

#19

现在应该没有业务流量啊，租户刚新建的

#20

55个节点，再加上这么多租户，这个ocp monitordb的资源有点吃紧。业务集群有没有在搞迁移的

另外有个排查思路：查询monitordb的sql采集数据，确定是否业务有数据激增，以及那张表的数据量比较大


select cluster_name,count(1) from ob_hist_sql_audit_stat_0 partition(P20240205) group by cluster_name;

select cluster_name,count(1) from ob_hist_sql_audit_stat_0 partition(P20240206) group by cluster_name;

select cluster_name,count(1) from ob_hist_sql_audit_stat_0 partition(P20240207) group by cluster_name;

select cluster_name,count(1) from ob_hist_sql_audit_sample partition(P20240205) group by cluster_name;

select cluster_name,count(1) from ob_hist_sql_audit_sample partition(P20240206) group by cluster_name;

select cluster_name,count(1) from ob_hist_sql_audit_sample partition(P20240207) group by cluster_name;

select cluster_name,count(1) from ob_hist_sqltext partition(P20240205) group by cluster_name;

select cluster_name,count(1) from ob_hist_sqltext partition(P20240206) group by cluster_name;

select cluster_name,count(1) from ob_hist_sqltext partition(P20240207) group by cluster_name;

select cluster_name,count(1) from ob_hist_plan_cache_plan_stat_0 partition(P20240205) group by cluster_name;

select cluster_name,count(1) from ob_hist_plan_cache_plan_stat_0 partition(P20240206) group by cluster_name;

select cluster_name,count(1) from ob_hist_plan_cache_plan_stat_0 partition(P20240207) group by cluster_name;

select cluster_name,count(1) from ob_hist_plan_cache_plan_explain partition(P20240205) group by cluster_name;

select cluster_name,count(1) from ob_hist_plan_cache_plan_explain partition(P20240206) group by cluster_name;

select cluster_name,count(1) from ob_hist_plan_cache_plan_explain partition(P20240207) group by cluster_name;

select cluster_name,count(1) from ob_hist_plan_cache_plan partition(P20240205) group by cluster_name;

select cluster_name,count(1) from ob_hist_plan_cache_plan partition(P20240206) group by cluster_name;

select cluster_name,count(1) from ob_hist_plan_cache_plan partition(P20240207) group by cluster_name;

select cluster_name,count(1) from ob_hist_sql_plan_monitor partition(P20240205) group by cluster_name;

select cluster_name,count(1) from ob_hist_sql_plan_monitor partition(P20240206) group by cluster_name;

select cluster_name,count(1) from ob_hist_sql_plan_monitor partition(P20240207) group by cluster_name;

select cluster_name,count(1) from ob_hist_trans_stat partition(P20240205) group by cluster_name;

select cluster_name,count(1) from ob_hist_trans_stat partition(P20240206) group by cluster_name;

select cluster_name,count(1) from ob_hist_trans_stat partition(P20240207) group by cluster_name;
#21

先别着急扩资源，先查查原因。

#22

好的，我查查

#23

这一整个环境都没人用，现在只是在创建租户，没有迁移啥的

#24

上面的sql查了，也没采集到很多数据吗，比如单日超过100W以上数据那种集群？也没有的话，那确实不太符合预期。用上面发的obdiag工具捞点ocp monitordb集群的诊断信息上来吧。

#25

大佬，我刚查出来，有百万的数据

#26

但百万数据是5号的，今天只有几千

#27

如果上面sql查出来的数据每个集群都不多的话，那可能就不是因为业务sql采集导致的cpu和内存高。
换一个排查思路：你提到的创建租户等任务慢，发起的租户创建任务大概是什么时间呢，耗时多久？另外合并的时间你没改过吧，默认是凌晨2点会合并，可以从ocp的界面去看合并的时间点。

#28

我把monitor租户资源扩起来了，没扩起来之前建一个租户大概要10分钟左右，且monitor cpu占用在90%左右，下午我重新试了一下建一个租户2分钟左右就结束了，monitor只在建租户的时候,cpu占用到40%左右，建完就降下去了。
合并没有调整，是默认值

#29

这样看，感觉像是monitor资源没给够

#30

现在monitordb扩大到多大的cpu和内存, 现在恢复到2分钟创建一个租户的任务属于是合理的时间，上午的几个怀疑点都被证实掉没啥问题，如果还需要继续跟进的话，建议还是用obdiag将上午创建租户慢的那段时间的ob日志收集一下发出来。

1 个赞
#31

我把cpu扩到32c了，上午不只是建租户慢，整个ocp都很慢，白屏操作延时比较高

#32

现在就好很多了

#33

感谢大佬的帮助