OCP重启不成功

[ootoo]

【 使用环境 】测试环境

【 OB or 其他组件 】observer、ocp、obd

【 使用版本 】observer 4.3.1.0 、 ocp 4.3.0

【问题描述】
我使用 obd cluster stop OCP_NAME 正常关闭 ocp
再使用 obd cluster start OCP_NAME 无法启动 ocp
抛出错误 "Access denied for user 'root@ocp_meta'@'xx.xx.xx.xx (using password: YES)")
于是我尝试用ocp使用的租户登录 observer ，继续抛出以下错误：

[admin@db-xx log]$ obclient -h  1xx.xx.xx.xx -P2881 -uroot@ocp_meta -pxxx
ERROR 8001 (08004): Server is initializing
[admin@db-xx log]$ obclient -h  1xx.xx.xx.xx -P2881 -uroot@ocp_monitor -pxxx
ERROR 8001 (08004): Server is initializing

看意思应该还是在初始化中，但测试 sys 租户是可以正常进入的，在继续等待了很长一段时间后，从命令行终于可以登录这两个租户了。但问题是，如果重启OCP obd cluster start/restart OCP_NAME 又会进入初始化中，很明显，这两个租户无法在 ocp 启动的过程中准备完毕

【复现路径】当前没有找到类似问题，但在我当前测试环境是必定出现的，无法成功启动

【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接（右键跳转查看）：
期间查看 observer.log 发现了两个错误，分别是 log out of disk space 和 tenant memory has reached the upper limit 分别调大了日志空间和内存大小，日志问题没有看到再出现了，内存问题仍然会出现，但最近的 observer.log 使用 obdiag 分析是 通过的，不清楚到底有没有影响

---

能否有人能指导一下如何继续排查observer 租户启动缓慢的原因，或者能否在等observer 正常之后，单独启动 OCP 进程

[辞霜]

是否手动在数据库里修改过ocp_monitor和ocp_meta的密码

[ootoo]

没有的，前面说的等待长时间初始化完毕后是可以手动通过密码登录进入的，应该就是初始化时间太长了。目前查了下社区关于OCP的问题，了解到通过 obd cluster start 重复启动应该可以恢复（不会再次重启observer），但是初始化这么慢应该如何排查呢

[ootoo]

还有就是这个租户，之前给的60G内存，日志中报错 No memory or reach tenant memory limit([OOPS]="alloc failed reason"，现在增加了内存，给到了75G左右，仍然会有这个报错。这个库数据量并不大，能否确定此错误是配置导致的问题

[辞霜]

您是说ocp的系统租户么，这两个租户实际消耗不了多少内存的（需要看管理的集群规模），关闭集群是使用obd关闭的么，下次关闭前做一次转储试试呢

[ootoo]

是的，但是现在就是与ocp其中一个租户的内存报错，详细如下:

observer.log:[2024-08-14 14:13:25.306927] WARN  common_realloc (ob_tenant_ctx_allocator.cpp:492) [3525078][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4013] No memory or reach tenant memory limit([OOPS]
="alloc failed reason", msg=tenant memory has reached the upper limit(tenant_id: 1004, tenant_hold: 72470298624, tenant_limit: 72477573120, alloc_size: 38862848))

[ootoo]

目前测试进入这个租户是成功的，但是执行show databases就报错内存不足了，另外进入数据库也非常慢，大概需要5秒以上

[admin@db-control1 log]$ obclient -h1xx.xx.xx.xx-P2881 -uroot@ocp_monitor -pxxx -A
Welcome to the OceanBase.  Commands end with ; or \g.
Your OceanBase connection id is 3221702330
Server version: OceanBase_CE 4.3.1.0 (r100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4) (Built May 16 2024 15:29:35)

Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

obclient [(none)]> show databases;
ERROR 4013 (HY001): No memory or reach tenant memory limit

[辞霜]

看着像是内存一直不释放ocp那边有什么告警么
手动发起转储试试
ALTER SYSTEM MINOR FREEZE TENANT = all;

[辞霜]

查看observer日志，通过关键词 ‘malloc_allocator.*tenant: 1004’ 可以获取租户的内存元信息，麻烦帮忙查询下发出来

[ootoo]

目前OCP启动不成功，无法查看，这条sql需要在这个问题的租户下执行吗，我使用root用户执行了，但是还没有效果

[ootoo]

这是日志检索内容：

[admin@db-control1 log]$ grep 'malloc_allocator.*tenant: 1004' observer.log*
observer.log:[2024-08-14 14:51:31.610708] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 69,838,372,864 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814144139950:[2024-08-14 14:41:08.137747] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,401,496,064 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144139950:[2024-08-14 14:41:18.279283] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 64,952,467,456 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814144139950:[2024-08-14 14:41:28.409153] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 72,258,945,024 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144139950:[2024-08-14 14:41:38.427526] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 63,110,316,032 rpc_hold: 0 cache_hold: 56,623,104 cache_used: 56,623,104 cache_item_count: 27
observer.log.20240814144212953:[2024-08-14 14:41:48.432349] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 65,703,510,016 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814144212953:[2024-08-14 14:41:58.614044] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 62,596,644,864 rpc_hold: 0 cache_hold: 50,331,648 cache_used: 50,331,648 cache_item_count: 24
observer.log.20240814144212953:[2024-08-14 14:42:08.539112] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 60,640,854,016 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814144246665:[2024-08-14 14:42:18.684988] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 69,197,955,072 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144246665:[2024-08-14 14:42:28.642398] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 66,400,681,984 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814144246665:[2024-08-14 14:42:38.793987] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,960,714,752 rpc_hold: 0 cache_hold: 39,845,888 cache_used: 39,845,888 cache_item_count: 19
observer.log.20240814144322093:[2024-08-14 14:42:48.778958] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 66,177,400,832 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144322093:[2024-08-14 14:42:58.813302] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 63,557,206,016 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814144322093:[2024-08-14 14:43:08.836495] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 69,074,223,104 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144322093:[2024-08-14 14:43:18.931932] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 71,056,359,424 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814144356492:[2024-08-14 14:43:28.865433] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 70,176,210,944 rpc_hold: 0 cache_hold: 50,331,648 cache_used: 50,331,648 cache_item_count: 24
observer.log.20240814144356492:[2024-08-14 14:43:38.938466] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 72,331,362,304 rpc_hold: 0 cache_hold: 39,845,888 cache_used: 39,845,888 cache_item_count: 19
observer.log.20240814144356492:[2024-08-14 14:43:49.020344] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 71,815,528,448 rpc_hold: 0 cache_hold: 39,845,888 cache_used: 39,845,888 cache_item_count: 19
observer.log.20240814144431480:[2024-08-14 14:43:58.958316] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 72,267,005,952 rpc_hold: 0 cache_hold: 39,845,888 cache_used: 39,845,888 cache_item_count: 19
observer.log.20240814144431480:[2024-08-14 14:44:09.124390] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 67,095,166,976 rpc_hold: 0 cache_hold: 48,234,496 cache_used: 48,234,496 cache_item_count: 23
observer.log.20240814144431480:[2024-08-14 14:44:19.220183] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 62,435,819,520 rpc_hold: 0 cache_hold: 50,331,648 cache_used: 50,331,648 cache_item_count: 24
observer.log.20240814144431480:[2024-08-14 14:44:29.150892] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 67,474,554,880 rpc_hold: 0 cache_hold: 48,234,496 cache_used: 48,234,496 cache_item_count: 23
observer.log.20240814144505869:[2024-08-14 14:44:39.303730] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 69,714,706,432 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814144505869:[2024-08-14 14:44:49.419979] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 70,158,778,368 rpc_hold: 0 cache_hold: 39,845,888 cache_used: 39,845,888 cache_item_count: 19
observer.log.20240814144505869:[2024-08-14 14:44:59.342737] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 70,675,267,584 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814144538629:[2024-08-14 14:45:09.593923] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 63,785,926,656 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814144538629:[2024-08-14 14:45:19.640681] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 62,774,509,568 rpc_hold: 0 cache_hold: 48,234,496 cache_used: 48,234,496 cache_item_count: 23
observer.log.20240814144538629:[2024-08-14 14:45:29.648791] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,313,022,464 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814144612731:[2024-08-14 14:45:39.774375] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 69,865,177,088 rpc_hold: 0 cache_hold: 39,845,888 cache_used: 39,845,888 cache_item_count: 19
observer.log.20240814144612731:[2024-08-14 14:45:49.636899] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 70,138,134,528 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814144612731:[2024-08-14 14:45:59.799042] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 71,579,009,024 rpc_hold: 0 cache_hold: 48,234,496 cache_used: 48,234,496 cache_item_count: 23
observer.log.20240814144612731:[2024-08-14 14:46:09.862975] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 67,616,505,856 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814144647551:[2024-08-14 14:46:19.916417] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 69,418,287,104 rpc_hold: 0 cache_hold: 50,331,648 cache_used: 50,331,648 cache_item_count: 24
observer.log.20240814144647551:[2024-08-14 14:46:29.961504] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 72,131,149,824 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814144647551:[2024-08-14 14:46:39.956670] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 69,156,864,000 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144723358:[2024-08-14 14:46:50.082647] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 69,984,649,216 rpc_hold: 0 cache_hold: 50,331,648 cache_used: 50,331,648 cache_item_count: 24
observer.log.20240814144723358:[2024-08-14 14:47:00.125403] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 70,774,685,696 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814144723358:[2024-08-14 14:47:10.051690] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 66,347,925,504 rpc_hold: 0 cache_hold: 48,234,496 cache_used: 48,234,496 cache_item_count: 23
observer.log.20240814144723358:[2024-08-14 14:47:20.199107] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,929,388,544 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144755504:[2024-08-14 14:47:30.206119] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,416,372,736 rpc_hold: 0 cache_hold: 54,525,952 cache_used: 54,525,952 cache_item_count: 26
observer.log.20240814144755504:[2024-08-14 14:47:40.279085] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,129,390,592 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144755504:[2024-08-14 14:47:50.485621] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,216,029,184 rpc_hold: 0 cache_hold: 50,331,648 cache_used: 50,331,648 cache_item_count: 24
observer.log.20240814144830545:[2024-08-14 14:48:00.549832] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,825,972,736 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144830545:[2024-08-14 14:48:10.571606] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 64,905,084,928 rpc_hold: 0 cache_hold: 52,428,800 cache_used: 52,428,800 cache_item_count: 25
observer.log.20240814144830545:[2024-08-14 14:48:20.542947] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 61,255,843,840 rpc_hold: 0 cache_hold: 52,428,800 cache_used: 52,428,800 cache_item_count: 25
observer.log.20240814144830545:[2024-08-14 14:48:30.539549] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 63,772,688,384 rpc_hold: 0 cache_hold: 56,623,104 cache_used: 56,623,104 cache_item_count: 27
observer.log.20240814144908866:[2024-08-14 14:48:40.626227] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,876,566,528 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814144908866:[2024-08-14 14:48:50.790218] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 65,595,179,008 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144908866:[2024-08-14 14:49:00.807159] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 69,662,605,312 rpc_hold: 0 cache_hold: 41,943,040 cache_used: 41,943,040 cache_item_count: 20
observer.log.20240814144944514:[2024-08-14 14:49:10.804232] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 65,399,160,832 rpc_hold: 0 cache_hold: 48,234,496 cache_used: 48,234,496 cache_item_count: 23
observer.log.20240814144944514:[2024-08-14 14:49:20.896636] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,450,844,672 rpc_hold: 0 cache_hold: 48,234,496 cache_used: 48,234,496 cache_item_count: 23
observer.log.20240814144944514:[2024-08-14 14:49:30.883694] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 66,401,927,168 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814144944514:[2024-08-14 14:49:40.914915] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 67,920,592,896 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814145019638:[2024-08-14 14:49:50.871273] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 70,976,077,824 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814145019638:[2024-08-14 14:50:01.145582] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 64,863,600,640 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814145019638:[2024-08-14 14:50:11.060497] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 67,071,377,408 rpc_hold: 0 cache_hold: 46,137,344 cache_used: 46,137,344 cache_item_count: 22
observer.log.20240814145055142:[2024-08-14 14:50:21.219441] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 67,222,831,104 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814145055142:[2024-08-14 14:50:31.082073] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 67,665,395,712 rpc_hold: 0 cache_hold: 48,234,496 cache_used: 48,234,496 cache_item_count: 23
observer.log.20240814145055142:[2024-08-14 14:50:41.199897] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 71,395,115,008 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814145055142:[2024-08-14 14:50:51.219151] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 71,704,641,536 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814145130660:[2024-08-14 14:51:01.284745] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,632,707,072 rpc_hold: 0 cache_hold: 48,234,496 cache_used: 48,234,496 cache_item_count: 23
observer.log.20240814145130660:[2024-08-14 14:51:11.331924] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 62,964,760,576 rpc_hold: 0 cache_hold: 44,040,192 cache_used: 44,040,192 cache_item_count: 21
observer.log.20240814145130660:[2024-08-14 14:51:21.412262] INFO  [LIB] operator() (ob_malloc_allocator.cpp:542) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1004, limit: 72,477,573,120 hold: 68,095,967,232 rpc_hold: 0 cache_hold: 50,331,648 cache_used: 50,331,648 cache_item_count: 24

这个是库内查询的内存情况:

obclient [oceanbase]> select /* MONITOR_AGENT */ con_id tenant_id, stat_id, value from v$sysstat where stat_id IN (140003,140002) and (con_id > 1000 or con_id = 1) and class < 1000
    ->
    -> ;
+-----------+---------+-------------+
| tenant_id | stat_id | value       |
+-----------+---------+-------------+
|         1 |  140002 |  7516192768 |
|         1 |  140003 |  4208918528 |
|      1001 |  140002 |  1073741824 |
|      1001 |  140003 |  1073217536 |
|      1002 |  140002 |  4294967296 |
|      1002 |  140003 |  2781216768 |
|      1003 |  140002 |  8053063680 |
|      1003 |  140003 |  4611112960 |
|      1004 |  140002 | 72477573120 |
|      1004 |  140003 | 72441331712 |
+-----------+---------+-------------+
10 rows in set (0.051 sec)

obclient [oceanbase]> select * from GV$OB_MEMORY order by USED desc limit 10;
+-----------+--------------+----------+-------------------+-----------------+-------+-------------+-------------+
| TENANT_ID | SVR_IP       | SVR_PORT | CTX_NAME          | MOD_NAME        | COUNT | HOLD        | USED        |
+-----------+--------------+----------+-------------------+-----------------+-------+-------------+-------------+
|      1004 | 172.22.5.124 |     2882 | MEMSTORE_CTX_ID   | Memstore        | 14744 | 30678843392 | 30664807104 |
|      1004 | 172.22.5.124 |     2882 | DEFAULT_CTX_ID    | SqlExecutor     | 74116 | 25809059840 | 25171144770 |
|      1004 | 172.22.5.124 |     2882 | RPC_CTX_ID        | RpcDefault      |   426 |  3728302080 |  3721111487 |
|      1004 | 172.22.5.124 |     2882 | PLAN_CACHE_CTX_ID | SqlPhyPlan      |  1238 |  1683524224 |  1677160703 |
|      1004 | 172.22.5.124 |     2882 | DEFAULT_CTX_ID    | MysqlRequesReco |   762 |  1661419520 |  1660619136 |
|      1003 | 172.22.5.124 |     2882 | MEMSTORE_CTX_ID   | Memstore        |   759 |  1579302912 |  1578580344 |
|         1 | 172.22.5.124 |     2882 | MEMSTORE_CTX_ID   | Memstore        |   442 |   919699456 |   919278672 |
|      1004 | 172.22.5.124 |     2882 | TX_DATA_TABLE     | TX_DATA_SLICE   | 89004 |   712032000 |   706335744 |
|      1002 | 172.22.5.124 |     2882 | MEMSTORE_CTX_ID   | Memstore        |   306 |   636715008 |   636423696 |
|      1003 | 172.22.5.124 |     2882 | TX_DATA_TABLE     | TX_DATA_SLICE   | 57899 |   463192000 |   459486464 |
+-----------+--------------+----------+-------------------+-----------------+-------+-------------+-------------+
10 rows in set (0.008 sec)

我看官方说这个是工作区内存不足，需要调整参数 ob_sql_work_area_percentage，但是我这边设置无效果，使用 show parameters 也查不到这个参数

[辞霜]

通过关键词 ‘1004 ctx_id=observer.log* MEMSTORE_CTX_ID’ 可以获取1004租户的MEMSTORE_CTX_ID的内存元信息（hold、used、limit、内存碎片清理信息），-A可以显示该CTX_ID的各内存模块的内存占用。帮忙执行下面这个
grep ‘1004 ctx_id= MEMSTORE_CTX_ID’ observer.log* -C 1 -A 10

[辞霜]

这好像是个变量

[ootoo]

搜索到的内容如下:

[admin@db-control1 log]$ grep -i -E '1004.*ctx_id=.*MEMSTORE_CTX_ID'  observer.log* -C 1 -A 10 --color
observer.log.20240814150149088-[2024-08-14 15:01:14.295369] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=42]
observer.log.20240814150149088:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,297,896,448 used= 31,053,381,632 limit= 9,223,372,036,854,775,807
observer.log.20240814150149088-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150149088-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150149088-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150149088-[MEMORY] hold= 31,053,381,632 used= 31,039,173,984 count=  14,924 avg_used=      2,079,816 block_cnt=  14,924 chunk_cnt=  14,924 mod=Memstore
observer.log.20240814150149088-[MEMORY] hold= 31,053,381,632 used= 31,039,173,984 count=  14,924 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150149088-[2024-08-14 15:01:14.295388] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11]
observer.log.20240814150149088-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,308,416 used=      2,141,120 limit= 9,223,372,036,854,775,807
observer.log.20240814150149088-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150149088-[MEMORY] wash_related_chunks=         5 washed_blocks=        16 washed_size= 9,371,648
observer.log.20240814150149088-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150149088-[2024-08-14 15:01:24.403124] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=49]
observer.log.20240814150149088:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,308,382,208 used= 31,063,785,472 limit= 9,223,372,036,854,775,807
observer.log.20240814150149088-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150149088-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150149088-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150149088-[MEMORY] hold= 31,063,785,472 used= 31,049,573,064 count=  14,929 avg_used=      2,079,816 block_cnt=  14,929 chunk_cnt=  14,929 mod=Memstore
observer.log.20240814150149088-[MEMORY] hold= 31,063,785,472 used= 31,049,573,064 count=  14,929 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150149088-[2024-08-14 15:01:24.403136] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5]
observer.log.20240814150149088-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,439,488 used=      1,355,264 limit= 9,223,372,036,854,775,807
observer.log.20240814150149088-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150149088-[MEMORY] wash_related_chunks=         6 washed_blocks=        19 washed_size= 11,337,728
observer.log.20240814150149088-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150149088-[2024-08-14 15:01:34.312964] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=33]
observer.log.20240814150149088:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,318,867,968 used= 31,074,189,312 limit= 9,223,372,036,854,775,807
observer.log.20240814150149088-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150149088-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150149088-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150149088-[MEMORY] hold= 31,074,189,312 used= 31,059,972,144 count=  14,934 avg_used=      2,079,816 block_cnt=  14,934 chunk_cnt=  14,934 mod=Memstore
observer.log.20240814150149088-[MEMORY] hold= 31,074,189,312 used= 31,059,972,144 count=  14,934 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150149088-[2024-08-14 15:01:34.312978] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8]
observer.log.20240814150149088-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,439,488 used=      1,518,720 limit= 9,223,372,036,854,775,807
observer.log.20240814150149088-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150149088-[MEMORY] wash_related_chunks=         6 washed_blocks=        19 washed_size= 11,337,728
observer.log.20240814150149088-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150149088-[2024-08-14 15:01:44.520718] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=36]
observer.log.20240814150149088:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,327,256,576 used= 31,082,512,384 limit= 9,223,372,036,854,775,807
observer.log.20240814150149088-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150149088-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150149088-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150149088-[MEMORY] hold= 31,082,512,384 used= 31,068,291,408 count=  14,938 avg_used=      2,079,816 block_cnt=  14,938 chunk_cnt=  14,938 mod=Memstore
observer.log.20240814150149088-[MEMORY] hold= 31,082,512,384 used= 31,068,291,408 count=  14,938 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150149088-[2024-08-14 15:01:44.520734] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8]
observer.log.20240814150149088-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,242,880 used=      2,034,432 limit= 9,223,372,036,854,775,807
observer.log.20240814150149088-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150149088-[MEMORY] wash_related_chunks=         5 washed_blocks=        19 washed_size= 9,437,184
observer.log.20240814150149088-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150222769-[2024-08-14 15:01:54.545682] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=25]
observer.log.20240814150222769:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,337,742,336 used= 31,092,916,224 limit= 9,223,372,036,854,775,807
observer.log.20240814150222769-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150222769-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150222769-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150222769-[MEMORY] hold= 31,092,916,224 used= 31,078,690,488 count=  14,943 avg_used=      2,079,816 block_cnt=  14,943 chunk_cnt=  14,943 mod=Memstore
observer.log.20240814150222769-[MEMORY] hold= 31,092,916,224 used= 31,078,690,488 count=  14,943 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150222769-[2024-08-14 15:01:54.545695] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7]
observer.log.20240814150222769-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,242,880 used=      1,977,088 limit= 9,223,372,036,854,775,807
observer.log.20240814150222769-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150222769-[MEMORY] wash_related_chunks=         5 washed_blocks=        19 washed_size= 9,437,184
observer.log.20240814150222769-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150222769-[2024-08-14 15:02:04.651303] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=45]
observer.log.20240814150222769:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,346,130,944 used= 31,101,239,296 limit= 9,223,372,036,854,775,807
observer.log.20240814150222769-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150222769-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150222769-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150222769-[MEMORY] hold= 31,101,239,296 used= 31,087,009,752 count=  14,947 avg_used=      2,079,816 block_cnt=  14,947 chunk_cnt=  14,947 mod=Memstore
observer.log.20240814150222769-[MEMORY] hold= 31,101,239,296 used= 31,087,009,752 count=  14,947 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150222769-[2024-08-14 15:02:04.651317] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7]
observer.log.20240814150222769-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,242,880 used=      1,919,936 limit= 9,223,372,036,854,775,807
observer.log.20240814150222769-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150222769-[MEMORY] wash_related_chunks=         5 washed_blocks=        19 washed_size= 9,437,184
observer.log.20240814150222769-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150222769-[2024-08-14 15:02:14.604997] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=37]
observer.log.20240814150222769:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,358,713,856 used= 31,113,723,904 limit= 9,223,372,036,854,775,807
observer.log.20240814150222769-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150222769-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150222769-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150222769-[MEMORY] hold= 31,113,723,904 used= 31,099,488,648 count=  14,953 avg_used=      2,079,816 block_cnt=  14,953 chunk_cnt=  14,953 mod=Memstore
observer.log.20240814150222769-[MEMORY] hold= 31,113,723,904 used= 31,099,488,648 count=  14,953 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150222769-[2024-08-14 15:02:14.605024] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19]
observer.log.20240814150222769-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,832,704 used=      1,739,712 limit= 9,223,372,036,854,775,807
observer.log.20240814150222769-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150222769-[MEMORY] wash_related_chunks=         6 washed_blocks=        20 washed_size= 10,944,512
observer.log.20240814150222769-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150256078-[2024-08-14 15:02:24.689344] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=35]
observer.log.20240814150256078:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,373,393,920 used= 31,128,289,280 limit= 9,223,372,036,854,775,807
observer.log.20240814150256078-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150256078-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150256078-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150256078-[MEMORY] hold= 31,128,289,280 used= 31,114,047,360 count=  14,960 avg_used=      2,079,816 block_cnt=  14,960 chunk_cnt=  14,960 mod=Memstore
observer.log.20240814150256078-[MEMORY] hold= 31,128,289,280 used= 31,114,047,360 count=  14,960 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150256078-[2024-08-14 15:02:24.689358] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7]
observer.log.20240814150256078-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      2,424,832 used=      1,568,832 limit= 9,223,372,036,854,775,807
observer.log.20240814150256078-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150256078-[MEMORY] wash_related_chunks=         7 washed_blocks=        21 washed_size= 12,255,232
observer.log.20240814150256078-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150256078-[2024-08-14 15:02:34.701309] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20]
observer.log.20240814150256078:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,381,782,528 used= 31,136,612,352 limit= 9,223,372,036,854,775,807
observer.log.20240814150256078-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150256078-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150256078-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150256078-[MEMORY] hold= 31,136,612,352 used= 31,122,366,624 count=  14,964 avg_used=      2,079,816 block_cnt=  14,964 chunk_cnt=  14,964 mod=Memstore
observer.log.20240814150256078-[MEMORY] hold= 31,136,612,352 used= 31,122,366,624 count=  14,964 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150256078-[2024-08-14 15:02:34.701320] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5]
observer.log.20240814150256078-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,963,776 used=      1,715,520 limit= 9,223,372,036,854,775,807
observer.log.20240814150256078-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150256078-[MEMORY] wash_related_chunks=         5 washed_blocks=        18 washed_size= 8,716,288
observer.log.20240814150256078-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150256078-[2024-08-14 15:02:44.800476] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=54]
observer.log.20240814150256078:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,394,365,440 used= 31,149,096,960 limit= 9,223,372,036,854,775,807
observer.log.20240814150256078-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150256078-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150256078-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150256078-[MEMORY] hold= 31,149,096,960 used= 31,134,845,520 count=  14,970 avg_used=      2,079,816 block_cnt=  14,970 chunk_cnt=  14,970 mod=Memstore
observer.log.20240814150256078-[MEMORY] hold= 31,149,096,960 used= 31,134,845,520 count=  14,970 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150256078-[2024-08-14 15:02:44.800490] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8]
observer.log.20240814150256078-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      3,866,624 used=      2,075,392 limit= 9,223,372,036,854,775,807
observer.log.20240814150256078-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150256078-[MEMORY] wash_related_chunks=         5 washed_blocks=        18 washed_size= 8,716,288
observer.log.20240814150256078-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150256078-[2024-08-14 15:02:54.785610] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=55]
observer.log.20240814150256078:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,398,559,744 used= 31,153,258,496 limit= 9,223,372,036,854,775,807
observer.log.20240814150256078-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150256078-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150256078-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150256078-[MEMORY] hold= 31,153,258,496 used= 31,139,005,152 count=  14,972 avg_used=      2,079,816 block_cnt=  14,972 chunk_cnt=  14,972 mod=Memstore
observer.log.20240814150256078-[MEMORY] hold= 31,153,258,496 used= 31,139,005,152 count=  14,972 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150256078-[2024-08-14 15:02:54.785625] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9]
observer.log.20240814150256078-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,046,272 used=      2,123,008 limit= 9,223,372,036,854,775,807
observer.log.20240814150256078-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150256078-[MEMORY] wash_related_chunks=         4 washed_blocks=        17 washed_size= 7,536,640
observer.log.20240814150256078-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150334403-[2024-08-14 15:03:04.826082] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=36]
observer.log.20240814150334403:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,409,045,504 used= 31,163,662,336 limit= 9,223,372,036,854,775,807
observer.log.20240814150334403-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150334403-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150334403-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150334403-[MEMORY] hold= 31,163,662,336 used= 31,149,404,232 count=  14,977 avg_used=      2,079,816 block_cnt=  14,977 chunk_cnt=  14,977 mod=Memstore
observer.log.20240814150334403-[MEMORY] hold= 31,163,662,336 used= 31,149,404,232 count=  14,977 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150334403-[2024-08-14 15:03:04.826097] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8]
observer.log.20240814150334403-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,046,272 used=        962,624 limit= 9,223,372,036,854,775,807
observer.log.20240814150334403-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150334403-[MEMORY] wash_related_chunks=         4 washed_blocks=        17 washed_size= 7,536,640
observer.log.20240814150334403-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150334403-[2024-08-14 15:03:14.982445] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=61]
observer.log.20240814150334403:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,430,017,024 used= 31,184,470,016 limit= 9,223,372,036,854,775,807
observer.log.20240814150334403-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150334403-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150334403-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150334403-[MEMORY] hold= 31,184,470,016 used= 31,170,202,392 count=  14,987 avg_used=      2,079,816 block_cnt=  14,987 chunk_cnt=  14,987 mod=Memstore
observer.log.20240814150334403-[MEMORY] hold= 31,184,470,016 used= 31,170,202,392 count=  14,987 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150334403-[2024-08-14 15:03:14.982461] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8]
observer.log.20240814150334403-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      5,046,272 used=      2,033,472 limit= 9,223,372,036,854,775,807
observer.log.20240814150334403-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150334403-[MEMORY] wash_related_chunks=         4 washed_blocks=        17 washed_size= 7,536,640
observer.log.20240814150334403-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150334403-[2024-08-14 15:03:24.983391] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=23]
observer.log.20240814150334403:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,444,697,088 used= 31,199,035,392 limit= 9,223,372,036,854,775,807
observer.log.20240814150334403-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150334403-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150334403-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150334403-[MEMORY] hold= 31,199,035,392 used= 31,184,761,104 count=  14,994 avg_used=      2,079,816 block_cnt=  14,994 chunk_cnt=  14,994 mod=Memstore
observer.log.20240814150334403-[MEMORY] hold= 31,199,035,392 used= 31,184,761,104 count=  14,994 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150334403-[2024-08-14 15:03:24.983405] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6]
observer.log.20240814150334403-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      6,029,312 used=      1,485,568 limit= 9,223,372,036,854,775,807
observer.log.20240814150334403-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150334403-[MEMORY] wash_related_chunks=         5 washed_blocks=        18 washed_size= 8,650,752
observer.log.20240814150334403-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150410116-[2024-08-14 15:03:35.072101] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=42]
observer.log.20240814150410116:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,453,085,696 used= 31,207,358,464 limit= 9,223,372,036,854,775,807
observer.log.20240814150410116-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150410116-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150410116-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150410116-[MEMORY] hold= 31,207,358,464 used= 31,193,080,368 count=  14,998 avg_used=      2,079,816 block_cnt=  14,998 chunk_cnt=  14,998 mod=Memstore
observer.log.20240814150410116-[MEMORY] hold= 31,207,358,464 used= 31,193,080,368 count=  14,998 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150410116-[2024-08-14 15:03:35.072115] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7]
observer.log.20240814150410116-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      4,980,736 used=      1,862,400 limit= 9,223,372,036,854,775,807
observer.log.20240814150410116-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150410116-[MEMORY] wash_related_chunks=         5 washed_blocks=        17 washed_size= 7,602,176
observer.log.20240814150410116-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150410116-[2024-08-14 15:03:45.082635] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19]
observer.log.20240814150410116:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,465,668,608 used= 31,219,843,072 limit= 9,223,372,036,854,775,807
observer.log.20240814150410116-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150410116-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150410116-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150410116-[MEMORY] hold= 31,219,843,072 used= 31,205,559,264 count=  15,004 avg_used=      2,079,816 block_cnt=  15,004 chunk_cnt=  15,004 mod=Memstore
observer.log.20240814150410116-[MEMORY] hold= 31,219,843,072 used= 31,205,559,264 count=  15,004 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150410116-[2024-08-14 15:03:45.082647] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6]
observer.log.20240814150410116-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      4,980,736 used=      1,944,896 limit= 9,223,372,036,854,775,807
observer.log.20240814150410116-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150410116-[MEMORY] wash_related_chunks=         5 washed_blocks=        17 washed_size= 7,602,176
observer.log.20240814150410116-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150410116-[2024-08-14 15:03:55.056983] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28]
observer.log.20240814150410116:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,476,154,368 used= 31,230,246,912 limit= 9,223,372,036,854,775,807
observer.log.20240814150410116-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150410116-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150410116-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150410116-[MEMORY] hold= 31,230,246,912 used= 31,215,958,344 count=  15,009 avg_used=      2,079,816 block_cnt=  15,009 chunk_cnt=  15,009 mod=Memstore
observer.log.20240814150410116-[MEMORY] hold= 31,230,246,912 used= 31,215,958,344 count=  15,009 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150410116-[2024-08-14 15:03:55.056996] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7]
observer.log.20240814150410116-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      4,915,200 used=      2,238,080 limit= 9,223,372,036,854,775,807
observer.log.20240814150410116-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150410116-[MEMORY] wash_related_chunks=         5 washed_blocks=        18 washed_size= 7,667,712
observer.log.20240814150410116-[MEMORY] request_cached_chunk_cnt=    0
--
observer.log.20240814150410116-[2024-08-14 15:04:05.286983] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=51]
observer.log.20240814150410116:[MEMORY] tenant_id= 1004 ctx_id=          MEMSTORE_CTX_ID hold= 31,484,542,976 used= 31,238,569,984 limit= 9,223,372,036,854,775,807
observer.log.20240814150410116-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150410116-[MEMORY] wash_related_chunks=         0 washed_blocks=         0 washed_size=         0
observer.log.20240814150410116-[MEMORY] request_cached_chunk_cnt=    0
observer.log.20240814150410116-[MEMORY] hold= 31,238,569,984 used= 31,224,277,608 count=  15,013 avg_used=      2,079,816 block_cnt=  15,013 chunk_cnt=  15,013 mod=Memstore
observer.log.20240814150410116-[MEMORY] hold= 31,238,569,984 used= 31,224,277,608 count=  15,013 avg_used=      2,079,816 mod=SUMMARY
observer.log.20240814150410116-[2024-08-14 15:04:05.286997] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8]
observer.log.20240814150410116-[MEMORY] tenant_id= 1004 ctx_id=           EXECUTE_CTX_ID hold=      4,521,984 used=      1,984,512 limit= 9,223,372,036,854,775,807
observer.log.20240814150410116-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814150410116-[MEMORY] wash_related_chunks=         6 washed_blocks=        20 washed_size= 10,158,080
observer.log.20240814150410116-[MEMORY] request_cached_chunk_cnt=    0

[辞霜]

这个也帮忙查一下，谢谢
grep ‘1004 ctx_id= DEFAULT_CTX_ID’ observer.log* -C 1 -A 10

[ootoo]

您好，以下是内容:

observer.log.20240814153405460-[2024-08-14 15:34:04.715085] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7]
observer.log.20240814153405460:[MEMORY] tenant_id= 1004 ctx_id=           DEFAULT_CTX_ID hold= 25,811,025,920 used= 24,940,610,672 limit= 9,223,372,036,854,775,807
observer.log.20240814153405460-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814153405460-[MEMORY] wash_related_chunks=       719 washed_blocks=     4,859 washed_size= 987,168,768
observer.log.20240814153405460-[MEMORY] request_cached_chunk_cnt=    1
observer.log.20240814153405460-[MEMORY] hold= 22,973,644,800 used= 22,393,772,009 count=  66,904 avg_used=        334,714 block_cnt=  66,904 chunk_cnt=   9,506 mod=SqlExecutor
observer.log.20240814153405460-[MEMORY] hold=    560,693,248 used=    560,434,560 count=     233 avg_used=      2,405,298 block_cnt=     233 chunk_cnt=     233 mod=MysqlRequesReco
observer.log.20240814153405460-[MEMORY] hold=    461,208,000 used=    455,892,694 count=  18,112 avg_used=         25,170 block_cnt=   3,702 chunk_cnt=     125 mod=SplitMultiStmt
observer.log.20240814153405460-[MEMORY] hold=     84,017,152 used=     83,886,080 count=       2 avg_used=     41,943,040 block_cnt=       2 chunk_cnt=       2 mod=LogGroupBuffer
observer.log.20240814153405460-[MEMORY] hold=     73,457,664 used=     72,751,744 count=     103 avg_used=        706,327 block_cnt=     103 chunk_cnt=       7 mod=ServerObjecPool
observer.log.20240814153405460-[MEMORY] hold=     71,663,616 used=     63,700,992 count=     972 avg_used=         65,536 block_cnt=     972 chunk_cnt=     455 mod=DagWarnHisMgr
observer.log.20240814153405460-[MEMORY] hold=     56,623,104 used=     50,331,648 count=     768 avg_used=         65,536 block_cnt=     768 chunk_cnt=     172 mod=SuspectInfoMgr
--
observer.log.20240814153440151-[2024-08-14 15:34:14.631934] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12]
observer.log.20240814153440151:[MEMORY] tenant_id= 1004 ctx_id=           DEFAULT_CTX_ID hold= 30,419,910,656 used= 29,388,846,416 limit= 9,223,372,036,854,775,807
observer.log.20240814153440151-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814153440151-[MEMORY] wash_related_chunks=       718 washed_blocks=     4,861 washed_size= 986,120,192
observer.log.20240814153440151-[MEMORY] request_cached_chunk_cnt=    1
observer.log.20240814153440151-[MEMORY] hold= 27,700,142,080 used= 27,005,840,401 count=  80,351 avg_used=        336,098 block_cnt=  80,351 chunk_cnt=  11,357 mod=SqlExecutor
observer.log.20240814153440151-[MEMORY] hold=    575,258,624 used=    574,992,768 count=     240 avg_used=      2,395,803 block_cnt=     240 chunk_cnt=     240 mod=MysqlRequesReco
observer.log.20240814153440151-[MEMORY] hold=    163,345,968 used=    162,605,991 count=   6,784 avg_used=         23,969 block_cnt=   1,112 chunk_cnt=      38 mod=SplitMultiStmt
observer.log.20240814153440151-[MEMORY] hold=     84,017,152 used=     83,886,080 count=       2 avg_used=     41,943,040 block_cnt=       2 chunk_cnt=       2 mod=LogGroupBuffer
observer.log.20240814153440151-[MEMORY] hold=     73,457,664 used=     72,751,744 count=     103 avg_used=        706,327 block_cnt=     103 chunk_cnt=       7 mod=ServerObjecPool
observer.log.20240814153440151-[MEMORY] hold=     71,663,616 used=     63,700,992 count=     972 avg_used=         65,536 block_cnt=     972 chunk_cnt=     455 mod=DagWarnHisMgr
observer.log.20240814153440151-[MEMORY] hold=     56,623,104 used=     50,331,648 count=     768 avg_used=         65,536 block_cnt=     768 chunk_cnt=     172 mod=SuspectInfoMgr
--
observer.log.20240814153440151-[2024-08-14 15:34:24.811839] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8]
observer.log.20240814153440151:[MEMORY] tenant_id= 1004 ctx_id=           DEFAULT_CTX_ID hold= 28,545,515,520 used= 24,727,484,352 limit= 9,223,372,036,854,775,807
observer.log.20240814153440151-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814153440151-[MEMORY] wash_related_chunks=       715 washed_blocks=     4,844 washed_size= 981,336,064
observer.log.20240814153440151-[MEMORY] request_cached_chunk_cnt=    1
observer.log.20240814153440151-[MEMORY] hold= 23,180,402,688 used= 22,613,736,911 count=  66,528 avg_used=        339,913 block_cnt=  66,528 chunk_cnt=   9,528 mod=SqlExecutor
observer.log.20240814153440151-[MEMORY] hold=    598,147,072 used=    597,869,952 count=     251 avg_used=      2,381,952 block_cnt=     251 chunk_cnt=     251 mod=MysqlRequesReco
observer.log.20240814153440151-[MEMORY] hold=     84,017,152 used=     83,886,080 count=       2 avg_used=     41,943,040 block_cnt=       2 chunk_cnt=       2 mod=LogGroupBuffer
observer.log.20240814153440151-[MEMORY] hold=     73,457,664 used=     72,751,744 count=     103 avg_used=        706,327 block_cnt=     103 chunk_cnt=       7 mod=ServerObjecPool
observer.log.20240814153440151-[MEMORY] hold=     71,663,616 used=     63,700,992 count=     972 avg_used=         65,536 block_cnt=     972 chunk_cnt=     455 mod=DagWarnHisMgr
observer.log.20240814153440151-[MEMORY] hold=     56,623,104 used=     50,331,648 count=     768 avg_used=         65,536 block_cnt=     768 chunk_cnt=     172 mod=SuspectInfoMgr
observer.log.20240814153440151-[MEMORY] hold=     46,555,136 used=     41,484,288 count=     619 avg_used=         67,018 block_cnt=     619 chunk_cnt=     320 mod=ReplaceRow
--
observer.log.20240814153440151-[2024-08-14 15:34:34.799235] INFO  [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [3524892][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6]
observer.log.20240814153440151:[MEMORY] tenant_id= 1004 ctx_id=           DEFAULT_CTX_ID hold= 31,000,494,080 used= 28,606,518,048 limit= 9,223,372,036,854,775,807
observer.log.20240814153440151-[MEMORY] idle_size=         0 free_size=         0
observer.log.20240814153440151-[MEMORY] wash_related_chunks=       716 washed_blocks=     4,846 washed_size= 980,090,880
observer.log.20240814153440151-[MEMORY] request_cached_chunk_cnt=    1
observer.log.20240814153440151-[MEMORY] hold= 27,060,174,848 used= 26,366,467,765 count=  80,538 avg_used=        327,379 block_cnt=  80,538 chunk_cnt=  11,263 mod=SqlExecutor
observer.log.20240814153440151-[MEMORY] hold=    600,227,840 used=    599,949,696 count=     252 avg_used=      2,380,752 block_cnt=     252 chunk_cnt=     252 mod=MysqlRequesReco
observer.log.20240814153440151-[MEMORY] hold=     84,017,152 used=     83,886,080 count=       2 avg_used=     41,943,040 block_cnt=       2 chunk_cnt=       2 mod=LogGroupBuffer
observer.log.20240814153440151-[MEMORY] hold=     73,457,664 used=     72,751,744 count=     103 avg_used=        706,327 block_cnt=     103 chunk_cnt=       7 mod=ServerObjecPool
observer.log.20240814153440151-[MEMORY] hold=     71,663,616 used=     63,700,992 count=     972 avg_used=         65,536 block_cnt=     972 chunk_cnt=     455 mod=DagWarnHisMgr
observer.log.20240814153440151-[MEMORY] hold=     56,623,104 used=     50,331,648 count=     768 avg_used=         65,536 block_cnt=     768 chunk_cnt=     172 mod=SuspectInfoMgr
observer.log.20240814153440151-[MEMORY] hold=     45,580,288 used=     40,566,784 count=     612 avg_used=         66,285 block_cnt=     612 chunk_cnt=     338 mod=ReplaceRow

[辞霜]

之前使用root@sys执行手动转储失败了么？
（ALTER SYSTEM MINOR FREEZE; 当前租户root执行）
root@sys执行查看
SELECT * FROM oceanbase.GV$OB_TABLET_COMPACTION_HISTORY WHERE TYPE=‘MINI_MERGE’ \G
参考配置一下使用obdiag抓取一下堆栈信息
https://www.oceanbase.com/docs/common-obdiag-cn-1000000001214410

[ootoo]

您好，执行之后没有搜索到 TYPE=‘MINI_MERGE’ 的内容，但是 like 是有不少的

obclient [oceanbase]> SELECT count(1) FROM oceanbase.GV$OB_TABLET_COMPACTION_HISTORY WHERE TYPE='MINI_MERGE'\G;
*************************** 1. row ***************************
count(1): 0
1 row in set (0.117 sec)

ERROR:
No query specified

obclient [oceanbase]> SELECT * FROM oceanbase.GV$OB_TABLET_COMPACTION_HISTORY WHERE TYPE='MINI_MERGE'\G;
Empty set (0.542 sec)

ERROR:
No query specified

obclient [oceanbase]> SELECT count(1) FROM oceanbase.GV$OB_TABLET_COMPACTION_HISTORY WHERE TYPE like '%MINI_MERGE%'\G;
*************************** 1. row ***************************
count(1): 114805
1 row in set (0.205 sec)

都是加了MDS前缀的

obclient [oceanbase]>  SELECT count(1),TYPE FROM oceanbase.GV$OB_TABLET_COMPACTION_HISTORY WHERE TYPE like '%MINI_MERGE%';
+----------+----------------+
| count(1) | TYPE           |
+----------+----------------+
|   147346 | MDS_MINI_MERGE |
+----------+----------------+
1 row in set (0.193 sec)

[ootoo]

执行stack提示架构不支持，当前服务器是 arm 架构的

[admin@db-control1 log]$ obd obdiag gather stack xxx_ocp
Get local repositories and plugins ok
The tool oceanbase-diagnostic-tool is already installed the latest version 2.1.0
Open ssh connection ok
gather_obstack start ...
Gather Ob stack Summary:
+--------------+------------------------------------------------------------------+--------+--------+------------+
| Node         | Status                                                           | Size   | Time   | PackPath   |
+==============+==================================================================+========+========+============+
| 1xx.xx.xx.xx | Error:remote server 1xx.xx.xx.xx arch not support gather obstack | 0.000B | 0 s    |            |
+--------------+------------------------------------------------------------------+--------+--------+------------+
Trace ID: 7683b91a-5a14-11ef-96f5-fa163e403b9f
If you want to view detailed obdiag logs, please run: obdiag display-trace 7683b91a-5a14-11ef-96f5-fa163e403b9f

[辞霜]

GV$OB_TABLET_COMPACTION_HISTORY 这个是历史表
SELECT * FROM oceanbase.GV$OB_TABLET_COMPACTION_PROGRESS WHERE TYPE=‘MINI_MERGE’\G。使用这种可以查看目前是否有正在转储的任务。