4013, 'No memory or reach tenant memory limit'

OceanBase
#1

【产品名称】

OCEANBASE

【产品版本】

3.1.3

【问题描述】

我用python程序向OB里面导入数据，遇到了上面错误。在没有修改memory_limit参数的情况下，我做了下转储，好像没啥用。这个问题是不是system_memory快用没有了？怎么查啊？谢谢

1 个赞
#2

| tenant_id | svr_ip      | svr_port | ctx_id | label      | ctx_name                      | mod_type | mod_id | mod_name   | zone | hold     | used     | count | alloc_count | free_count |

+-----------+--------------+----------+--------+-------------+--------------------------------+----------+--------+-------------+-------+-----------+-----------+-------+-------------+------------+

|        1 | 192.168.56.4 |    2882 |     1 | OB_MEMSTORE | MEMSTORE_CTX_ID               | user    |     0 | OB_MEMSTORE | zone1 | 167731200 | 167696000 |   80 |          0 |         0 |

|     1001 | 192.168.56.4 |    2882 |     1 | OB_MEMSTORE | MEMSTORE_CTX_ID               | user    |     0 | OB_MEMSTORE | zone1 | 96445440 | 96425200 |   46 |          0 |         0 |


这个是租户内存情况


1 个赞
#4

Q：运行 OBLOADER 过程中报错：Over tenant memory limits 或 No memory or reach tenant memory limit。 A：调大租户内存的占比（或指定 --thread 降低并发数）。

set global ob_sql_work_area_percentage=30; -- Default 5

1 个赞
#5

内存超限了

1 个赞
#6

我看了observer.log,下面说ob_kvcache_store的内存不够。调高ob_sql_work_area_percentage变量可以扩大KVCACHE吗？谢谢

[2022-04-24 15:17:39.321534] ERROR [COMMON] sync_wash_mbs (ob_kvcache_store.cpp:446) [12632][58][YB42C0A83804-0005DD60C9875192] [lt=59] [dc=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=2097152) BACKTRACE:0x99be10e 0x9793491 0x22b00f6 0x22afd3b 0x22afab3 0x22a3c39 0x228662e 0x2285b2b 0x97a6619 0x976a0a1 0x9769bae 0x9769930 0x977b89d 0x9755bfb 0x977c1d4 0x977ae6d 0x976cf02 0x22ad33d 0x22aebb8 0x8833ab8 0x87bfc4d 0x87be35b 0x87bd350 0x87bc8a6 0x87bbce5 0x87b40f2 0x7a9c062 0x7aa993d 0x7a8450a 0x8770bb8 0x8774f04 0x87726ae 0x8775d59 0x875b896 0x875b0b8 0x87651d2 0x87780a7 0x7bd0bd6 0x33acf2a 0x33b03a0 0x3385bef 0x2c70eb2 0x9746965 0x9745692 0x974214f

[2022-04-24 15:17:39.321675] ERROR [LIB] ob_malloc (ob_malloc.h:38) [12632][58][YB42C0A83804-0005DD60C9875192] [lt=6] [dc=0] allocate memory fail(attr=tenant_id=500, label=SstableReader, ctx_id=0, prio=0, nbyte=10272) BACKTRACE:0x99be10e 0x9793491 0x22ae05f 0x22adcab 0x22ada72 0x22ad6aa 0x22ad3cf 0x22aebb8 0x8833ab8 0x87bfc4d 0x87be35b 0x87bd350 0x87bc8a6 0x87bbce5 0x87b40f2 0x7a9c062 0x7aa993d 0x7a8450a 0x8770bb8 0x8774f04 0x87726ae 0x8775d59 0x875b896 0x875b0b8 0x87651d2 0x87780a7 0x7bd0bd6 0x33acf2a 0x33b03a0 0x3385bef 0x2c70eb2 0x9746965 0x9745692 0x974214f

[2022-04-24 15:17:59.364834] ERROR [COMMON] sync_wash_mbs (ob_kvcache_store.cpp:446) [12622][40][YB42C0A83804-0005DD60C9875194] [lt=25] [dc=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=2097152) BACKTRACE:0x99be10e 0x9793491 0x22b00f6 0x22afd3b 0x22afab3 0x22a3c39 0x228662e 0x2285b2b 0x97a6619 0x976a0a1 0x9769bae 0x9769930 0x977b89d 0x9755bfb 0x977c1d4 0x977ae6d 0x976cf02 0x22ad33d 0x22aebb8 0x8833ab8 0x87bfc4d 0x87be35b 0x87bd350 0x87bc8a6 0x87bbce5 0x87b40f2 0x7a9c062 0x7aa993d 0x7a8450a 0x8770bb8 0x8774f04 0x8777f21 0x7bd0bd6 0x33acf2a 0x33b03a0 0x3385bef 0x2c70eb2 0x9746965 0x9745692 0x974214f

[2022-04-24 15:17:59.365022] ERROR [LIB] ob_malloc (ob_malloc.h:38) [12622][40][YB42C0A83804-0005DD60C9875194] [lt=8] [dc=0] allocate memory fail(attr=tenant_id=500, label=SstableReader, ctx_id=0, prio=0, nbyte=10272) BACKTRACE:0x99be10e 0x9793491 0x22ae05f 0x22adcab 0x22ada72 0x22ad6aa 0x22ad3cf 0x22aebb8 0x8833ab8 0x87bfc4d 0x87be35b 0x87bd350 0x87bc8a6 0x87bbce5 0x87b40f2 0x7a9c062 0x7aa993d 0x7a8450a 0x8770bb8 0x8774f04 0x8777f21 0x7bd0bd6 0x33acf2a 0x33b03a0 0x3385bef 0x2c70eb2 0x9746965 0x9745692 0x974214f


#7

设置了set global ob_sql_work_area_percentage=30;也不行。数据导入还是报同样的错误。除了增大memory_limit没有其他办法？

日志里面说KVCAHCE不足。tenant_id 500

#8

重启了下OBSERVER也不管用。

#9

500租户内存使用情况：

MySQL [oceanbase]> select svr_ip,mod_name,sum(hold) system_memory_sum

   -> from __all_virtual_memory_info

   -> where tenant_id=500 and hold<>0

   -> group by svr_ip,mod_name

   -> order by system_memory_sum desc;

+--------------+-------------------+-------------------+

| svr_ip      | mod_name         | system_memory_sum |

+--------------+-------------------+-------------------+

| 192.168.56.4 | CO_STACK         |       1780719616 |

| 192.168.56.4 | glibc_malloc     |        572798016 |

| 192.168.56.4 | OB_KVSTORE_CACHE |        285636224 |

| 192.168.56.4 | ConcurObjPool    |        165575296 |

| 192.168.56.4 | Sstable          |        159743408 |

| 192.168.56.4 | LinearHashMap    |        154632768 |

| 192.168.56.4 | CallbackTask     |        146800640 |

| 192.168.56.4 | FixeSizeBlocAll  |        142614592 |

| 192.168.56.4 | LogHotCache      |        138412032 |

| 192.168.56.4 | ClogMgr          |        127984320 |

| 192.168.56.4 | LightyQueue      |        123872896 |

| 192.168.56.4 | PartitTableTask  |        123769216 |

| 192.168.56.4 | OB_TEST2_PCODE   |        100567968 |

| 192.168.56.4 | SstableAio       |         96468480 |

| 192.168.56.4 | OmtTenant        |         94373040 |

| 192.168.56.4 | FixedQueue       |         89889264 |

| 192.168.56.4 | HashNodeInteChe  |         81788928 |

| 192.168.56.4 | IdMap            |         79691776 |

| 192.168.56.4 | SstablMergeInfo  |         75092736 |

| 192.168.56.4 | LogArchiLineCac  |         69231488 |

| 192.168.56.4 | ClogExtLineCach  |         69231488 |



#10

kvcache是动态调整的，手动调整不了，kvcache不足有可能是其他模块抢占资源导致的，tenant_id=500租户是ob内部租户，grep MEMORY observer.log |grep 500用这个命令排查一下日志

#11

我在测试导入数据，也遇到了报错，你插入的数据量大约多大

#12

出现这个问题， 不一定是memstore（转储）的问题。 也有可能是大sql 大事务导致的， sql不能改的情况下， 调整ob_sql_work_area_percentage是正解

#13

mark

#14

手动把SQL copy 出来执行下试试，有可能是查询解析器，执行器对大事务的支持不太友好 。。

#15

如果是SQL 问题，尝试绑定outline，hit 等等试试

#16

这个问题解决了吗