allocate memory fail

刘标 · 2021 年8 月 30 日 10:18

【产品名称】oceanbase

【产品版本】3.1.0

【问题描述】

本地单节点部署社区版3.1.0，运行日志一直抛出分配内存失败，如下：

[2021-08-30 10:12:08.010106] ERROR [LIB] ob_malloc (ob_malloc.h:38) [11573][394][Y0-0000000000000000] [lt=30] [dc=0] allocate memory fail(attr=tenant_id=500, label=IlogFileBuilder, ctx_id=0, prio=0, nbyte=26781080) BACKTRACE:0x91613ee 0x912f6c1 0x21562ef 0x2155f3b 0x2155d02 0x215593a 0x215565f 0x219a087 0x6d51e28 0x6d519ee 0x6d516f7 0x6d72aff 0x6d6f87e 0x9162d87 0x331e22f 0x2caaec4 0x8fc84c5 0x8fc7152 0x8fc3c7f

[2021-08-30 10:12:08.010115] ERROR [CLOG.CSR] prepare_raw_array_ (ob_ilog_file_builder.cpp:251) [11573][394][Y0-0000000000000000] [lt=7] [dc=0] raw_array alloc failed(ret=-4013, raw_array.count_=371959) BACKTRACE:0x91613ee 0x912f6c1 0x21562ef 0x2155f3b 0x2155d02 0x2142634 0x6d520fd 0x6d519ee 0x6d516f7 0x6d72aff 0x6d6f87e 0x9162d87 0x331e22f 0x2caaec4 0x8fc84c5 0x8fc7152 0x8fc3c7f

[2021-08-30 10:12:08.010122] ERROR [CLOG.CSR] get_file_buffer (ob_ilog_file_builder.cpp:208) [11573][394][Y0-0000000000000000] [lt=6] [dc=0] build_file_ failed(ret=-4013) BACKTRACE:0x91613ee 0x912f6c1 0x211b3c4 0x211aeab 0x211ac11 0x2119ba8 0x6d518b6 0x6d72aff 0x6d6f87e 0x9162d87 0x331e22f 0x2caaec4 0x8fc84c5 0x8fc7152 0x8fc3c7f

[2021-08-30 10:12:08.010127] ERROR [CLOG.CSR] build_and_write_file_ (ob_ilog_store.cpp:1282) [11573][394][Y0-0000000000000000] [lt=5] [dc=0] builder get_file_buffer failed(ret=-4013) BACKTRACE:0x91613ee 0x912f6c1 0x211b3c4 0x211aeab 0x211ac11 0x2119ba8 0x6d72cbc 0x6d6f87e 0x9162d87 0x331e22f 0x2caaec4 0x8fc84c5 0x8fc7152 0x8fc3c7f

[2021-08-30 10:12:08.010133] ERROR [CLOG] runTimerTask (ob_ilog_store.cpp:970) [11573][394][Y0-0000000000000000] [lt=5] [dc=0] build_and_write_file_ failed(ret=-4013) BACKTRACE:0x91613ee 0x912f6c1 0x211b3c4 0x211aeab 0x211ac11 0x2119ba8 0x6d6fddb 0x9162d87 0x331e22f 0x2caaec4 0x8fc84c5 0x8fc7152 0x8fc3c7f

^C

[ober@progs log]$ free -g

total used free shared buff/cache available

Mem: 8 5 3 0 0 3

Swap: 3 0 3

[ober@progs log]$ free -h

total used free shared buff/cache available

Mem: 8.8G 5.1G 3.3G 11M 358M 3.5G

Swap: 4.0G 0B 4.0G

[ober@progs log]$

余安 · 2021 年8 月 30 日 10:38

除了observer机器系统占用内存，最好留足不低于8G的内存给observer。如果是虚拟机，调大点试试。

刘标 · 2021 年8 月 30 日 10:38

启动进程

[ober@progs log]$ ps -ef|grep observer

root 11340 1 99 09:23 ? 02:02:20 /ups/app/oceanbase/bin/observer -r 192.168.10.181:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=8G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4 -z zone1 -p 2881 -P 2882 -c 1 -d /ups/app/oceanbase/store -i ens32 -l ERROR

2.配置文件

刘标 · 2021 年8 月 30 日 10:39

没有那么多的内存可以分配给虚机

余安 · 2021 年8 月 30 日 15:34

8G内存是observe最低门槛了

obpilot · 2021 年8 月 30 日 16:30

确实是这个原因。虚拟机total才8g，实际可用内存低于7G，所以这个observer启动不来。

obpilot · 2021 年8 月 30 日 19:08

如果你是linux服务器centos或redhat系统，可用内存大于8G那不要用虚拟机和docker了，直接原参数启动也行。

刘标 · 2021 年8 月 31 日 12:44

observer进程已经启动了，但observer.log日志一直都在刷ERROR

c7f

[2021-08-31 12:42:32.481948] ERROR [LIB] direct_alloc (achunk_mgr.cpp:120) [40683][426][Y0-0000000000000000] [lt=6] [dc=0] low alloc fail(size=6291456, orig_errno=12, errno=12) BACKTRACE:0x91613ee 0x912f6c1 0x2158336 0x2157f7b 0x2157cf3 0x3fe37f9 0x90138e8 0x9014143 0x901562c 0x8feb701 0x8feb20e 0x8ffc0b0 0x8fd739b 0x8ffc9d4 0x8ffb6ad 0x8fee092 0x21555cd 0x8fe236c 0x6ed6d6a 0x6ee5676 0x6ee4620 0x6ee32df 0x6ef16be 0x6cfe686 0x6d0225a 0x9162d87 0x331e22f 0x2caaec4 0x8fc84c5 0x8fc7152 0x8fc3c7f

[2021-08-31 12:42:32.481978] ERROR [LIB] ob_malloc (ob_malloc.h:38) [40683][426][Y0-0000000000000000] [lt=27] [dc=0] allocate memory fail(attr=tenant_id=500, label=ClogInfoBlkHndl, ctx_id=0, prio=0, nbyte=4198400) BACKTRACE:0x91613ee 0x912f6c1 0x21562ef 0x2155f3b 0x2155d02 0x215593a 0x215565f 0x8fe236c 0x6ed6d6a 0x6ee5676 0x6ee4620 0x6ee32df 0x6ef16be 0x6cfe686 0x6d0225a 0x9162d87 0x331e22f 0x2caaec4 0x8fc84c5 0x8fc7152 0x8fc3c7f

[2021-08-31 12:42:32.481987] ERROR [CLOG] init (ob_log_direct_reader.cpp:51) [40683][426][Y0-0000000000000000] [lt=7] [dc=0] ob_malloc fail(ret=-4013, align_size=4096, size=4194304) BACKTRACE:0x91613ee 0x912f6c1 0x2158336 0x2157f7b 0x2157cf3 0x214be79 0x6ed6ebb 0x6ee5676 0x6ee4620 0x6ee32df 0x6ef16be 0x6cfe686 0x6d0225a 0x9162d87 0x331e22f 0x2caaec4 0x8fc84c5 0x8fc7152 0x8fc3c7f

[ober@progs log]$ ps -ef|grep observer

ober 20559 12131 0 06:11 pts/1 00:00:00 tail -100f observer.log

root 40462 1 99 12:32 ? 00:17:57 /ups/app/oceanbase/bin/observer -r 192.168.10.181:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=8G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4 -z zone1 -p 2881 -P 2882 -c 1 -d /ups/app/oceanbase/store -i ens32 -l ERROR

ober 41696 41420 0 12:42 pts/6 00:00:00 grep --color=auto observer

[ober@progs log]$

刘标 · 2021 年8 月 31 日 12:45

查询内存信息：

MySQL [oceanbase]> SELECT * FROM __all_virtual_memory_info WHERE tenant_id = 500 and label='RsLeaderCoordin';

+-----------+----------------+----------+--------+-----------------+----------------+----------+--------+-----------------+-------+------+------+-------+-------------+------------+

+-----------+----------------+----------+--------+-----------------+----------------+----------+--------+-----------------+-------+------+------+-------+-------------+------------+

| 500 | 192.168.10.181 | 2882 | 0 | RsLeaderCoordin | DEFAULT_CTX_ID | user | 0 | RsLeaderCoordin | zone1 | 848 | 776 | 1 | 0 | 0 |

+-----------+----------------+----------+--------+-----------------+----------------+----------+--------+-----------------+-------+------+------+-------+-------------+------------+

1 row in set (0.099 sec)

MySQL [oceanbase]> SELECT * FROM __all_virtual_memory_info WHERE tenant_id = 500 and label='ClogInfoBlkHndl';

+-----------+----------------+----------+--------+-----------------+----------------+----------+--------+-----------------+-------+---------+---------+-------+-------------+------------+

+-----------+----------------+----------+--------+-----------------+----------------+----------+--------+-----------------+-------+---------+---------+-------+-------------+------------+

| 500 | 192.168.10.181 | 2882 | 0 | ClogInfoBlkHndl | DEFAULT_CTX_ID | user | 0 | ClogInfoBlkHndl | zone1 | 8263296 | 6164456 | 2 | 0 | 0 |

+-----------+----------------+----------+--------+-----------------+----------------+----------+--------+-----------------+-------+---------+---------+-------+-------------+------------+

1 row in set (0.018 sec)

MySQL [oceanbase]>

刘标 · 2021 年9 月 2 日 14:54

8G内存还不够用呀？学习环境有点难搞