这种我们需要如何调整？
这个租户单机内存是512G。 可用总内存我们设置了512G
为了加大缓存可用内存， memstore我们只设置了25%
3台机器分配资源如下
Sep 20 01:19:36 ep-ob-server02 kernel: aliyun-service invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
aliyun-service触发OOM
Sep 20 01:19:36 ep-ob-server02 kernel: aliyun-service cpuset=/ mems_allowed=0
Sep 20 01:19:36 ep-ob-server02 kernel: CPU: 0 PID: 17435 Comm: aliyun-service Tainted: G OE ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Sep 20 01:19:36 ep-ob-server02 kernel: Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 2221b89 04/01/2014
Sep 20 01:19:36 ep-ob-server02 kernel: Call Trace:
Sep 20 01:19:36 ep-ob-server02 kernel: [] dump_stack+0x19/0x1f
Sep 20 01:19:36 ep-ob-server02 kernel: [] dump_header+0x90/0x22d
Sep 20 01:19:36 ep-ob-server02 kernel: [] ? ktime_get_ts64+0x52/0xf0
Sep 20 01:19:36 ep-ob-server02 kernel: [] ? delayacct_end+0x8f/0xc0
Sep 20 01:19:36 ep-ob-server02 kernel: [] ? virtballoon_oom_notify+0x2a/0x80 [virtio_balloon]
Sep 20 01:19:36 ep-ob-server02 kernel: [] oom_kill_process+0x2d5/0x4a0
Sep 20 01:19:36 ep-ob-server02 kernel: [] ? oom_unkillable_task+0xcd/0x120
Sep 20 01:19:36 ep-ob-server02 kernel: [] out_of_memory+0x31a/0x500
Sep 20 01:19:36 ep-ob-server02 kernel: [] __alloc_pages_nodemask+0xae4/0xbf0
Sep 20 01:19:36 ep-ob-server02 kernel: [] alloc_pages_current+0x98/0x110
Sep 20 01:19:36 ep-ob-server02 kernel: [] __page_cache_alloc+0x97/0xb0
Sep 20 01:19:36 ep-ob-server02 kernel: [] filemap_fault+0x270/0x420
Sep 20 01:19:36 ep-ob-server02 kernel: [] ext4_filemap_fault+0x36/0x50 [ext4]
Sep 20 01:19:36 ep-ob-server02 kernel: [] __do_fault.isra.61+0x8a/0x100
Sep 20 01:19:36 ep-ob-server02 kernel: [] do_read_fault.isra.63+0x4c/0x1b0
Sep 20 01:19:36 ep-ob-server02 kernel: [] handle_mm_fault+0xa33/0x1190
Sep 20 01:19:36 ep-ob-server02 kernel: [] __do_page_fault+0x213/0x510
Sep 20 01:19:36 ep-ob-server02 kernel: [] trace_do_page_fault+0x56/0x150
Sep 20 01:19:36 ep-ob-server02 kernel: [] do_async_page_fault+0x22/0x100
Sep 20 01:19:36 ep-ob-server02 kernel: [] async_page_fault+0x28/0x30
Sep 20 01:19:36 ep-ob-server02 kernel: Mem-Info:
Sep 20 01:19:36 ep-ob-server02 kernel: active_anon:126940042 inactive_anon:112 isolated_anon:0#012 active_file:2739 inactive_file:7965 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:27791 slab_unreclaimable:50905#012 mapped:8761 shmem:182 pagetables:252976 bounce:0#012 free:1033039 free_pcp:1575 free_cma:0
Sep 20 01:19:36 ep-ob-server02 kernel: Node 0 DMA free:15908kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Sep 20 01:19:36 ep-ob-server02 kernel: lowmem_reserve[]: 0 2782 507713 507713
Sep 20 01:19:36 ep-ob-server02 kernel: Node 0 DMA32 free:2030856kB min:11492kB low:14364kB high:17236kB active_anon:767844kB inactive_anon:12kB active_file:200kB inactive_file:656kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3111612kB managed:2849436kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:12kB slab_reclaimable:200kB slab_unreclaimable:1360kB kernel_stack:320kB pagetables:1760kB unstable:0kB bounce:0kB free_pcp:1096kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:1513 all_unreclaimable? yes
Sep 20 01:19:36 ep-ob-server02 kernel: lowmem_reserve[]: 0 0 504930 504930
Sep 20 01:19:36 ep-ob-server02 kernel: Node 0 Normal free:2085392kB min:2085592kB low:2606988kB high:3128388kB active_anon:506992324kB inactive_anon:436kB active_file:10756kB inactive_file:31204kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:525336576kB managed:517051592kB mlocked:0kB dirty:0kB writeback:0kB mapped:35040kB shmem:716kB slab_reclaimable:110964kB slab_unreclaimable:202260kB kernel_stack:45280kB pagetables:1010144kB unstable:0kB bounce:0kB free_pcp:5204kB local_pcp:204kB free_cma:0kB writeback_tmp:0kB pages_scanned:63071 all_unreclaimable? yes
Sep 20 01:19:36 ep-ob-server02 kernel: lowmem_reserve[]: 0 0 0 0
Sep 20 01:19:36 ep-ob-server02 kernel: Node 0 DMA: 14kB (U) 08kB 016kB 132kB (U) 264kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15908kB
Sep 20 01:19:36 ep-ob-server02 kernel: Node 0 DMA32: 4584kB (UEM) 4278kB (UE) 39116kB (UEM) 28332kB (UEM) 17464kB (UE) 74128kB (UEM) 35256kB (UEM) 27512kB (UEM) 121024kB (EM) 12048kB (U) 4774096kB (UM) = 2032080kB
Sep 20 01:19:36 ep-ob-server02 kernel: Node 0 Normal: 95914kB (UEM) 22268kB (UEM) 141316kB (UE) 190432kB (EM) 145664kB (UEM) 885128kB (UE) 465256kB (EM) 182512kB (UE) 14911024kB (EM) 32048kB (UEM) 0*4096kB = 2091324kB
Sep 20 01:19:36 ep-ob-server02 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Sep 20 01:19:36 ep-ob-server02 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 20 01:19:36 ep-ob-server02 kernel: 13002 total pagecache pages
Sep 20 01:19:36 ep-ob-server02 kernel: 0 pages in swap cache
Sep 20 01:19:36 ep-ob-server02 kernel: Swap cache stats: add 0, delete 0, find 0/0
Sep 20 01:19:36 ep-ob-server02 kernel: Free swap = 0kB
Sep 20 01:19:36 ep-ob-server02 kernel: Total swap = 0kB
Sep 20 01:19:36 ep-ob-server02 kernel: 132116045 pages RAM
Sep 20 01:19:36 ep-ob-server02 kernel: 0 pages HighMem/MovableOnly
Sep 20 01:19:36 ep-ob-server02 kernel: 2136811 pages reserved
Sep 20 01:19:36 ep-ob-server02 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Sep 20 01:19:36 ep-ob-server02 kernel: [ 798] 0 798 14141 113 31 0 0 systemd-journal
Sep 20 01:19:36 ep-ob-server02 kernel: [ 828] 0 828 11351 157 24 0 -1000 systemd-udevd
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1168] 0 1168 13883 101 27 0 -1000 auditd
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1227] 999 1227 153087 2160 62 0 0 polkitd
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1229] 81 1229 14530 174 35 0 -900 dbus-daemon
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1231] 32 1231 17314 135 37 0 0 rpcbind
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1257] 0 1257 48802 118 35 0 0 gssproxy
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1266] 0 1266 6653 167 19 0 0 systemd-logind
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1587] 0 1587 25753 514 49 0 0 dhclient
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1648] 0 1648 143572 2837 95 0 0 tuned
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1807] 0 1807 52300 3032 64 0 0 argusagent
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1852] 0 1852 22452 266 43 0 0 master
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1856] 89 1856 22495 263 46 0 0 qmgr
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1865] 0 1865 651281 5525 130 0 0 argusagent
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1897] 0 1897 61829 351 52 0 0 rsyslogd
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1898] 0 1898 28251 278 59 0 -1000 sshd
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1900] 0 1900 31597 171 19 0 0 crond
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1903] 0 1903 6477 50 18 0 0 atd
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1912] 0 1912 27552 42 10 0 0 agetty
Sep 20 01:19:36 ep-ob-server02 kernel: [ 1913] 0 1913 27552 41 11 0 0 agetty
Sep 20 01:19:36 ep-ob-server02 kernel: [ 4377] 0 4377 289484 5879 52 0 0 ocp_agentd
Sep 20 01:19:36 ep-ob-server02 kernel: [ 4394] 0 4394 324543 10137 70 0 0 ocp_mgragent
Sep 20 01:19:36 ep-ob-server02 kernel: [ 4395] 0 4395 865646 412605 1139 0 0 ocp_monagent
Sep 20 01:19:36 ep-ob-server02 kernel: [ 5931] 1000 5931 130236320 126463929 250557 0 0 observer
Sep 20 01:19:36 ep-ob-server02 kernel: [21732] 0 21732 12973 386 23 0 0 AliYunDunUpdate
Sep 20 01:19:36 ep-ob-server02 kernel: [21841] 0 21841 28860 536 55 0 0 AliYunDun
Sep 20 01:19:36 ep-ob-server02 kernel: [21852] 0 21852 47622 10512 90 0 0 AliYunDunMonito
Sep 20 01:19:36 ep-ob-server02 kernel: [17386] 0 17386 171907 2160 19 0 0 aliyun-service
Sep 20 01:19:36 ep-ob-server02 kernel: [17520] 0 17520 4852 202 12 0 0 assist_daemon
Sep 20 01:19:36 ep-ob-server02 kernel: [29097] 38 29097 6433 159 17 0 0 ntpd
Sep 20 01:19:36 ep-ob-server02 kernel: [37207] 89 37207 22478 251 45 0 0 pickup
Sep 20 01:19:36 ep-ob-server02 kernel: Out of memory: Kill process 5931 (observer) score 974 or sacrifice child
Sep 20 01:19:36 ep-ob-server02 kernel: Killed process 5931 (observer), UID 1000, total-vm:520945280kB, anon-rss:505865392kB, file-rss:0kB, shmem-rss:0kB
observer被内核kill
observer固定分配的470G内存。 按道理，不会出现内存溢出的情况的···
是哪不对吗
机器内存是512G，memory_limit也设置了512G吗？这样就100%会出问题
默认是90%的物理机内存，生产环境一般建议设置为物理内存的80%，最高不超过90%，需要冗余一部分给OS等
system_memory 默认为 0M
这个应该设置多少
我一直以为， 设置512G. ob只用90%分给租户。 还有10%是给系统用的···
