【 使用环境 】 测试环境
【 OB or 其他组件 】
./obdumper -h xxxxxxx -P 2883 -u root -p test@@321 --sys-user root --sys-password test@@321 -t test -D yongcepro --ddl --all -f ./test 导数据
【 使用版本 】
4.0.0
【问题描述】
【复现路径】问题出现前后相关操作
【问题现象及影响】
oceanbase 服务core 了
Oct 17 18:09:59 oceanbase01 kernel: lowmem_reserve[]: 0 0 13034 13034
Oct 17 18:09:59 oceanbase01 kernel: lowmem_reserve[]: 0 0 0 0
Oct 17 18:09:59 oceanbase01 kernel: Node 0 DMA: 14kB (U) 08kB 016kB 032kB 264kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15876kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 DMA32: 10564kB (UEM) 2908kB (UEM) 18716kB (UEM) 13432kB (UEM) 7564kB (UE) 32128kB (UE) 31256kB (UEM) 26512kB (UEM) 211024kB (UEM) 02048kB 04096kB = 65472kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 Normal: 9984kB (UEM) 3648kB (UEM) 35916kB (UEM) 27232kB (UEM) 24564kB (UEM) 91128kB (UE) 16256kB (EM) 5512kB (EM) 11024kB (M) 02048kB 04096kB = 56360kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 17 18:09:59 oceanbase01 kernel: 210776 total pagecache pages
Oct 17 18:09:59 oceanbase01 kernel: 0 pages in swap cache
Oct 17 18:09:59 oceanbase01 kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 17 18:09:59 oceanbase01 kernel: Free swap = 0kB
Oct 17 18:09:59 oceanbase01 kernel: Total swap = 0kB
Oct 17 18:09:59 oceanbase01 kernel: 4194162 pages RAM
Oct 17 18:09:59 oceanbase01 kernel: 0 pages HighMem/MovableOnly
Oct 17 18:09:59 oceanbase01 kernel: 127831 pages reserved
Oct 17 18:09:59 oceanbase01 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Oct 17 18:09:59 oceanbase01 kernel: [ 423] 0 423 11909 2262 30 0 0 systemd-journal
Oct 17 18:09:59 oceanbase01 kernel: [ 455] 0 455 11417 206 23 0 -1000 systemd-udevd
Oct 17 18:09:59 oceanbase01 kernel: [ 660] 0 660 13883 113 28 0 -1000 auditd
Oct 17 18:09:59 oceanbase01 kernel: [ 684] 0 684 4977 57 14 0 0 qemu-ga
Oct 17 18:09:59 oceanbase01 kernel: [ 685] 81 685 14555 216 34 0 -900 dbus-daemon
Oct 17 18:09:59 oceanbase01 kernel: [ 688] 32 688 17314 135 36 0 0 rpcbind
Oct 17 18:09:59 oceanbase01 kernel: [ 691] 999 691 153060 1874 65 0 0 polkitd
Oct 17 18:09:59 oceanbase01 kernel: [ 699] 998 699 29452 178 28 0 0 chronyd
Oct 17 18:09:59 oceanbase01 kernel: [ 707] 0 707 6596 124 18 0 0 systemd-logind
Oct 17 18:09:59 oceanbase01 kernel: [ 720] 0 720 48801 119 36 0 0 gssproxy
Oct 17 18:09:59 oceanbase01 kernel: [ 731] 0 731 102805 1034 31 0 0 ctcss-agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 737] 0 737 137519 5013 47 0 0 eShield-modules
Oct 17 18:09:59 oceanbase01 kernel: [ 743] 0 743 1097 35 9 0 0 acpid
Oct 17 18:09:59 oceanbase01 kernel: [ 799] 0 799 119094 498 88 0 0 NetworkManager
Oct 17 18:09:59 oceanbase01 kernel: lowmem_reserve[]: 0 2830 15865 15865
Oct 17 18:09:59 oceanbase01 kernel: lowmem_reserve[]: 0 0 13034 13034
Oct 17 18:09:59 oceanbase01 kernel: lowmem_reserve[]: 0 0 0 0
Oct 17 18:09:59 oceanbase01 kernel: Node 0 DMA: 14kB (U) 08kB 016kB 032kB 264kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15876kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 DMA32: 10564kB (UEM) 2908kB (UEM) 18716kB (UEM) 13432kB (UEM) 7564kB (UE) 32128kB (UE) 31256kB (UEM) 26512kB (UEM) 211024kB (UEM) 02048kB 04096kB = 65472kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 Normal: 9984kB (UEM) 3648kB (UEM) 35916kB (UEM) 27232kB (UEM) 24564kB (UEM) 91128kB (UE) 16256kB (EM) 5512kB (EM) 11024kB (M) 02048kB 04096kB = 56360kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 17 18:09:59 oceanbase01 kernel: 210776 total pagecache pages
Oct 17 18:09:59 oceanbase01 kernel: 0 pages in swap cache
Oct 17 18:09:59 oceanbase01 kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 17 18:09:59 oceanbase01 kernel: Free swap = 0kB
Oct 17 18:09:59 oceanbase01 kernel: Total swap = 0kB
Oct 17 18:09:59 oceanbase01 kernel: 4194162 pages RAM
Oct 17 18:09:59 oceanbase01 kernel: 0 pages HighMem/MovableOnly
Oct 17 18:09:59 oceanbase01 kernel: 127831 pages reserved
Oct 17 18:09:59 oceanbase01 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Oct 17 18:09:59 oceanbase01 kernel: [ 423] 0 423 11909 2262 30 0 0 systemd-journal
Oct 17 18:09:59 oceanbase01 kernel: [ 455] 0 455 11417 206 23 0 -1000 systemd-udevd
Oct 17 18:09:59 oceanbase01 kernel: [ 660] 0 660 13883 113 28 0 -1000 auditd
Oct 17 18:09:59 oceanbase01 kernel: [ 684] 0 684 4977 57 14 0 0 qemu-ga
Oct 17 18:09:59 oceanbase01 kernel: [ 685] 81 685 14555 216 34 0 -900 dbus-daemon
Oct 17 18:09:59 oceanbase01 kernel: [ 688] 32 688 17314 135 36 0 0 rpcbind
Oct 17 18:09:59 oceanbase01 kernel: [ 691] 999 691 153060 1874 65 0 0 polkitd
Oct 17 18:09:59 oceanbase01 kernel: [ 699] 998 699 29452 178 28 0 0 chronyd
Oct 17 18:09:59 oceanbase01 kernel: [ 707] 0 707 6596 124 18 0 0 systemd-logind
Oct 17 18:09:59 oceanbase01 kernel: [ 720] 0 720 48801 119 36 0 0 gssproxy
Oct 17 18:09:59 oceanbase01 kernel: [ 731] 0 731 102805 1034 31 0 0 ctcss-agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 737] 0 737 137519 5013 47 0 0 eShield-modules
Oct 17 18:09:59 oceanbase01 kernel: [ 743] 0 743 1097 35 9 0 0 acpid
Oct 17 18:09:59 oceanbase01 kernel: [ 799] 0 799 119094 498 88 0 0 NetworkManager
Oct 17 16:20:01 oceanbase01 systemd: Started Session 3422 of user root.
Oct 17 16:28:32 oceanbase01 systemd: Starting Cleanup of Temporary Directories…
Oct 17 16:28:33 oceanbase01 systemd: Started Cleanup of Temporary Directories.
Oct 17 16:30:01 oceanbase01 systemd: Started Session 3423 of user root.
Oct 17 17:20:01 oceanbase01 systemd: Started Session 3429 of user root.
Oct 17 18:00:01 oceanbase01 systemd: Started Session 3433 of user root.
Oct 17 18:09:59 oceanbase01 kernel: CPU: 1 PID: 6587 Comm: IO_SCHEDULE0 Kdump: loaded Not tainted 3.10.0-1160.95.1.el7.x86_64 #1
Oct 17 18:09:59 oceanbase01 kernel: Hardware name: %{distro} OpenStack Compute, BIOS 1.13.0-2.ctl2 04/01/2014
Oct 17 18:09:59 oceanbase01 kernel: Call Trace:
Oct 17 18:09:59 oceanbase01 kernel: [] dump_stack+0x19/0x1f
Oct 17 18:09:59 oceanbase01 kernel: [] dump_header+0x90/0x22d
Oct 17 18:09:59 oceanbase01 kernel: [] ? ktime_get_ts64+0x52/0xf0
Oct 17 18:09:59 oceanbase01 kernel: [] ? delayacct_end+0x8f/0xc0
Oct 17 18:09:59 oceanbase01 kernel: [] oom_kill_process+0x2d5/0x4a0
Oct 17 18:09:59 oceanbase01 kernel: [] ? oom_unkillable_task+0xcd/0x120
Oct 17 18:09:59 oceanbase01 kernel: [] out_of_memory+0x31a/0x500
Oct 17 18:09:59 oceanbase01 kernel: [] __alloc_pages_nodemask+0xae4/0xbf0
Oct 17 18:09:59 oceanbase01 kernel: [] alloc_pages_current+0x98/0x110
Oct 17 18:09:59 oceanbase01 kernel: [] __page_cache_alloc+0x97/0xb0
Oct 17 18:09:59 oceanbase01 kernel: [] filemap_fault+0x270/0x420
Oct 17 18:09:59 oceanbase01 kernel: [] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Oct 17 18:09:59 oceanbase01 kernel: [] xfs_filemap_fault+0x2c/0x40 [xfs]
Oct 17 18:09:59 oceanbase01 kernel: [] __do_fault.isra.61+0x8a/0x100
Oct 17 18:09:59 oceanbase01 kernel: [] do_read_fault.isra.63+0x4c/0x1b0
Oct 17 18:09:59 oceanbase01 kernel: [] handle_mm_fault+0xa33/0x1190
Oct 17 18:09:59 oceanbase01 kernel: [] __do_page_fault+0x213/0x510
Oct 17 18:09:59 oceanbase01 kernel: [] trace_do_page_fault+0x56/0x150
Oct 17 18:09:59 oceanbase01 kernel: [] do_async_page_fault+0x22/0x100
Oct 17 18:09:59 oceanbase01 kernel: [] async_page_fault+0x28/0x30
Oct 17 18:09:59 oceanbase01 kernel: Mem-Info:
Oct 17 18:09:59 oceanbase01 kernel: lowmem_reserve[]: 0 2830 15865 15865
Oct 17 18:09:59 oceanbase01 kernel: lowmem_reserve[]: 0 0 13034 13034
Oct 17 18:09:59 oceanbase01 kernel: lowmem_reserve[]: 0 0 0 0
Oct 17 18:09:59 oceanbase01 kernel: Node 0 DMA: 14kB (U) 08kB 016kB 032kB 264kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15876kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 DMA32: 10564kB (UEM) 2908kB (UEM) 18716kB (UEM) 13432kB (UEM) 7564kB (UE) 32128kB (UE) 31256kB (UEM) 26512kB (UEM) 211024kB (UEM) 02048kB 04096kB = 65472kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 Normal: 9984kB (UEM) 3648kB (UEM) 35916kB (UEM) 27232kB (UEM) 24564kB (UEM) 91128kB (UE) 16256kB (EM) 5512kB (EM) 11024kB (M) 02048kB 0*4096kB = 56360kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Oct 17 18:09:59 oceanbase01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 17 18:09:59 oceanbase01 kernel: 210776 total pagecache pages
Oct 17 18:09:59 oceanbase01 kernel: 0 pages in swap cache
Oct 17 18:09:59 oceanbase01 kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 17 18:09:59 oceanbase01 kernel: Free swap = 0kB
Oct 17 18:09:59 oceanbase01 kernel: Total swap = 0kB
Oct 17 18:09:59 oceanbase01 kernel: 4194162 pages RAM
Oct 17 18:09:59 oceanbase01 kernel: 0 pages HighMem/MovableOnly
Oct 17 18:09:59 oceanbase01 kernel: 127831 pages reserved
Oct 17 18:09:59 oceanbase01 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Oct 17 18:09:59 oceanbase01 kernel: [ 423] 0 423 11909 2262 30 0 0 systemd-journal
Oct 17 18:09:59 oceanbase01 kernel: [ 455] 0 455 11417 206 23 0 -1000 systemd-udevd
Oct 17 18:09:59 oceanbase01 kernel: [ 660] 0 660 13883 113 28 0 -1000 auditd
Oct 17 18:09:59 oceanbase01 kernel: [ 684] 0 684 4977 57 14 0 0 qemu-ga
Oct 17 18:09:59 oceanbase01 kernel: [ 685] 81 685 14555 216 34 0 -900 dbus-daemon
Oct 17 18:09:59 oceanbase01 kernel: [ 688] 32 688 17314 135 36 0 0 rpcbind
Oct 17 18:09:59 oceanbase01 kernel: [ 691] 999 691 153060 1874 65 0 0 polkitd
Oct 17 18:09:59 oceanbase01 kernel: [ 699] 998 699 29452 178 28 0 0 chronyd
Oct 17 18:09:59 oceanbase01 kernel: [ 707] 0 707 6596 124 18 0 0 systemd-logind
Oct 17 18:09:59 oceanbase01 kernel: [ 720] 0 720 48801 119 36 0 0 gssproxy
Oct 17 18:09:59 oceanbase01 kernel: [ 731] 0 731 102805 1034 31 0 0 ctcss-agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 737] 0 737 137519 5013 47 0 0 eShield-modules
Oct 17 18:09:59 oceanbase01 kernel: [ 743] 0 743 1097 35 9 0 0 acpid
Oct 17 18:09:59 oceanbase01 kernel: [ 799] 0 799 119094 498 88 0 0 NetworkManager
Oct 17 18:09:59 oceanbase01 kernel: [ 1033] 0 1033 25751 513 49 0 0 dhclient
Oct 17 18:09:59 oceanbase01 kernel: [ 1100] 0 1100 3177 46 11 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 1102] 0 1102 143571 2835 95 0 0 tuned
Oct 17 18:09:59 oceanbase01 kernel: [ 1113] 0 1113 3183 267 11 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 1114] 0 1114 3183 161 12 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 1115] 0 1115 3177 93 11 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 1249] 0 1249 22451 260 43 0 0 master
Oct 17 18:09:59 oceanbase01 kernel: [ 1251] 89 1251 22521 303 44 0 0 qmgr
Oct 17 18:09:59 oceanbase01 kernel: [ 1252] 0 1252 158282 1844 120 0 0 rsyslogd
Oct 17 18:09:59 oceanbase01 kernel: [ 1260] 0 1260 28250 280 57 0 -1000 sshd
Oct 17 18:09:59 oceanbase01 kernel: [ 1347] 0 1347 31596 211 20 0 0 crond
Oct 17 18:09:59 oceanbase01 kernel: [ 1399] 0 1399 27551 32 12 0 0 agetty
Oct 17 18:09:59 oceanbase01 kernel: [ 1404] 0 1404 27551 32 10 0 0 agetty
Oct 17 18:09:59 oceanbase01 kernel: [ 2256] 0 2256 3177 43 12 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 2259] 0 2259 3177 249 12 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 2260] 0 2260 3177 120 12 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 2348] 0 2348 3177 42 11 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 2353] 0 2353 3177 248 11 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 2354] 0 2354 3177 125 11 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 2470] 0 2470 3177 42 11 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 2471] 0 2471 3177 248 11 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 2472] 0 2472 3177 119 11 0 0 ctcm_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 6410] 0 6410 4024171 3652270 7656 0 0 observer
Oct 17 18:09:59 oceanbase01 kernel: [ 8181] 0 8181 243746 3007 34 0 0 ob_agentd
Oct 17 18:09:59 oceanbase01 kernel: [ 8186] 0 8186 225719 4272 39 0 0 ob_mgragent
Oct 17 18:09:59 oceanbase01 kernel: [ 8190] 0 8190 252782 5467 49 0 0 ob_monagent
Oct 17 18:09:59 oceanbase01 kernel: [ 9377] 89 9377 22477 308 47 0 0 pickup
Oct 17 18:09:59 oceanbase01 kernel: [12940] 0 12940 31917 758 20 0 0 python
Oct 17 18:09:59 oceanbase01 kernel: [12945] 0 12945 4668 387 13 0 0 python
Oct 17 18:09:59 oceanbase01 kernel: [12950] 0 12950 2366 30 9 0 0 sh
Oct 17 18:09:59 oceanbase01 kernel: [12951] 0 12951 2366 27 9 0 0 sh
Oct 17 18:09:59 oceanbase01 kernel: Out of memory: Kill process 6410 (observer) score 873 or sacrifice child
Oct 17 18:09:59 oceanbase01 kernel: Killed process 6410 (observer), UID 0, total-vm:16096684kB, anon-rss:14609080kB, file-rss:0kB, shmem-rss:0kB
Oct 17 18:10:01 oceanbase01 systemd: Started Session 3435 of user root.
Oct 17 18:20:01 oceanbase01 systemd: Started Session 3436 of user root.
Oct 17 18:28:21 oceanbase01 systemd-logind: New session 3437 of user root.
【附件】