貌似之前内存256MB,后来调整为800MB?
Dec 28 21:38:19 db4 kernel: Free swap = 2047992kB
Dec 28 21:38:19 db4 kernel: Total swap = 2047992kB
Dec 28 21:38:19 db4 kernel: 65519 pages RAM 《---- 65519*4096=256MB
Dec 28 21:38:19 db4 kernel: 4474 pages reserved
Dec 28 21:38:19 db4 kernel: 9318 pages shared
Dec 28 21:38:19 db4 kernel: 51480 pages non-shared
出现物理内存不足linux把进程干掉了:
ec 28 21:38:19 db4 kernel: sdbbp invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Dec 28 21:38:19 db4 kernel: sdbbp cpuset=/ mems_allowed=0
Dec 28 21:38:19 db4 kernel: Pid: 1252, comm: sdbbp Tainted: G --------------- H 2.6.32-431.el6.x86_64 #1
调用堆栈:
Dec 28 21:38:19 db4 kernel: Call Trace:
Dec 28 21:38:19 db4 kernel: [] ? cpuset_print_task_mems_allowed+0x91/0xb0
Dec 28 21:38:19 db4 kernel: [] ? dump_header+0x90/0x1b0
Dec 28 21:38:19 db4 kernel: [] ? security_real_capable_noaudit+0x3c/0x70
Dec 28 21:38:19 db4 kernel: [] ? oom_kill_process+0x82/0x2a0
Dec 28 21:38:19 db4 kernel: [] ? select_bad_process+0xe1/0x120
Dec 28 21:38:19 db4 kernel: [] ? out_of_memory+0x220/0x3c0
Dec 28 21:38:19 db4 kernel: [] ? __alloc_pages_nodemask+0x8ac/0x8d0
Dec 28 21:38:19 db4 kernel: [] ? alloc_pages_current+0xaa/0x110
Dec 28 21:38:19 db4 kernel: [] ? __page_cache_alloc+0x87/0x90
Dec 28 21:38:19 db4 kernel: [] ? find_get_page+0x1e/0xa0
Dec 28 21:38:19 db4 kernel: [] ? filemap_fault+0x1a7/0x500
Dec 28 21:38:19 db4 kernel: [] ? __do_fault+0x54/0x530
Dec 28 21:38:19 db4 kernel: [] ? handle_pte_fault+0xf7/0xb00
Dec 28 21:38:19 db4 kernel: [] ? futex_wake+0x10e/0x120
Dec 28 21:38:19 db4 kernel: [] ? handle_mm_fault+0x22a/0x300
Dec 28 21:38:19 db4 kernel: [] ? __do_page_fault+0x138/0x480
Dec 28 21:38:19 db4 kernel: [] ? cpumask_any_but+0x31/0x50
也就是说在分配内存的时候内核里面发生OOM,产生了干掉进程的请求。
可以看到,在DMA32 Zone里面内存已经爆了:
Dec 28 21:38:19 db4 kernel: Node 0 DMA32 free:96kB min:92kB low:112kB high:136kB active_anon:64084kB inactive_anon:64284kB active_file:12712kB inactive_file:19696kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:242336kB mlocked:0kB dirty:0kB writeback:19208kB mapped:2708kB shmem:200kB slab_reclaimable:7628kB slab_unreclaimable:49104kB kernel_stack:688kB pagetables:2996kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:250112 all_unreclaimable? yes
通过计算RSS可以看到sequoiadb的安装进程组总共占用了大概130MB左右:
Dec 28 21:38:19 db4 kernel: [ 1203] 0 1203 132696 28106 0 0 0 sequoiadb-1.10-
Dec 28 21:38:19 db4 kernel: [ 1249] 0 1249 26516 338 0 0 0 install_om.sh
Dec 28 21:38:19 db4 kernel: [ 1250] 0 1250 7341 903 0 0 0 sdb
Dec 28 21:38:19 db4 kernel: [ 1251] 0 1251 26812 903 0 0 0 sdbbp
Dec 28 21:38:19 db4 kernel: [ 1254] 500 1254 4990 400 0 0 0 sdbstart
Dec 28 21:38:19 db4 kernel: [ 1255] 500 1255 80666 1863 0 0 0 sequoiadb
其他的是一些系统进程占用的。
不知道是不是你在kernel设置里面禁止了swap空间的使用?我看到你这里:
Dec 28 21:38:19 db4 kernel: Free swap = 2047992kB
Dec 28 21:38:19 db4 kernel: Total swap = 2047992kB
貌似swap没有被使用就直接触发OOM了。
28号21:42以后貌似内存调整为800MB了:
Dec 28 21:42:49 db4 kernel: Memory: 773392k/819136k available (5325k kernel code, 392k absent, 45352k reserved, 7013k data, 1276k init)
但是同样出现OOM,这次内存爆貌似是shared memory导致:
Dec 28 21:46:02 db4 kernel: 204783 pages RAM
Dec 28 21:46:02 db4 kernel: 7040 pages reserved
Dec 28 21:46:02 db4 kernel: 139890 pages shared
Dec 28 21:46:02 db4 kernel: 61656 pages non-shared
然后22:58再次发生OOM,同样swap没有使用:
Dec 28 22:58:21 db4 kernel: Free swap = 2047992kB
Dec 28 22:58:21 db4 kernel: Total swap = 2047992kB
Dec 28 22:58:21 db4 kernel: 204783 pages RAM
Dec 28 22:58:21 db4 kernel: 7040 pages reserved
Dec 28 22:58:21 db4 kernel: 168159 pages shared
Dec 28 22:58:21 db4 kernel: 52449 pages non-shared
不过这次撞枪口上的是flush-8:0进程,系统也没有杀进程,应该可能是这一把最后成功了。
我这里看不到你的内核参数,是不是你设置了
/proc/sys/vm/swappiness=0