服务器内存不足导致LINUX KILL MYSQLD 故障

环境说明

OS操作系统：SUSE Linux 11

数据库版 本：MYSQL 5.6

故障问题描述

客户反映MYSQL数据库因为内存不足被重启。

故障分析

1、根据MYSQL报警日志确认问题：

2016-10-24 12:28:49 29226 [Note] Plugin 'FEDERATED' is disabled.

2016-10-24 12:28:49 29226 [Note] InnoDB: Using atomics to ref count buffer pool pages

2016-10-24 12:28:49 29226 [Note] InnoDB: The InnoDB memory heap is disabled

2016-10-24 12:28:49 29226 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2016-10-24 12:28:49 29226 [Note] InnoDB: Compressed tables use zlib 1.2.3

2016-10-24 12:28:49 29226 [Note] InnoDB: Using Linux native AIO

2016-10-24 12:28:49 29226 [Note] InnoDB: Using CPU crc32 instructions

2016-10-24 12:28:49 29226 [Note] InnoDB: Initializing buffer pool, size = 2.0G

2016-10-24 12:28:50 29226 [Note] InnoDB: Completed initialization of buffer pool

2016-10-24 12:28:50 29226 [Note] InnoDB: Highest supported file format is Barracuda.

2016-10-24 12:28:50 29226 [Note] InnoDB: Log scan progressed past the checkpoint lsn 27852365485

2016-10-24 12:28:50 29226 [Note] InnoDB: Database was not shutdown normally!

2016-10-24 12:28:50 29226 [Note] InnoDB: Starting crash recovery.

2016-10-24 12:28:50 29226 [Note] InnoDB: Reading tablespace information from the .ibd files...

2016-10-24 12:28:50 29226 [Note] InnoDB: Restoring possible half-written data pages

2016-10-24 12:28:50 29226 [Note] InnoDB: from the doublewrite buffer...

InnoDB: Doing recovery: scanned up to log sequence number 27852370215

2016-10-24 12:28:51 29226 [Note] InnoDB: Starting an apply batch of log records to the database...

InnoDB: Progress in percent: 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

InnoDB: Apply batch completed

InnoDB: Last MySQL binlog file position 0 401322767, file name TSPTONLDB01-it_mgmad-bin.000023

2016-10-24 12:28:52 29226 [Note] InnoDB: 128 rollback segment(s) are active.

2016-10-24 12:28:52 29226 [Note] InnoDB: 5.6.19 started; log sequence number 27852370215

2016-10-24 12:28:52 29226 [Note] Recovering after a crash using /dbdata/it_mgmad/binlog/it_mgmad/TSPTONLDB01-it_mgmad-bin

2016-10-24 12:29:00 29226 [Note] Starting crash recovery...

2016-10-24 12:29:00 29226 [Note] Crash recovery finished.

2016-10-24 12:29:06 29226 [Note] Server hostname (bind-address): '*'; port: 60002

2016-10-24 12:29:06 29226 [Note] IPv6 is available.

2016-10-24 12:29:06 29226 [Note] - '::' resolves to '::';

2016-10-24 12:29:06 29226 [Note] Server socket created on IP: '::'.

2016-10-24 12:29:06 29226 [Warning] 'proxies_priv' entry '@ root@tsptonldb01' ignored in --skip-name-resolve mode.

2016-10-24 12:29:06 29226 [Note] Event Scheduler: Loaded 0 events

2016-10-24 12:29:06 29226 [Note] /usr/local/mysql/bin/mysqld: ready for connections.

Version: '5.6.19-upsql-0.1.1-log' socket: '/etc/mysock/it_mgmad.sock' port: 60002 UpSQL Server

2016-10-24 12:30:21 29226 [ERROR] /usr/local/mysql/bin/mysqld: Out of memory (Needed 8160 bytes)

2016-10-24 12:30:21 29226 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space

161024 12:31:35 mysqld_safe Number of processes running now: 0

161024 12:31:35 mysqld_safe mysqld restarted

2016-10-24 12:31:37 29659 [Warning] Buffered warning: Could not increase number of max_open_files to more than 10240 (request: 25000)

从上面的信息，我们看到， MSYQL报内存不足的错误无法启动。

2. 操作系统日志提示错误

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996482] Mem-Info:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996483] Node 0 DMA per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996485] CPU 0: hi: 0, btch: 1 usd: 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996486] CPU 1: hi: 0, btch: 1 usd: 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996487] Node 0 DMA32 per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996488] CPU 0: hi: 186, btch: 31 usd: 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996489] CPU 1: hi: 186, btch: 31 usd: 5

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996490] Node 0 Normal per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996491] CPU 0: hi: 186, btch: 31 usd: 30

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996492] CPU 1: hi: 186, btch: 31 usd: 144

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996495] active_anon:658762 inactive_anon:260562 isolated_anon:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996496] active_file:353 inactive_file:1716 isolated_file:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996496] unevictable:0 dirty:6 writeback:5 unstable:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996497] free:21800 slab_reclaimable:2214 slab_unreclaimable:10742

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996498] mapped:765 shmem:456 pagetables:6664 bounce:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996499] Node 0 DMA free:15880kB min:256kB low:320kB high:384kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996505] lowmem_reserve[]: 0 3000 4010 4010

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996507] Node 0 DMA32 free:54356kB min:50368kB low:62960kB high:75552kB active_anon:2275764kB inactive_anon:569032kB active_file:12kB inactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:12kB slab_reclaimable:1896kB slab_unreclaimable:7060kB kernel_stack:2904kB pagetables:6588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:611 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996513] lowmem_reserve[]: 0 0 1010 1010

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996515] Node 0 Normal free:16964kB min:16956kB low:21192kB high:25432kB active_anon:359284kB inactive_anon:473216kB active_file:1400kB inactive_file:6776kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:0kB dirty:24kB writeback:20kB mapped:3028kB shmem:1812kB slab_reclaimable:6960kB slab_unreclaimable:35908kB kernel_stack:3624kB pagetables:20068kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12878 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996520] lowmem_reserve[]: 0 0 0 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996522] Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996544] Node 0 DMA32: 1308*4kB 973*8kB 448*16kB 181*32kB 91*64kB 43*128kB 11*256kB 8*512kB 8*1024kB 1*2048kB 0*4096kB = 54456kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996549] Node 0 Normal: 439*4kB 303*8kB 169*16kB 83*32kB 18*64kB 5*128kB 4*256kB 3*512kB 3*1024kB 0*2048kB 0*4096kB = 16964kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996553] 38343 total pagecache pages

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996554] 35726 pages in swap cache

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996556] Swap cache stats: add 5506519, delete 5470793, find 22920240/23085658

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996557] Free swap = 0kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996557] Total swap = 2097148kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006955] 1048560 pages RAM

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006957] 67010 pages reserved

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006958] 1978 pages shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006959] 953768 pages non-shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006960] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006974] [ 571] 0 571 4223 1 1 -17 -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006981] [ 1689] 0 1689 998 1 0 0 0 acpid

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006983] [ 1704] 100 1704 5444 117 0 0 0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006986] [ 1722] 0 1722 4824 62 0 0 0 syslog-ng

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006988] [ 1725] 0 1725 1065 0 0 0 0 klogd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006991] [10038] 0 10038 31835 168 1 0 0 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006994] [10096] 0 10096 15193 1 0 0 0 VGAuthService

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006996] [10185] 101 10185 8608 146 0 0 0 hald

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006999] [10188] 0 10188 27278 0 0 0 0 console-kit-dae

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007001] [10189] 0 10189 4505 2 0 0 0 hald-runner

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007004] [10284] 0 10284 8690 0 0 0 0 hald-addon-inpu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007007] [10308] 0 10308 8690 17 0 0 0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007009] [10312] 101 10312 9129 0 1 0 0 hald-addon-acpi

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007012] [10313] 0 10313 8690 16 0 0 0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007014] [12046] 0 12046 7409 8 0 -17 -1000 auditd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007017] [12048] 0 12048 3606 0 0 0 0 audispd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007020] [12081] 0 12081 4723 16 0 0 0 rpcbind

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007022] [12083] 0 12083 2088 17 0 0 0 haveged

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007025] [12140] 0 12140 4216 1 1 -17 -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007027] [12141] 0 12141 4216 1 0 -17 -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007029] [12224] 0 12224 6759 1 1 0 0 mcelog

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007031] [12701] 0 12701 17296 28 0 -17 -1000 sshd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007034] [12715] 0 12715 17933 0 1 0 0 cupsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007036] [12730] 0 12730 59557 71 0 0 0 nscd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007038] [12798] 0 12798 9552 28 0 0 0 master

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007041] [12825] 51 12825 10052 31 1 0 0 qmgr

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007043] [12864] 0 12864 16711 47 1 0 0 gdm

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007046] [12878] 0 12878 27377 121 0 0 0 X

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007048] [12885] 0 12885 3697 15 0 0 0 cron

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007051] [12924] 0 12924 1121 2 1 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007055] [12925] 0 12925 1121 2 1 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007058] [12926] 0 12926 1121 2 0 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007061] [12927] 0 12927 1121 2 1 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007064] [12928] 0 12928 1121 2 1 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007067] [12929] 0 12929 1121 2 0 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007071] [13064] 107 13064 5548 0 1 0 0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007074] [13065] 107 13065 3362 0 1 0 0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007077] [13066] 107 13066 39551 82 0 0 0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007080] [13079] 107 13079 9863 67 0 0 0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007083] [13083] 107 13083 31861 0 1 0 0 at-spi-registry

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007086] [13084] 107 13084 70831 2912 0 0 0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007090] [13086] 107 13086 20207 0 1 0 0 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007092] [13094] 107 13094 57536 135 0 0 0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007095] [13096] 107 13096 10854 0 0 0 0 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007098] [13097] 107 13097 54246 0 0 0 0 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007101] [21349] 1018 21349 2857 2 1 0 0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007105] [12777] 1016 12777 2857 2 0 0 0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007109] [18450] 0 18450 13384 49 1 0 0 gdm-simple-slav

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007112] [18452] 0 18452 66809 13462 1 0 0 X

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007115] [18489] 107 18489 5548 0 1 0 0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007118] [18518] 0 18518 24613 0 1 0 0 gdm-session-wor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007121] [18540] 0 18540 39584 89 1 0 0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007124] [18613] 0 18613 4178 19 0 0 0 gpg-agent

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007126] [18619] 0 18619 5548 0 1 0 0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007129] [18620] 0 18620 3429 0 0 0 0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007132] [18623] 0 18623 9887 123 1 0 0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007135] [18627] 0 18627 91696 332 0 0 0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007138] [18631] 0 18631 23814 1 1 0 0 gnome-keyring-d

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007141] [18635] 0 18635 10908 0 1 0 0 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007144] [18639] 0 18639 16822 19 0 0 0 gvfs-fuse-daemo

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007147] [18645] 0 18645 51782 400 1 0 0 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007149] [18650] 0 18650 93659 621 1 0 0 gnome-panel

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007152] [18651] 0 18651 135905 13563 1 0 0 nautilus

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007155] [18653] 0 18653 36624 0 0 0 0 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007160] [18659] 0 18659 82105 455 1 0 0 main-menu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007171] [18662] 0 18662 12083 0 1 0 0 gvfsd-trash

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007174] [18666] 0 18666 13609 1 0 0 0 gvfs-hal-volume

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007178] [18668] 0 18668 15774 1 1 0 0 gvfs-gphoto2-vo

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007181] [18677] 0 18677 75456 1 1 0 0 gnome-volume-co

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007184] [18681] 0 18681 56947 1 0 0 0 python

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007187] [18686] 0 18686 39298 113 0 0 0 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007189] [18688] 0 18688 57564 125 0 0 0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007192] [18692] 0 18692 38020 114 1 0 0 gnome-screensav

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007195] [18693] 0 18693 48186 0 1 0 0 pulseaudio

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007198] [18718] 0 18718 10962 0 0 0 0 gvfsd-burn

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007201] [21417] 0 21417 59168 142 1 0 0 application-bro

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007204] [21420] 0 21420 57129 199 0 0 0 baobab

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007208] [21463] 0 21463 35883 1 0 0 0 gnomesu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007211] [21464] 0 21464 13445 4 0 0 0 gnomesu-pam-bac

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007214] [21471] 0 21471 3221 1 0 0 0 yast2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007217] [21501] 0 21501 118060 114 0 0 0 y2base

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007221] [21621] 0 21621 55974 123 0 0 0 gnome-terminal

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007224] [21622] 0 21622 1616 0 0 0 0 gnome-pty-helpe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007227] [21623] 0 21623 3505 1 0 0 0 bash

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007230] [21649] 0 21649 9244 1 0 0 0 gvfsd-metadata

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007233] [31142] 1016 31142 1100222 219744 1 0 0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007236] [31181] 1018 31181 1039509 163992 1 0 0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007239] [ 699] 1017 699 2856 98 0 0 0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007242] [ 1625] 1017 1625 1076677 414298 1 0 0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007246] [ 1861] 1019 1861 641372 48223 1 0 0 java

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007284] [14971] 51 14971 10014 128 0 0 0 pickup

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007287] Out of memory: Kill process 31142 (mysqld) score 283 or sacrifice child

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007358] Killed process 31142 (mysqld) total-vm:4400888kB, anon-rss:878976kB, file-rss:0kB

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351765] vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351772] vmtoolsd cpuset=/ mems_allowed=0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351777] Pid: 10038, comm: vmtoolsd Tainted: G X 3.0.13-0.27-default #1

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351780] Call Trace:

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351803] [] dump_trace+0x75/0x300

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351814] [] dump_stack+0x69/0x6f

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351825] [] dump_header+0x8f/0x1f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351834] [] oom_kill_process+0x95/0x180

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351841] [] out_of_memory+0x14c/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351848] [] __alloc_pages_slowpath+0x7ab/0x7f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351854] [] __alloc_pages_nodemask+0x1b1/0x1c0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351863] [] alloc_pages_current+0xa5/0x120

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351871] [] filemap_fault+0x172/0x3d0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351878] [] __do_fault+0x7a/0x5d0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351884] [] handle_pte_fault+0x9d/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351893] [] do_page_fault+0x207/0x4c0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351900] [] page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.353040] DWARF2 unwinder stuck at page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.353042]

通过日志发现发现在10月24日到11月份出现多次内存不足，操作系统 KILL 掉MYSQLD进程的情况。

在该服务器上有3个MYSQL实例，每个MYSQL实例的 INNODB 缓冲池设置为2GB，但操作系统的物理内存

只有4GB，另外在服务器上还有一个数据同步的 JAVA进程。结合日志分析，MYSQL 内存分配不合理，建议把服务器内存扩展到10GB。

建议MYSQL INNODB池计算方法： INNODB 缓冲池内存= 操作系统物理内存 - （预留给操作系统的内存1-2GB）