2021-07-14 08:45:12
2021-07-06 09:51:14
2021-07-06 09:49:50
156067
Hello !Friendica Support
can someone help me to shed a bit more light onto the #OOM kill log for my #Friendica server, I am not sure how to read it and am not sure if really all the RAM was used when #MariaDB got killed.
This happens since a week or so. I decreased some of the number of worker processes for php-fpm, as this get's killed too sometimes, some PHP RAM limit for scripts, and decreased some of the MariaDB setting. I am not good in tuning MariaDB, so at some time I decided to buy me some time there and threw some #RAM on to the problem.
This server only runs Friendica for 10 accounts or so and has 19.5 GB RAM and 20 GB swap, so in my opinion the OOM kills should only happen when almost all of this is used.
Can someone highlight the important information to troubleshoot this?
cc @utzer
can someone help me to shed a bit more light onto the #OOM kill log for my #Friendica server, I am not sure how to read it and am not sure if really all the RAM was used when #MariaDB got killed.
This happens since a week or so. I decreased some of the number of worker processes for php-fpm, as this get's killed too sometimes, some PHP RAM limit for scripts, and decreased some of the MariaDB setting. I am not good in tuning MariaDB, so at some time I decided to buy me some time there and threw some #RAM on to the problem.
This server only runs Friendica for 10 accounts or so and has 19.5 GB RAM and 20 GB swap, so in my opinion the OOM kills should only happen when almost all of this is used.
Can someone highlight the important information to troubleshoot this?
Jul 06 10:03:36 social kernel: mariadbd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jul 06 10:03:37 social kernel: Call Trace:
Jul 06 10:03:37 social kernel: dump_stack+0x6b/0x83
Jul 06 10:03:37 social kernel: dump_header+0x4a/0x1f3
Jul 06 10:03:37 social kernel: oom_kill_process.cold+0xb/0x10
Jul 06 10:03:37 social kernel: out_of_memory+0x1a5/0x490
Jul 06 10:03:37 social kernel: __alloc_pages_slowpath.constprop.0+0xbcb/0xca0
Jul 06 10:03:37 social kernel: __alloc_pages_nodemask+0x2e4/0x320
Jul 06 10:03:37 social kernel: pagecache_get_page+0xdd/0x380
Jul 06 10:03:37 social kernel: filemap_fault+0x6a1/0x900
Jul 06 10:03:37 social kernel: ? xas_find+0x161/0x1a0
Jul 06 10:03:37 social kernel: ? filemap_map_pages+0x223/0x410
Jul 06 10:03:37 social kernel: __do_fault+0x38/0xd0
Jul 06 10:03:37 social kernel: handle_mm_fault+0x1455/0x1990
Jul 06 10:03:37 social kernel: do_user_addr_fault+0x1b8/0x3f0
Jul 06 10:03:37 social kernel: exc_page_fault+0x64/0x150
Jul 06 10:03:37 social kernel: ? asm_exc_page_fault+0x8/0x30
Jul 06 10:03:37 social kernel: asm_exc_page_fault+0x1e/0x30
Jul 06 10:03:37 social kernel: RIP: 0033:0x5594d4826836
Jul 06 10:03:37 social kernel: Code: Unable to access opcode bytes at RIP 0x5594d482680c.
Jul 06 10:03:37 social kernel: RSP: 002b:00007fa0895eec20 EFLAGS: 00010202
Jul 06 10:03:37 social kernel: RAX: 0000000000000038 RBX: 00007fa0895eede8 RCX: 0000000000000038
Jul 06 10:03:37 social kernel: RDX: 0000000000000001 RSI: 0000000000000064 RDI: 00005594d4e75f60
Jul 06 10:03:37 social kernel: RBP: 00007fa0895efa40 R08: 0000000000000030 R09: 00005594d4e48d54
Jul 06 10:03:37 social kernel: R10: 00007fa082e9fd58 R11: 00007fa347e64018 R12: 00007fa0895eed02
Jul 06 10:03:37 social kernel: R13: 00007fa082e9fd58 R14: 0000000000000066 R15: 00000000ffffeb1b
Jul 06 10:03:37 social kernel: Mem-Info:
Jul 06 10:03:37 social kernel: active_anon:3532225 inactive_anon:362219 isolated_anon:113
active_file:36 inactive_file:1269 isolated_file:7
unevictable:0 dirty:32 writeback:62
slab_reclaimable:21227 slab_unreclaimable:124058
mapped:108 shmem:9 pagetables:20761 bounce:0
free:36699 free_pcp:1107 free_cma:0
Jul 06 10:03:37 social kernel: Node 0 active_anon:14128900kB inactive_anon:1448876kB active_file:144kB inactive_file:5076kB unevictable:0kB isolated(anon):452kB isolated(file):28kB mapped:432kB dirty:128kB writeback:248kB shmem:36kB shmem>
Jul 06 10:03:37 social kernel: Node 0 DMA free:11796kB min:52kB low:64kB high:76kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908k>
Jul 06 10:03:37 social kernel: lowmem_reserve[]: 0 2984 19983 19983 19983
Jul 06 10:03:37 social kernel: Node 0 DMA32 free:77712kB min:10084kB low:13136kB high:16188kB reserved_highatomic:0KB active_anon:2566244kB inactive_anon:223200kB active_file:0kB inactive_file:2064kB unevictable:0kB writepending:0kB prese>
Jul 06 10:03:37 social kernel: lowmem_reserve[]: 0 0 16999 16999 16999
Jul 06 10:03:37 social kernel: Node 0 Normal free:57288kB min:57444kB low:74848kB high:92252kB reserved_highatomic:0KB active_anon:11561892kB inactive_anon:1225344kB active_file:0kB inactive_file:3752kB unevictable:0kB writepending:100kB >
Jul 06 10:03:37 social kernel: lowmem_reserve[]: 0 0 0 0 0
Jul 06 10:03:37 social kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11796kB
Jul 06 10:03:37 social kernel: Node 0 DMA32: 3*4kB (ME) 62*8kB (UME) 80*16kB (UME) 43*32kB (UM) 97*64kB (UM) 41*128kB (UM) 6*256kB (UM) 91*512kB (UM) 11*1024kB (M) 2*2048kB (M) 0*4096kB = 78108kB
Jul 06 10:03:37 social kernel: Node 0 Normal: 281*4kB (UME) 130*8kB (UE) 95*16kB (UME) 1077*32kB (UME) 130*64kB (UM) 58*128kB (UM) 17*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 58756kB
Jul 06 10:03:37 social kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 06 10:03:37 social kernel: 17375 total pagecache pages
Jul 06 10:03:37 social kernel: 15694 pages in swap cache
Jul 06 10:03:37 social kernel: Swap cache stats: add 33996402, delete 33976575, find 7302845/9534817
Jul 06 10:03:37 social kernel: Free swap = 0kB
Jul 06 10:03:37 social kernel: Total swap = 20971516kB
Jul 06 10:03:37 social kernel: 5242741 pages RAM
Jul 06 10:03:37 social kernel: 0 pages HighMem/MovableOnly
Jul 06 10:03:37 social kernel: 119104 pages reserved
Jul 06 10:03:37 social kernel: 0 pages cma reserved
Jul 06 10:03:37 social kernel: 0 pages hwpoisoned
Jul 06 10:03:37 social kernel: Tasks state (memory values in pages):
Jul 06 10:03:37 social kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jul 06 10:03:37 social kernel: [ 250] 0 250 36054 0 290816 263 -250 systemd-journal
Jul 06 10:03:37 social kernel: [ 263] 0 263 7296 0 77824 403 -1000 systemd-udevd
Jul 06 10:03:37 social kernel: [ 267] 981 267 4903 0 77824 253 0 systemd-network
Jul 06 10:03:37 social kernel: [ 320] 81 320 3065 1 65536 287 -900 dbus-daemon
Jul 06 10:03:37 social kernel: [ 322] 973 322 102044 0 110592 800 0 memcached
Jul 06 10:03:37 social kernel: [ 326] 0 326 2241 1 57344 196 -1000 sshd
Jul 06 10:03:37 social kernel: [ 327] 0 327 3714 0 65536 241 0 systemd-logind
Jul 06 10:03:37 social kernel: [ 329] 0 329 2733 13 61440 108 0 agetty
Jul 06 10:03:37 social kernel: [ 331] 87 331 19230 11 69632 206 0 ntpd
Jul 06 10:03:37 social kernel: [ 334] 0 334 2804 0 61440 337 0 nginx
Jul 06 10:03:37 social kernel: [ 335] 33 335 2968 57 65536 506 0 nginx
Jul 06 10:03:37 social kernel: [ 336] 33 336 2857 0 61440 444 0 nginx
Jul 06 10:03:37 social kernel: [ 337] 33 337 2893 0 61440 456 0 nginx
Jul 06 10:03:37 social kernel: [ 339] 33 339 2914 0 65536 545 0 nginx
Jul 06 10:03:37 social kernel: [ 340] 33 340 2890 0 65536 486 0 nginx
Jul 06 10:03:37 social kernel: [ 341] 33 341 2857 0 61440 451 0 nginx
Jul 06 10:03:37 social kernel: [ 342] 33 342 2926 0 65536 543 0 nginx
Jul 06 10:03:37 social kernel: [ 343] 33 343 2890 0 65536 483 0 nginx
Jul 06 10:03:37 social kernel: [ 432333] 1000 432333 4039 29 69632 342 0 systemd
Jul 06 10:03:37 social kernel: [ 432334] 1000 432334 43762 12 102400 729 0 (sd-pam)
Jul 06 10:03:37 social kernel: [ 432395] 0 432395 3108 14 69632 304 0 sshd
Jul 06 10:03:37 social kernel: [ 432397] 1000 432397 3108 47 69632 272 0 sshd
Jul 06 10:03:37 social kernel: [ 432398] 1000 432398 2856 21 57344 623 0 zsh
Jul 06 10:03:37 social kernel: [ 432488] 974 432488 3148091 58796 19836928 2118350 0 mariadbd
Jul 06 10:03:37 social kernel: [ 432526] 33 432526 70728 392 204800 3665 0 php
Jul 06 10:03:37 social kernel: [ 432534] 33 432534 90108 731 237568 4057 0 php
Jul 06 10:03:37 social kernel: [ 453567] 0 453567 69251 80 221184 1584 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 510552] 33 510552 93139 3988 278528 2671 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 511063] 33 511063 2448209 825518 13422592 819532 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 516530] 33 516530 98241 2766 315392 8975 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 518145] 33 518145 93196 844 278528 5843 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 518204] 33 518204 92432 3354 266240 2614 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 526863] 33 526863 93117 1729 274432 4917 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 528492] 33 528492 93032 2855 274432 3721 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 529013] 33 529013 93284 779 278528 5999 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 529934] 33 529934 93208 3542 278528 2953 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 529953] 33 529953 2447919 1005324 13332480 628811 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 530657] 33 530657 2447309 1230344 14516224 550499 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 530834] 33 530834 92722 587 274432 5567 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 531721] 33 531721 105500 4383 380928 14318 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 534001] 0 534001 1900 1 49152 111 0 sh
Jul 06 10:03:37 social kernel: [ 534611] 33 534611 2447233 697236 13426688 947553 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 535337] 33 535337 92913 1497 274432 4952 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 537013] 33 537013 92869 1984 274432 4421 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 538699] 0 538699 1900 1 57344 116 0 borgbackup.sh
Jul 06 10:03:37 social kernel: [ 538702] 0 538702 52347 12719 466944 34129 0 borg
Jul 06 10:03:37 social kernel: [ 538709] 0 538709 4072 392 69632 541 0 ssh
Jul 06 10:03:37 social kernel: [ 538750] 33 538750 92575 2590 278528 3470 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 538831] 33 538831 92858 4286 274432 2018 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 539584] 33 539584 92450 3123 270336 2868 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 539622] 33 539622 93377 714 278528 6194 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 539648] 33 539648 93459 1085 274432 5731 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 539754] 33 539754 93830 2672 282624 4704 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 543681] 33 543681 92433 452 266240 5460 0 php-fpm7
Jul 06 10:03:37 social kernel: [ 563640] 33 563640 94672 1159 278528 7201 0 php
Jul 06 10:03:37 social kernel: [ 565310] 33 565310 91653 1145 253952 5204 0 php
Jul 06 10:03:37 social kernel: [ 598206] 33 598206 90786 939 253952 4629 0 php
Jul 06 10:03:37 social kernel: [ 598246] 33 598246 91853 487 266240 5950 0 php
Jul 06 10:03:37 social kernel: [ 610639] 33 610639 94846 11 278528 8788 0 php
Jul 06 10:03:37 social kernel: [ 611433] 33 611433 92272 12 270336 6706 0 php
Jul 06 10:03:37 social kernel: [ 611554] 33 611554 93439 12 274432 7855 0 php
Jul 06 10:03:37 social kernel: [ 611734] 33 611734 92280 2451 262144 4705 0 php
Jul 06 10:03:37 social kernel: [ 612234] 33 612234 90587 12 241664 5292 0 php
Jul 06 10:03:37 social kernel: [ 612279] 33 612279 71109 2396 217088 1546 0 php-fpm7
Jul 06 10:03:37 social kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=432488,uid=974
Jul 06 10:03:37 social kernel: Out of memory: Killed process 432488 (mariadbd) total-vm:12592364kB, anon-rss:235184kB, file-rss:0kB, shmem-rss:0kB, UID:974 pgtables:19372kB oom_score_adj:0
Jul 06 10:03:37 social kernel: oom_reaper: reaped process 432488 (mariadbd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 06 10:03:37 social kernel: audit: type=1131 audit(1625558617.191:1682): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mariadb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jul 06 10:03:37 social audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mariadb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jul 06 10:03:37 social systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Jul 06 10:03:37 social systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Jul 06 10:03:37 social systemd[1]: mariadb.service: Failed with result 'oom-kill'.cc @utzer
elrido
The total_vm sizes are in KiB. So, OOM killed mariadb because it was the largest at ~3.1 GiB memory used, but it is surprising to see 4 php-fpm7 processes using ~2.4 GiB each. My guess would be that either the php-fpm allows too large of a memory consumption or there are too many processes allowed. You may have reasons to let your php processes use more than 256 MiB (the default for "memory_limit" in... show more
The total_vm sizes are in KiB. So, OOM killed mariadb because it was the largest at ~3.1 GiB memory used, but it is surprising to see 4 php-fpm7 processes using ~2.4 GiB each. My guess would be that either the php-fpm allows too large of a memory consumption or there are too many processes allowed. You may have reasons to let your php processes use more than 256 MiB (the default for "memory_limit" in php.ini - maybe you added a zero to that?) but than you need to adjust the "pm.max_children" setting of your php-fpm worker to not exceed the RAM.
Therefore, "pm.max_children" times "memory_limit" should be less than 19.5 GiB total RAM minus 3.5 GiB for mariadb and however much the other processes need.
nginx has a similar setting for the "worker_processes" number and you can see your nginx using up to 3 MiB per worker at the time of the OOM took the memory usage snapshot.
Admin (of friendica.utzer.de) likes this.
Admin (of friendica.utzer.de)
•I was also wondering why MariaDB got killed with that little RAM and somehow was thinking it was the reason, but it seem PHP-FPM is. So I will look into this again. But I can't decrease the number of worker so far, that is not possible, the server would need too much time to process all the jobs when it is limited to 15 php processes or even 30 taking swap into account (rough estimations only).
utzer [Friendica]
•utzer [Friendica]
•utzer [Friendica]
•elrido
Admin (of friendica.utzer.de)
•I have now only 20 php-fpm processes, but I will try what happens if I set it to 15.
journal entries:
... show more
I have now only 20 php-fpm processes, but I will try what happens if I set it to 15.
journal entries:
Admin (of friendica.utzer.de)
•https://zerobin.utzer.de/?97457187f6bd86f8#6873CQibcBscsVFY8LFktdgwb5xn9vBynLsurB9MLA2C
Hans Wolters
•Tuning a mysql/mariadb is often something that depends on the load. There is a script that can be used that will look at a lot of settings.
https://github.com/major/MySQLTuner-perl
Straight from my brain I would tune the query cache and key memory, for starters but maybe that is already in place. Different howto's will describe how to tune it
utzer [Friendica]
•Hans Wolters likes this.
Admin (of friendica.utzer.de)
•Admin (of friendica.utzer.de)
•The server OOM happens always between 3:15 and 3:45 in the morning. No process runs at that time, so it might be some higher external load. I think next I will try to block all crawlers. Now the memory footprint it very low, fortunately #Friendica still works well like that. But I am not sure why MariaDB, still uses that much RAM, I thought I set it to use 9 GB max, but the DB is probably 30 GB now and if some setting permits it will use lots of RAM of course. So I need to check that.
elrido
•Finally, since I don't use the performance schema, I switch it off to save a few extra 100 MiBs of RAM.
You can of course do the calculations manually and replace the parts in double curly brackets with your results, when not using ansible. Hope it can help you tailor your DBs memory consumption to your environment.
... show more
Finally, since I don't use the performance schema, I switch it off to save a few extra 100 MiBs of RAM.
You can of course do the calculations manually and replace the parts in double curly brackets with your results, when not using ansible. Hope it can help you tailor your DBs memory consumption to your environment.
utzer [Friendica] likes this.