10 GB of #ImageMagic temp files filled up my /tmp. Anyone know why this lately happens? Didn't you mention something like this @Steffen K9 🐰 ?
!Friendica Support
!Friendica Support
Friendica Support reshared this.
Friendica Support reshared this.
Steffen K9 🐰
•*) around 10-15 GB RAM!
Steffen K9 🐰
•utzer [Friendica]
•utzer [Friendica]
•Ermanno
•https://www.imagemagick.org/script/architecture.php#tera-pixel
(1) ImageMagick allocates memory for the images while processing them since it doesn't know in advance how much memory it needs
(2) eventually, because the image is big or because of server load, it cannot allocate more memory
(3) at this point it will start to use the pixel cache on disk
(4) the default path specified in policy.xml is to create it in /tmp
(5) if /tmp is a ram based tmpfs like on my server it is full (or almost full) when the ram is exhausted
(6) php-imagick coredumps while allocating memory according to gdb
(7) some temp files in /tmp are not cleaned making the problem even worse
I did set the temporary path to a location on disk and did not have problems so far, but maybe I was just lucky.
imagemagick is configured in /etc/ImageMagick-7/policy.xml
utzer [Friendica]
•Steffen K9 🐰
•utzer [Friendica]
•Steffen K9 🐰
•W/o working animated gif support it is useless because I can simply use libgd (php-gd) instead. Which doesn't have any issues with memory usage but also no animated gif support.
utzer [Friendica] likes this.
Steffen K9 🐰
•Ingo Jürgensmann
•I think that the issue should be resolved somehow, because... it's annoying somehow to repeatedly cleanup /tmp:
It's all the same file, many times...
My /tmp is a 2 GB tmpfs mount (red line):
Steffen K9 🐰
•Btw. for me this mostly happens when someone is uploading and sharing a large animated gif locally. I recently did that myself with a 5MB gif and the server was swapping around 15GB of data.
Ermanno
•<policy domain="resource" name="memory" value="2GiB"/>
I think the 2G default limit is for every frame of the animated gif
Ingo Jürgensmann
•<policy domain="resource" name="memory" value="256MiB"/>
Comparing that to the 704K from above this is still within the limit. My impression is that it is over and over repeated in tryting to convert something and maybe the return code is not considered when it failed. Or maybe it is, and this is why it is always retried forever. Maybe this should only retried 3 times or so and then the job should fail and be removed. I don't know...
utzer [Friendica] likes this.
Ermanno
•the php-imagick extension runs ImageMagick as a library inside the same address space as php, if ImageMagick segfaults or otherwise abnormally ends the entire php thread dumps core and is terminated. There is no return code since the call never returns and even the code that should check the return cose no longer runs.
To check the return code ImageMagick commands should be run externally and not using a php extension.
Ingo Jürgensmann
•Ermanno
•Steffen K9 🐰
•Ingo Jürgensmann
•The result seems to be this:
95G /var/tmp/
¯_(ツ)_/¯
like this
utzer [Friendica] and Steffen K9 🐰 like this.
utzer [Friendica]
•Ingo Jürgensmann
•405M total
I have no idea how to solve this issue for good...
Ingo Jürgensmann
•So, it's not like that much disk space makes it better and imagick happily consumes disk space as long as there is free space... :)
Can we get rid off of imagick please? I cannot remove it from system because other stuff may rely on it, so maybe a config option for GIF suppot/imagick would help here?
Steffen K9 🐰
•If you ask me GIF should die (again). It died once together with the marquee HTML element back in the 90s. Then some asshole invented memes and brought that shit back to life.
Ingo Jürgensmann
•Disk IO may not be an issue if you run 1 or 2 small VMs on your host server, but when you do run a dozen VMs with lots of different apps and websites, disk IO is actually a real issue.
Steffen K9 🐰
•... show more
utzer [Friendica]
•Steffen K9 🐰
•utzer [Friendica] likes this.
Steffen K9 🐰
•This setting seems to be too aggressive...
<policy domain="resource" name="area" value="16KP"/>
I get broken images when the area value is set very low like this.
So I raised that to a more reasonable value.
<policy domain="resource" name="area" value="5MP"/>
Also, I gave I*M*ck some more space on disk.
<policy domain="resource" name="disk" value="2GiB"/>
These settings work pretty well. Images are ok.
I don't have the problem with the massive temp files I*M*ck leaves behind when an image processing crashed or couldn't processed properly. (1) After the image processing the temp files are deleted automatically.
But I still have the issue with massive memory usage, swapping, cpu load when a large animated GIF is resized/processed via PHP/imagick/ImageMagick. This is what happens...
... show moreload 80
This setting seems to be too aggressive...
<policy domain="resource" name="area" value="16KP"/>
I get broken images when the area value is set very low like this.
So I raised that to a more reasonable value.
<policy domain="resource" name="area" value="5MP"/>
Also, I gave I*M*ck some more space on disk.
<policy domain="resource" name="disk" value="2GiB"/>
These settings work pretty well. Images are ok.
I don't have the problem with the massive temp files I*M*ck leaves behind when an image processing crashed or couldn't processed properly. (1) After the image processing the temp files are deleted automatically.
But I still have the issue with massive memory usage, swapping, cpu load when a large animated GIF is resized/processed via PHP/imagick/ImageMagick. This is what happens...
load 80
swapping hell
(1) Just a guess: I think this happens when the resizing process couldn't be finished due to memory issues. It could be that ImageMagick is crashing w/o a proper exit code. So it doesn't clean the temp files at the end of the processing. Maybe there are some out of memory errors in PHP error log or syslog. It's worth looking for something like this, I guess.
utzer [Friendica]
•And what changed in Friendica, I think the current stable does not have such problems, or do you guys see the same there?
Ingo Jürgensmann
•utzer [Friendica]
•Also about the clean up script, I didn't find the time yet. Aber aufgeschoben ist nicht aufgehoben.
Ingo Jürgensmann
•utzer [Friendica]
•utzer [Friendica]
•