So this week after a version upgrade on GraphicsMagick we got some segfaults on our servers. Nothing terrible, twelve segfaults or close to that on a 24 hour period. The only information was a line on
No core dumps since
ulimit -c is zeroed. What to do to at least have an idea of what is happening?
Well luckily I build the packages for our internal use so I had he build directory available with the unstripped binaries, with that it’s trivial to use the GNU Debugger (gdb) and find what is going on.
First, notice that the segfault is happening on a shared lib, this is per se a complication. You see, when you have the segfault to happen on a non-shared lib binary the ip (instruction pointer) value points to the instruction on the binary, in this case it is pointing to a shared lib, dynamically linked on the
To find the instruction, then, subtract the offset given on the segfault message (it’s the 7fd1379b9000 part after the lib’s name) from the ip:
Finally, using GDB you can check what is happening @ that addres on the library, provided you have an unstripped object (you can get it with -dbg packages on debian/ubuntu):
There’s the culprit. You can also find some info on the stripped library using
nm, remember that
nm will not show anything on shared libs if not used with the
-D option (showing just part of the output):
1 2 3 4 5 6 7 8 9 10 11
You can see that there are some PNG related symbols around the address 0x21xxxx. If you check the code for GraphicsMagic PNG support you will see that
WritePNGImage is part of the
In this case I correlated the logs and found that the request that caused the segfault completed without problems and the PNG image was correctly generated, so my conclusion is that the segfault is happening on some non-crucial part of those functions, but there’s not a lot of things to do exactly pinpoint the problem.
ldd are powerful tools when debugging or trying to do a postmortem on a segfault. It would be easier to find what exactly is going on with a core dump and maybe more info.