I ran into an interesting issue the other day. Something I had not seen before and I thought I'd mention it here.
One of the servers I manage was having occasional issues with processes being killed. Syslog entries indicated processes were being killed because the system was out of memory. This on a system that has 32GB of RAM and monitoring shows has several gigs of RAM free which had me a little puzzled at first.
The out of memory killer kills processes if the system is out of memory, but the system did not appear to be out of memory. So what was going on?
Well, as it turns out the system was out of low memory.
The issue was that this system had 32bit Linux installed on it and should have had 64bit Linux installed. The reason that is a problem is that 32bit operating systems manage memory using memory space below 1024k. The more memory the OS manages the more low memory space is needed to manage it. In this case managing 32GB of memory consumed too much low memory.
For 64bit Linux all memory is essentially low memory so this issue does not exist.
A short term solution was to setup a bunch of huge pages which reduced the number of memory pages needing to be managed in low memory. The long term solution was to reinstall the server with 64bit Linux.
