thread.error: can't start new thread

Today I was puzzled for a short while by a machine on which Python seemed to be acting up:

$ python
Python 2.5.2 (r252:60911, Jan 20 2010, 23:14:04) 
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
>>> import thread
>>> thread.start_new(lambda: None, ())
Traceback (most recent call last):
 File "", line 1, in 
thread.error: can't start new thread

The machine had 8G of RAM free and was running only about 200 processes (~500 threads total). So, WTF? While running the command above, I straced Python and here's what I saw:

mmap(NULL, 17592186048512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|0x40, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 17592186048512, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
write(2, "Traceback (most recent call last"..., 35) = 35

The second argument to mmap is the amount of memory you want to allocate, and yes that's 16GB. I checked /etc/security/limits.conf and I saw that the problem came from there. Someone put this line:

*       -       stack   17179869184  # 16GB

Now when Python creates a thread, it first allocates* memory for the stack with mmap before calling clone to start the thread. If you're also puzzled by this can't start new thread error, make sure it's not a ulimit problem!

* Of course mmap doesn't actually allocate the memory, it just "reserves" it. Actual memory isn't allocated until you use it, but in this case the OS was refusing the "reservation request" because it was more than the maximum amount of RAM that a process is allowed to have according to other ulimits