Comments on Tsuna's blog: The "Out of socket memory" error

Hi there. I'm having the "Out of socket ...

2011-11-24T10:28:30.411-08:00

Hi there.

I'm having the "Out of socket memory" error in one of my servers.

I raised the orphans from 62144 to 1062144 but still getting the error.

$ cat /proc/net/sockstat
sockets: used 2298
TCP: inuse 7 orphan 0 tw 239 alloc 522 mem 66

66 its a very low memory use.

$ cat /proc/sys/net/ipv4/tcp_mem
3071488 3075584 3079680

Should I raise the tcp_mem limit also?

$ cat /proc/version
Linux version 2.6.32-5-openvz-amd64

Sorry for my english.
You have a very nice blog, thanks.

tsuna - 2.6.32-131.0.15.el6.x86_64 (RHEL6 stock) ...

2011-10-18T16:50:31.469-07:00

tsuna - 2.6.32-131.0.15.el6.x86_64 (RHEL6 stock)

I'm building openTSDB now to get some graphs - it happened a few minutes ago and again on both counts it looked like I had loads of spare capacity!

What's your kernel version?

2011-10-05T19:31:50.449-07:00

What's your kernel version?

Hi, I seem to have a problem with no orphans but ...

2011-10-03T16:33:53.096-07:00

Hi,

I seem to have a problem with no orphans but some limit - but following your analysis I have 100216 sockets used which is still far less than 4607424; yet I am still getting "Out of socket memory" errors:

[root@frontend2 log]# cat /proc/net/sockstat | grep orphan
TCP: inuse 134376 orphan 0 tw 87 alloc 150249 mem 100217
[root@frontend2 log]#
[root@frontend2 log]#
[root@frontend2 log]# cat /proc/sys/net/ipv4/tcp_mem
2303712 3071616 4607424
[root@frontend2 log]# cat /proc/net/sockstat
sockets: used 150458
TCP: inuse 134376 orphan 1 tw 69 alloc 150250 mem 100217
UDP: inuse 12 mem 3
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
[root@frontend2 log]# cat /proc/sys/net/ipv4/tcp_max_orphans
65536
[root@frontend2 log]# cat /proc/net/sockstat
sockets: used 150458
TCP: inuse 134375 orphan 1 tw 76 alloc 150251 mem 100216
UDP: inuse 12 mem 3
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

If I were you I'd look at other metrics such a...

2011-04-18T16:01:18.963-07:00

If I were you I'd look at other metrics such as number of sockets, number of connections opened, whether the box is sending SYN cookies (which are enabled by default in a lot of kernels these days), the packet rate, and such.

PS: all these metrics and many more are exposed by OpenTSDB (http://opentsdb.net) with a data point every 15 seconds (by default). Give it a try and I promise you'll never wanna use Cacti again :)

Hi, I saw this error on a server over the weekend ...

2011-04-11T01:45:47.107-07:00

Hi, I saw this error on a server over the weekend so I started monitoring the orphans count from sockstat via cacti to try to understand what was happening.

The error recurred but rather than a gradually climbing line what I saw was a near vertical ascent from the low hundreds to 30K in a fraction of a second.

This suggests to me that we have some sort of software issue, most likely with php-fastcgi, but I don't have an immediate clue how to diagnose it or find a solution.

Any thoughts?