Comments on Tsuna's blog: How long does it take to make a context switch?

pankaj@tux: this article shows that context switch...

2011-02-14T21:57:15.403-08:00

pankaj@tux: this article shows that context switching takes a lot more time than just a simple user-kernel switch (aka mode switch). System calls only do a mode switch.

What should take more time a context switch or a u...

2011-02-14T21:28:26.475-08:00

What should take more time a context switch or a user-kernel mode switch?

@James Aguilar: I'm not sure to follow your ca...

2010-11-29T17:44:51.970-08:00

@James Aguilar: I'm not sure to follow your calculation that leads you to the conclusion that you can't spend more than 3% of your time context switching. I have a MySQL database server running on a dual E7220 = 4 actual cores, it's doing 25k context switches per second, meaning it's doing on average 25000/4=6250 switches per core per second, if each switch takes 30µs, each core is spending 187500µs = 187ms per second doing context switches, which translates in 18.75% of the CPU cycles being wasted switching. In practice though, I'm guessing that this DB doesn't get the full 30µs penalty thanks to shared working set and saved TLB flushes (most of the active threads are part of the same process, so they share the same address space), so we can see 18.75% as the theoretical upper bound of the percentage of CPU cycles wasted to context switching.

@Anonymous 5 and James Aguilar: async servers are ...

2010-11-29T17:44:45.938-08:00

@Anonymous 5 and James Aguilar: async servers are more likely to perform better than non-async servers. YMWV depending on the type of server we're talking about. At least not doing context switches saves TLB flushes. If you design and implement your multi-threaded server appropriately, you can maximize the performance by using dedicated threads per task type and CPU affinity in order to keep the high cache hit rate that boosts performance so much. Yes it's more work to implement servers this way, but it's the price to pay for the best performance. I changed a server application at StumbleUpon to be fully async non-blocking and I gained from 40% to a full order of magnitude performance boost, depending on the workload.

@Anonymous 4: I left HyperThreading on because tha...

2010-11-29T17:44:36.743-08:00

@Anonymous 4: I left HyperThreading on because that's what we use on our servers at StumbleUpon (for a variety of reasons). The only "true numbers" are the ones that actually matter for your environment. I know how HT works and the limitations it has, but this isn't relevant to this post.

@Anonymous 1: All the tests happened in an x86_64 ...

2010-11-29T17:44:31.558-08:00

@Anonymous 1: All the tests happened in an x86_64 environment. I will update the post to reflect this.

@Adrian Cockcroft: I didn't use lmbench becaus...

2010-11-29T17:44:23.407-08:00

@Adrian Cockcroft: I didn't use lmbench because I was wondering how to write such a benchmark in the first place, and you always learn more by building things yourself than by using something already existing. My little benchmarks aren't supposed to be alternatives to lmbench, I simply provided the source code so others could unambiguously see how I did the benchmarks and reproduce them.

Your conclusion about writing servers in async sty...

2010-11-28T11:03:15.112-08:00

Your conclusion about writing servers in async style versus sync style is not justified by the data. You said yourself that the problem with a context switch is that it trashes your cache. What do you think happens when you submit one async operation and the thread switches to working on another one? Because all the data for the second operation is not in the cache, it has to be faulted in. The effect is the same as the effect of switching to another thread. If data for the second op does not have to be faulted in (i.e. it is already in the cache), then the same would hold for thread-switching and the switch would be inexpensive.

I can buy that you would not want to have many more CPU-bound threads than hardware threads, but even that conclusion is dubious. If the cost of a context switch is 30 microseconds and the Linux kernel has 1000 time-slices per second, then the most you could be spending on context switches is 30 ms per second, or about three percent of the computer's power. Maybe through other mechanisms the situation is actually worse than this, but without experimentation it will not be easy to be sure.

An async server still has context switches, and I ...

2010-11-27T19:30:29.456-08:00

An async server still has context switches, and I mean from a conceptual angle, not a processor angle. A select server still has to change which data structures it is working on. During those events you are switching the context and L2 will need to be switched out.

And don't forget the other cost, which would be code cleanliness and complexity.

Leaving on HyperThreading does not give a good ind...

2010-11-27T18:27:22.280-08:00

Leaving on HyperThreading does not give a good indication. HyperThreading does not have a true doubling of hardware threads. There is a probability of contention of the same rare resources. This test should be re-run with HyperThreading disabled to get a look at true numbers. HyperThreading works best when your issue is total throughput, where lazy workloads can better make use of idle areas of the CPU. In a contentious environment with latency sensitivity, HT works against predictability and overall performance when measured as lower latency.

very interesting blog post, thanks

2010-11-27T15:47:31.179-08:00

very interesting blog post, thanks

Benoit, I sent you an email to the address listed...

2010-11-27T15:01:12.399-08:00

Benoit,

I sent you an email to the address listed in your bio. Let me know if you're interested! Thanks,

Adam

You don't appear to mention if you're test...

2010-11-27T14:43:27.087-08:00

You don't appear to mention if you're testing on a x86 or x86_64 system (as those CPUs can run both). Syscalls are _much_ faster on x86_64.

Thanks! I've always wondered what the performa...

2010-11-27T14:20:16.331-08:00

Thanks! I've always wondered what the performance hit would be in a virtual machine environment.

Context switch timing benchmarks have been around ...

2010-11-27T11:43:51.708-08:00

Context switch timing benchmarks have been around for a long time, why not run something like lmbench?
http://lmbench.sourceforge.net/cgi-bin/man?keyword=lmbench&section=8