Tsuna's blog

Hardware Growler for Mac OS X Lion

2011-10-08T21:16:00.000+02:00

Just in case this could be of any use to someone else, I compiled Growl 1.2.2 for Lion with the fix for HardwareGrowler crash on Lion that happens when disconnecting from a wireless network or waking up the Mac.You can download it here. The binary should work on Snow Leopard too. It's only compiled for x86_64 CPUs.

ext4 2x faster than XFS?

2011-09-13T20:12:00.000+02:00

For a lot of people, the conventional wisdom is that XFS outperforms ext4. I'm not sure whether this is just because XFS use to be a lot faster than ext2 or ext3 or what. I don't have anything against XFS, and actually I would like to see it outperform ext4, unfortunately my benchmarks show otherwise. I'm wondering whether I'm doing something wrong.

In the benchmark below, the same machine and same HDDs were tested with 2 different RAID controllers. In most tests, ext4 has better results than XFS. In some tests, the difference is as much as 2x. Here are the details of the config:

CPU: 2 x Intel L5630 (Westmere microarchitecture, so 2x4x2 = 16 hardware threads and lots of caches)
RAM: 2 x 6 x 8GB = 96GB DDR3 ECC+Reg Dual-Rank DIMMs
Disks: 12 x Western Digital (WD) RE4 (model: WD2003FYYS – 2TB SATA 7200rpm)
RAID controllers: Adaptec 51645 and LSI MegaRaid 9280-16i4e

Both RAID controllers are equipped with 512MB of RAM and are in their respective default factory config, except that WriteBack mode was enabled on the LSI because it's disabled by default (!). One other notable difference between the default configurations is that the Adaptec uses a strip size of 256k whereas the LSI uses 64k – this was left unchanged. Both arrays were created as RAID10 (6 pairs of 2 disks, so no spares). One controller was tested at a time, in the same machine and with the same disks. The OS (Linux 2.6.32) was on a separate RAID1 of 2 drives. The IO scheduler in use was "deadline". SysBench was using O_DIRECT on 64 files, for a total of 100GB of data.

Some observations:

Formatting XFS with the optimal values for sunit and swidth doesn't lead to much better performance. The gain is about 2%, except for sequential writes where it actually makes things worse. Yes, there was no partition table, the whole array was formatted directly as one single big filesystem.
Creating more allocation groups in XFS than physical threads doesn't lead to better performance.
XFS has much better random write throughput at low concurrency levels, but quickly degrades to the same performance level as ext4 with more than 8 threads.
ext4 has consistently better random read/write throughput and latency, even at high concurrency levels.
Similarly, for random reads ext4 also has much better throughput and latency.
By default XFS creates too few allocation groups, which artificially limits its performance at high concurrency levels. It's important to create as many AGs as hardware threads. ext4, on the other hand, doesn't really need any tuning as it performs well out of the box.

See the benchmark results in full screen or look at the raw outputs of SysBench.See the benchmark results

Hitachi 7K3000 vs WD RE4 vs Seagate Constellation ES

2011-08-28T05:13:00.003+02:00

These days, the Hitachi 7K3000 seems like the best bang for your bucks. You can get 2TB disks for around US$100. The 7K3000 isn't an "enterprise disk", so many people wouldn't buy it for their servers.
It's not clear what disks sold with the Enterprise™©® label really do to justify the big price difference. Often it seems like the hardware is exactly the same, but the firmware behaves differently, notably to report errors faster. In desktop environments, you want the disk to try hard to read bad sectors, but in RAID arrays it's better to give up quickly and let the RAID controller know, otherwise the disks might timeout from the controller's point of view, and the whole disk might be incorrectly considered dead and trigger a spurious rebuild.
So I recently benchmarked the Hitachi 7K3000 against two other "enterprise" disks, the Western Digital RE4 and the Seagate Constellation ES.

The line up

Hitachi 7K3000 model: HDS723020BLA642 – the baseline
Western Digital (WD) RE4 model: WD2003FYYS
Seagate Constellation ES model: ST2000NM0011

All disks are 3.5" 2TB SATA 7200rpm with 64MB of cache, all but the WD are 6Gb/s SATA. The WD is 3Gb/s – not that this really matters, as I have yet to see a spinning disk of this grade exceed 2Gb/s.
Both enterprise disks cost about $190, so about 90% more (almost double the price) than the Hitachi. Are they worth the extra money?

The test

I ended up using SysBench to compare the drives. I had all 3 drives connected to the motherboard of the same machine, a dual L5630 with 96GB of RAM, running Linux 2.6.32. Drives and OS were using their default config, except the "deadline" IO scheduler was in effect (whereas vanilla Linux uses CFQ by default since 2.6.18). SysBench used O_DIRECT for all its accesses. Each disk was formatted with ext4 – no partition table, the whole disk was used directly. Default formatting and mount options were used. SysBench was told to use 64 files, for a total of 100GB of data. Every single test was repeated 4 times and then averages were plotted. Running all the tests takes over 20h.
SysBench produces some kind of a free-form output which isn't very easy to use. So I wrote a Python script to parse the results and a bit of JavaScript to visualize them. The code is available on GitHub: tsuna/sysbench-tools.

Results

A picture is worth a thousand words, so take a look at the graphs. Overall the WD RE4 is a clear winner for me, as it outperforms its 2 buddies on all tests involving random accesses. The Seagate doesn't seem worth the money. Although it's the best at sequential reads, the Hitachi is pretty much on par with it while being almost twice cheaper.
So I'll buy the Hitachi 7K3000 for everything, and pay the extra premium for the WD RE4 for MySQL servers, because MySQL isn't a cheap bastard and needs every drop of performance it can get out of the IO subsystem. No, I don't want to buy ridiculously expensive and power-hungry 15k RPM SAS drives, thank you.
The raw outputs of SysBench are available here: http://tsunanet.net/~tsuna/benchmarks/7K3000-RE4-ConstellationES

Formatting XFS for optimal performance on RAID10

2011-08-20T00:20:00.006+02:00

XFS has terribly bad performance out of the box, especially on large RAID arrays. Unlike ext4, the filesystem needs to be formatted with the right parameters to perform well. If you don't get the parameters right, you need to reformat the filesystem as they can't be changed later.

The 3 main parameters are:

agcount: Number of allocation groups
sunit: Stripe size (as configured on your RAID controller)
swidth: Stripe width (number of data disks, excluding parity / spare disks)

Let's take an example: you have 12 disks configured in a RAID 10 (so 6 pairs of disks in RAID 1, and RAID 0 across the 6 pairs). Let's assume the RAID controller was instructed to use a stripe size of 256k. Then we have:

sunit = 256k / 512 = 512, because sunit is in multiple of 512 byte sectors
swidth = 6 * 512 = 3072, because in a RAID10 with 12 disks we have 6 data disks excluding parity disks (and no hot spares in this case)

Now XFS internally split the filesystem into "allocation groups" (AG). Essentially an AG is like a filesystem on its own. XFS splits the filesystem into multiple AGs in order to help increase parallelism, because each AG has its own set of locks. My rule of thumb is to create as many AGs as you have hardware threads. So if you have a dual-CPU configuration, with 4 cores with HyperThreading, then you have 2 x 4 x 2 = 16 hardware threads, so you should create 16 AGs.

$ sudo mkfs.xfs -f -d sunit=512,swidth=$((512*6)),agcount=16 /dev/sdb
Warning: AG size is a multiple of stripe width.  This can cause performance
problems by aligning all AGs on the same disk.  To avoid this, run mkfs with
an AG size that is one stripe unit smaller, for example 182845376.
meta-data=/dev/sdb               isize=256    agcount=16, agsize=182845440 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2925527040, imaxpct=5
         =                       sunit=64     swidth=384 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Now from the output above, we can see 2 problems:

There's this warning message we better pay attention to.
The values of sunit and swidth printed don't correspond to what we asked for.

The reason the values printed don't match what we wanted is because they're in multiples of "block size". We can see that bsize=4096, so sure enough the numbers match up: 4096 x 64 = 512 x 512 = our stripe size of 256k.

Now let's look at this warning message. It suggests us to use agsize=182845376 instead of agsize=182845440. When we specified the number of AGs we wanted, XFS automatically figured the size of each AG, but then it's complaining that this size is suboptimal. Yay. Now agsize is specified in blocks (so multiples of 4096), but the command line tool expects the value in bytes. At this point you're probably thinking like me: "you must be kidding me, right? Some options are in bytes, some in sectors, some in blocks?!" Yes.

So to make it all work:

$ sudo mkfs.xfs -f -d sunit=512,swidth=$((512*6)),agsize=$((182845376*4096)) /dev/sdb
meta-data=/dev/sdb               isize=256    agcount=16, agsize=182845376 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2925526016, imaxpct=5
         =                       sunit=64     swidth=384 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

It's critical that you get this right before you start using the filesystem. There's no way to change them later. You might be tempted to try using mount -o remount,sunit=X,swidth=Y, and the command will succeed but do nothing. The only XFS parameter you can change at runtime is nobarrier (see the source code of XFS's remount support in the Linux kernel), which you should use if you have a battery-backup unit (BBU) on your RAID card, although the performance boost seems pretty small on DB-type workloads, even with 512MB of RAM on the controller.

Next post: how much of a performance difference is there when you give XFS the right sunit/swidth parameters, and does this allow XFS to beat ext4's performance.

e1000e scales a lot better than bnx2

2011-08-16T01:36:00.005+02:00

At StumbleUpon we've had a never ending string of problems with Broadcom's cards that use the bnx2 driver. The machine cannot handle more than 100kpps (packets/s), the driver has bugs that will lock up the NIC until it gets reset manually when you use jumbo frames and/or TSO (TCP Segmentation Offloading).

So we switched everything to Intel NICs. Not only they don't have these nasty bugs, but also they scale better. They can do up to 170kpps each way before they start discarding packets. Graphs courtesy of OpenTSDB:

Packets/s vs. packets dropped/s

Packets/s vs. interrupts/s

We can also see how the NIC is doing interrupt coalescing at high packet rates. Yay.
Kernel tested: 2.6.32-31-server x86_64 from Lucid, running on 2 L5630 with 48GB of RAM.

VM warning: GC locker is held; pre-dump GC was skipped

2011-07-28T09:22:00.002+02:00

If you ever run into this message while using the Sun JVM / OpenJDK:

Java HotSpot(TM) 64-Bit Server VM warning: GC locker is held; pre-dump GC was skipped

then I wouldn't worry too much about it as it seems like it's printed when running a jmap -histo:live while the GC is already running or holding a certain lock in the jVM.

Clarifications on Linux's NUMA stats

2011-06-03T21:52:00.003+02:00

After reading the excellent post on The MySQL “swap insanity” problem and the effects of the NUMA architecture, I remembered about the existence of /sys/devices/system/node/node*/numastat and decided to add these numbers to a collector for OpenTSDB. But whenever I add a collector that reads metrics from /proc or /sys, I always need to go read the Linux kernel's source code, because most metrics tend to be misleading and under-documented (when they're documented at all).

In this case, if you RTFM, you'll get this:

Numa policy hit/miss statistics

/sys/devices/system/node/node*/numastat

All units are pages. Hugepages have separate counters.

numa_hit      A process wanted to allocate memory from this node, and succeeded.
numa_miss     A process wanted to allocate memory from another node, but ended up with memory from this node.
numa_foreign  A process wanted to allocate on this node, but ended up with memory from another one.
local_node    A process ran on this node and got memory from it.
other_node    A process ran on this node and got memory from another node.
interleave_hit  Interleaving wanted to allocate from this node and succeeded.

I was very confused about the last one, about the exact difference between the second and the third one, and about the difference between the first 3 metrics and the next 2.

After RTFSC, the relevant part of the code appeared to be in mm/vmstat.c:

void zone_statistics(struct zone *preferred_zone, struct zone *z, gfp_t flags)
{       
        if (z->zone_pgdat == preferred_zone->zone_pgdat) {
                __inc_zone_state(z, NUMA_HIT);
        } else {
                __inc_zone_state(z, NUMA_MISS);
                __inc_zone_state(preferred_zone, NUMA_FOREIGN);
        }
        if (z->node == ((flags & __GFP_OTHER_NODE) ?
                        preferred_zone->node : numa_node_id()))
                __inc_zone_state(z, NUMA_LOCAL);
        else
                __inc_zone_state(z, NUMA_OTHER);
}

So here's what it all really means:

numa_hit: Number of pages allocated from the node the process wanted.
numa_miss: Number of pages allocated from this node, but the process preferred another node.
numa_foreign: Number of pages allocated another node, but the process preferred this node.
local_node: Number of pages allocated from this node while the process was running locally.
other_node: Number of pages allocated from this node while the process was running remotely (on another node).
interleave_hit: Number of pages allocated successfully with the interleave strategy.

I was originally confused about numa_foreign but this metric can actually be useful to see what happens when a node runs out of free pages. If a process attempts to get a page from its local node, but this node is out of free pages, then the numa_miss of that node will be incremented (indicating that the node is out of memory) and another node will accomodate the process's request. But in order to know which nodes are "lending memory" to the out-of-memory node, you need to look at numa_foreign. Having a high value for numa_foreign for a particular node indicates that this node's memory is under-utilized so the node is frequently accommodating memory allocation requests that failed on other nodes.

JVM u24 segfault in clearerr on Jaunty

2011-05-08T07:12:00.004+02:00

At StumbleUpon we've been tracking down a weird problem with one of our application servers written in Java. We run Sun's jdk1.6.0_24 on Ubuntu Jaunty (9.04 – yes, these servers are old and due for an upgrade) and this application seems to do something that causes the JVM to segfault:

[6972247.491417] hbase_regionser[32760]: segfault at 8 ip 00007f26cabd608b sp 00007fffb0798270 error 4 in libc-2.9.so[7f26cab66000+168000]
[6972799.682147] hbase_regionser[30904]: segfault at 8 ip 00007f8878fb608b sp 00007fff09b69900 error 4 in libc-2.9.so[7f8878f46000+168000]

What's odd is that the problem always happens on different hosts, and almost always around 6:30 - 6:40 am. Go figure.

Understanding segfault messages from the Linux kernel

Let's try to make sense of the messages shown above, logged by the Linux kernel. Back in Linux v2.6.28, it was logged by do_page_fault, but since then this big function has been refactored into multiple smaller functions, so look for show_signal_msg now.

 791                         printk(
 792                         "%s%s[%d]: segfault at %lx ip %p sp %p error %lx",
 793                         task_pid_nr(tsk) > 1 ? KERN_INFO : KERN_EMERG,
 794                         tsk->comm, task_pid_nr(tsk), address,
 795                         (void *) regs->ip, (void *) regs->sp, error_code);
 796                         print_vma_addr(" in ", regs->ip);

From the above, we see that segfault at 8 means that the code attempted to access the address "8", which is what caused the segfault (because there is no page ever mapped at address 0 (normally)). ip stands for instruction pointer, so the code that triggered the segfault was mapped at the address 0x00007f8878fb608b. sp is stack pointer and isn't very relevant here. error 4 means that this was a read access (4 = PF_USER, which used to be a #define but is now part of enum x86_pf_error_code). The rest of the message tells us that the address of the instruction pointer falls inside the memory region mapped for the code of the libc, and it tells us in square brackets that the libc is mapped at the base address 0x7f8878f46000 and that there's 168000 bytes of code mapped. So that means that we were at 0x00007f8878fb608b - 0x7f8878f46000 = 0x7008b into the libc when the segfault occurred.

So where did the segfault occur exactly?

Since now we know what offset into the libc we were while the segfault happened, we can fire gdb and see what's up with that code:

$ gdb -q /lib/libc.so.6
(no debugging symbols found)
(gdb) x/i 0x7008b
0x7008b <clearerr+27>: cmp    %r8,0x8(%r10)

Interesting... So the JVM is segfaulting in clearerr. We're 27 bytes into this function when the segfault happens. Let's see what the function does up to here:

(gdb) disas clearerr
Dump of assembler code for function clearerr:
0x0000000000070070 <clearerr+0>:        push   %rbx
0x0000000000070071 <clearerr+1>:        mov    (%rdi),%eax
0x0000000000070073 <clearerr+3>:        mov    %rdi,%rbx
0x0000000000070076 <clearerr+6>:        test   %ax,%ax
0x0000000000070079 <clearerr+9>:        js     0x700c7 <clearerr+87>
0x000000000007007b <clearerr+11>:       mov    0x88(%rdi),%r10
0x0000000000070082 <clearerr+18>:       mov    %fs:0x10,%r8
0x000000000007008b <clearerr+27>:       cmp    %r8,0x8(%r10)
0x000000000007008f <clearerr+31>:       je     0x700c0 <clearerr+80>
0x0000000000070091 <clearerr+33>:       xor    %edx,%edx
0x0000000000070093 <clearerr+35>:       mov    $0x1,%esi
0x0000000000070098 <clearerr+40>:       mov    %edx,%eax
0x000000000007009a <clearerr+42>:       cmpl   $0x0,0x300fa7(%rip)        # 0x371048
0x00000000000700a1 <clearerr+49>:       je     0x700ac <clearerr+60>
0x00000000000700a3 <clearerr+51>:       lock cmpxchg %esi,(%r10)
[...]

Reminder: the prototype of the function is void clearerr(FILE *stream); so there's one pointer argument and no return value. The code above starts by saving rbx (because it's the callee's responsibility to save this register), then dereferences the first (and only) argument (passed in rdi) and saves the dereferenced address in eax. Then it copies the pointer passed in argument in rbx. It then tests whether low 16 bits in eax are negative and jumps over some code if it is, because they contain the _flags field of the FILE* passed in argument. At this point it helps to know what a FILE looks like. This structure is opaque so it depends on the libc implementation. In this case, it's the glibc's:

 271 struct _IO_FILE {
 272   int _flags;           /* High-order word is _IO_MAGIC; rest is flags. */
[...]
 310   _IO_lock_t *_lock;
 311 #ifdef _IO_USE_OLD_IO_FILE
 312 };

Then it's looking 0x88 = 136 bytes into the FILE* passed in argument and storing this in r10. If you look at the definition of FILE* and add up the offsets, 136 bytes into the FILE* you'll find the _IO_lock_t *_lock; member of the struct, the mutex that protects this FILE*. Then we're loading address 0x10 from the FS segment in r8. On Linux x86_64, the F segment is used for thread-local data. In this case it's loading a pointer to a structure that corresponds to the local thread. Finally, we're comparing r8 to the value 8 bytes into the value pointed to by r10, and kaboom, we get a segfault. This suggest that r10 is a NULL pointer, meaning that the _lock of the FILE* given in argument is NULL. Now that's weird. I'm not sure how this happened. So the assembly code above is essentially doing:

void clearerr(FILE *stream) {
  if (stream->_flags & 0xFFFF >= 0) {
    struct pthread* self = /* mov %fs:0x10,%r8 -- (can't express this in C, but you can use arch_prctl) */;
    struct lock_t lock = *stream->_lock;
    if (lock.owner != self) {  // We segfault here, when doing lock->owner
      mutex_lock(lock.lock);
      lock.owner = self;
    }
    // ...
  }
  // ...
}

What's odd is that the return value of the JVM is 143 (128+SIGTERM) and not 139 (=128+SIGSEGV). Maybe it's because the JVM is always catching and handling SIGSEGV (they do this to allow the JIT to optimize away some NULL-pointer checks and translate them into NullPointerExceptions, among other things). But even then, normally the JVM will write a file where it complains about the segfault, asks you to file a bug, and dumps all the registers and whatnot... We should see that file somewhere. Yet it's nowhere to be found in the JVM's current working directory or anywhere else I looked.

So this segfault remains a mystery so far. Next step: run the application server with ulimit -c unlimited and analyze a core dump.

The "Out of socket memory" error

2011-03-15T04:49:00.012+01:00

I recently did some work on some of our frontend machines (on which we run Varnish) at StumbleUpon and decided to track down some of the errors the Linux kernel was regularly throwing in kern.log such as:

Feb 25 08:23:42 foo kernel: [3077014.450011] Out of socket memory

Before we get started, let me tell you that you should NOT listen to any blog or forum post without doing your homework, especially when the post recommends that you tune up virtually every TCP related knob in the kernel. These people don't know what they're doing and most probably don't understand much to TCP/IP. Most importantly, their voodoo won't help you fix your problem and might actually make it worse.

Dive in the Linux kernel

In order to best understand what's going on, the best thing is to go read the code of the kernel. Unfortunately, the kernel's error messages or counters are often imprecise, confusing, or even misleading. But they're important. And reading the kernel's code isn't nearly as hard as what people say.

The "Out of socket memory" error

The only match for "Out of socket memory" in the kernel's code (as of v2.6.38) is in net/ipv4/tcp_timer.c:

 66 static int tcp_out_of_resources(struct sock *sk, int do_reset)
 67 {
 68         struct tcp_sock *tp = tcp_sk(sk);
 69         int shift = 0;
 70 
 71         /* If peer does not open window for long time, or did not transmit
 72          * anything for long time, penalize it. */
 73         if ((s32)(tcp_time_stamp - tp->lsndtime) > 2*TCP_RTO_MAX || !do_reset)
 74                 shift++;
 75 
 76         /* If some dubious ICMP arrived, penalize even more. */
 77         if (sk->sk_err_soft)
 78                 shift++;
 79 
 80         if (tcp_too_many_orphans(sk, shift)) {
 81                 if (net_ratelimit())
 82                         printk(KERN_INFO "Out of socket memory\n");

So the question is: when does tcp_too_many_orphans return true? Let's take a look in include/net/tcp.h:

 268 static inline bool tcp_too_many_orphans(struct sock *sk, int shift)
 269 {
 270         struct percpu_counter *ocp = sk->sk_prot->orphan_count;
 271         int orphans = percpu_counter_read_positive(ocp);
 272 
 273         if (orphans << shift > sysctl_tcp_max_orphans) {
 274                 orphans = percpu_counter_sum_positive(ocp);
 275                 if (orphans << shift > sysctl_tcp_max_orphans)
 276                         return true;
 277         }
 278 
 279         if (sk->sk_wmem_queued > SOCK_MIN_SNDBUF &&
 280             atomic_long_read(&tcp_memory_allocated) > sysctl_tcp_mem[2])
 281                 return true;
 282         return false;
 283 }

So two conditions that can trigger this "Out of socket memory" error:

There are "too many" orphan sockets (most common).
The socket already has the minimum amount of memory and we can't give it more because TCP is already using more than its limit.

In order to remedy to your problem, you need to figure out which case you fall into. The vast majority of the people (especially those dealing with frontend servers like Varnish) fall into case 1.

Are you running out of TCP memory?

Ruling out case 2 is easy. All you need is to see how much memory your kernel is configured to give to TCP vs how much is actually being used. If you're close to the limit (uncommon), then you're in case 2. Otherwise (most common) you're in case 1. The kernel keeps track of the memory allocated to TCP in multiple of pages, not in bytes. This is a first bit of confusion that a lot of people run into because some settings are in bytes and other are in pages (and most of the time 1 page = 4096 bytes).

Rule out case 2: find how much memory the kernel is willing to give to TCP:

$ cat /proc/sys/net/ipv4/tcp_mem
3093984 4125312 6187968

The values are in number of pages. They get automatically sized at boot time (values above are for a machine with 32GB of RAM). They mean:

When TCP uses less than 3093984 pages (11.8GB), the kernel will consider it below the "low threshold" and won't bother TCP about its memory consumption.
When TCP uses more than 4125312 pages (15.7GB), enter the "memory pressure" mode.
The maximum number of pages the kernel is willing to give to TCP is 6187968 (23.6GB). When we go above this, we'll start seeing the "Out of socket memory" error and Bad Things will happen.

Now let's find how much of that memory TCP actually uses.

$ cat /proc/net/sockstat
sockets: used 14565
TCP: inuse 35938 orphan 21564 tw 70529 alloc 35942 mem 1894
UDP: inuse 11 mem 3
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

The last value on the second line (mem 1894) is the number of pages allocated to TCP. In this case we can see that 1894 is way below 6187968, so there's no way we can possibly be running out of TCP memory. So in this case, the "Out of socket memory" error was caused by the number of orphan sockets.

Do you have "too many" orphan sockets?

First of all: what's an orphan socket? It's simply a socket that isn't associated to a file descriptor. For instance, after you close() a socket, you no longer hold a file descriptor to reference it, but it still exists because the kernel has to keep it around for a bit more until TCP is done with it. Because orphan sockets aren't very useful to applications (since applications can't interact with them), the kernel is trying to limit the amount of memory consumed by orphans, and it does so by limiting the number of orphans that stick around. If you're running a frontend web server (or an HTTP load balancer), then you'll most likely have a sizeable number of orphans, and that's perfectly normal.

In order to find the limit on the number of orphan sockets, simply do:

$ cat /proc/sys/net/ipv4/tcp_max_orphans
65536

Here we see the default value, which is 64k. In order to find the number of orphan sockets in the system, look again in sockstat:

$ cat /proc/net/sockstat
sockets: used 14565
TCP: inuse 35938 orphan 21564 tw 70529 alloc 35942 mem 1894
[...]

So in this case we have 21564 orphans. That doesn't seem very close to 65536... Yet, if you look once more at the code above that prints the warning, you'll see that there is this shift variable that has a value between 0 and 2, and that the check is testing if (orphans << shift > sysctl_tcp_max_orphans). What this means is that in certain cases, the kernel decides to penalize some sockets more, and it does so by multiplying the number of orphans by 2x or 4x to artificially increase the "score" of the "bad socket" to penalize. The problem is that due to the way this is implemented, you can see a worrisome "Out of socket memory" error when in fact you're still 4x below the limit and you just had a couple "bad sockets" (which happens frequently when you have an Internet facing service). So unfortunately that means that you need to tune up the maximum number of orphan sockets even if you're 2x or 4x away from the threshold. What value is reasonable for you depends on your situation at hand. Observe how the count of orphans in /proc/net/sockstat is changing when your server is at peak traffic, multiply that value by 4, round it up a bit to have a nice value, and set it. You can set it by doing a echo of the new value in /proc/sys/net/ipv4/tcp_max_orphans, and don't forget to update the value of net.ipv4.tcp_max_orphans in /etc/sysctl.conf so that your change persists across reboots.

That's all you need to get rid of these "Out of socket memory" errors, most of which are "false alarms" due to the shift variable of the implementation.

Java IO: slowest readLine ever

2010-12-11T03:30:00.005+01:00

I have a fairly simple problem: I want to count the number of lines in a file, then seek back to after the first line, and then read the file line by line. Easy heh? Not in Java. Enter the utterly retarded world of the JDK.

So if you're n00b, you'll start with a FileInputStream, but quickly you'll realize that seeking around with it isn't really possible... Indeed, the only way to go back to a previous position in the file is to call reset(), which will take you back to the previous location you marked with mark(int). The argument to mark is "the maximum limit of bytes that can be read before the mark position becomes invalid". OK WTF.

If you dig around some more, you'll see that you should really be using a RandomAccessFile – so much for good OO design. The other seemingly cool thing about RandomAccessFile is that it's got a readLine() method. Unfortunately, this method was implemented by a 1st year CS student who probably dropped out before understanding the basics of systems programming.

Believe it or not, but readLine() reads the file one byte at a time. It does one system call to read per byte. As such, it's 2 orders of magnitude slower than it could be... In fact, you can't really implement a readline function that's much slower than that. facepalm.

PS: This is with Sun's JRE version 1.6.0_22-b04. JDK7/OpenJDK has the same implementation. Apache's Harmony implementation is the same, so Android has the same retarded implementation.

OpenTSDB at Strata'11

2010-12-10T04:52:00.004+01:00

I will be speaking about OpenTSDB at the Strata conference, Wednesday, February 02, 2011, in Santa Clara, CA. You can sign up with this promo code and get a 25% discount: str11fsd.
Strata is a new conference about large scale systems put together by O'Reilly.

How long does it take to make a context switch?

2010-11-15T05:53:00.020+01:00

That's a interesting question I'm willing to waste some of my time on. Someone at StumbleUpon emitted the hypothesis that with all the improvements in the Nehalem architecture (marketed as Intel i7), context switching would be much faster. How would you devise a test to empirically find an answer to this question? How expensive are context switches anyway? (tl;dr answer: very expensive)

The lineup

April 21, 2011: update: I added an "extreme" Nehalem and a low-voltage Westmere.
I've put 4 different generations of CPUs to test:

A dual Intel 5150 (Woodcrest, based on the old "Core" architecture, 2.67GHz). The 5150 is a dual-core, and so in total the machine has 4 cores available. Kernel: 2.6.28-19-server x86_64.
A dual Intel E5440 (Harpertown, based on the Penrynn architecture, 2.83GHz). The E5440 is a quad-core so the machine has a total of 8 cores. Kernel: 2.6.24-26-server x86_64.
A dual Intel E5520 (Gainestown, based on the Nehalem architecture, aka i7, 2.27GHz). The E5520 is a quad-core, and has HyperThreading enabled, so the machine has a total of 8 cores or 16 "hardware threads". Kernel: 2.6.28-18-generic x86_64.
A dual Intel X5550 (Gainestown, based on the Nehalem architecture, aka i7, 2.67GHz). The X5550 is a quad-core, and has HyperThreading enabled, so the machine has a total of 8 cores or 16 "hardware threads". Note: the X5550 is in the "server" product line. This CPU is 3x more expensive than the previous one. Kernel: 2.6.28-15-server x86_64.
A dual Intel L5630 (Gulftown, based on the Westmere architecture, aka i7, 2.13GHz). The L5630 is a quad-core, and has HyperThreading enabled, so the machine has a total of 8 cores or 16 "hardware threads". Note: the L5630 is a "low-voltage" CPU. At equal price, this CPU is in theory 16% less powerful than a non-low-voltage CPU. Kernel: 2.6.32-29-server x86_64.

As far as I can say, all CPUs are set to a constant clock rate (no Turbo Boost or anything fancy). All the Linux kernels are those built and distributed by Ubuntu.

First idea: with syscalls (fail)

My first idea was to make a cheap system call many times in a row, time how long it took, and compute the average time spent per syscall. The cheapest system call on Linux these days seems to be gettid. Turns out, this was a naive approach since system calls don't actually cause a full context switch anymore nowadays, the kernel can get away with a "mode switch" (go from user mode to kernel mode, then back to user mode). That's why when I ran my first test program, vmstat wouldn't show a noticeable increase in number of context switches. But this test is interesting too, although it's not what I wanted originally.

Source code: timesyscall.c Results:

Intel 5150: 105ns/syscall
Intel E5440: 87ns/syscall
Intel E5520: 58ns/syscall
Intel X5550: 52ns/syscall
Intel L5630: 58ns/syscall

Now that's nice, more expensive CPUs perform noticeably better. But that's not really what we wanted to know. So to test the cost of a context switch, we need to force the kernel to de-schedule the current process and schedule another one instead. And to benchmark the CPU, we need to get the kernel to do nothing but this in a tight loop. How would you do this?

Second idea: with `futex`

The way I did it was to abuse futex (RTFM). futex is the low level Linux-specific primitive used by most threading libraries to implement blocking operations such as waiting on a contended mutexes, semaphores that run out of permits, condition variables and friends. If you would like to know more, go read Futexes Are Tricky by Ulrich Drepper. Anyways, with a futex, it's easy to suspend and resume processes. What my test does is that it forks off a child process, and the parent and the child take turn waiting on the futex. When the parent waits, the child wakes it up and goes on to wait on the futex, until the parent wakes it and goes on to wait again. Some kind of a ping-pong "I wake you up, you wake me up...".

Source code: timectxsw.c Results:

Intel 5150: ~4300ns/context switch
Intel E5440: ~3600ns/context switch
Intel E5520: ~4500ns/context switch
Intel X5550: ~3000ns/context switch
Intel L5630: ~3000ns/context switch

Note: those results include the overhead of the futex system calls.

Now you must take those results with a grain of salt. The micro-benchmark does nothing but context switching. In practice context switching is expensive because it screws up the CPU caches (L1, L2, L3 if you have one, and the TLB – don't forget the TLB!).

CPU affinity

Things are harder to predict in an SMP environment, because the performance can vary wildly depending on whether a task is migrated from one core to another (especially if the migration is across physical CPUs). I ran the benchmarks again but this time I pinned the processes/threads on a single core (or "hardware thread"). The performance speedup is dramatic.

Source code: cpubench.sh Results:

Intel 5150: ~1900ns/process context switch, ~1700ns/thread context switch
Intel E5440: ~1300ns/process context switch, ~1100ns/thread context switch
Intel E5520: ~1400ns/process context switch, ~1300ns/thread context switch
Intel X5550: ~1300ns/process context switch, ~1100ns/thread context switch
Intel L5630: ~1600ns/process context switch, ~1400ns/thread context switch

Performance boost: 5150: 66%, E5440: 65-70%, E5520: 50-54%, X5550: 55%, L5630: 45%.

The performance gap between thread switches and process switches seems to increase with newer CPU generations (5150: 7-8%, E5440: 5-15%, E5520: 11-20%, X5550: 15%, L5630: 13%). Overall the penalty of switching from one task to another remains very high. Bear in mind that those artificial tests do absolutely zero computation, so they probably have 100% cache hit in L1d and L1i. In the real world, switching between two tasks (threads or processes) typically incurs significantly higher penalties due to cache pollution. But we'll get back to this later.

Threads vs. processes

After producing the numbers above, I quickly criticized Java applications, because it's fairly common to create shitloads of threads in Java, and the cost of context switching becomes high in such applications. Someone retorted that, yes, Java uses lots of threads but threads have become significantly faster and cheaper with the NPTL in Linux 2.6. They said that normally there's no need to do a TLB flush when switching between two threads of the same process. That's true, you can go check the source code of the Linux kernel (switch_mm in mmu_context.h):

static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,
                             struct task_struct *tsk)
{
       unsigned cpu = smp_processor_id();

       if (likely(prev != next)) {
               [...]
               load_cr3(next->pgd);
       } else {
               [don't typically reload cr3]
       }
}

In this code, the kernel expects to be switching between tasks that have different memory structures, in which cases it updates CR3, the register that holds a pointer to the page table. Writing to CR3 automatically causes a TLB flush on x86.

In practice though, with the default kernel scheduler and a busy server-type workload, it's fairly infrequent to go through the code path that skips the call to load_cr3. Plus, different threads tend to have different working sets, so even if you skip this step, you still end up polluting the L1/L2/L3/TLB caches. I re-ran the benchmark above with 2 threads instead of 2 processes (source: timetctxsw.c) but the results aren't significantly different (this varies a lot depending on scheduling and luck, but on average on many runs it's typically only 100ns faster two switch between threads if you don't set a custom CPU affinity).

Indirect costs in context switches: cache pollution

The results above are in line with a paper published a bunch of guys from University of Rochester: Quantifying The Cost of Context Switch. On an unspecified Intel Xeon (the paper was written in 2007, so the CPU was probably not too old), they end up with an average time of 3800ns. They use another method I thought of, which involves writing / reading 1 byte to / from a pipe to block / unblock a couple of processes. I thought that (ab)using futex would be better since futex is essentially exposing some scheduling interface to userland.

The paper goes on to explain the indirect costs involved in context switching, which are due to cache interference. Beyond a certain working set size (about half the size of the L2 cache in their benchmarks), the cost of context switching increases dramatically (by 2 orders of magnitude).

I think this is a more realistic expectation. Not sharing data between threads leads to optimal performance, but it also means that every thread has its own working set and that when a thread is migrated from one core to another (or worse, across physical CPUs), the cache pollution is going to be costly. Unfortunately, when an application has many more active threads than hardware threads, this is happening all the time. That's why not creating more active threads than there are hardware threads available is so important, because in this case it's easier for the Linux scheduler to keep re-scheduling the same threads on the core they last used ("weak affinity").

Having said that, these days, our CPUs have much larger caches, and can even have an L3 cache.

5150: L1i & L1d = 32K each, L2 = 4M
E5440: L1i & L1d = 32K each, L2 = 6M
E5520: L1i & L1d = 32K each, L2 = 256K, L3 = 8M (same for the X5550)
L5630: L1i & L1d = 32K each, L2 = 256K, L3 = 12M

Note that in the case of the E5520/X5550/L5630 (the ones marketed as "i7"), the L2 cache is tiny but there's one L2 cache per core (with HT enabled, this gives us 128K per hardware thread). The L3 cache is shared by the 4 cores that are on each physical CPU.

Having more cores is great, but it also increases the chance that your task be rescheduled onto a different core. The cores have to "migrate" cache lines around, which is expensive. I recommend reading What Every Programmer Should Know About Main Memory by Ulrich Drepper (yes, him again!) to understand more about how this works and the performance penalties involved.

So how does the cost of context switching increases with the size of the working set? This time we'll use another micro-benchmark, timectxswws.c that takes in argument the number of pages to use as a working set. This benchmark is exactly the same as the one used earlier to test the cost of context switching between two processes except that now each process does a memset on the working set, which is shared across both processes. Before starting, the benchmark times how long it takes to write over all the pages in the working set size requested. This time is then discounted from the total time taken by the test. This attempts to estimate the overhead of overwriting pages across context switches.

Here are the results for the 5150:As we can see, the time needed to write a 4K page more than doubles once our working set is bigger than what we can fit in the L1d (32K). The time per context switch keeps going up and up as the working set size increases, but beyond a certain point the benchmark becomes dominated by memory accesses and is no longer actually testing the overhead of a context switch, it's simply testing the performance of the memory subsystem.

Same test, but this time with CPU affinity (both processes pinned on the same core):Oh wow, watch this! It's an order of magnitude faster when pinning both processes on the same core! Because the working set is shared, the working set fits entirely in the 4M L2 cache and cache lines simply need to be transfered from L2 to L1d, instead of being transfered from core to core (potentially across 2 physical CPUs, which is far more expensive than within the same CPU).

Now the results for the i7 processor:Note that this time I covered larger working set sizes, hence the log scale on the X axis.

So yes, context switching on i7 is faster, but only for so long. Real applications (especially Java applications) tend to have large working sets so typically pay the highest price when undergoing a context switch. Other observations about the Nehalem architecture used in the i7:

Going from L1 to L2 is almost unnoticeable. It takes about 130ns to write a page with a working set that fits in L1d (32K) and only 180ns when it fits in L2 (256K). In this respect, the L2 on Nehalem is more of a "L1.5", since its latency is simply not comparable to that of the L2 of previous CPU generations.
As soon as the working set increases beyond 1024K, the time needed to write a page jumps to 750ns. My theory here is that 1024K = 256 pages = half of the TLB of the core, which is shared by the two HyperThreads. Because now both HyperThreads are fighting for TLB entries, the CPU core is constantly doing page table lookups.

Speaking of TLB, the Nehalem has an interesting architecture. Each core has a 64 entry "L1d TLB" (there's no "L1i TLB") and a unified 512 entry "L2TLB". Both are dynamically allocated between both HyperThreads.

Virtualization

I was wondering how much overhead there is when using virtualization. I repeated the benchmarks for the dual E5440, once a normal Linux install, once while running the same install inside VMware ESX Server. The result is that, on average, it's 2.5x to 3x more expensive to do a context switch when using virtualization. My guess is that this is due to the fact that the guest OS can't update the page table itself, so when it attempts to change it, the hypervisor intervenes, which causes an extra 2 context switches (one to get inside the hypervisor, one to get out, back to the guest OS).

This probably explains why Intel added the EPT (Extended Page Table) on the Nehalem, since it enables the guest OS to modify its own page table without help of the hypervisor, and the CPU is able to do the end-to-end memory address translation on its own, entirely in hardware (virtual address to "guest-physical" address to physical address).

Parting words

Context switching is expensive. My rule of thumb is that it'll cost you about 30µs of CPU overhead. This seems to be a good worst-case approximation. Applications that create too many threads that are constantly fighting for CPU time (such as Apache's HTTPd or many Java applications) can waste considerable amounts of CPU cycles just to switch back and forth between different threads. I think the sweet spot for optimal CPU use is to have the same number of worker threads as there are hardware threads, and write code in an asynchronous / non-blocking fashion. Asynchronous code tends to be CPU bound, because anything that would block is simply deferred to later, until the blocking operation completes. This means that threads in asynchronous / non-blocking applications are much more likely to use their full time quantum before the kernel scheduler preempts them. And if there's the same number of runnable threads as there are hardware threads, the kernel is very likely to reschedule threads on the same core, which significantly helps performance.

Another hidden cost that severely impacts server-type workloads is that after being switched out, even if your process becomes runnable, it'll have to wait in the kernel's run queue until a CPU core is available for it. Linux kernels are often compiled with HZ=100, which entails that processes are given time slices of 10ms. If your thread has been switched out but becomes runnable almost immediately, and there are 2 other threads before it in the run queue waiting for CPU time, your thread may have to wait up to 20ms in the worst scenario to get CPU time. So depending on the average length of the run queue (which is reflected in load average), and how long your threads typically run before getting switched out again, this can considerably impact performance.

It is illusory to imagine that NPTL or the Nehalem architecture made context switching cheaper in real-world server-type workloads. Default Linux kernels don't do a good job at keeping CPU affinity, even on idle machines. You must explore alternative schedulers or use taskset or cpuset to control affinity yourself. If you're running multiple different CPU-intensive applications on the same server, manually partitioning cores across applications can help you achieve very significant performance gains.

thread.error: can't start new thread

2010-10-17T04:51:00.003+02:00

Today I was puzzled for a short while by a machine on which Python seemed to be acting up:

$ python
Python 2.5.2 (r252:60911, Jan 20 2010, 23:14:04) 
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
>>> import thread
>>> thread.start_new(lambda: None, ())
Traceback (most recent call last):
 File "", line 1, in 
thread.error: can't start new thread

The machine had 8G of RAM free and was running only about 200 processes (~500 threads total). So, WTF? While running the command above, I straced Python and here's what I saw:

mmap(NULL, 17592186048512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|0x40, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 17592186048512, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
write(2, "Traceback (most recent call last"..., 35) = 35

The second argument to mmap is the amount of memory you want to allocate, and yes that's 16GB. I checked /etc/security/limits.conf and I saw that the problem came from there. Someone put this line:

*       -       stack   17179869184  # 16GB

Now when Python creates a thread, it first allocates* memory for the stack with mmap before calling clone to start the thread. If you're also puzzled by this can't start new thread error, make sure it's not a ulimit problem!

* Of course mmap doesn't actually allocate the memory, it just "reserves" it. Actual memory isn't allocated until you use it, but in this case the OS was refusing the "reservation request" because it was more than the maximum amount of RAM that a process is allowed to have according to other ulimits

IOException: well-known file is not secure

2010-10-05T08:33:00.004+02:00

I implemented a command-line tool to discover and read JMX attributes on Java processes to make it easier to write monitoring scripts for things that, unfortunately, use JMX. The tool uses Sun's Attach API, which is a Sun specific API to attach to a running, local VM, which allows you to load a JMX agent in an existing JVM without requiring that JVM to be started with all the -Djmx flags crap.

It worked like a charm, except when run as root. As root, it would instead fail most of the time (but not always!) with this perfectly intelligible exception:

java.io.IOException: well-known file is not secure
 at sun.tools.attach.LinuxVirtualMachine.checkPermissions(Native Method)
 at sun.tools.attach.LinuxVirtualMachine.(LinuxVirtualMachine.java:93)
 at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:46)
 at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:195)

I Googled that to find it was bug #6649594, but the bug report was completely unhelpful. It mentions a race condition but doesn't explain what the race condition is or how to work around it. Awesome.

So, Google Code Search to the rescue! This exception is thrown by Java_sun_tools_attach_LinuxVirtualMachine_checkPermissions, which is checking that the permissions and mode of a so-called "well-known file" match that of the running JVM. I couldn't trace the code to the location that produces the path, as it's the usual Java-style code with a bazillion layers of abstractions and indirections, but I'm pretty certain this is due to the /tmp/hsperfdata_$USER/$PID files. Sure enough, one of my files was owned by the group root instead of the group of the user (this is because before forking the JVM, I was calling setuid to drop the privileges, but I forgot to call setgid). Fixing the permissions on the file solved the error.

So if you too are scratching your head over this mysterious "well-known file", make sure the permissions on all the /tmp/hsperfdata_$USER/$PID files are consistent.

The jLOL of the day: it's impossible, in Java, to portably find the PID of the current JVM. So when iterating on all the VMs running on the localhost, here's a workaround to detect whether a VirtualMachine instance corresponds to the current JVM. Insert something in the system properties (System.setProperty(..)) and then check whether the VirtualMachine you have at hand has this property. There's a bug filed in — I kid you not — 1999 about this: Bug #4244896 with a whopping 109 votes for it. The bug is in state "Fix Understood", I guess I'm too dumb to understand the fix.

JPMP (Java's Poor Man's Profiler)

2010-08-28T07:45:00.011+02:00

I recently came across PMP and found it surprisingly useful to troubleshoot and analyze live production systems, despite it being so ridiculously simple and naive. Unfortunately for me, I've had to deal with quite a bit of Java lately. So here we go, JPMP:

$ cat jpmp
#!/bin/bash
pid=$1
nsamples=$2
sleeptime=$3

for x in $(seq 1 $nsamples)
  do
    jstack $pid
    sleep $sleeptime
  done | \
awk 'BEGIN { s = "" }
/^"/ { if (s) print s; s = "" }
/^ at / { sub(/\([^)]*\)?$/, "", $2); sub(/^java/, "j", $2);
               if (s) s = s "," $2; else s = $2 }
END { if(s) print s }' | \
sort | uniq -c | sort -rnk1

Output:

$ jpmp 28881 5 0
    120 sun.misc.Unsafe.park,j.util.concurrent.locks.LockSupport.park,j.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await,j.util.concurrent.LinkedBlockingQueue.take,j.util.concurrent.ThreadPoolExecutor.getTask,j.util.concurrent.ThreadPoolExecutor$Worker.run,j.lang.Thread.run
     15 sun.nio.ch.EPollArrayWrapper.epollWait,sun.nio.ch.EPollArrayWrapper.poll,sun.nio.ch.EPollSelectorImpl.doSelect,sun.nio.ch.SelectorImpl.lockAndDoSelect,sun.nio.ch.SelectorImpl.select,org.simpleframework.transport.reactor.ActionDistributor.distribute,org.simpleframework.transport.reactor.ActionDistributor.execute,org.simpleframework.transport.reactor.ActionDistributor.run,j.lang.Thread.run
      5 sun.nio.ch.EPollArrayWrapper.epollWait,sun.nio.ch.EPollArrayWrapper.poll,sun.nio.ch.EPollSelectorImpl.doSelect,sun.nio.ch.SelectorImpl.lockAndDoSelect,sun.nio.ch.SelectorImpl.select,org.apache.zookeeper.ClientCnxn$SendThread.run
      5 sun.misc.Unsafe.park,j.util.concurrent.locks.LockSupport.park,j.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await,j.util.concurrent.LinkedBlockingQueue.take,org.apache.zookeeper.ClientCnxn$EventThread.run
      5 sun.misc.Unsafe.park,j.util.concurrent.locks.LockSupport.park,j.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await,j.util.concurrent.DelayQueue.take,org.simpleframework.util.lease.LeaseCleaner.clean,org.simpleframework.util.lease.LeaseCleaner.run,j.lang.Thread.run
      5 j.lang.Thread.sleep,org.simpleframework.util.buffer.FileManager.run,j.lang.Thread.run
      5 j.lang.Object.wait,j.lang.ref.ReferenceQueue.remove,j.lang.ref.ReferenceQueue.remove,j.lang.ref.Finalizer$FinalizerThread.run
      5 j.lang.Object.wait,j.lang.Object.wait,j.lang.ref.Reference$ReferenceHandler.run

Java NIO and overengineering

2010-08-28T03:23:00.004+02:00

In an earlier post about some of the most horrible APIs in the JDK (the date related APIs), I mentioned how the Java's IO APIs were equally (if not more) retarded. Today I had the joy to dig in the code of Java NIO try to understand why NIO doesn't have "built-in" proxy support like OIO through -D socksProxyHost and -DsocksProxyPort. I wanted to see if I could find a way to work around this limitation, possibly by doing evil things with reflection.

The problems boils down to the fact that with NIO, you use a SocketChannel, and under the hood, some kind of a Socket must be used. Here the fun starts, because as the javadoc says, "The actual work of the socket is performed by an instance of the SocketImpl class", which is itself an abstract class (it's an abstract Impl, yeah, right...).

So let's see how the code goes from a SocketChannel to a Socket or a SocketImpl:

People typically create a SocketChannel like so: SocketChannel.open()
SocketChannel.open() does return SelectorProvider.provider().openSocketChannel();
SelectorProvider.provider() checks to see if the system property java.nio.channels.spi.SelectorProvider is set and a few other things. Unless you specifically do something manually, none of that stuff will end up finding a provider, so the code will end up doing sun.nio.ch.DefaultSelectorProvider.create(); and returning that.
Now here things get a little bit blurry. I think we end up in here (on Linux) doing return new sun.nio.ch.EPollSelectorProvider();
So in step 2., when openSocketChannel() is called on the provider, we end up in here, doing return new SocketChannelImpl(this);
SocketChannelImpl then does all the work for NIO sockets, mostly using sun.nio.ch.Net which is chiefly made of native methods.
If you want to view a SocketChannelImpl as a Socket by calling socket(), the SocketChannelImpl wraps itself in a SocketAdaptor, but the adaptor simply transforms all the calls to calls on the underlying SocketChannelImpl which uses the native functions in sun.nio.ch.Net, so there's nothing really you can do to make it use a SOCKS proxy.

So the bottom line of the story is that there's no way to use a SOCKS proxy with NIO unless you implement the SOCKS proxy protocol yourself (and if you do, make sure you use enough abstract factories implementations and other layers of indirections, so that your code will fit well in the Java IO philosophy).

Also, just for the fun of it, I was curious to see how many objects were involved under the hood of a single SocketChannel. The NIO code is so bloated that I was expecting quite a bit of overhead. I used the following not-very-scientific method (which is kinda similar to PMP) to try to get an answer:

$ cat t.java
import java.net.InetSocketAddress;
import java.nio.channels.SocketChannel;

final class t {
  public static void main(String[]a) throws Exception {
    System.out.println("ready");
    Thread.sleep(5000);
    SocketChannel socket = SocketChannel.open();
    InetSocketAddress addr = new InetSocketAddress("www.google.com", 80);
    socket.connect(addr);
    System.out.println("connected");
    Thread.sleep(5000);
  }
}

then I run this code and use jmap -histo:live right after seeing the "ready" message and right after seeing the "connected" message. In between both of those messages, 2933 objects occupying 371744 bytes of RAM (on my MBP with the JDK 1.6.0_20-b02-279-10M3065 and HotSpot JVM 16.3-b01-279, both shipped by Apple with OS X 10.6.4). I don't know if I'm supposed to laugh or cry.

<methodKlass>                                         +615 +74264
<constMethodKlass>                                    +615 +72416
<constantPoolKlass>                                    +65 +47288
<instanceKlassKlass>                                   +65 +46264
<symbolKlass>                                         +705 +33312
<constantPoolCacheKlass>                               +67 +32024
java.lang.Class                                        +69 +12696
[C                                                     +85 +10664
[[I                                                   +121  +8760
[I                                                     +73  +5928
[S                                                     +96  +5232
[B                                                     +58  +4600
java.lang.String                                       +94  +3760
java.util.LinkedList$Entry                             +65  +2600
<objArrayKlassKlass>                                    +4  +2336
<methodDataKlass>                                       +4  +1800
java.net.URL                                           +15  +1560
java.util.HashMap$Entry                                +29  +1392
[Ljava.util.HashMap$Entry;                              +8  +1344
java.util.LinkedHashMap$Entry                          +15   +960
java.util.LinkedList                                   +23   +920
sun.misc.URLClassPath$JarLoader                         +6   +432
java.util.HashMap                                       +6   +384
java.io.ExpiringCache$Entry                            +11   +352
java.util.jar.JarFile$JarFileEntry                      +3   +336
java.lang.reflect.Constructor                           +2   +240
[Ljava.lang.Object;                                     +2   +208
java.net.Inet4Address                                   +5   +200
sun.nio.ch.SocketChannelImpl                            +1   +176
java.util.LinkedHashMap                                 +2   +160
java.lang.Object                                        +8   +128
java.lang.ThreadLocal                                   +5   +120
java.lang.ClassLoader$NativeLibrary                     +2    +96
[Ljava.net.InetAddress;                                 +2    +88
java.util.ArrayList                                     +2    +80
sun.misc.JarIndex                                       +2    +80
[Ljava.lang.String;                                     +2    +64
java.net.InetAddress$CacheEntry                         +2    +64
java.net.InetAddress$Cache                              +2    +64
java.net.InetAddress$Cache$Type                         +2    +64
sun.misc.URLClassPath                                   +1    +56
java.lang.ref.SoftReference                             +1    +56
[Ljava.lang.ThreadLocal;                                +1    +48
java.util.Stack                                         +1    +40
sun.reflect.NativeConstructorAccessorImpl               +1    +40
java.net.InetSocketAddress                              +1    +40
[Ljava.net.InetAddress$Cache$Type;                      +1    +40
[Ljava.lang.reflect.Constructor;                        +1    +32
java.net.Inet6AddressImpl                               +1    +32
sun.reflect.DelegatingConstructorAccessorImpl           +1    +24
java.util.HashMap$KeySet                                +1    +24
java.nio.channels.spi.AbstractInterruptibleChannel$1    +1    +24
[Ljava.lang.Class;                                      +1    +24
java.net.InetAddress$1                                  +1    +16
sun.net.www.protocol.jar.Handler                        +1    +16
sun.nio.ch.KQueueSelectorProvider                       +1    +16
sun.nio.ch.SocketDispatcher                             +1    +16
java.io.FileDescriptor                                  -1    -24
java.util.Vector                                        -1    -40
java.io.FileInputStream                                 -2    -64
java.util.jar.JarFile                                   -1    -80
java.util.zip.ZStreamRef                                -4    -96
java.util.zip.Inflater                                  -4   -192
java.util.zip.ZipFile$ZipFileInputStream               -12   -672
java.lang.ref.Finalizer                                -17  -1088

Er.. OK I guess that wasn't entirely fair, since 2286 objects (taking 328928 bytes of RAM) have been created by the VM for classes that have been loaded and compiled. So if we discount those, we're still left with 647 objects taking 42816 bytes of RAM... Ahem... Just for a little SocketChannel.

PS: I'm half-ashamed but to produce the output above, I used what could possibly be the most horrible "one-liner" I ever wrote in Python. I put the output of jmap in /tmp/a for the first run and /tmp/b for the second run:

python -c 'load = lambda path: dict((klass, (int(instances), int(bytes))) for (num, instances, bytes, klass) in (line.split() for line in open(path) if line[4] == ":")); a = load("/tmp/a"); b = load("/tmp/b"); print "\n".join("%s\033[55G%+4d\t%+6d" % (klass, diffinst, diffbytes) for (klass, diffinst, diffbytes) in sorted(((klass, b[klass][0] - a.get(klass, (0, 0))[0], b[klass][1] - a.get(klass, (0, 0))[1]) for klass in b), key=lambda i: i[2], reverse=True) if diffbytes != 0).replace(">", "&lt;").replace(">", "&gt;")'

Linux's dentry_cache takes all available memory

2010-06-23T03:48:00.002+02:00

So for some reason one of my servers had 12G of memory allocated to the dentry_cache that wasn't being reclaimed. Probably a bug in the old version of the Linux kernel that this server is using. The solution that worked for me was:

# sync && echo 2 >/proc/sys/vm/drop_caches

If you're curious as to what this does, RTFM.

SimpleDateFormat and "AssertionError: cache control: inconsictency"

2010-06-22T22:50:00.005+02:00

Oh boy, the JDK was really poorly """designed""". Today's fail is:

java.lang.AssertionError: cache control: inconsictency, cachedFixedDate=733763, computed=733763, date=2009-12-22T00:00:00.000Z
        at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2070)
        at java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2472)
        at java.util.Calendar.updateTime(Calendar.java:2468)
        at java.util.Calendar.getTimeInMillis(Calendar.java:1087)
        at java.util.Calendar.getTime(Calendar.java:1060)
        at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1368)
        at java.text.DateFormat.parse(DateFormat.java:335)

(Courtesy of -enablesystemassertions). Because, yeah, SimpleDateFormat is not thread-safe.

Date formats are not synchronized. It is recommended to create separate format instances for each thread. If multiple threads access a format concurrently, it must be synchronized externally.

Like many things in Java, this is just retarded. If SimpleDateFormat was written in a more functional way instead of storing the result of the last call to parse in itself, this problem wouldn't exist.
So yeah, I'm probably a JDK n00b and did not RTFM properly, but this just adds up to the already overflowing list of crap in the JDK.

In the next episode we'll review the OMGWTFBBQ code you have to write to use Java's crappy IO library – and I'm not even talking about the fact that the library exclusively allows 1975-style programming... which is generally the case for everything written in Java anyway.

Also, it's spelled "inconsistency" not "inconsictency", dumbass.

The moral of the story is that if grep 'static .* SimpleDateFormat' on your code returns anything, you have a bug!

Edit 2010-09-27: This can also manifest itself if parsing a date fails randomly with weird error messages such as: ParseException: For input string: "" (even though the string passed in argument was definitely not empty). With such concurrency bugs, a number of weird things can happen...

Go fix the bug yourself, KTHXBYE

2010-06-13T18:42:00.000+02:00

Have you ever wondered why, often, small open-source projects led by a handful of bearded hackers become more successful than large, entreprise-class software? They don't necessarily become more successful in terms of adoption (actually, this rarely happens), but technically speaking, and for those who adopt them, they're much, much, much better.

I'm sure a number of people have already observed how open source communities work, how people interact together, how projects move forward, and so on and so forth, to try to understand what makes the successful ones. But so far I've yet to see a "definitive guide" on "How To Make The Engineers In Your Company As Efficient As In An Awesome Open Source Community".

One of the differences that struck me today is the following. Someone wrote a piece of code that someone else uses. The user complains to the original author "hey, your code is broken, go fix it!".

In an open-source setting, frequently the user will be told "Oh, I see, it's probably something in file foo.bar... hmm, I'm pretty busy right now, how about you fix it and send us the patch? [KTHXBYE]".

And that's fine. Hardly anyone will be shocked by that. Hey, it's an open-source project after all. The author is probably not getting paid for this and doing it on his free time. Now if the user really cares about this bug, and there's no easy workaround, it's actually rather likely that he'll bite the bullet, dive into the code, and fix the bug. And when he does so, it's even more likely that he'll contribute the fix back. Great.

But in a corporate setting, some parameters are different. If the author refuses to fix the bug found, often the user will complain that they're being blocked by the author. It's like, "Hey, it's gonna take you just 10 minutes to fix it, whereas if I fix it myself I'll have to learn X/Y/Z first and it's gonna take me hours. Plus, it's your code.". To the defense of the user, this argument is probably almost true, except it's probably gonna take 30 minutes to the guy to fix the bug and just 1-2 hours for the reporter to learn whatever is required and fix the bug.

Depending on the likelihood that the user to re-use this piece of code where the bug was, getting the user to fix the bug themselves is actually a lot more beneficial, I think. If he does it, he'll know at least a little bit about the project and its implementation. Should another bug crop up, he'll be able to get the fix in place a lot faster since he'll be familiar with the code and won't need to ask anyone else for help (asking for help is the equivalent of a huge cache miss + TLB miss + page fault and incurs a significant performance penalty).

There are other nice side effects of getting the user to fix the code themselves. Maybe, by going through the code, they'll stumble upon a feature they didn't know was there, and they'll re-use it instead of re-implementing something similar. Maybe they'll find that the feature isn't exactly what they need, but it's half-way there, so they'll improve the code, which in turns will benefit everyone else already using that piece of code, or looking for a similar-ish feature, so they won't have to make a third implementation.

Now what I've seen so far is that in practice this frequently doesn't happen in companies. Each individual owns a number of projects, libraries, systems and such. And they're the only one knowing everything inside out about what they own. And when there's a problem, they're asked to fix it because, hey, they're the ones who know how the damn thing works (or doesn't). And they don't want to block their co-workers, do they? Right? Right?! So there's no incentive for others to learn and start sharing the ownership of those systems. And when the person leaves, gets fired, takes some extended time off, no one's left to deal with the problems. So what happens is that, after a while, once everyone is fed up with this goddamn thing that doesn't-work-and-no-one-knows-how-to-fix-it, someone gets assigned to find a durable solution. "Hey, this has been causing a number of problems, others are blocked by it or it's affecting their productivity, you gotta do something about it.", says the manager / project lead. And then what happens is that either the victim starts taking ownership of the system, and we're back with the original problem with exclusive ownership, or the victim rewrites the entire thing and .. still has exclusive ownership over it.

I think this is one of the many many factors that explain why, in a corporate environment, it's so frequent to have lots of "projects" exclusively owned by a single individual and why the projects aren't moving as fast as they could. There's a significant amount of time spent re-inventing the wheel, ditching and rewriting existing stuff, going back and forth with another engineer/team about how the problem wasn't fixed properly, etc. In the open-source world those inefficiencies don't exist so much because engineers maintain different relationships.

It's probably not a coincidence that this is something that happens a lot at Google. I mean, people are encouraged to fix the bugs they find. Every engineer has access to virtually all the code (crazy heh? yes this model works, and it's marvelous) so there are no barriers to prevent you from fixing something you see is wrong. It's so easy, you check out the code, learn what you need to learn about it (hopefully it's slightly better documented than your average corporate code as code reviews are mandatory at Google, and hopefully people complain when they're asked to review undocumented code), you fix the problem, and send a code review to the owner of the code. With a little bit of luck, they'll quickly get back to you with an LGTM (Looks Good To Me), et voilà, your fix/improvement has been submitted and is available to ~10k other engineers. Google is effectively the large corporation that, internally, looks the most like a gigantic open-source community. And the fact that they are where they are today is probably not a coincidence when you mix this with an engineering- & data-driven culture.

Beamer's semiverbatim, spaces and newlines

2010-06-10T04:27:00.002+02:00

If like me, you're puzzled because the semiverbatim environment isn't respecting your spaces and newlines like verbatim does, make sure you haven't forgotten to declare the frame [fragile]!

It took me a little while to realize this.

How to disable Bonjour on OSX 10.6 (Snow Leopard)

2010-02-22T02:32:00.003+01:00

Bonjour is Apple's zeroconf protocol and iTunes (among others) uses it to broadcast the existence of its shared library, which I find annoying because it's quite chatty on the network. In the past shutting down mDNSResponder was enough to keep it quiet but this doesn't work as intended anymore on Snow Leopard as all DNS activity goes through mDNSResponder now.

Apple's KB explains how to properly disable those broadcast advertisements:

sudo vim /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
Add <string>-NoMulticastAdvertisements</string> at the end of the ProgramArguments array.
Save the file
Restart mDNSResponder:
sudo launchctl unload /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
sudo launchctl load /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist

There's no need to reboot unlike what the KB says, as you long as you restart mDNSResponder.

Linux pipe count

2010-02-06T06:57:00.003+01:00

I recently needed to keep track of the number of pipes opened by a server and I was using lsof -nw | fgrep FIFO but this was rather slow.

Here's a better way of counting the number of pipes opened:

# find /proc/[0-9]*/fd -lname 'pipe:*' 2>/dev/null -printf "%l\n" | wc -l
1294
# find /proc/[0-9]*/fd -lname 'pipe:*' 2>/dev/null -printf "%l\n" | sort -u | wc -l
16

So in this case there are 16 different pipes opened on this system and 1294 FDs associated with one or the other end of each pipe. Let's see how many FDs are associated with each pipe:

# find /proc/[0-9]*/fd -lname 'pipe:*' 2>/dev/null -printf "%l\n" | sort | uniq -c
      1 pipe:[10420]
      4 pipe:[1752247694]
      4 pipe:[1752247702]
      2 pipe:[1752247883]
      2 pipe:[234248737]
      1 pipe:[2597396375]
    504 pipe:[4194986548]
    255 pipe:[4194986549]
    252 pipe:[4194986551]
    252 pipe:[4194986552]
      2 pipe:[4213391983]
      1 pipe:[4214019609]
      2 pipe:[480771901]
      8 pipe:[480771902]
      2 pipe:[6087]
      2 pipe:[6288]

So certain pipes are much more popular than others.

Bonus script to find out which pipe is used by which process:

# find /proc/[0-9]*/fd -lname 'pipe:*' -printf "%p/%l\n" 2>/dev/null | python -c 'import os
import sys
pid2cmd = {}
def cmdname(pid):
  cmd = os.path.basename(os.readlink("/proc/%s/exe" % pid))
  pid2cmd[pid] = cmd
  return cmd
pipes = {}
for line in sys.stdin:
  line = line.rstrip().split("/")
  pid, pipe = line[2], line[5]
  cmd = pid2cmd.get(pid) or cmdname(pid)
  pipes.setdefault(pipe, {}).setdefault(cmd, 0)
  pipes[pipe][cmd] += 1
n = 0
for pipe, cmds in sorted(pipes.iteritems()):
  print pipe,
  for cmd, cnt in cmds.iteritems():
    n += int(cnt)
    print "%s=%s" % (cmd, cnt),
  print
print len(pipes), "pipes using", n, "fds"'
pipe:[10420] rpc.statd=1
pipe:[1752247694] dsm_sa_datamgr32d.5.8.0.4961=4
pipe:[1752247702] dsm_sa_datamgr32d.5.8.0.4961=4
pipe:[1752247883] dsm_sa_datamgr32d.5.8.0.4961=2
pipe:[234248737] famd=2
pipe:[2597396375] ntpd=1
pipe:[4194986548] apache2=504
pipe:[4194986549] dash=1 rotatelogs=1 apache2=253
pipe:[4194986551] apache2=252
pipe:[4194986552] apache2=252
pipe:[4214440803] sshd=2
pipe:[4214877266] find=1
pipe:[480771901] master=2
pipe:[480771902] qmgr=2 pickup=2 master=2 tlsmgr=2
pipe:[6087] init=2
pipe:[6288] udevd=2
16 pipes using 1294 fds

So apache2 is the heavy pipe user here. Yay for mpm_prefork</irony>

Using the PUK code to unlock an Android G1

2009-08-01T10:20:00.003+02:00

If you enter your PIN wrong 3 times, your G1 becomes "PUK locked" and supposedly you have to call the customer service to get this famous PUK code (which unique to each SIM card and doesn't normally change, so write it down somewhere). Thing is, it's not obvious at all how to use the code once you have it. You have to hit the "emergency dial" button and "dial" this "number":

**05*<PUK Code>*<enter a new PIN>*<confirm the new PIN>#

Normally, as soon as you enter the #, Android should tell you that you have unblocked your SIM card and it will take a short while to reset the PIN (be patient).

MacPorts failing to upgrade gettext with +with_default_names

2009-04-18T23:22:00.005+02:00

MacPort 1.7.1 breaks itself when trying to upgrade from gettext 0.17_3 to 0.17_4 when coreutils are installed with +with_default_names, here's how to fix it.

First off, the symptom will be:

--->  Deactivating gettext @0.17_3
dyld: Library not loaded: /opt/local/lib/libintl.8.dylib
  Referenced from: /opt/local/bin/rm
  Reason: image not found
Error: Deactivating gettext 0.17_3 failed: 0
Error: Unable to upgrade port: dyld: Library not loaded: /opt/local/lib/libintl.8.dylib
  Referenced from: /opt/local/bin/ln
  Reason: image not found
Error: Unable to upgrade port: dyld: Library not loaded: /opt/local/lib/libintl.8.dylib
  Referenced from: /opt/local/bin/ln
  Reason: image not found
Error: Unable to upgrade port: dyld: Library not loaded: /opt/local/lib/libintl.8.dylib
  Referenced from: /opt/local/bin/ln
  Reason: image not found

A thread on macports-users explains how to fix this:

Edit /opt/local/etc/macports/macports.conf and add this line at the end of the file:
binpath /bin:/sbin:/usr/bin:/usr/sbin:/opt/local/bin:/opt/local/sbin:/usr/X11R6/bin

port deactivate gettext

port install gettext

Remove the line you added in step 1.

Re-run the initial command you were running to resume the upgrade of whatever you were upgrading.

Ryan Schmidt said that he was considering to remove +with_default_names but I disagree with his statement that "it's probably not good to override those default Mac OS X utilities". The GNU coreutils are vastly superior to the BSD ones and what's the point of installing them without +with_default_names? Having to prefix almost every single command with `g' is a waste of time.

When Ryanair says "Your session has been locked"

2009-03-14T23:12:00.002+01:00

For some reason, Ryanair's site screws up every once in a while when booking a flight and it will only show you a page saying "Your session has been locked", "if not redirected in 10 seconds click here" -- but obviously that doesn't work and you're stuck.

In order to start a new session, you need to remove the cookie ASP.NET_SessionId. If you don't know how to do this, here's the step by step with Firefox:

While on the "Your session has been locked" page, right click anywhere in the page and select "View Page Info"

Go to the "Security" tab and click the button "View Cookies"

You'll see a list with a few elements. Click on the line that has ASP.NET_SessionId in the "Cookie Name" column

Click "Remove Cookie"

Alternatively you can try to restart your browser.

More feedback on Android/G1

2009-01-11T15:17:00.007+01:00

Here's a more complete review of Android/G1 vs. the iPhone (after just 4 days using it).

The packaging is a bit cumbersome, and you have to install the battery yourself. Compared to the iPhone which works straight out of its nice packaging, this doesn't contribute to give a good 1st impression.

The global look'n'feel is not as good as the iPhone, although it's definitely orders of magnitudes better than alternatives I've already seen/used (such as Windows Mobile). I think the iPhone is more intuitive / easier to use, generally speaking. It's hard (impossible?) to beat Apple in terms of GUI.

The form factor of the G1 is not well thought. First off the handset is not flat (the bottom side is curved out a little bit). Secondly when you hold it in your right hand, you can't easily reach the buttons at the bottom (it's virtually impossible without relocating the device in your hand, which is very inconvenient). The iPhone has the same problem but to a much lesser extent since you can reach the only button below the screen more easily. The Camera button on the G1 is useless most of the time.

No multi-touch screen. I'm not sure whether this is a hardware limitation of the G1 or a software limitation of Android or yet something else. Also the screen of the G1 is a bit smaller than that of the iPhone. It's also a less responsive. Especially with slide bars (e.g. when playing a YouTube video) which I find rather hard to scroll. EDIT: Someone apparently found a way to have almost iPhone-style multi-touch with no kernel change!

The G1 has no standard headphone plug.

Unlike the iPhone, the G1 seems to be unable to automatically adjust the brightness of the screen depending on the amount of ambient light.

Overall Android is much more responsive than the iPhone OS. Most applications have a lower latency, even (surprise!) network-bound applications such as Maps or YouTube which are far more responsive on the Android.

Unlike the iPhone OS, Android has pretty good support for multi-tasking, background applications and daemons. For instance I was pleased to discover that my handset was notifying me and showing me the Gtalks messages I was receiving as I walked away from my desk. In a single tap I could reply to the messages that were showing up in the status bar.

Some apps have the same caching problems I reported in a previous post about the iPhone. The YouTube application for instance doesn't properly cache videos (if you watch a video and click "Play Again" at the end, it re-downloads the entire video!). However in several cases it was able to resume the video even after I used some other apps in the mean time.

Speaking of YouTube, the search results for "Britney Spears" don't match at all what I get when I search YouTube on my desktop. The search results on the Android are incomplete and have a much lower quality, weird. Also let's note the lack of auto-suggest (which has been enabled by default on YouTube for a while).

I often have problems finding my location on Maps. It just doesn't find me, even when I enable the GPS while outside. As I already noted in my previous post, however, the Maps application is at least one order of magnitude faster on the G1. That's hard to explain given that I use both my G1 and my iPhone on the same wifi network so they should both be able to load map tiles at the same speed. Yet on the iPhone I can see each tile loading slowly one after another, whereas on the Android it's blazzingly fast.

Hooray! The Android has 2-way synchronization over the air! You never need to plug it in a computer, except to recharge the battery (but then you can use a power outlet as well). The integration with Google services (Gmail, Calendar, Gtalk, etc) is just awesome. What I liked about the iPhone is its level of integration with Mac OSX (and iLife), however it's constantly behind (unless you sync it with iTunes very frequently). The Android beats the iPhone hands down here.

The Chrome-based browser in the Android is not nearly has good as MobileSafari.app. No double-tap to easily zoom in on an element of the page, that makes me sad. Past a certain zoom level, some images get heavily pixelized while you scroll (e.g. try on XKCD). The browser is missing the "find text in this page" feature, too bad (but then the iPhone doesn't have it either). Hopefully this should be fixable much more easily given the openness of both Android and Chrome.

The G1 (well, at least my G1) comes with a SD-card of only 1G and virtually no internal storage space. Moreover it seems you can't use the space on the SD card to install apps, so you're heavily limited by the internal storage. Compared to the 16G of my iPhone, that's nothing. I'll have to buy a much larger SD-card if I want to store a reasonable number of songs and movies in my Android.

The Android integrates with all the Google services by asking you to log in to your Google Account, however it does not support to have multiple Google Accounts! Very disappointing. I signed in with my gmail.com address but now I'm unable to easily access things with my google.com account. I hope they'll fix this huge oversight (although I reckon it may be something non-trivial).

The camera on the G1 is a joke. But that of the iPhone is a joke too anyway.

I played Quake on the G1 and I'm amazed by the quality of the 3D graphics. I think the iPhone already has Quake although I haven't tried it, so I can't compare. At least the G1 has a real keyboard, which makes that kind of app much more usable.

Gmote is pretty cool. It doesn't integrate as well with iTunes as Apple's Remote application but it has more cool features, such as the ability to stream the music to your G1 instead of playing it on your Mac. It can also control other apps or control the mouse pointer remotely (handy for presentations!). However you have to install the Gmote server on your Mac, which freaks me out since I know have an untrusted server listening to an open TCP port.

So all in all, I think Android is a pretty good platform and it's on the right track to become a real iPhone killer (although we're not there yet). I doubt Android (or anything else) will ever succeed in having the iPhone's elegance and usability, however it looks like to me that Android is already technically superior than the iPhone OS. And given that Android is an open platform, there's hope that it'll improve much faster than the iPhone.

The G1 handset is disappointing compared to the iPhone. It's not well designed (at least compared to the iPhone) and the touch-screen is not as good as that of the iPhone. I could be wrong but to me it also looks a bit more fragile than the iPhone (especially the "nipple" under the menu button). I'm probably being a bit harsh against the G1 here. Don't mistaken me, it's a great handset, but Apple has set the bar very high with both its handset and its system. The cool thing about it all is that unlike the iPhone OS and the iPhone, Android is not tied to the G1. I'm sure we'll see much superior handsets on the market pretty soon.

1st feedback on Android and the G1

2009-01-08T22:55:00.004+01:00

Some feedback on my shiny new G1 handset Dream-Edition. I've used it for a couple of hours only so I'm utterly unable to answer the question I had in mind ever since I heard about Android: Is it really an iPhone killer?

Anyways, after two hours of using it, I can tell:

It boots pretty darn fast. Much faster than the iPhone. I expect it's gonna get worse since early versions tend to be more lightweight. Let's hope this won't change.

I had troubles joining the open-wireless network at Google. It tried to grab an IP for several minutes and that really frustrated me. Hopefully it eventually got an IP and could become more useful than a brick. It's amazing how these new phones are useless without Internet connectivity. It was very frustrating to see the G1 failing to grab an IP when my iPhone worked just fine.

The Google Maps application kicks ass. It's much much better than that in the iPhone, especially in terms of latency. On the iPhone the map tiles load one by one and very slowly (even on the WiFi with a high-speed Internet link). On my Android it's blazingly fast.

The Alarm Clock application is a joke. That of the iPhone is much more complete and serves as a World Clock, Stopwatch and Timer.

There is already a whole bunch of cool free apps on the phone!

edit: When you hold the G1 in your hand, your thumb begs to have a button on the right side of the handset. Alas, there is no such button and the buttons below the screens are unreachable without changing the way you hold the handset. Too bad. This is really an usability oversight.

My main concern right now is that the G1 is unable to see the SSID of my home WiFi network and I have no explanation for this. It's a standard WEP network and works just fine on my 2 MacBooks and with my iPhone. Yet the G1 doesn't even show its SSID in the list of available networks (it does see the networks of my neighbors however). Very frustrating. Some other Googlers who use the same ADSL-modem-router seem to experience the same issue.

Anyways right now Android looks very promising to me!

BibTeX and url

2008-11-24T18:39:00.003+01:00

If you ever run into this:


(./foo.bbl
./foo.bbl:12: Undefined control sequence.
 \Hy@pstringURI 

l.12 ...refix\url{\url{http://tsunanet.net}}

./foo.bbl:12: Undefined control sequence.
\hyper@linkurl ...Action/S/URI/URI(\Hy@pstringURI 
                                                  )>>}\relax \Hy@colorlink \...
l.12 ...refix\url{\url{http://tsunanet.net}}

./foo.bbl:12: Undefined control sequence.
\Url Error ->\url used in a moving argument. 

l.12 ...refix\url{\url{http://tsunanet.net}}

and, just like me, you go like "WTF, I did \usepackage{url}?!". The answer is to NOT use \url in the URL field... The link will be created automatically anyway.

Fixing the sparsebundle error "Operation not supported on socket".

2008-10-02T20:57:00.003+02:00

If you're trying to mount a .sparsebundle (typically a directory that contains a FileVault $HOME) and get an unexpected Operation not supported on socket error (aka EOPNOTSUPP / ENOTSUP for those who know what errno is) then the fix is simple: fix the permissions of the file within the .sparsebundle file. At least it worked for me. Some of the files were owned by root. So a simple

sudo chown -R $USER my.sparsebundle

did the trick.

Oddly enough I couldn't reproduce this problem by manually trying to chmod some files in the .sparsebundle. But I'm 100% sure that in my case fixing the permissions fixed the problem.

Compile-time assertion to detect signed and unsigned types in C

2008-09-22T17:19:00.003+02:00

I was wondering how to tell whether a given typedef was a signed or unsigned integral type (e.g. unsigned int vs signed int, size_t vs ssize_t, etc.) in C. In C++ it would be easy with a little bit of meta-programing but in C there are far fewer options when it comes to static (aka compile-time) checking. And on this one, Google didn't help me much. So I thought I'd write something, hopefully it'll be helpful to someone searching what I was trying to find.

Here is a simple trick:

#define IS_UNSIGNED_TYPE(Type) \
  char ERROR_ ## Type ## _MUST_BE_UNSIGNED[((Type) -1 < 0) * -1]

Now if you call IS_UNSIGNED_TYPE(ssize_t); GCC will give you a nice error: size of array 'ERROR_ssize_t_MUST_BE_UNSIGNED' is negative, whereas with size_t it compiles fine.

For those who don't understand how this works, it's pretty simple: the two hashes (##) are used to concatenate tokens in the macro. So for instance for the case of ssize_t we have:

  char ERROR_ssize_t_MUST_BE_UNSIGNED[((ssize_t) -1 < 0) * -1]
  char ERROR_ssize_t_MUST_BE_UNSIGNED[(-2147483648 < 0) * -1]
  char ERROR_ssize_t_MUST_BE_UNSIGNED[1 * -1]
  char ERROR_ssize_t_MUST_BE_UNSIGNED[-1]  // Invalid

Whereas for size_t we have:

  char ERROR_size_t_MUST_BE_UNSIGNED[((size_t) -1 < 0) * -1]
  char ERROR_size_t_MUST_BE_UNSIGNED[(4294967295 < 0) * -1]
  char ERROR_size_t_MUST_BE_UNSIGNED[0 * -1]
  char ERROR_size_t_MUST_BE_UNSIGNED[0]  // Valid

Not particularly useful but ... Kind of neat, isn't it?

iPhone/iCal sync woes: data.syncdb getting HUGE

2008-04-03T09:26:00.004+02:00

Following up on my previous post where I explain how to "solve" the problem of uber-slow syncs with the iPhone. This problem seems to be a recurring problem and I suspect that it's caused by remote sync with iCal's calendars, somehow.

Personally, my data.syncdb file (under ~/Library/Application Support/SyncServices/Local) keeps growing, very quickly, even if I never sync my iPhone. This file is a SQLite database so it's fairly easy to see why it's this large. There's an analyzer to produce a report on the database (SQLite's download page - x86 binary direct download).

It's fairly easy to use:

cd /tmp
wget http://www.sqlite.org/sqlite3_analyzer-3.5.4-osx-x86.bin.gz
gunzip sqlite3_analyzer-3.5.4-osx-x86.bin.gz
cd ~/Library/Application\ Support/SyncServices/Local
/tmp/sqlite3_analyzer-3.5.4-osx-x86.bin data.syncdb >/tmp/sqlite.log

Have a look at the log. Mine was showing some really crazy numbers about fragmentation level and number of pages unused. If you also see these, consider cleaning up the db:

# Make sure you kill any SyncServer running
sqlite3 data.syncdb vacuum

I don't know if killing the SyncServer is actually necessary, but I know it uses this SQLite DB. Since I don't know whether SQLite properly locks the tables/rows/whatever that are in use, let's make sure nobody else fiddles with the DB while we vacuum it. Just to stay on the safe side. Update: If the database is effectively in use, you'll see a "SQL error: database is locked", so don't bother unless you see this message, in which case you know you have to kill SyncServer.

Personally, my data.syncdb went from a whopping big 825M down to 195M after vacuuming! I ran the analysis again after vacuuming and here are some of the differences between the reports before and after:

-Size of the file in bytes............. 871493632
-Bytes of user payload stored.......... 96907358    11.1%
+Size of the file in bytes............. 204472320
+Bytes of user payload stored.......... 96907358    47.4%

*** All tables and indices *******************************************

-Bytes of payload...................... 175282561   20.1%
+Bytes of payload...................... 175282561   85.7%

-Fragmentation.........................  69.4%
+Fragmentation.........................   2.1%

-Unused bytes on all pages............. 677992333   77.9%
+Unused bytes on all pages............. 14527468     7.1%

*** All tables *******************************************************

-Bytes of payload...................... 96909201    12.5%
+Bytes of payload...................... 96909201    92.2%

-Fragmentation.........................  66.2%
+Fragmentation.........................   0.74%

-Unused bytes on all pages............. 667043272   86.3%
+Unused bytes on all pages............. 2132671      2.0%

That's just incredible. How could the DB get to such a bad fragmentation level in only ONE WEEK?!

So yeah, I think this is a cleaner workaround that reseting the history of syncs with iSync, but it's still a workaround. I wonder why my DB "leaks" disk space so quickly... Which also seems to incur memory leaks in the SyncServer, because as time passes its memory footprint get bigger and bigger (up to ~300M!).

What to do if sync'ing your iPhone takes forever

2008-03-25T08:15:00.003+01:00

My iPhone was starting to take a really long time to synchronize. Seemingly forever... More than half an hour, and I only had less than 50 songs, less than 50 contacts, and no big calendar. iTunes spent most of its time "Syncing calendars with iPhone" and "Syncing contacts with iPhone". Well, this seems to be a known bug experienced by many others. The workaround is simple. Open iSync, go into preferences and click the button "Reset Sync History...". When you do this make sure your iPhone is not in the middle of a sync. This should also free some 2G occupied by the file Library/Application Support/SyncServices/Local/data.syncdb (under your home directory).

As usual: "no warranty, worked for me, hope this helps".

Some more TODO items for Apple's iPhone dev team.

2008-03-24T16:44:00.007+01:00

Following up on my previous post, here are some other things that I think Apple's iPhone development team should seriously fix. Note that all these critics do not mean that I don't like the iPhone, it just means that I hope that Apple will address the issues of this new immature product line.

Generic landscape mode. Already mentioned in my previous post. This is a serious mis-design issue. Go fix your code.

Total lack of select/copy/cut/paste. I know the iPhone UI doesn't lend itself very easily to these features, but they are critical. You must find a clever way of implementing them, just like you used the clever magnifying glass so we can move the cursor throughout the text easily.

Absolutely zero control on the underlying FS. I know you try to keep control over it, but come on! We need to use the iPhone as a storage device! We want to store documents on them! Right now if I browse the web and find what seems to be an interesting PDF, I can't even save it on my iPhone to transfer it to my MacBook later!

On a related note: I can't save attachments from Mail to my iPhone. Even if you decide you don't want to fix the previous item, you can still make it possible to import pictures attached to mails into the iPhone's photo library, maybe in a different roll though.

Fix your cache use! If I look at the stock or weather, go back to SpringBoard, and look at it again 2min later, it will refresh the data that's only 2min old! For stock refreshing the data after 2min barely makes sense, but if you really want to force an auto-refresh, don't refresh if you know that markets are closed, because this is utterly useless. And you know when markets are closed, because it's mentioned right under the stocks history. Too many apps tend to refresh their data too often due to poor caching policies. You should work to improve all apps on this aspect.

Make it possible to import ICS from the web to Calendar.app, FFS! Or at least sync up with the Google Calendar team to make it possible to use them read/write on the iPhone!

Add a video recorder! How on earth could you leave this out?

Do not resize pictures sent by email when connected on a WiFi network. I guess you resize them to save bandwidth and because EDGE is already slow enough when sending downscaled pictures, but on a WiFi connection there's no reason why you shouldn't send the full resolution one.

Rotate the fucking pictures! Currently the iPhone always stores the JPG data in landscape mode and only mentions the orientation of the picture in EXIF metadata, but if you upload the picture on the web for instance, then the browsers won't render it properly, because many apps still don't use EXIF metadata. Moreover there's no "easy" way to rotate the pictures on OSX, which is a real pain for the average user, I guess. Personally I used jpegtran to rotate the JPEG DCT losslessly but I won't expect normal users to do this.

No chat application. No port of iChat or whatever... For a phone with an always on Internet connection, that's ... weird, to say the least.

No Streetview in Google Maps.

It's not possible to plan a trip on Google Maps with multiple destinations.

(I reckon that the least two items about Google Maps are also Google's fault, I hope we'll try to do something about it despite the fact that a lot of effort is also being invested in Android.)

That's about it for now, I'll add more TODO items as they come to my mind. Please Apple, make the iPhone perfect! Currently it's only great. It ought to be perfect.

Getting synergy to work with a non-rectangular multiple screen setup.

2008-03-15T22:24:00.010+01:00

So I like to work with multiple screens. I really do. These days, my setup at work looks like this one:

Short story: This CANNOT work properly with synergy's current code. Below you'll find a patch (for synergys) and a config file with which everything works as expected. THE PATCH IS NOT GENERIC AND MUST BE ADJUSTED IF YOUR SETUP ISN'T EXACTLY IDENTICAL.

Long story:
I really like to use my Mac, I don't want to bother with another keyboard/mouse to use my Linux workstation. And yes, I'm happy with the track pad of the Mac and don't need an external mouse or anything else other than my Mac (exception being when playing Q3, then I really need a mouse). Of course, synergy is the tool I need. Now it's not that simple. First, let's just have a quick look at part of my xorg.conf:

Section "ServerLayout"
 Identifier "Default Layout"
       Screen          "Main Screen" Absolute 0 150
       Screen          "Secondary Screen" Absolute 1920 0
 InputDevice "Generic Keyboard"
 InputDevice "Configured Mouse"
 Option  "Xinerama" "on"
EndSection

Notice that Xinerama is on so I can have one huge desktop instead of two different window managers that can barely talk together. Notice also that the top-edges of both screens are not aligned.

Now let's try to setup synergys (s stands for "server") on my MacBook Pro and connect the client on the Linux workstation (with the synergyc command). I wrote the following naive configuration file:

section: screens
 workstation:
 macbook:
end
section: links
 workstation:
   down = macbook
 macbook:
   up = workstation
end

Which led me to the following setup:

This setup is uber-b0rken because of the way synergy seems to work. When I leave the screen of the MacBook from the upper edge, most of the time I will end up in the orange zone, where the mouse is actually considered to be on the workstation but isn't actually anywhere. It's just moving in a zone that can't be seen. So I have to move the mouse even further up to eventually see it appear on the 1st screen of the workstation. This is due to the non-rectangular nature of the setup, and due to the fact that synergy doesn't seem to be coded to handle that.

Another problem is that leaving the screen of the MacBook from the right-half-ish of the upper-edge makes the mouse enter the bottom of the 2nd screen of the workstation. This doesn't feel natural at all. Hopefully, in this case, we can adjust the behavior by telling synergy that when I leave the MacBook through the upper-edge, it should only enter the 1st screen. So, since synergy does not even seem to realize that the workstation has two screens, you tell it this by telling it that it should only enter the first 61% of the width of the bottom edge of the workstation "screen" (as seen by synergy, that is, in this case, a 3120x1920 virtual screen).

In a similar fashion, exiting through the bottom edge of the 2nd screen of the workstation isn't natural, so we can setup synergy with a similar behavior. So now the configuration file looks like this one:

section: screens
  workstation:
    xtestIsXineramaUnaware = false
  macbook:
end
section: links
  workstation:
    down(0,60) = macbook
  macbook:
    up = workstation(1,61)
end

Notice that I don't start at 0% when leving the upper edge of the MacBook screen because I activated the top-left hot-corner of my Mac to enable the screen saver and I find it more convenient to just stuff the mouse pointer in there without worrying that pushing it too hard could let it slip through to the other screen. I also noticed that setting xtestIsXineramaUnaware to false helped a little bit (ref: synergy config guide).

It's getting better but it's still unusable. First off, there's this part of the screen where the pointer is just invisible. Admittedly, it's not that bad but it gets really annoying to have to move it around to eventually see where it was hidden. Personally, I can't live with that. The other major problem is that the 2nd screen of the work station has a huge band of 1200x150 unreachable pixels. You CAN'T move the mouse in there.

So, I really searched hard, I've read everything I could find on setting up synergy, but honestly, this setup just can't work with synergy's code. But I really want to use it this way. So I had a look at the code and tried to hack it to suit my needs. I eventually got it to work, and thought it could be useful to others. My patch, however, is NOT generic. It circumvents synergy's misbehavior only for my very specific setup with hard coded constants. I just don't have time to fix synergy properly. But. It could still be useful for others to modify my patch and achieve similar dirty-workarounds.

The patch is here. It also contains an additional workaround to simulate a border. This way, when trying to leave the 2nd screen from the left edge, you can't enter the invisible region. I find this handy.

Rebuild synergy with this patch, and use the binary cmd/synergys/synergys instead of you're distro's. The client (synergyc, on the workstation) does not need any change.

Hope synergy fixes their code. This isn't a trivial fix, it needs to completely change the way synergy parses its config file and handles remote screens. My suggestion for synergy developers (if this cool tool is still being developed) would be to change the config file format so that the user declares the size of the overall bounding box (in my case it's 3120x2250, 1200 + 900 + 150 = 2250) and then position each screen individually in within this huge bounding box and specify which range of pixels (no percentages please!) is mapped to which other edge of which screen on which range of pixels. And even though Xinerama might not help, please do consider each physical screen separately!

Hope this will be useful to others.

Recovering crashed iPhone

2008-03-13T22:41:00.006+01:00

So I was bored with the few apps that come with the iPhone and talked with some people who used ZiPhone to jailbreak it. As expected, it was really easy. ZiPhone will also installs Installer.app, the package manager that will enable you to install lots of third party software.

One of these is OpenSSH (client & server). Obviously, the first thing to do when opening port 22 on this kind of device is to change the default (factory) password (which is alpine on newer iPhones, including firmware 1.1.4). Which is what I did (with passwd).

The problem is that it seems that changing the root password breaks SpringBoard. It will enter an endless loop and will keep crashing forever and showing a message box "edit home screen".

The solution to this problem is simple, you must restore the iPhone with iTunes. The problem for me was that the iPhone didn't show up in iTunes. The solution is to put your iPhone in DFU mode (Device Firmware Upgrade) as follows:

And then let iTunes do the magic for you and you're done! (Done on a 1.1.4, no warranty, hope this helps, happy jailbreaking!)

My first impressions on the iPhone

2008-03-12T21:03:00.009+01:00

So I couldn't resist any longer and eventually bought an iPhone. 16G model of course. I waited for a long time, hoping that Apple would release a new version (with GPR and 3G) but it looks like it's not on their (public) roadmap for anytime soon.

I won't go over the details that everyone already knows (such as how lovely it is, and that "it's been designed and engineered by angles" as I've read on some blog somewhere). It's obviously a great product, like most (if not all) of the Apple products.

But. There are some limitations. So here is what annoys me (in no particular order):

Top issue: AT&T sux balls. Whoever is responsible for locking the iPhone on AT&T deserves to burn in hell forever.

Very limited tab support in Safari. This is a huge overlook from Apple. It greatly diminishes the comfort of browsing the Web from your iPhone.

SMS application does not display the number of characters while you're typing. Generally speaking, the SMS application of the iPhone encourages you to send many SMSes, because SMSes are presented like a chat, so this might be an intentional misfeature (probably requested by the same asshole who chose to lock the iPhone on these crooks called AT&T).

Poor calendar support. Honestly, even the Calendar application builtin Windows Mobile 5 is better. Of course the UI and look'n'feel is better in the iPhone, but feature-wise, it's very limited. You can't have more than one calendar in it (iTunes will squash them all in a single calendar when you import multiple calendars in your iPhone). You can't set any option about your calendar (such as setting a default alert 15 minutes before every event you add). The calendar application can only be sync'ed through iTunes (that's also a limitation in Windows Mobile -- not with iTunes of course -- but given the level of connectivity of the iPhone, I'd expect it to be able to read ICS files from an URL).

No week view in the Calendar application. That really sucks. Why didn't they just make a lightweight port of iCal?!

Geo-positioning on Google maps is too imprecise. This isn't a problem in the iPhone itself, but a shortcoming of the underlying technology (EDGE triangulation). Too bad that the iPhone doesn't have 3G heh :P. Also it's too bad that you can't import your "My Maps" from Google Maps and use them straight in the Maps application of the iPhone.

The Weather application should tell the current time and date of the city you're looking at.

The Calculator should have a more advanced mode with more maths functions.

The style of the Notes are weird and don't stick with the general look'n'feel of the iPhone.

There's no "My TODOs" application. You can write Notes, but if you don't keep consulting them, you forget. I'd love to have a "TODO" application where I can just quickly add stuff and always have them reminded to me. Hopefully this kind of app will be quickly available as a 3rd party app thanks to the SDK release.

From what I've understood, there's a $100 fee to be able to release iPhone apps with the SDK. This is going to severely limit the number of free apps on the iPhone, which is a real shame. (Note: a friend of mine mentioned that this is a one time fee to be able to sign your apps, so it's not that bad, but still...)

Lack of applications. Within the first day, you've seen more or less everything in the iPhone. Once again, let's hope new 3rd party apps will enhance this little gem that the iPhone is.

General lack of configurability. The Settings menus are really basic, there could be much more settings to adapt the iPhone to your personal preferences. I'd love to customize the welcome screen (to get rid of the useless iTunes icon for instance).

Non-generic landscape mode. Only some apps switch to landscape mode (e.g Safari. The iPod mode uses "cover flow" when in landscape mode, etc.). That sucks, there are many other situations where apps would be more convenient in landscape mode. For instance, a week view in the Calendar application would be hard to fit in straight mode, whereas it would surely be usable in landscape mode. I find this utterly ridiculous that app writers have to handle landscape mode specifically. This should be generic for all iPhone apps (and handled specially on a per-need basis).

Sync'ing the iPhone takes waaaaaaay too much time. When I only add a couple of contacts and calendar events, it takes minutes to sync!

Edit: no video recording program! OMG how could I not notice that in the first place?! OMFG how could Apple not include that in the Picture app?!

So yeah, that's about it. Admittedly, some concerns are more important than others (AT&T being the top one -- I hate you AT&T! Even more than I hate Comcast!). Overall, I'm rather satisfied with the iPhone. These concerns are all software-related, so there's hope that Apple addresses them one day or another. I was only (very) disappointed by the various shortcomings of the standard apps. This is probably due to the fact that Apple has set the level high and my expectations were also high. Also, I'm probably a power user, so I might want more out of the iPhone than the average customer (I need an SSH client on it FFS! Edit: got one with ZiPhone, see next post). But yeah, for some reason I was expecting Apple to perform better than that.

git cvs in 5min

2007-07-22T13:51:00.000+02:00

Tired of CVS? Use git instead.

cvs -d cvs-myproj@cvs.foo.com:/cvs co projectwd
mv projectwd myproj && cd myproj
git-cvsimport -v -C ../git-myproj -a
cd ../git-myproj && git status

This will checkout the CVS repo and convert it in git. You will need cvsps >= 2.1.
For some reason, when I've done this, I ended up with all my files marked as deleted in my working copy... If you have the same problem, a simple git reset --hard will undo the deletion.

Now do some work, and commit in your git repo.

It is now time to export these commits in the CVS repo. You can review your commits with git log origin..master (This assumes that you worked in the master branch which is the case by default. The origin branch was created by cvsimport and should not be modified unless when you actually sync it with the CVS repo). Now go back to your CVS working copy:

cd ../myproj
cvs up -d
for i in `GIT_DIR=../git-piscine-j01/.git git-cherry origin work | sed -n 's/^+ //p'`; do echo Exporting $i; GIT_DIR=../git-piscine-j01/.git git-cvsexportcommit -c -p -v $i || break; done
cvs diff -u
git-cvsimport -v -C ../git-myproj -a
cd ../git-myproj
git checkout origin
git merge master
git checkout master

First, make sure your CVS working copy is up to date. Then we ask git-cherry to give us all the commits that are in master and not in origin, and we pass them to git-cvsexportcommit. If the patch applies cleanly, the commit will be automatically sent to the CVS server. It happened to me once that git-cvsexportcommit failed to properly add the new directories I commited in git and I had to checkout another fresh CVS working copy and then it worked. Weird. Then (step 4) review your CVS working copy to make sure nothing was left behind. And then you update your git repository so that it takes into account the new commits you just exported in the CVS repo. Finally, go back to your git repository and enter the origin branch. Sync it with your master branch (step 8) and finally switch back to the master branch where you can do some more work and repeat the whole process.

At first I did not expect to have to do steps 7 and 8 but since the sha1 sums generated by the cvsimport for the commits exported by cvsexportcommit are different, these are required so that branches stay in sync. This extra merge step should however not be of a problem since it should be a simple "fast forward" merge. The only minor inconvenience is that commits made in git, exported to CVS and imported back to git will appear twice with different sha1 IDs.

This should be enough to work with git instead of CVS.

Learning git-svn in 5min

2007-07-18T23:17:00.006+02:00

You are a SVN user and you don't have time to learn new things, here is a 5min course to get started with Git and git-svn.

Import your SVN repository in Git:
git svn clone -s https://svn.foo.com/svn/proj
Make your own Git branch:
git checkout -b work trunk
git add the files you changed.
git commit
Want to sync with the remote master SVN repos?
git svn dcommit

There you go! And guess what, svn-wrapper supports Git!

Some more details now:

The various -s argument is simply here to tell git svn that you use the standard SVN-style layout (trunk/branches/tags).
You must not work in a remote branch. That's why the 2nd step is to setup a local branch where your work will happen. I checked-out trunk but you could also checkout branch "1.0" or whatever.
This is the only notable difference with SVN for basic usage. With subversion, you svn add your files once and then SVN will track them automatically. Git does not track files but content. I know this might sound weird and hard to digest when you start using Git, but that's how Git is and there are many reasons why things are such. So for now, just don't forget that you must git add the files you want to commit. Alternatively, you can use git commit -a to commit all the files. Be warned though that if you do this, git will schedule all the files for the next commit. It will remove all files that disappeared in the mean time too.
Don't forget that the commit will be done in your local repository only.
git-svn will push each commit you made in the remote SVN repository. Each commit will be pushed separately with the log message you gave to Git.
Someone committed in the SVN repository? Fetch the new revisions with git svn fetch

Now you can read the previous post in order to find some useful references that are worth reading. You will quickly see that the time you invest learning Git will pay off (for those of you who are very concerned with ROI). git-svn is ironically the best SVN client IMO.

Some questions you might ask:

Where does git store its stuff?
In the single .git folder at the root of the working copy. Everything is there, you don't have .git folders all over the place like .svn folders.

Where are my branches and stuff?
They are in .git/refs/remotes/, you can easily switch between branches with git checkout.

What about my svn:ignore?
You can import them in git with:
(echo; git-svn show-ignore) >> .git/info/exclude

What about my svn:externals?
Sorry, they are not yet supported by git-svn. But you're not lost! Create another git repository with the svn:external repository and put that repository where it's meant to be and checkout the revision that was pinned in the SVN. Look near the end of .git/svn//unhandled.log, you'll see:
rREVISION1
+dir_prop: trunk svn:externals external_name%20-r%20REVISION2%20URL
This tells you that at REVISION1 in the branch , the svn:externals pinned the revision REVISION2 of URL as external_name.
You need to find the sha1 hash of this revision in git, enter the "external" repository and do:
grep -r rREVISION2 .git
.git/logs/refs/remotes/trunk: [...] rREVISION2
There you go, you can simply issue:
git checkout

Still not convinced by Git?

Git is way faster than SVN
Git is distributed (you can work offline which is a great advantage for laptop users)
Git makes branching and tagging extremely cheap and convenient.
Most important: Git makes merging a trivial yet powerful operation. Subversion is flawed in this respect, merging is a pain, you have to manually track the last revision that was merged, you loose the history, etc. Git does not have all these disadvantages. Merging is done right: fast, easy, reliable. As a bonus, you even get less conflicts.
Git has tons of sexy features that SVN will probably never have and SVN users can only dream about. For instance, Git can instantly tell you where does THIS LINE come from, even if this line moved across different files over years.
Git is safe and reliable. We also use versioning systems because we want to keep a safe backup of all the history of a project. But servers crash, filesystems get corrupted, we all have this kind of problem. Sometimes malicious people try to fiddle with the history on well-known public servers. Git checks every single thing it controls with sha1 sums, there is no way you can screw things up without noticing. This reason is actually good enough that everyone actually carrying about their code should switch to Git right now. With a single 40-bytes sha1 hash, you can make sure that, not only a single revision is OK, but that the entire history, all the files and stuff straight from the beginning until this revision are OK.
Git makes it a lot easier for everyone to contribute. No endless delicate political discussions about commit access, people pull changes from each other, usually from the people they trust. Everyone maintains their own branches and publish only what they want to publish.
Git is highly optimized. The first thing people usually worry about is "OMG, this is gonna take a hell lot of space if I gotta import the entire history on my local hard drive". First off, you don't have to import everything if you're wrapping SVN repositories with Git (see the links in my previous post). Second thing, most of my Git repos are actually smaller than the same working copy in SVN (even though the Git one actually has the entire history it its .git!) I have an example at hand, a 1340 revision SVN working copy freshly checked out. It's 7.6MB. The same working copy in Git but with the entire history of the 1340 revisions (for all branches and tags) is only 6.6MB.

scm_type git = ++svn;

2007-07-15T12:08:00.000+02:00

(Hmm, I need to personnalize this "blog" a lil bit but I'm too lazy)

If you really enjoy SVN, you'll love Git.
I won't go into all the details because there are already plenty of pages explaining how to get started and why Git kicks ass, but here are some useful links (in the suggested order of reading):

I also added (beta) Git-support in SVN-wrapper which is very handy in order to automatically generate template ChangeLog entries.

I strongly encourage anyone working with SCMs on a daily basis to give Git a try, it's really worth it. Especially because Git is very good at wrapping other SCMs (I only wrapped SVN repositories so far, had some troubles with big CVS repositories, most probably because of cvsps).
Anyways, thanks to the KDE people I met at the aKademy (pics) for sorta convincing me to try Git.

PS: QGit is cool.

Test

2007-07-05T09:12:00.000+02:00

This is a test.

Tsuna's blog

Hardware Growler for Mac OS X Lion

ext4 2x faster than XFS?

Hitachi 7K3000 vs WD RE4 vs Seagate Constellation ES

The line up

The test

Results

Formatting XFS for optimal performance on RAID10

e1000e scales a lot better than bnx2

VM warning: GC locker is held; pre-dump GC was skipped

Clarifications on Linux's NUMA stats

JVM u24 segfault in clearerr on Jaunty

Understanding segfault messages from the Linux kernel

So where did the segfault occur exactly?

The "Out of socket memory" error

Dive in the Linux kernel

The "Out of socket memory" error

Are you running out of TCP memory?

Do you have "too many" orphan sockets?

Java IO: slowest readLine ever

OpenTSDB at Strata'11

How long does it take to make a context switch?

The lineup

First idea: with syscalls (fail)

Second idea: with futex

CPU affinity

Threads vs. processes

Indirect costs in context switches: cache pollution

Virtualization

Parting words

thread.error: can't start new thread

IOException: well-known file is not secure

JPMP (Java's Poor Man's Profiler)

Java NIO and overengineering

Linux's dentry_cache takes all available memory

SimpleDateFormat and "AssertionError: cache control: inconsictency"

Go fix the bug yourself, KTHXBYE

Beamer's semiverbatim, spaces and newlines

How to disable Bonjour on OSX 10.6 (Snow Leopard)

Linux pipe count

Using the PUK code to unlock an Android G1

MacPorts failing to upgrade gettext with +with_default_names

When Ryanair says "Your session has been locked"

More feedback on Android/G1

1st feedback on Android and the G1

BibTeX and url

Fixing the sparsebundle error "Operation not supported on socket".

Compile-time assertion to detect signed and unsigned types in C

iPhone/iCal sync woes: data.syncdb getting HUGE

What to do if sync'ing your iPhone takes forever

Some more TODO items for Apple's iPhone dev team.

Getting synergy to work with a non-rectangular multiple screen setup.

Recovering crashed iPhone

My first impressions on the iPhone

git cvs in 5min

Learning git-svn in 5min

scm_type git = ++svn;

Test

Second idea: with `futex`