Linux's dentry_cache takes all available memory

So for some reason one of my servers had 12G of memory allocated to the dentry_cache that wasn't being reclaimed. Probably a bug in the old version of the Linux kernel that this server is using. The solution that worked for me was:

# sync && echo 2 >/proc/sys/vm/drop_caches

If you're curious as to what this does, RTFM.

I like it!

SimpleDateFormat and "AssertionError: cache control: inconsictency"

Oh boy, the JDK was really poorly """designed""". Today's fail is:

java.lang.AssertionError: cache control: inconsictency, cachedFixedDate=733763, computed=733763, date=2009-12-22T00:00:00.000Z
        at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2070)
        at java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2472)
        at java.util.Calendar.updateTime(Calendar.java:2468)
        at java.util.Calendar.getTimeInMillis(Calendar.java:1087)
        at java.util.Calendar.getTime(Calendar.java:1060)
        at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1368)
        at java.text.DateFormat.parse(DateFormat.java:335)

(Courtesy of -enablesystemassertions). Because, yeah, SimpleDateFormat is not thread-safe.

Date formats are not synchronized. It is recommended to create separate format instances for each thread. If multiple threads access a format concurrently, it must be synchronized externally.

Like many things in Java, this is just retarded. If SimpleDateFormat was written in a more functional way instead of storing the result of the last call to parse in itself, this problem wouldn't exist.
So yeah, I'm probably a JDK n00b and did not RTFM properly, but this just adds up to the already overflowing list of crap in the JDK.

In the next episode we'll review the OMGWTFBBQ code you have to write to use Java's crappy IO library – and I'm not even talking about the fact that the library exclusively allows 1975-style programming... which is generally the case for everything written in Java anyway.

Also, it's spelled "inconsistency" not "inconsictency", dumbass.

The moral of the story is that if grep 'static .* SimpleDateFormat' on your code returns anything, you have a bug!

I like it!

Go fix the bug yourself, KTHXBYE

Have you ever wondered why, often, small open-source projects led by a handful of bearded hackers become more successful than large, entreprise-class software? They don't necessarily become more successful in terms of adoption (actually, this rarely happens), but technically speaking, and for those who adopt them, they're much, much, much better.

I'm sure a number of people have already observed how open source communities work, how people interact together, how projects move forward, and so on and so forth, to try to understand what makes the successful ones. But so far I've yet to see a "definitive guide" on "How To Make The Engineers In Your Company As Efficient As In An Awesome Open Source Community".

One of the differences that struck me today is the following. Someone wrote a piece of code that someone else uses. The user complains to the original author "hey, your code is broken, go fix it!".

In an open-source setting, frequently the user will be told "Oh, I see, it's probably something in file foo.bar... hmm, I'm pretty busy right now, how about you fix it and send us the patch? [KTHXBYE]".

And that's fine. Hardly anyone will be shocked by that. Hey, it's an open-source project after all. The author is probably not getting paid for this and doing it on his free time. Now if the user really cares about this bug, and there's no easy workaround, it's actually rather likely that he'll bite the bullet, dive into the code, and fix the bug. And when he does so, it's even more likely that he'll contribute the fix back. Great.

But in a corporate setting, some parameters are different. If the author refuses to fix the bug found, often the user will complain that they're being blocked by the author. It's like, "Hey, it's gonna take you just 10 minutes to fix it, whereas if I fix it myself I'll have to learn X/Y/Z first and it's gonna take me hours. Plus, it's your code.". To the defense of the user, this argument is probably almost true, except it's probably gonna take 30 minutes to the guy to fix the bug and just 1-2 hours for the reporter to learn whatever is required and fix the bug.

Depending on the likelihood that the user to re-use this piece of code where the bug was, getting the user to fix the bug themselves is actually a lot more beneficial, I think. If he does it, he'll know at least a little bit about the project and its implementation. Should another bug crop up, he'll be able to get the fix in place a lot faster since he'll be familiar with the code and won't need to ask anyone else for help (asking for help is the equivalent of a huge cache miss + TLB miss + page fault and incurs a significant performance penalty).

There are other nice side effects of getting the user to fix the code themselves. Maybe, by going through the code, they'll stumble upon a feature they didn't know was there, and they'll re-use it instead of re-implementing something similar. Maybe they'll find that the feature isn't exactly what they need, but it's half-way there, so they'll improve the code, which in turns will benefit everyone else already using that piece of code, or looking for a similar-ish feature, so they won't have to make a third implementation.

Now what I've seen so far is that in practice this frequently doesn't happen in companies. Each individual owns a number of projects, libraries, systems and such. And they're the only one knowing everything inside out about what they own. And when there's a problem, they're asked to fix it because, hey, they're the ones who know how the damn thing works (or doesn't). And they don't want to block their co-workers, do they? Right? Right?! So there's no incentive for others to learn and start sharing the ownership of those systems. And when the person leaves, gets fired, takes some extended time off, no one's left to deal with the problems. So what happens is that, after a while, once everyone is fed up with this goddamn thing that doesn't-work-and-no-one-knows-how-to-fix-it, someone gets assigned to find a durable solution. "Hey, this has been causing a number of problems, others are blocked by it or it's affecting their productivity, you gotta do something about it.", says the manager / project lead. And then what happens is that either the victim starts taking ownership of the system, and we're back with the original problem with exclusive ownership, or the victim rewrites the entire thing and .. still has exclusive ownership over it.

I think this is one of the many many factors that explain why, in a corporate environment, it's so frequent to have lots of "projects" exclusively owned by a single individual and why the projects aren't moving as fast as they could. There's a significant amount of time spent re-inventing the wheel, ditching and rewriting existing stuff, going back and forth with another engineer/team about how the problem wasn't fixed properly, etc. In the open-source world those inefficiencies don't exist so much because engineers maintain different relationships.

It's probably not a coincidence that this is something that happens a lot at Google. I mean, people are encouraged to fix the bugs they find. Every engineer has access to virtually all the code (crazy heh? yes this model works, and it's marvelous) so there are no barriers to prevent you from fixing something you see is wrong. It's so easy, you check out the code, learn what you need to learn about it (hopefully it's slightly better documented than your average corporate code as code reviews are mandatory at Google, and hopefully people complain when they're asked to review undocumented code), you fix the problem, and send a code review to the owner of the code. With a little bit of luck, they'll quickly get back to you with an LGTM (Looks Good To Me), et voilà, your fix/improvement has been submitted and is available to ~10k other engineers. Google is effectively the large corporation that, internally, looks the most like a gigantic open-source community. And the fact that they are where they are today is probably not a coincidence when you mix this with an engineering- & data-driven culture.

I like it!

Beamer's semiverbatim, spaces and newlines

If like me, you're puzzled because the semiverbatim environment isn't respecting your spaces and newlines like verbatim does, make sure you haven't forgotten to declare the frame [fragile]!

It took me a little while to realize this.

I like it!

How to disable Bonjour on OSX 10.6 (Snow Leopard)

Bonjour is Apple's zeroconf protocol and iTunes (among others) uses it to broadcast the existence of its shared library, which I find annoying because it's quite chatty on the network. In the past shutting down mDNSResponder was enough to keep it quiet but this doesn't work as intended anymore on Snow Leopard as all DNS activity goes through mDNSResponder now.

Apple's KB explains how to properly disable those broadcast advertisements:

1. sudo vim /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
2. Add <string>-NoMulticastAdvertisements</string> at the end of the ProgramArguments array.
3. Save the file
4. Restart mDNSResponder:
   sudo launchctl unload /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
sudo launchctl load /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist

There's no need to reboot unlike what the KB says, as you long as you restart mDNSResponder.

I like it!

Linux pipe count

I recently needed to keep track of the number of pipes opened by a server and I was using lsof -nw | fgrep FIFO but this was rather slow.

Here's a better way of counting the number of pipes opened:

# find /proc/[0-9]*/fd -lname 'pipe:*' 2>/dev/null -printf "%l\n" | wc -l
1294
# find /proc/[0-9]*/fd -lname 'pipe:*' 2>/dev/null -printf "%l\n" | sort -u | wc -l
16

So in this case there are 16 different pipes opened on this system and 1294 FDs associated with one or the other end of each pipe. Let's see how many FDs are associated with each pipe:

# find /proc/[0-9]*/fd -lname 'pipe:*' 2>/dev/null -printf "%l\n" | sort | uniq -c
      1 pipe:[10420]
      4 pipe:[1752247694]
      4 pipe:[1752247702]
      2 pipe:[1752247883]
      2 pipe:[234248737]
      1 pipe:[2597396375]
    504 pipe:[4194986548]
    255 pipe:[4194986549]
    252 pipe:[4194986551]
    252 pipe:[4194986552]
      2 pipe:[4213391983]
      1 pipe:[4214019609]
      2 pipe:[480771901]
      8 pipe:[480771902]
      2 pipe:[6087]
      2 pipe:[6288]

So certain pipes are much more popular than others.

Bonus script to find out which pipe is used by which process:

# find /proc/[0-9]*/fd -lname 'pipe:*' -printf "%p/%l\n" 2>/dev/null | python -c 'import os
import sys
pid2cmd = {}
def cmdname(pid):
  cmd = os.path.basename(os.readlink("/proc/%s/exe" % pid))
  pid2cmd[pid] = cmd
  return cmd
pipes = {}
for line in sys.stdin:
  line = line.rstrip().split("/")
  pid, pipe = line[2], line[5]
  cmd = pid2cmd.get(pid) or cmdname(pid)
  pipes.setdefault(pipe, {}).setdefault(cmd, 0)
  pipes[pipe][cmd] += 1
n = 0
for pipe, cmds in sorted(pipes.iteritems()):
  print pipe,
  for cmd, cnt in cmds.iteritems():
    n += int(cnt)
    print "%s=%s" % (cmd, cnt),
  print
print len(pipes), "pipes using", n, "fds"'
pipe:[10420] rpc.statd=1
pipe:[1752247694] dsm_sa_datamgr32d.5.8.0.4961=4
pipe:[1752247702] dsm_sa_datamgr32d.5.8.0.4961=4
pipe:[1752247883] dsm_sa_datamgr32d.5.8.0.4961=2
pipe:[234248737] famd=2
pipe:[2597396375] ntpd=1
pipe:[4194986548] apache2=504
pipe:[4194986549] dash=1 rotatelogs=1 apache2=253
pipe:[4194986551] apache2=252
pipe:[4194986552] apache2=252
pipe:[4214440803] sshd=2
pipe:[4214877266] find=1
pipe:[480771901] master=2
pipe:[480771902] qmgr=2 pickup=2 master=2 tlsmgr=2
pipe:[6087] init=2
pipe:[6288] udevd=2
16 pipes using 1294 fds

So apache2 is the heavy pipe user here. Yay for mpm_prefork</irony>

I like it!

Using the PUK code to unlock an Android G1

If you enter your PIN wrong 3 times, your G1 becomes "PUK locked" and supposedly you have to call the customer service to get this famous PUK code (which unique to each SIM card and doesn't normally change, so write it down somewhere). Thing is, it's not obvious at all how to use the code once you have it. You have to hit the "emergency dial" button and "dial" this "number":

**05*<PUK Code>*<enter a new PIN>*<confirm the new PIN>#

Normally, as soon as you enter the #, Android should tell you that you have unblocked your SIM card and it will take a short while to reset the PIN (be patient).

I like it!

MacPorts failing to upgrade gettext with +with_default_names

MacPort 1.7.1 breaks itself when trying to upgrade from gettext 0.17_3 to 0.17_4 when coreutils are installed with +with_default_names, here's how to fix it.

First off, the symptom will be:

--->  Deactivating gettext @0.17_3
dyld: Library not loaded: /opt/local/lib/libintl.8.dylib
  Referenced from: /opt/local/bin/rm
  Reason: image not found
Error: Deactivating gettext 0.17_3 failed: 0
Error: Unable to upgrade port: dyld: Library not loaded: /opt/local/lib/libintl.8.dylib
  Referenced from: /opt/local/bin/ln
  Reason: image not found
Error: Unable to upgrade port: dyld: Library not loaded: /opt/local/lib/libintl.8.dylib
  Referenced from: /opt/local/bin/ln
  Reason: image not found
Error: Unable to upgrade port: dyld: Library not loaded: /opt/local/lib/libintl.8.dylib
  Referenced from: /opt/local/bin/ln
  Reason: image not found

A thread on macports-users explains how to fix this:

1. Edit /opt/local/etc/macports/macports.conf and add this line at the end of the file:
   binpath /bin:/sbin:/usr/bin:/usr/sbin:/opt/local/bin:/opt/local/sbin:/usr/X11R6/bin


2. port deactivate gettext


3. port install gettext


4. Remove the line you added in step 1.


5. Re-run the initial command you were running to resume the upgrade of whatever you were upgrading.

Ryan Schmidt said that he was considering to remove +with_default_names but I disagree with his statement that "it's probably not good to override those default Mac OS X utilities". The GNU coreutils are vastly superior to the BSD ones and what's the point of installing them without +with_default_names? Having to prefix almost every single command with `g' is a waste of time.

I like it!

When Ryanair says "Your session has been locked"

For some reason, Ryanair's site screws up every once in a while when booking a flight and it will only show you a page saying "Your session has been locked", "if not redirected in 10 seconds click here" -- but obviously that doesn't work and you're stuck.

In order to start a new session, you need to remove the cookie ASP.NET_SessionId. If you don't know how to do this, here's the step by step with Firefox:

1. While on the "Your session has been locked" page, right click anywhere in the page and select "View Page Info"


2. Go to the "Security" tab and click the button "View Cookies"


3. You'll see a list with a few elements. Click on the line that has ASP.NET_SessionId in the "Cookie Name" column


4. Click "Remove Cookie"

Alternatively you can try to restart your browser.

I like it!

More feedback on Android/G1

Here's a more complete review of Android/G1 vs. the iPhone (after just 4 days using it).

• The packaging is a bit cumbersome, and you have to install the battery yourself. Compared to the iPhone which works straight out of its nice packaging, this doesn't contribute to give a good 1st impression.


• The global look'n'feel is not as good as the iPhone, although it's definitely orders of magnitudes better than alternatives I've already seen/used (such as Windows Mobile). I think the iPhone is more intuitive / easier to use, generally speaking. It's hard (impossible?) to beat Apple in terms of GUI.


• The form factor of the G1 is not well thought. First off the handset is not flat (the bottom side is curved out a little bit). Secondly when you hold it in your right hand, you can't easily reach the buttons at the bottom (it's virtually impossible without relocating the device in your hand, which is very inconvenient). The iPhone has the same problem but to a much lesser extent since you can reach the only button below the screen more easily. The Camera button on the G1 is useless most of the time.


• No multi-touch screen. I'm not sure whether this is a hardware limitation of the G1 or a software limitation of Android or yet something else. Also the screen of the G1 is a bit smaller than that of the iPhone. It's also a less responsive. Especially with slide bars (e.g. when playing a YouTube video) which I find rather hard to scroll. EDIT: Someone apparently found a way to have almost iPhone-style multi-touch with no kernel change!
  


• The G1 has no standard headphone plug.


• Unlike the iPhone, the G1 seems to be unable to automatically adjust the brightness of the screen depending on the amount of ambient light.


• Overall Android is much more responsive than the iPhone OS. Most applications have a lower latency, even (surprise!) network-bound applications such as Maps or YouTube which are far more responsive on the Android.


• Unlike the iPhone OS, Android has pretty good support for multi-tasking, background applications and daemons. For instance I was pleased to discover that my handset was notifying me and showing me the Gtalks messages I was receiving as I walked away from my desk. In a single tap I could reply to the messages that were showing up in the status bar.


• Some apps have the same caching problems I reported in a previous post about the iPhone. The YouTube application for instance doesn't properly cache videos (if you watch a video and click "Play Again" at the end, it re-downloads the entire video!). However in several cases it was able to resume the video even after I used some other apps in the mean time.


• Speaking of YouTube, the search results for "Britney Spears" don't match at all what I get when I search YouTube on my desktop. The search results on the Android are incomplete and have a much lower quality, weird. Also let's note the lack of auto-suggest (which has been enabled by default on YouTube for a while).


• I often have problems finding my location on Maps. It just doesn't find me, even when I enable the GPS while outside. As I already noted in my previous post, however, the Maps application is at least one order of magnitude faster on the G1. That's hard to explain given that I use both my G1 and my iPhone on the same wifi network so they should both be able to load map tiles at the same speed. Yet on the iPhone I can see each tile loading slowly one after another, whereas on the Android it's blazzingly fast.


• Hooray! The Android has 2-way synchronization over the air! You never need to plug it in a computer, except to recharge the battery (but then you can use a power outlet as well). The integration with Google services (Gmail, Calendar, Gtalk, etc) is just awesome. What I liked about the iPhone is its level of integration with Mac OSX (and iLife), however it's constantly behind (unless you sync it with iTunes very frequently). The Android beats the iPhone hands down here.


• The Chrome-based browser in the Android is not nearly has good as MobileSafari.app. No double-tap to easily zoom in on an element of the page, that makes me sad. Past a certain zoom level, some images get heavily pixelized while you scroll (e.g. try on XKCD). The browser is missing the "find text in this page" feature, too bad (but then the iPhone doesn't have it either). Hopefully this should be fixable much more easily given the openness of both Android and Chrome.


• The G1 (well, at least my G1) comes with a SD-card of only 1G and virtually no internal storage space. Moreover it seems you can't use the space on the SD card to install apps, so you're heavily limited by the internal storage. Compared to the 16G of my iPhone, that's nothing. I'll have to buy a much larger SD-card if I want to store a reasonable number of songs and movies in my Android.


• The Android integrates with all the Google services by asking you to log in to your Google Account, however it does not support to have multiple Google Accounts! Very disappointing. I signed in with my gmail.com address but now I'm unable to easily access things with my google.com account. I hope they'll fix this huge oversight (although I reckon it may be something non-trivial).


• The camera on the G1 is a joke. But that of the iPhone is a joke too anyway.


• I played Quake on the G1 and I'm amazed by the quality of the 3D graphics. I think the iPhone already has Quake although I haven't tried it, so I can't compare. At least the G1 has a real keyboard, which makes that kind of app much more usable.


• Gmote is pretty cool. It doesn't integrate as well with iTunes as Apple's Remote application but it has more cool features, such as the ability to stream the music to your G1 instead of playing it on your Mac. It can also control other apps or control the mouse pointer remotely (handy for presentations!). However you have to install the Gmote server on your Mac, which freaks me out since I know have an untrusted server listening to an open TCP port.

So all in all, I think Android is a pretty good platform and it's on the right track to become a real iPhone killer (although we're not there yet). I doubt Android (or anything else) will ever succeed in having the iPhone's elegance and usability, however it looks like to me that Android is already technically superior than the iPhone OS. And given that Android is an open platform, there's hope that it'll improve much faster than the iPhone.

The G1 handset is disappointing compared to the iPhone. It's not well designed (at least compared to the iPhone) and the touch-screen is not as good as that of the iPhone. I could be wrong but to me it also looks a bit more fragile than the iPhone (especially the "nipple" under the menu button). I'm probably being a bit harsh against the G1 here. Don't mistaken me, it's a great handset, but Apple has set the bar very high with both its handset and its system. The cool thing about it all is that unlike the iPhone OS and the iPhone, Android is not tied to the G1. I'm sure we'll see much superior handsets on the market pretty soon.

I like it!