iPhone/iCal sync woes: data.syncdb getting HUGE

Following up on my previous post where I explain how to "solve" the problem of uber-slow syncs with the iPhone. This problem seems to be a recurring problem and I suspect that it's caused by remote sync with iCal's calendars, somehow.

Personally, my data.syncdb file (under ~/Library/Application Support/SyncServices/Local) keeps growing, very quickly, even if I never sync my iPhone. This file is a SQLite database so it's fairly easy to see why it's this large. There's an analyzer to produce a report on the database (SQLite's download page - x86 binary direct download).

It's fairly easy to use:

cd /tmp
wget http://www.sqlite.org/sqlite3_analyzer-3.5.4-osx-x86.bin.gz
gunzip sqlite3_analyzer-3.5.4-osx-x86.bin.gz
cd ~/Library/Application\ Support/SyncServices/Local
/tmp/sqlite3_analyzer-3.5.4-osx-x86.bin data.syncdb >/tmp/sqlite.log

Have a look at the log. Mine was showing some really crazy numbers about fragmentation level and number of pages unused. If you also see these, consider cleaning up the db:

# Make sure you kill any SyncServer running
sqlite3 data.syncdb vacuum

I don't know if killing the SyncServer is actually necessary, but I know it uses this SQLite DB. Since I don't know whether SQLite properly locks the tables/rows/whatever that are in use, let's make sure nobody else fiddles with the DB while we vacuum it. Just to stay on the safe side. Update: If the database is effectively in use, you'll see a "SQL error: database is locked", so don't bother unless you see this message, in which case you know you have to kill SyncServer.

Personally, my data.syncdb went from a whopping big 825M down to 195M after vacuuming! I ran the analysis again after vacuuming and here are some of the differences between the reports before and after:

-Size of the file in bytes............. 871493632
-Bytes of user payload stored.......... 96907358    11.1%
+Size of the file in bytes............. 204472320
+Bytes of user payload stored.......... 96907358    47.4%

*** All tables and indices *******************************************

-Bytes of payload...................... 175282561   20.1%
+Bytes of payload...................... 175282561   85.7%

-Fragmentation.........................  69.4%
+Fragmentation.........................   2.1%

-Unused bytes on all pages............. 677992333   77.9%
+Unused bytes on all pages............. 14527468     7.1%

*** All tables *******************************************************

-Bytes of payload...................... 96909201    12.5%
+Bytes of payload...................... 96909201    92.2%

-Fragmentation.........................  66.2%
+Fragmentation.........................   0.74%

-Unused bytes on all pages............. 667043272   86.3%
+Unused bytes on all pages............. 2132671      2.0%

That's just incredible. How could the DB get to such a bad fragmentation level in only ONE WEEK?!

So yeah, I think this is a cleaner workaround that reseting the history of syncs with iSync, but it's still a workaround. I wonder why my DB "leaks" disk space so quickly... Which also seems to incur memory leaks in the SyncServer, because as time passes its memory footprint get bigger and bigger (up to ~300M!).

What to do if sync'ing your iPhone takes forever

My iPhone was starting to take a really long time to synchronize. Seemingly forever... More than half an hour, and I only had less than 50 songs, less than 50 contacts, and no big calendar. iTunes spent most of its time "Syncing calendars with iPhone" and "Syncing contacts with iPhone". Well, this seems to be a known bug experienced by many others. The workaround is simple. Open iSync, go into preferences and click the button "Reset Sync History...". When you do this make sure your iPhone is not in the middle of a sync. This should also free some 2G occupied by the file Library/Application Support/SyncServices/Local/data.syncdb (under your home directory).

As usual: "no warranty, worked for me, hope this helps".

Some more TODO items for Apple's iPhone dev team.

Following up on my previous post, here are some other things that I think Apple's iPhone development team should seriously fix. Note that all these critics do not mean that I don't like the iPhone, it just means that I hope that Apple will address the issues of this new immature product line.

• Generic landscape mode. Already mentioned in my previous post. This is a serious mis-design issue. Go fix your code.


• Total lack of select/copy/cut/paste. I know the iPhone UI doesn't lend itself very easily to these features, but they are critical. You must find a clever way of implementing them, just like you used the clever magnifying glass so we can move the cursor throughout the text easily.


• Absolutely zero control on the underlying FS. I know you try to keep control over it, but come on! We need to use the iPhone as a storage device! We want to store documents on them! Right now if I browse the web and find what seems to be an interesting PDF, I can't even save it on my iPhone to transfer it to my MacBook later!


• On a related note: I can't save attachments from Mail to my iPhone. Even if you decide you don't want to fix the previous item, you can still make it possible to import pictures attached to mails into the iPhone's photo library, maybe in a different roll though.


• Fix your cache use! If I look at the stock or weather, go back to SpringBoard, and look at it again 2min later, it will refresh the data that's only 2min old! For stock refreshing the data after 2min barely makes sense, but if you really want to force an auto-refresh, don't refresh if you know that markets are closed, because this is utterly useless. And you know when markets are closed, because it's mentioned right under the stocks history. Too many apps tend to refresh their data too often due to poor caching policies. You should work to improve all apps on this aspect.


• Make it possible to import ICS from the web to Calendar.app, FFS! Or at least sync up with the Google Calendar team to make it possible to use them read/write on the iPhone!


• Add a video recorder! How on earth could you leave this out?


• Do not resize pictures sent by email when connected on a WiFi network. I guess you resize them to save bandwidth and because EDGE is already slow enough when sending downscaled pictures, but on a WiFi connection there's no reason why you shouldn't send the full resolution one.


• Rotate the fucking pictures! Currently the iPhone always stores the JPG data in landscape mode and only mentions the orientation of the picture in EXIF metadata, but if you upload the picture on the web for instance, then the browsers won't render it properly, because many apps still don't use EXIF metadata. Moreover there's no "easy" way to rotate the pictures on OSX, which is a real pain for the average user, I guess. Personally I used jpegtran to rotate the JPEG DCT losslessly but I won't expect normal users to do this.


• No chat application. No port of iChat or whatever... For a phone with an always on Internet connection, that's ... weird, to say the least.


• No Streetview in Google Maps.


• It's not possible to plan a trip on Google Maps with multiple destinations.

(I reckon that the least two items about Google Maps are also Google's fault, I hope we'll try to do something about it despite the fact that a lot of effort is also being invested in Android.)

That's about it for now, I'll add more TODO items as they come to my mind. Please Apple, make the iPhone perfect! Currently it's only great. It ought to be perfect.

Recovering crashed iPhone

So I was bored with the few apps that come with the iPhone and talked with some people who used ZiPhone to jailbreak it. As expected, it was really easy. ZiPhone will also installs Installer.app, the package manager that will enable you to install lots of third party software.

One of these is OpenSSH (client & server). Obviously, the first thing to do when opening port 22 on this kind of device is to change the default (factory) password (which is alpine on newer iPhones, including firmware 1.1.4). Which is what I did (with passwd).

The problem is that it seems that changing the root password breaks SpringBoard. It will enter an endless loop and will keep crashing forever and showing a message box "edit home screen".


The solution to this problem is simple, you must restore the iPhone with iTunes. The problem for me was that the iPhone didn't show up in iTunes. The solution is to put your iPhone in DFU mode (Device Firmware Upgrade) as follows:


And then let iTunes do the magic for you and you're done! (Done on a 1.1.4, no warranty, hope this helps, happy jailbreaking!)

My first impressions on the iPhone

So I couldn't resist any longer and eventually bought an iPhone. 16G model of course. I waited for a long time, hoping that Apple would release a new version (with GPR and 3G) but it looks like it's not on their (public) roadmap for anytime soon.

I won't go over the details that everyone already knows (such as how lovely it is, and that "it's been designed and engineered by angles" as I've read on some blog somewhere). It's obviously a great product, like most (if not all) of the Apple products.

But. There are some limitations. So here is what annoys me (in no particular order):

• Top issue: AT&T sux balls. Whoever is responsible for locking the iPhone on AT&T deserves to burn in hell forever.


• Very limited tab support in Safari. This is a huge overlook from Apple. It greatly diminishes the comfort of browsing the Web from your iPhone.


• SMS application does not display the number of characters while you're typing. Generally speaking, the SMS application of the iPhone encourages you to send many SMSes, because SMSes are presented like a chat, so this might be an intentional misfeature (probably requested by the same asshole who chose to lock the iPhone on these crooks called AT&T).


• Poor calendar support. Honestly, even the Calendar application builtin Windows Mobile 5 is better. Of course the UI and look'n'feel is better in the iPhone, but feature-wise, it's very limited. You can't have more than one calendar in it (iTunes will squash them all in a single calendar when you import multiple calendars in your iPhone). You can't set any option about your calendar (such as setting a default alert 15 minutes before every event you add). The calendar application can only be sync'ed through iTunes (that's also a limitation in Windows Mobile -- not with iTunes of course -- but given the level of connectivity of the iPhone, I'd expect it to be able to read ICS files from an URL).


• No week view in the Calendar application. That really sucks. Why didn't they just make a lightweight port of iCal?!


• Geo-positioning on Google maps is too imprecise. This isn't a problem in the iPhone itself, but a shortcoming of the underlying technology (EDGE triangulation). Too bad that the iPhone doesn't have 3G heh :P. Also it's too bad that you can't import your "My Maps" from Google Maps and use them straight in the Maps application of the iPhone.


• The Weather application should tell the current time and date of the city you're looking at.


• The Calculator should have a more advanced mode with more maths functions.


• The style of the Notes are weird and don't stick with the general look'n'feel of the iPhone.


• There's no "My TODOs" application. You can write Notes, but if you don't keep consulting them, you forget. I'd love to have a "TODO" application where I can just quickly add stuff and always have them reminded to me. Hopefully this kind of app will be quickly available as a 3rd party app thanks to the SDK release.


• From what I've understood, there's a $100 fee to be able to release iPhone apps with the SDK. This is going to severely limit the number of free apps on the iPhone, which is a real shame. (Note: a friend of mine mentioned that this is a one time fee to be able to sign your apps, so it's not that bad, but still...)


• Lack of applications. Within the first day, you've seen more or less everything in the iPhone. Once again, let's hope new 3rd party apps will enhance this little gem that the iPhone is.


• General lack of configurability. The Settings menus are really basic, there could be much more settings to adapt the iPhone to your personal preferences. I'd love to customize the welcome screen (to get rid of the useless iTunes icon for instance).


• Non-generic landscape mode. Only some apps switch to landscape mode (e.g Safari. The iPod mode uses "cover flow" when in landscape mode, etc.). That sucks, there are many other situations where apps would be more convenient in landscape mode. For instance, a week view in the Calendar application would be hard to fit in straight mode, whereas it would surely be usable in landscape mode. I find this utterly ridiculous that app writers have to handle landscape mode specifically. This should be generic for all iPhone apps (and handled specially on a per-need basis).


• Sync'ing the iPhone takes waaaaaaay too much time. When I only add a couple of contacts and calendar events, it takes minutes to sync!


• Edit: no video recording program! OMG how could I not notice that in the first place?! OMFG how could Apple not include that in the Picture app?!

So yeah, that's about it. Admittedly, some concerns are more important than others (AT&T being the top one -- I hate you AT&T! Even more than I hate Comcast!). Overall, I'm rather satisfied with the iPhone. These concerns are all software-related, so there's hope that Apple addresses them one day or another. I was only (very) disappointed by the various shortcomings of the standard apps. This is probably due to the fact that Apple has set the level high and my expectations were also high. Also, I'm probably a power user, so I might want more out of the iPhone than the average customer (I need an SSH client on it FFS! Edit: got one with ZiPhone, see next post). But yeah, for some reason I was expecting Apple to perform better than that.