Comments on Tsuna's blog: Java IO: slowest readLine ever

If you can help it, always use mapped files with D...

2012-02-08T09:07:08.066-08:00

If you can help it, always use mapped files with DirectBuffers. RandomAccessFile is retarded in reading all primitives, incl. readInt(), readLong(). Full JNI switch for each bytes, magic!!

Of course mapping comes w/ a price of being unable ( to unmap (baring hacks)

For any speedy file reading on Android you need to...

2011-02-26T03:56:18.597-08:00

For any speedy file reading on Android you need to build some class that uses an InputStream like DataInputStream then use chunked buffers and parse.

Buffering the whole file obv gives you random access - into the buffer.

If you have a size problem (most of us do! ;) ) then
Performant random access and seeking makes things more complicated.

you posted this on my birthday! YO BEN! you in s...

2011-01-05T23:09:39.759-08:00

you posted this on my birthday!

YO BEN! you in sf now? come visit!

@Anonymous: the javadoc of RandomAccessFile doesn&...

2010-12-15T17:23:57.806-08:00

@Anonymous: the javadoc of RandomAccessFile doesn't mention anything about reading byte by byte, it just says "This method successively reads bytes from the file". Either way, there's no reason whatsoever to have such an utterly stupid implementation of readLine. BufferedReader doesn't have a seek method. So you can either seek around or have a non-stupid readLine, but not both. Yay.

And what about using java BufferReader, or read(by...

2010-12-15T15:22:28.229-08:00

And what about using java BufferReader, or read(byte[] b, int off, int len) if you need to use a RandomAccessFile ?
If I'm not wrong, javadocs tell us that it'll be read byte per byte, and might be inefficient...

@Anonymous: you could do this if you know the file...

2010-12-11T19:09:58.933-08:00

@Anonymous: you could do this if you know the file isn't too large and can fit in memory – but still, you'd have to implement readLine yourself because the implementation in the JDK is unacceptable. In my case though, there's another reason why I require two passes through the file, the first pass pre-computes some data that the second pass needs.

I'm still new to programming, but wouldn't...

2010-12-11T16:25:19.728-08:00

I'm still new to programming, but wouldn't it be better to use

String s = "";
ArrayList a = new ArrayList();
while ((s=in.readline())!=null) a.add(s);

to get the entire file which you intend to read line-by-line anyway. Then you'd have a.size() as the number of lines in the file, and could use a for loop to access them quickly for whatever processing needs done. I don't know much about speed or memory usage yet, so please don't yell at me if I said something stupid. T.T

@royans: with a RandomAccessFile you can seek arou...

2010-12-11T13:30:40.376-08:00

@royans: with a RandomAccessFile you can seek around freely, no need to reopen the file.

@Adrien: I always read the code of library functions before calling them from my code. I know not to trust the JDK especially, because there are a lot of dumb things in it. I spend a lot of time reading the code of the JDK on Google Code Search, but in this case I also double-checked that the JRE I was using had the problem with strace indeed. A number of packages in the JDK are very poorly implemented. Some packages are pretty awesome however, such as everything that Doug Lea wrote (no surprise there) for java.util.concurrent.

Nice history ! How did you dive that deep into re...

2010-12-11T09:17:45.048-08:00

Nice history !

How did you dive that deep into readLine() implementation ? Strace ?

What about readFully() ? Did it also fill the byte array... byte after byte ?

Wow, thats very interesting (and sort of dumb). Ho...

2010-12-11T01:36:47.491-08:00

Wow, thats very interesting (and sort of dumb). How expensive is to reopen the file if its huge file, or cache the contents of small files to avoid the "re-seeking start of file" issue.