Seconds_Behind_Master in SHOW SLAVE STATUS) would sometimes suddenly jump to 7 hours and then come back, and jump again, and come back.
STOP SLAVE; START SLAVE;
try:
# some code
except Exception, e: # Bad
log.error("Uncaught exception!", e)
Yet you need to do something like that, typically in the event loop of an application server, or when one library is calling into another library and needs to make sure that no exception escapes from the call, or that all exceptions are re-packaged in another type of exception.
__builtin__.object
BaseException
Exception
StandardError
ArithmeticError
AssertionError
AttributeError
BufferError
EOFError
EnvironmentError
ImportError
LookupError
MemoryError
NameError
UnboundLocalError
ReferenceError
RuntimeError
NotImplementedError
SyntaxError
IndentationError
TabError
SystemError
TypeError
ValueError
Meaning, if you try to catch all Exceptions, you're also hiding real problems like syntax errors (!!), typoed imports, etc. But then what are you gonna do? Even if you wrote something silly such as:try:
# some code
except (ArithmeticError, ..., ValueError), e:
log.error("Uncaught exception!", e)
You still wouldn't catch the many cases where people define new types of exceptions that inherit directly from Exception. So it looks like your only option is to catch Exception and then filter out things you really don't want to catch, e.g.:
try:
# some code
except Exception, e:
if isinstance(e, (AssertionError, ImportError, NameError, SyntaxError, SystemError)):
raise
log.error("Uncaught exception!", e)
But then nobody does this. And pylint still complains.SystemExit, KeyboardInterrupt, and GeneratorExit to be subclasses of BaseException but that's all.
IOException.
IOException.
IOException is for:
"Signals that an I/O exception of some sort has occurred. This class is the general class of exceptions produced by failed or interrupted I/O operations."In Hadoop everything is an
IOException. Everything. Some assertion fails, IOException. A number exceeds the maximum allowed by the config, IOException. Some protocol versions don't match, IOException. Hadoop needs to fart, IOException.
throws IOException and everything is catching, wrapping, re-throwing, logging, eating, and ignoring IOExceptions. Impossible. No matter what goes wrong, you're left clueless. And it's not like there is a nice exception hierarchy to help you handle them. No, virtually everything is just a bare IOException.
String) to try to figure out what's wrong and what to do with it. A friend of mine was recently explaining to me how Apache Kafka was "stringly typed" (a new cutting-edge paradigm whereby you show the middle finger to the type system and stuff everything in Strings). Well Hadoop has invented better than checked exceptions, they have stringed exceptions. Unfortunately, half of the time you can't even leverage this awesome new idiom because the message of the exception itself is useless. For example when a MapReduce chokes on a corrupted file, it will just throw an IOException without telling you the path of the problematic file. This way it's more fun, once you nail it down (with a binary search of course), you feel like you accomplished something. Or you'll get messages like "IOException: Split metadata size exceeded 10000000.". Figuring out what was the actual value is left as an exercise to the reader.
IOException!IOException alone!
IOExceptions. HBase (0.92.1) has over 400. Apache committers should consider every single one of these lines as a code smell that needs to be fixed, that's begging to be fixed. Please introduce a new base exception type, and create a sound exception hierarchy.
IOException (HBASE-5796).const (C/C++) and final (Java/Scala) are truly here to help the compiler help you. Many things aren't supposed to change. References in a given scope are often not made point to another object, various methods aren't supposed to be overridden, most classes aren't designed to be subclassed, etc. In C/C++ const also helps avoid doing unintentional pointer arithmetic. So when something isn't supposed to happen, if you state it explicitly, you allow the compiler to catch and report any violation of this otherwise implicit assumption.
The other aspect of const correctness is that you also help the compiler itself. Often the extra bit of information enables it to produce more efficient code. In Java especially, final plays an important role in thread safety, and when used on Strings as well as built-in types. Here's an example of the latter:
1 final class concat {
2 public static void main(final String[] _) {
3 String a = "a";
4 String b = "b";
5 System.out.println(a + b);
6 final String X = "X";
7 final String Y = "Y";
8 System.out.println(X + Y);
9 }
10 }
Which gets compiled to:
public static void main(java.lang.String[]); Code: 0: ldc #2; //String a 2: astore_1 3: ldc #3; //String b 5: astore_2 6: getstatic #4; //Field java/lang/System.out:Ljava/io/PrintStream; 9: new #5; //class java/lang/StringBuilder 12: dup 13: invokespecial #6; //Method java/lang/StringBuilder."In the original code, lines 3-4-5 are identical to lines 6-7-8 modulo the presence of two":()V 16: aload_1 17: invokevirtual #7; //Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder; 20: aload_2 21: invokevirtual #7; //Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder; 24: invokevirtual #8; //Method java/lang/StringBuilder.toString:()Ljava/lang/String; 27: invokevirtual #9; //Method java/io/PrintStream.println:(Ljava/lang/String;)V 30: getstatic #4; //Field java/lang/System.out:Ljava/io/PrintStream; 33: ldc #10; //String XY 35: invokevirtual #9; //Method java/io/PrintStream.println:(Ljava/lang/String;)V 38: return }
final keywords. Yet, lines 3-4-5 get compiled to 14 byte code instructions (lines 0 through 27), whereas 6-7-8 turn into only 3 (lines 30 through 35). I find it kind of amazing that the compiler doesn't even bother optimizing such a simple piece of code, even when used with the -O flag which, most people say, is almost a no-op as of Java 1.3 – at least I checked in OpenJDK6, and it's truly a no-op there, the flag is only accepted for backwards compatibility. OpenJDK6 has a -XO flag instead, but the Sun Java install that comes on Mac OS X doesn't recognize it...
There was another thing that I thought was a side effect of final. I thought any method marked final, or any method in a class marked final would allow the compiler to devirtualize method calls. Well, it turns out that I was wrong. Not only it doesn't do this, but also the JVM considers this compile-time optimization downright illegal! Only the JIT compiler is allowed to do it.
All method calls in Java are compiled to an invokevirtual byte code instruction, except:
invokespecial.invokestatic.invokeinterface.
Anyway, I always imagined that having a final method meant that the compiler would compile all calls to it using invokespecial instead of invokevirtual, to "devirtualize" the method calls since it already knows for sure at compile-time where to transfer execution. Doing this at compile time seems like a trivial optimization, while leaving this up to the JIT is far more complex. But no, the compiler doesn't do this. It's not even legal to do it!
interface iface {
int foo();
}
class base implements iface {
public int foo() {
return (int) System.nanoTime();
}
}
final class sealed extends base { // Implies that foo is final
}
final class sealedfinal extends base {
public final int foo() { // Redefine it to be sure / help the compiler.
return super.foo();
}
}
public final class devirt {
public static void main(String[] a) {
int n = 0;
final iface i = new base();
n ^= i.foo(); // invokeinterface
final base b = new base();
n ^= b.foo(); // invokevirtual
final sealed s = new sealed();
n ^= s.foo(); // invokevirtual
final sealedfinal s = new sealedfinal();
n ^= s.foo(); // invokevirtual
}
}
A simple Caliper benchmark also shows that in practice all 4 calls above have exactly the same performance characteristic (see full microbenchmark). This seems to indicate that the JIT compiler is able to devirtualize the method calls in all these cases.
To try to manually devirtualize one of the last two calls, I applied a binary patch (courtesy of xxd) on the .class generated by javac. After doing this, javap correctly shows an invokespecial instruction. To my dismay the JVM then rejects the byte code: Exception in thread "main" java.lang.VerifyError: (class: devirt, method: timeInvokeFinalFinal signature: (I)I) Illegal use of nonvirtual function call
I find the wording of the JLS slightly ambiguous as to whether or not this is truly illegal, but in any case the Sun JVM rejects it, so it can't be used anyway.
The moral of the story is that javac is really only translating Java code into pre-parsed Java code. Nothing interesting happens at all in the "compiler", which should really be called the pre-parser. They don't even bother doing any kind of trivial optimization. Everything is left up to the JIT compiler. Also Java byte code is bloated, but then it's normal, it's Java :)