220 comments

[ 2.8 ms ] story [ 238 ms ] thread
This would be immensely beneficial, and long overdue in my opinion.
Imagine how many thousand man-hours would be saved in debugging if this existed a decade ago. Little things like this over time adds up.
If nothing else, it is really interesting to see exactly why this is hard.
Historic reasons, Java used to be bloated so things like these were not included, not computers are faster, have more memory and object metadata can be expanded.
> have more memory

The L in Java stands for Low Memory.

Nice joke from 2002
Last time someone was let down by Java...
pretty much since early 2000s java has always had 'free' null derefencing and OS traps for addressing the zeroth page.

each dereference where NPE can occur has had 'metadata'. It's compiling metadata not object one.

It is almost 2020 so it is reasonable to know what the actual error is.

This reminds me of old C/C++ parsers that couldn't tell you what line was expecting a bracket or semicolon and you got to manually find the parse error.

I don't know why i find this so funny but it's definitely something that would be greatly appreciated!
I don't see that many NPE in kotlin these days but pretty sure that on java/Android I get a useful error message.

Is that an ART/Dalvik feature ?

I don't remember getting any useful error message, but I've been using Kotlin for 3 years now, so I might not remember correctly. Can you post a stacktrace of what you're referring to?
as stated in another comment, you get : s exception: Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String java.lang.Integer.toString()' on a null object reference

the name of the method causing the crash (toString in that sample) + the line number and location thanks to the stacktrace (although in some rare case it is 100% accurate, but still pretty close, maybe that's due to runtime inlining) and it is very trivial to know where the exception comes from.

> I don't see that many NPE in kotlin these days

Nice humble brag. :D

> Is that an ART/Dalvik feature ?

Yes, ART has had this for a few years now. For example in Oreo this code:

    static Integer nums[] = new Integer[100];
    public static String stabSelf() {
        return nums[new Random().nextInt(100)].toString();
    }
Will produce this exception:

    Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String java.lang.Integer.toString()' on a null object reference
This sounds like a good addition.

It happens quite often that we get a bug report with "We got a NPE". Our users often attach a screenshot, just showing the standard "NPE error" message, which does not really help.

That's like every bug report I get from non-technical users at my company. "The device doesn't work. Can you log in and take a look at the logs?" Yeah, if you can give me some information about which device it is I'd be happy to.
Meanwhile, for as long as I can remember, Visual Studio could easily jump straight to the offending null variable when debugging C#.
There’s a difference between an IDE offering it, or the JDK
(comment deleted)
I seem to remember that too, null pointer exception coughs up more than enough information to figure out where it barfed.

Makes me feel that complaints about null aren't really about null itself but languages and OS's not generating useful error messages.

This would be good, but I'd prefer that Java have something like the null operators that C# has.

https://docs.microsoft.com/en-us/dotnet/csharp/language-refe...

Adding Elvis and safe navigation operators to Java was discussed before Java 7, over 10 years ago. Sadly they never made it even to the official proposal lists and with Optional introduced in Java 8 I don't think we'll get them any time soon.

See also Brian Goetz opinion on that subject: https://youtu.be/FdkPHShh628?t=50m17s

(comment deleted)
Plus C# is introducing non-nullable reference types (null becomes opt in).

https://msdn.microsoft.com/en-us/magazine/mt829270.aspx?f=25...

If I read that correctly, what they're actually introducing is a compiler setting (defaulted to off) to have it issue a warning when you assign a null or possibly-null value to a reference that wasn't explicitly annotated to indicate that it expects null.

But even with the setting turned on, you're still free to ignore the compiler warnings (assuming you haven't also told it to treat warnings as errors) and blithely assign null to a reference that wasn't flagged as nullable. So it's not quite non-nullability.

Which isn't to say that it's a misfeature. It's a huge step in the right direction that will be immensely helpful, and they managed to do it in a way that doesn't break the mountains of C# code that already exist. These sorts of methodical, well-considered design decisions that keep the language moving steadily forward without causing chaos for its users are exactly what's so great about the C# team.

I'm a bit torn on this because I always avoided having too many method calls on the same line. Code Complete recommended this a long time ago.

However now I see just about everyone stacking this method calls one after the other. Is this just acceptable now?

state = stateFromPostcode(customer.getAddress().getPostCode());

Depends really. Can getAddress() return null or is it guaranteed to be there per your system design? If it's not then that code is wrong, but if you know your invariants and enforce them higher up the chain then perhaps not.

I realize that all sounds very obvious, but like most things... it really does depend. It can certainly make debugging harder if you need to step through. Personally I try to write code which simply doesn't allow nulls below the first layer they may be encountered. Not always possible though (e.g. nullable fields in entities and whatnot.)

> Is this just acceptable now?

No, but where before it was a hard requirement in Java to deal with NPEs, newer libs are less likely to return nulls, so it's usually a bit safer than it was. You also avoid leaving references around that could potentially leak.

There is also a bit of fashion with streams and JS-inspired call-chaining.

Isn't this what you get when you drink heavily of the Lambda Kool-Aid?
Yes, that and also when you aren't working with code that is older than three months.
> However now I see just about everyone stacking this method calls one after the other. Is this just acceptable now?

Not only acceptable but a great idea. Vertical space is a much more limited resource than anything else.

> Is this just acceptable now?

That’s a matter of opinion, and my opinion is hell yes that’s not only acceptable but recommended! If creating a variable makes the code more readable, then do it; if it doesn’t, then don’t. But certainly don’t worry about NPEs.

A similar piece of advice is that you shouldn’t do this:

    if (a.equals("hello"))
You should instead, the advice goes, do this:

    if ("hello".equals(a))
because this doesn’t throw an NPE if `a` happens to be null. The only problem is that you want to be aware that your value is null; you don’t want to hide the error, then wonder why your program blows up further down the call stack. You haven’t solved the problem. You haven’t fixed a bug. All you’ve done is make that particular error message go away.

It turns out that the real advice is not “structure your code to avoid NullPointerExceptions”, but “avoid null”. With this in mind, rules like this one, as well as guidelines telling you not to have too many function calls on the same line, are rendered useless.

In Scala, no (that's what val is for) but I think Java it's ok. For builder patterns with long method names I'll just put some of the methods on their own lines
Can we add column numbers to the stack traces? Seems like that would solve this in the general case.
(comment deleted)
no, not really. Java Class format doesn't support columns, just lines.
Just imagine the request read "Can we add column numbers to the stack traces? And to whatever code that relies on to add column numbers to stack traces?"
All the existing code and libraries have to be recompiled. Non-javac compilers have to be updated, etc.

Pretty much all byte code editing tools would be obsolete as the constant pool needs change and so on. Also the generated byte code would be more verbose (worse load times)

The current change is easy to implement and it'd work on code compiled for/with java 1.0.

Other than supporting and working with legacy software, I cannot muster any reason to write code in Java at this point... it’s just a terrible language to work with.
Downvote if you must... but deep down you know it’s true.
The problem is that "I don't like Java a lot." does not contribute to this discussion at all.
for one, after some scala exposure, I am missing the "simplicity" of java.
Java sucks in a lot of ways, but the reasons why it sucks and the methods for mitigating the parts that suck are well-known. On the plus side, you get an ecosystem with high-quality libraries and tooling that actually come with documentation and examples, you get a good-enough type system to allow reasonably safe and strict development, and you can easily find solutions for any pitfalls you fall into because it's likely 1000+ people have already done the same. As a member of the http://boringtechnology.club/, I embrace Java's warts and enjoy its usefulness.

My advice:

- avoid frameworks like the plague

- write in a functional style

- lean on the type system as much as possible

- don't chase the trendy stuff

- avoid annotations

- use code generation

I am not really that familiar with Java, but I don't understand how to get a JVM heap dump if there is an exception that brings down the JVM or even a thread. Is it even possible?

I work on mainframe (z/OS), and it is completely normal there that when an application fails with exception (they're called ABENDs - from ABnormal ENDing), you can get a dump of memory of the application. From that, you can see all the values of the offending variables and all the relevant system areas and so on.

You can turn this on but the dumps are kind of huge. Also, you can attach to a running JVM and get a memory dump that way. NPEs don't bring down the JVM though unless that happens in your main method of course, in which case the process exits normally.
What I mean, to produce the dump in the case of exceptions that percolate to top level and exit the process or thread (such as NPE). I am aware that I can attach, but that kind of defeats the ability to get the information in case of unexpected error (in production). Dumps might be huge, but is that really a problem on modern hardware? On z/OS this is doable (and common practice).
>What I mean, to produce the dump in the case of exceptions that percolate to top level and exit the process or thread (such as NPE).

Why not have a top level try catch then?

If you have a top level catch, then you lose the local context of the error (local variables, parameters, stack trace). I mean seriously, this is a solved problem on z/OS (the ABEND condition bypasses normal stack, so to speak), it has been solved for over 50 years. I don't see why JVM couldn't do the same thing - is it a technical problem? It seems like a purely cultural problem to me.
Maybe because in Java it's usually considered a very bad practice to halt your program. All the web containers that run your code will catch any error, to prevent bringing down the whole web service if only a small piece of it fails (for example, cannot serve one URL, but can others).

So instead, in Java, it's your responsibility to catch and correctly handle all the exceptions.

Also, variables in the last function (where exception was raised from) are unlikely to give me understanding of the bug, because it's usually thrown somewhere deep in others code, and also I'm more interested in _why_ it happened, not just _what_ happened.

We could dump all the variables in the stack trace, but I'm not sure I would like to read the whole dump of every variable in stack trace, that's just too much. I'd better write my code with proper logging, so I would be able to read the logs in order to understand not only _what_ happened, but _why_ it happened. With good logs you rarely even need to debug anything, I'd say 95% of the time it's pretty clear where to fix.

Again, on z/OS, this is a solved problem. If the application runs inside an "application server", for instance CICS, you can still get the dump and the CICS can continue or even restart your app (or it can be restarted through other things). You can get the dump of the process even if just one thread will fail.

I know from experience on z/OS how useful full dump can be, and I sorely miss it in Java. You wouldn't have to read the full dump yourself - a good postmortem debugger can do that work for you - displaying stack trace, locals, etc.

Good logs can certainly save you.. if you have them, and you happen to log exactly what you need. But it's much easier to look at the memory and see what went wrong. (Personally, I think logs are great on the boundary, to describe inputs and outputs.)

I also think that the exceptions are kind of wrong approach (compared to z/OS ABEND conditions), and it's not entirely true that failing loudly is a bad practice, but that's for another discussion.

Indeed a cultural thing in the sense that a JVM crashing is rare and exceptions are both common and non-fatal.

In short, that's what logging is for and there are some awesome options for that in Java. We log all exceptions with stacktrace + full MDC (mapped diagnostic context) to a central logging cluster. Also, we start alerting if that happens. Most application servers will have a top level catch and a way of logging/handling exceptions (e.g. sending a 500 response for unhandled exceptions).

The JVM crashing usually means you are hitting a jvm bug; it's not supposed to do that. It's been a while since I had that going on. Even out of memory exceptions tend to happen in a somewhat orderly fashion. It will print some message probably. Setting -XX:+HeapDumpOnOutOfMemoryError does what you want here: dumping the memory. Of course the use of that on an ephemeral disk in e.g. Docker is kind of limited. These heap dumps are super detailed though and you can dig through them using special tools. A common issue with docker is not setting limits correctly and then docker killing your process instead of it exiting the normal way.

You can dump a stack trace of all threads to std out by sending kill -QUIT to the process (does not kill the process). Great for debugging deadlocks.

Yes, you can enable dumping the heap via command line arguments when starting the JVM: https://docs.oracle.com/javase/8/docs/technotes/guides/troub...
Well, it seems to me that XX:HeapDumpOnOutOfMemoryError is only useful if you run out of memory, not for other errors such as NPE. The other one XX:OnError I am not really sure how to use with the JVM.. I mean can probably get the dump of the process on certain systems but I am not clear how to then proceed to get the actual dump of Java heap out of that. (And also, it's not clear what kind of cleanup happens inside JVM between the error and the execution of the command.)
Better than nothing, but worse than having a type system than disallows the exception it in the first place.
Maybe or optional types don't really help because your "get value" functions also throw exceptions, or have undefined behavior, if the value doesn't exist.

Examples:

https://en.cppreference.com/w/cpp/utility/optional/value

https://docs.oracle.com/javase/8/docs/api/java/util/Optional...

Nonsense. Obviously if you make every single value in your program with an option then you have the same problem as if your language had null. But the difference is that in a language that has optional types, it becomes practical to have values that aren't optional. And for values that really are meant to be optional, you don't use the "get value" functions (which really shouldn't exist at all), you use functions that handle both cases.
You are not supposed to do a get on an Option type; it's an antipattern. You're supposed to map over them, and when you really need the value, to pattern match the Some and Nothing cases, which cannot blow up in your face.
Bad advice.

If you're given an Optional that's guaranteed to have a value, it's better to call get and blow up if someone violates this invariant than to silently ignore the invalid input via pattern matching or optional chaining.

Crashing is better than silently ignoring logic errors.

Of course, one could ask what's the point of an Optional that's guaranteed to be non-nil. Well, it often happens that there's some relationship between a set of values that's hard/awkward to express via the type system. E.g. a toy example would be that A and B can both be nil, but it also holds that (A == nil) == (B == nil).

I don't know what to tell you. It's how they are meant to be used [0], it's not some advice I made up. It's the whole point of using optionals, otherwise you're left with a verbose way of saying "this may be null".

There are unsafe methods to extract the value in those rare occasions when you absolutely know it's there, but you seldom need them. You're supposed to map and pattern match. Neither is silent, so I don't understand what you meant.

> Crashing is better than silently ignoring logic errors.

Map and pattern matching are not silent.

----

[0] https://www.oracle.com/technetwork/articles/java/java8-optio... (search for "this is not the recommended way of using Optional", and bear in mind this was for Java 8, which doesn't have pattern matching like Scala or an ML language)

>Map and pattern matching are not silent.

I'm writing a complicated algorithm and am at a point where I know for certain that a given value must be non-nil. Sometimes it can be nil, but at this point it's provably non-nil (E.g. at the end of Dijkstra's algorithm every entry in the distance vector must be non-nil - assuming the graph is connected). The type system can't prove this, but I know that it's non-nil at this point.

So, when I access it, I want the program to fail loud and clear, because it means that I've messed up somewhere. Handling it via map or pattern matching makes no sense here. What would you write in the nil case of the pattern matching anyway? return -1? That's absurd. abort()? That's exactly what get() does.

There are unsafe escape hatches for the rare cases where you do need them, but they aren't recommended. It's not the idiomatic usage, which is why the original post in this thread saying "it'll blow up when I do Optional.get" is mistaken.

If it's provably non-nil, use get. But you must provide the proof, and your first reaction must always be "I'm not sure I can assume this".

> What would you write in the nil case of the pattern matching anyway? return -1? That's absurd.

Yes, it's absurd. You don't return this type of sentinel values in statically typed languages with optional types. If you don't need to discriminate the error, sometimes returning Nothing will suffice. If you need the error, you can use an Either type (where traditionally Left is used for the error).

Again, this isn't something I made up. It's how Optionals are used. I suggest you read the link I posted above.

If you have something that might be nil earlier, then some stuff happens and you know it definitely isn't nil anymore then why is it still in an option, isn't the logical path split that diverted those "might be nil" cases where a case handling the nil would peel off leaving you not with a maybe but with a real value?

I think that sort of pattern is a bad code smell, you are propagating doubt where there is none leading to misleading code that has assumptions that differ from what's on the page - obviously I don't have a real example in front of me so there may be good allowances for understanding but I don't find your core premise to be sound and I feel like any counter example would have a lot of work to do to appear convincing.

> If you're given an Optional that's guaranteed to have a value, it's better to call get and blow up if someone violates this invariant than to silently ignore the invalid input via pattern matching or optional chaining.

Nope that's bad programming. If you know the Maybe is always there, PROVE IT! Step 1 to proving anything: Make a proposition. In other words, swap out `Maybe a` for `a`. Then prove it upstream. Maybe at some point you want to crash, but you definitely want to do it at the boundary and not within your function. Writing a function that takes a Maybe but assumes it's there just seems like asking to break things.

> a toy example would be that A and B can both be nil, but it also holds that (A == nil) == (B == nil).

Isn't this just `Maybe (A, B)`? If you have both sums and products in your type system, these sorts of situations go away.

A more complicated example: You have A and B and A, B, or both are non-null. But both are never null at the same time.

The way to lift this invariant into your type system (i.e. make a proposition about your program) is:

    data IOr a b = ILeft a | IRight b | IBoth a b
Let’s say Math.sqrt returns Optional because we haven’t invented complex numbers yet. You’re given an integer between 1 and 10, you square it, and then take the square root. The result is an Optional, but it’s provably non-nil. On the other hand, you can’t prove this within the type system. (unless we’re talking about Idris or coq or whatever exotic crap that noone’s using outside of academia)

>If you have both sums and products in your type system, these sorts of situations go away.

Like I said, it’s a toy example. If you need a different example, here’s one: A, B, C and D are Optionals, but at most two of them are nil.

If you depend on the square root existing, why not have a (very short) positive_sqrt function?

> A, B, C and D are Optionals, but at most two of them are nil.

That sounds like something that's quite fragile and should either be wrapped up in its own type or not depended on.

It's quite easy to work with type-system-checked positive numbers outside of academia. I've done it in production in Haskell and made real money ;)

You can write a verbose sum-of-products to encode that ABCD example although if it's really true that only two tops can be nil and it can be any combo and they're all disparate type, I'd love to see what real problem that would be about! Seems nuts.

Lemme see if I can have some fun and encode it more elegantly than "unrolling the permutations" :)

I just winged it and got this. It was an extremely mechanical GADT to write. Once you've seen one correct-by-construction dependent type, you've seen them all. Just follow the north star (Curry-Howard)!

    {-# LANGUAGE GADTs #-}
    {-# LANGUAGE RankNTypes #-}
    {-# LANGUAGE TypeOperators #-}
    {-# LANGUAGE DataKinds #-}
    {-# LANGUAGE KindSignatures #-}

    module ABCD where

    import Data.Kind (Type)

    data N = Z | S N

    data AtMostN (n :: N) (ts :: [Type]) where
      AMN_Nil :: forall m. AtMostN m '[]
      AMN_Pass :: forall t m rest. AtMostN m rest -> AtMostN m (t ': rest)
      AMN_Cons :: forall t m rest. t -> AtMostN m rest -> AtMostN (S m) (t ': rest)
Let's try to have two optional booleans but say that at most one can be present:

    λ: AMN_Cons True (AMN_Cons True AMN_Nil) :: AtMostN (S Z) '[Bool, Bool]

    <interactive>:7:1: error:
        • Couldn't match type ‘'S m0’ with ‘'Z’
          Expected type: AtMostN ('S 'Z) '[Bool, Bool]
            Actual type: AtMostN ('S ('S m0)) '[Bool, Bool]
        • In the expression:
              AMN_Cons True (AMN_Cons True AMN_Nil) ::
                AtMostN (S Z) '[Bool, Bool]
          In an equation for ‘it’:
              it
                = AMN_Cons True (AMN_Cons True AMN_Nil) ::
                    AtMostN (S Z) '[Bool, Bool]
The type error requires some interpretation, but it sees that we set two. Note the ('S ('S m0))

We can say at most two are set and now it compiles:

    λ: AMN_Cons True (AMN_Cons True AMN_Nil) :: AtMostN (S (S Z)) '[Bool, Bool]
And we can set just one and compile fine as well:

    λ: AMN_Pass (AMN_Cons False AMN_Nil) :: AtMostN (S Z) '[Bool, Bool]
This is a bare-bones example. We can easily add a lot of sugar on top to make it more ergonomic (maybe even custom type errors)
The distinction here is that those implementations aren't a type system, they are types in the existing c++/java type system.

In many languages that support Optional types as first-class citizens, it is a compile-time error to ignore the fact that it may be null/none. Your code must deal with this, and you won't have any surprise exceptions, even if you are lazy.

Which languages? In the languages that I'm somewhat familiar that support Optionals (Kotlin and Swift) you can still get NPEs, there's just some nice syntax that makes it harder to do accidentally (e.g. by using !! or ! on a null/nil optional)
Most ML(Ocaml) languages I use will tell you if you haven't handled all the cases.
Elm is an example of a language that literally can not produce NPEs. I use Rust on a daily basis and while it "technically" can produce one, I've never seen one.
The idiomatic way to use optional types is pattern matching (e.g. when in kotlin, switch in Swift). These pattern matching constructs tend to have a requirement that your cases are exhaustive. This makes it impossible to miss the None case.

However, they also tend to provide escape hatches for getting at the type directly, which defeats that and will still let you NPE.

Rust does not allow null values without resorting to unsafe code.
Technically, it does: std::ptr::null and std::ptr::null_mut are not unsafe. (Of course, you cannot try to dereference the resulting pointer without using unsafe.)
To that end, I find that the value of optional types depends heavily on the language they're being used in.

Might as well beat up on Java: One of the best belly laughs I had during a debugging session was tracing a `NullPointerException` to the source, and finding that the reference was to an `Optional<T>` that was being used to handle a value that might not be present. Of course, the null reference here was to the `Optional<T>` itself.

There are also also some ergonomic issues. Obvious ones like the lack of pattern matching making them a bit more verbose to work with, of course. But also deeper, more annoying problems, such as Java's standard library lacking a really well-thought-out equivalent to the various abstractions on sequences that are standard in functional languages, and, by extension, functions for operating on them. Though it does have plenty of bits and bobs that were diligently cargo culted on without really capturing the essence of what this stuff is supposed to be about. So, for example, while (at least in Java 8) `Optional<T>` does have its own method called `flatMap`, it doesn't implement `Stream<T>` or offer an easy method for producing a stream. So the idiom of flatmapping a function that returns `Optional<T>` over a sequence to yield a (possibly shorter) sequence of items for which the function returned something, which is so simple and readable in an ML-like language, becomes hopelessly verbose, awkward, and even error-prone in Java.

(Also, Java Streams are, by design, not referentially transparent, so much so that they cannot be made to behave in a referentially transparent way. Which is a total tangent from what I'm talking about here, but does at least give another angle from which to see the depths to which Java misses the point on these sorts of things, and possibly isn't even able to catch it.)

All of which kind of leaves me thinking, "When in Rome, do as the Romans do." Yeah, in hindsight, the ALGOL family of languages' standard method of dealing with cases where a value might not be available is hopelessly fraught with problems. But importing a second approach from a separate language family that split off about 40 years back and started going in a whole different semantic direction into an existing language that already has its own ways of doing things doesn't appreciably improve that situation. It just introduces two competing ways of doing things that end up interacting poorly with each other.

So, to wrap it up: I'm inclined to say that keeping null and just improving the error message when it happens unexpectedly probably is the best band-aid for this problem.

> Maybe or optional types don't really help because your "get value" functions also throw exceptions, or have undefined behavior, if the value doesn't exist.

That's a big leap. That's like saying Haskell isn't pure because of unsafePerformIO. You still get nice static checks with Maybes. If people cheat with fromJust, it's at least obvious they are cheating. Both big and obvious improvements over null :)

Finally someone noticed it!
This is too much of a hack; I predict it will be unreliable, at times sending programmers on wild goose chases more time-wasting than an uninformed investigation.
Because of some track record of Java, or because you just have a feeling about it?

I don't program in Java at all, and frankly, I was flabbergasted to see a change as obvious and useful as this wasn't already in there a decade ago. Or at least 3-4 years ago where a lot more languages started advertising better compilation error messages.

> Because of some track record of Java, or because you just have a feeling about it?

It's proposing to reverse-engineer the corresponding source code from the bytecode when encountering the exception. That does sound like a potentially unreliable hack.

Not necessarily. The sort of dataflow analysis they're talking about is an inherent part of bytecode verification already. A more advanced version of it is done every time a class is loaded. JVM bytecode is designed to make this sort of thing possible.
this is good stuff. please come back 10 years ago and do it!
Maybe 20 - I graduated high school in 1998 and in 1999 Java had taken over a lot of the more "industry oriented" CS schools in my country (Australia).
I am so frustrated at the slow pace of java but then I remember that for some reason everyone is using JS now and laugh in at-least-some-semblance-of-typesafety (also stacktraces, a module system not invented by a SF hobo, a real compiler and build systems that aren't a complete disaster)
I don't write Java, but support an application written in Java that gets null pointer issues and aborts all the time. It's also annoying that there is precious little information for me to go on.
Blame it on the poorly written code, not Java.
It generally means that an object was supposed contain data, but it doesnt. This typically could mean the network disconnected or a file was not read. You could be missing .jar files that contain the appropriate classes. It can also be a programmer mistake.

Its not a very clear error, but you can narrow down the issue to a few things even when doing application support.

In our case it is reading from an antiquated non relational database and usually one of the millions of fields that were being parsed had a "null" value. It is easy to fix, but only if I know which ones the app needs.
How long do you support the application?

I believe that if you support it long enough, it's your application now, so no one else is to blame. How long is long enough? I'd say 1k-2k of lines of code per month of supporting it should be pretty reasonable. So if you supported a project of 10k lines for a year -- it's yours now, stop blaming original authors.

Good engineer would start improving it little-by-little, or convince manager that they need to spend some time on refactoring some piece of it. That differentiates a coder from an engineer.

You are assuming that they have access to the original source code and an ability to edit that source as well, which may not be the case. If it were me and I didn't have the original source I would decompile it out of curiosity.
(comment deleted)
Bingo....this one I do not have the source for. It is a vendor app. Sadly, I could rewrite everything it does in a fraction of the code and it would be more maintainable and cheaper, but it would never be approved.
That moment when you step out of your safe Scala world just to realize that people still face NPEs.
I write exclusively in Scala, but NPEs still exist. The moment you introduce a java library to your code, you become vulnerable to them. And if you have an inexperienced java developer switching over to scala, you'll get nulls in scala code as well.

And in performance critical code you might find them as well. Sometimes dereferencing an option is too costly, so you put up with an anti-pattern in order to get lower latency or higher throughput.

After 20 years, first in Java and then in Scala I've been tortured by this error message. Some years ago they fixed ClassCastException which for a long time also didn't tell you what was wrong. There are some others if I remember correctly with bad error messages, like NumberFormatException. In general error messages on the JVM are very bad, wish they would take a look at Elm error messages.

I've left this year the JVM for TS, but glad for everyone who uses the JVM.

The default compiler options for java provide a lot of information on null pointer exceptions. I'm sort of blown away that so many on this thread are confounded by NPEs, in my view they are dead simple to track down if you have access to the source.
As I'm running a legacy larger Scala app, what would that compiler options be? e.g. for the line

    service.withName(customer.getName().firstName);
with a NPE in line 25?
Break out the line into multiple lines...

String s = customer.getName().firstName;

service.withName(s);

So the answer is rewrite your code to work around tooling deficiencies?

I'd prefer this be fixed in the JVM, so you can decide to write your code in the way that's most readable, and not be forced to write it a certain way because the tools suck.

Yes, this is what I'll need to do every time, which is crazy to adapt your code to the tools and then back again.

And it's even

   val n = customer.getName()
   val f = n.firstName
   service.withname(f)
Gladly with Some/None NPEs happen seldom with Scala but they do from interfacing with Java libs from time to time.
> Computation overhead > NullPointerExceptions are thrown frequently.

"Frequently" is obviously a relative term, but are NullPointerExceptions really common enough to be a performance concern? It's good that they are taking performance overhead into consideration of course, I'm just surprised it's even an issue.

Agreed. I feel the key word in dealing with exceptions is "exception," meaning something uncommon, which is a foreign concept in some codebases, where exceptions are used as routine flow-control.

Throwing an exception in Java is a non-trivial expense since the constructor for Throwable populates a stack-trace structure for the current thread. If you're using exceptions as a routine flow-control mechanism, you're incurring that cost too frequently.

Like you said, sure, tune the performance of this new feature because it's part of the standard library and should be reasonably well-tuned. But if people are frequently throwing NPEs, they should consider alternatives such as simple null checks.

> exceptions are used as routine flow-control Unfortunately, this approach is encouraged by some popular frameworks, for example a ResponseStatusException in Spring 5
And even the JDK itself. java.lang.InterruptedException comes to mind.
InterruptedException is sort of necessary evil, esp. when it comes to concurrent data structures.

InterruptedException are rare (very) and used mostly to signal termination attempt. LockSupport.parkNanos doesn't throw the exception (unlike sleep, and sleep should never be used in code) but the interruption signal/flag has to be respected somehow. Most of the time is rethrow in library code or bailing out with IllegalStateException in 'userspace'

I think that it could be done better. .NET's CancellationTokens are a step in the right direction, though sadly just one step. You can check if cancellation is requested, but there's also a method for asking the token to throw a CancellationException if cancellation has been requested, and this is how all the abstractions around threading do it.

But it's reasonably easy to imagine ways to produce an API that only uses tokens like that instead of using exceptions to signal cancellation/termination requests. And I suspect, given how many bits have been spilled trying to explain to people just how they're supposed to deal with InterruptedExceptions, that the resulting system would have better ergonomics.

Long story short, I don't know that it's that InterruptedException was a necessary evil so much as that people were still fairly new to this whole exceptions thing when Java came out, and it's really hard to get subtle things like this right on the first try.

> Throwing an exception in Java is a non-trivial expense since the constructor for Throwable populates a stack-trace structure for the current thread. If you're using exceptions as a routine flow-control mechanism, you're incurring that cost too frequently.

Unfortunately, the JVM has special code to detect this case (!) and stop recording the stack trace after a while (!!). Then when you go look at your log to see why the system isn't working, you see only `NullPointerException` without the expected stack trace, and you have to guess where it's coming from (after wondering for a few hours why the stack trace went missing, only to find the explanation at https://stackoverflow.com/questions/2411487/nullpointerexcep...).

The fact that anyone ever thought it would be worthwhile to implement an optimization like this is very telling.

I'm not exactly sure what it tells us, but it definitely tells us something.

I've worked with developers who use this pattern frequently for code execution.

try{ .. business logic .. } catch (NullPointerException e) { .. else .. }

Rather than a null guard. That's what occurred to me.

This is arguably more robust, because "foo.bar.quux.doTheThing()" is three potential null pointer exceptions in a row, and the code to do consecutive testing is ugly and verbose.
Having a try..catch round such code would seem sensible but to rely on that over normal checking seems spectacularly horrible.
This is why Rust's '?' operator for error propagation is quite nice.
Or kotlin's '.?' operator for null propagation.
Or C#'s '?.' operator for null propagation.
It’s a code smell if you are calling a method chain that long at all.
Cannot agree with you more. I personally hate chained expressions. Awful for debugging too.
It's only awful for debugging because the tools are awful.
The tools for Java are greate (try C++ if you disagree).

It's the code that is the problem here.

The tools for Java are much better than those for C++. However, claiming the code is the problem is silly. I'm not going to introduce a bunch of local variables just to appease a debugger.

If splitting on separate lines makes code more readable, then sure, do that. But often it just makes the code longer and harder to follow.

Java debugging tools are beyond excellent for debugging such code (not that it should be written that way)
I still have to select the sub-expression whose value I wish to inspect. This involves precisely aiming the cursor at text, which is a lot of mental burden (x millions).

A task I thank the developer who creates a variable to hold such a reference, for it creates a much better experience.

This is largely not the case outside of the C-like languages that inherited nulls.

Long method chains with a strong type system or a functional (also called "fluent") style are extremely expressive and common in languages that obviate the need to handle null values at every member.

Regardless of safety guarantees (or lack thereof), a method chain that long indicates a lot of knowledge of the structure and hierarchy of the dependency. It makes refactoring/testing/mocking more difficult.
What does the length of the method chain have to with anything? Let's say I define an interface Foo.bar.xyz with method call, and I call the implementation foo.bar.xyzimpl.method()

Everything is just as testable and decoupled, it's just a different project structure.

Method chains have nothing to do with the things you stated.

It's usually a property chain, not a method chain - I don't think get...() is meaningfully a method in most cases, even if implemented as such.
I guess there are extremely few people in the topic who understand how NPE works internally.

There is no null check in the generated assembly code, when a null dereference occurs - it's an effective kernel trap (as 0 is not mapped to the process). The latter uses the code execution pointer to understand what has been attempted to execute and throws the exception. This may or may not involve stack crawl (which is very expensive) depending on JIT ability to prove if the stack would be unused.

Nulls should be avoided, all fields should be initialized, etc. Nulls are great for =very= high performance code as null checks are virtually free.

I remember seeing sample code from WebLogic that returned values from methods using exceptions. Mind you this was in documentation rather than in production code - but hardly a good example to follow!
I've seen a horrifying adaptation of this pattern in C++, where a piece of code was using catch(...) to detect dereferencing an invalid (not even necessarily null, just an object that's gone!) pointer.
That's strange, as dereferencing an invalid pointer in C++ will not cause anything to be thrown. In the best case you immediately get a segfault; if you're unlucky the code just goes on reading random data from memory and crashing some time after that.
The standard is U.B., so throwing is a valid response.

On Win32, a segfault / access violation - and similar low-level errors, like division by zero - is represented as something called "structured exception". These have a standard OS ABI such that they can be caught, and stack can be unwound, across different languages.

In MSVC, normally, they are not treated as C++ exceptions for the purpose of catching them, although it will still participate in stack unwinding (if something else below is catching them). However, there is an opt-in compiler switch that does make them look like C++ exceptions in a sense that catch(...) will catch them. It's not something you'll see in most code written in the past 15 years or so, for obvious reasons.

I've seen one dev manager (out of many) who failed code that threw NPEs. Every other one let them slide unabated. They are super common.
A lot of those NullExceptions (and a lot of boilerplate code) could be linted away if Java gave compiler-level semantics to @NotNull annotations.

Letting null values propagate through your program is frequently a code smell, you want to constrain it at the boundaries of your program logic if possible. Unfortunately, Java does not have a semantic way to do this, just documentation and discipline.

Checker framework [1] can be used to add compile time checks on such annotations. Unfortunately I found it non trivial to setup (that was some time ago though, I'll have to check again). I noticed also that a recent version of Wildfly application server can do those check at runtime, but I haven't investigated it enough to see if it is configurable and the performance impact of the checks [1] https://checkerframework.org
Well there is an JVM optimization (OmitStackTraceInFastThrow, which is on by default) which optimizes "built in" exceptions that are thrown frequently enough to be cached and have to stack trace. So somebody at one point thought that was a worthwhile optimization and allocated engineering resources to it.

https://www.oracle.com/technetwork/java/javase/relnotes-1391...

In JavaScript I often find my self running the code in both firefox and chrome to get the full picture of what is undefined: (firefox)

    window.foo.bar
    //=> TypeError: window.foo is undefined
and what I'm trying to get: (chrome)

    window.foo.bar
    //=> Uncaught TypeError: Cannot read property 'bar' of undefined
Personally I find the firefox error message more useful, but often I get a better understanding of whats wrong when I run it in chrome. I don’t know why they are mentioning undefined at all and don’t say something like:

   TypeError: Cannot read property 'bar' of 'window.foo' (undefined)
---

Edit: Formatting

On Chrome if you click the hyperlink on the far right side of the error it brings you to the line that caused the error. Not as easy as if it were displayed as part of the error but easier than switching between browsers.
This helps but isn't always sufficient, for example in conditionals where comparing property x of two different objects.

I like the solution proposed here.

The worst part is that Firefox tried to change the error message it throws to include the "bar" piece of information, and that had to be reverted because it broke sites that were parsing the exception message with regexps. :( See https://bugzilla.mozilla.org/show_bug.cgi?id=1498257 for the gory details, though it was not the only site affected: see also https://bugzilla.mozilla.org/show_bug.cgi?id=1490772 (fixed by site author) and https://bugzilla.mozilla.org/show_bug.cgi?id=1512401 (fixed by the backout).
Anyone know what the future of this is?

Sounds like they were considering trying to get sites to fix this and then redo the change, but it's not clear what happened to that in the last 5 months.

Seems to me that a developer relying on behavior like that should be left to fend for themselves. It's not like the behavior is already consistent between implementations of ES5, or a part of its spec.
In this case "left to fend for themselves" manifests as "users end up with a non-working (on some sites) browser, switch browsers". So it's not really an option, unless the browser involved is a monopoly and the site involved has a tiny audience.
Wow. One would generally not consider exceptions part of a public API but I suppose they really are; if one rewrites it as a sort of Haskell-ish `Input -> Either Error value` one can see this directly... then if one church-encodes one also gets the way JS promises work, `Input -> forall z. (Error -> z, value -> z) -> z`...

I suppose that the "normal" response is just to say "yeah people will write bad code in any language," but maybe there was something we missed out that could have signaled to someone that this was a very bad thing they were doing? It's something that could easily happen to a junior developer. So if we had the language of our dreams, is there some other option?

Erlang's "just let it crash, do not attempt recovery but rather monitoring" comes somewhat to mind. Standardizing the error message also comes to mind, but seems a bit banal. Error codes are handy but with C I have noticed that every single piece of software has some different idea of which integers mean what things, so that developers have to absorb a hodgepodge.

Really it's I guess that you don't want any sort of conditional dispatch to happen on such a value? You would like the ability in an hypothetical programming language to mark a field as "advisory", and then the programming language checks that it never performs any sort of conditional dispatch on an advisory field. That gives you a way to make the API contract, "this contains a string but I might change that string at any later date." So if a junior dev runs into that, they are intimidated and get that sort of instant feedback they need.

> Wow. One would generally not consider exceptions part of a public API

Interestingly this happened with Java exception stack traces too. Joshua Bloch remarked that one needs to provide access to all info through standard API, if it's not there people start using toString and others and it bacames a de facto API.

Even then exceptions are likely being logged and the 'API' may be getting consumed by other processes, reporting scripts, log formatters, email rules and god knows what else. Changing an exception string can ruin a lot of peoples day.
Yes, having a way to prevent conditional dispatch on known-likely-to-change values would be good for cases like this. Adding such a language feature to JS specifically is pretty unlikely, of course. :)

Also, you may still want dispatch on "what sort of exception was this?" so then you have to provide an alternate mechanism for that...

  Every change breaks someone's workflow. [1]

  With a sufficient number of users of an API,
  it does not matter what you promise in the contract:
  all observable behaviors of your system
  will be depended on by somebody. [2]
[1] https://xkcd.com/1172/

[2] http://www.hyrumslaw.com/

Even if people rely on implementation details it can be still ok to change them. Even java, not generally known for breaking APIs, did that several times in its history.
Yes, it can, but potential breakage must be taken into account. Windows is probably the ultimate example of backward compatibility including many arguably buggy or incorrect behaviors that clients have became dependent on.
You could do this reasonably straightforwardly within an existing programming language: make a string wrapper type that had the standard I/O functionality as a "friend" or equivalent, so that printing the string to standard error prints its actual contents, but any other use turns into "The value of this string is not part of a stable API." Or more simply, provide an exception type such that getting its string description isn't a public function, but if it's uncaught from main(), whatever in the language runtime invokes main() is allowed to get the description.

That said, it sounds like the "real" problem here is that there's no programmatic way to investigate details an exception, and the answer should likely be to make an extensible structure (to which you can add new fields later) that contains the meaningful information in the exception string.... Ideally the exception string should be a static, probably translatable template string that you can use with the language's printf equivalent to fill in the per-exception data in the structured fields.

I suppose one solution would be to add a secondary error message that is mozilla specific and print that error as well when the exception is uncaught?
This is the same exact reason why PHP double colon errors still include the `T_PAAMAYIM_NEKUDOTAYIM` string somewhere in the error, as various error parsers still look for it even though the error messages have been updated to be clearer. The PHP team can't remove it or it will break tons of error parsers.

https://philsturgeon.uk/php/2013/09/09/t-paamayim-nekudotayi...

    TypeError: Cannot read property 'bar' of undefined 'window.foo'
When Firefox tried to make the "X is undefined" JavaScript exception message user-friendlier, it broke flipkart.com. The website's JavaScript explicitly depended on the exact wording of exception messages. Simply loading the flipkart.com home page caused "X is undefined" exceptions, which it tried to parse with regular expressions. The new exception message had to be reverted. :(

https://www.fxsitecompat.com/en-CA/docs/2018/improved-javasc...

This is an unfortunate example of Hyrum's Law: "With a sufficient number of users of an API, all observable behaviors of your system will be depended on by somebody."

http://www.hyrumslaw.com/

Why in the world was Flipkart regexing for an "X is undefined" message? That seems as brittle as possible.
Why in the world did Firefox care enough to revert the behaviour, rather than telling them to go to hell?
Because it is the right thing to do. First, you shouldn’t brake the web. Second, you shouldn’t be mean to people. Some people do stupid things (perhaps they don’t know any better), accommodating for them and helping them do the right thing, is a far better option, then to ignore and shame them.
I would argue that coddling FlipKart's engineers is NOT the right thing to do. Educate them on why what they did was wrong, help them fix it, but don't punish the rest of the world for one company's mistakes.

Similarly, I would argue that by relying on undefined behavior it was the FlipKart engineers breaking the web.

People who would make that decision don’t end up making decisions at Firefox. Browser behavior consists mostly of arbitrary decisions which have been codified into rules to keep things from breaking.
Because Flipkart is one of the biggest sites in India. Firefox devs didn't want to ruin the day-to-day web browsing experience of so many people, so they reverted the change. They preserved the user experience at the expense of the developer experience, and I think that was the right call. Doesn't matter that it was Flipkart's fault.
Because:

1) It's not just Flipkart. It's a library that was being used on multiple sites. For example, lego.com was broken too, for the same reason.

2) Firefox can't tell Flipkart to go to hell per se. All it can do is tell its users to go to hell and not use Flipkart (or lego.com), or at least not use them in Firefox. Phrased that way, I think it's clearer why Firefox cared enough...

> When Firefox tried to make the "X is undefined" JavaScript exception message user-friendlier, it broke flipkart.com. The new exception message had to be reverted.

I am Jack's complete lack of surprise.

>all observable behaviors of your system will be depended on by somebody.

And by all means, break them. They're undefined behaviour.

(comment deleted)
Software relying on undefined behavior like that deserves to be broken and made fun of. I fully support breaking it and the idea that they had to revert the error message change horrifies me.

This is why we can't have nice things.

I wouldn't be so quick to jump to that conclusion. Was there a better, more defined way to catch that error type globally? if there is not, that is not the site creators fault, that is on mozilla
(comment deleted)
Not sure I follow, if your code throws an exception, you should be able to code against why that exception would have been able to occur, not against "the toString() output of the exception", especially in a landscape with multiple browsers that all have different exception text.
Unfortunately sometimes software or APIs don't give codified error messages or exceptions and the _only way_ to detect the cause of the error programmatically is to scrape the error message, as horrible as it is.
But if you do that, you can easily wrap it in try-catch, to make it robust to wording changes, not breaking the site.
I thought about this, but I think in the general case it's not always obvious whether the lesser of two evils is to risk getting false positives (by being too "robust to wording changes") or false negatives by being too strict. Either way, my point really is that I'm not about to condemn error message scraping wholesale.
Maybe the site deserves that, but users would just observe that Firefox doesn't work with the site and switch to Chrome.
Ah yes the undoable task of fixing it on there site.

How long would have taken to fix? How much time would we all now safe?

Firefox is not in a position these days where they can force other people to fix their websites.
Their code explicitly depended on the wording of exception messages because they were using a dynamic language. Java is a typed language, where we can match on exception types:

  if (ex instanceof NullPointerException)
  catch (NullPointerException ex) {
It is therefore rare for Java programmers to match on exception messages (it would be a mistake).
Funny thing, some JVM's already do exactly this for many years:

Failed to write HTTP message: org.springframework.http.converter.HttpMessageNotWritableException: Could not write JSON: while trying to invoke the method de.hybris.platform.catalog.model.CatalogModel.getId() of a null object returned from de.hybris.platform.catalog.model.CatalogVersionModel.getCatalog();

It would be useful to know what JVM is that.
Probably SAP's, since Hybris is an SAP product.
SAP's definitely does this, I just checked.

In fact I relied on it so much that I thought every JVM does this.

this is a json serializer, =NOTHING= to do with compiled dererences.
Better idea:

  Optional.ofNullable(foo).orElse(new Foo())
In kotlin: `foo ?: Foo()`

Although I'd rather creating objects if not needed to be easier on memory.

In Java that's

  Optional.ofNullable(foo).orElseGet(Foo::new)
to avoid side effects from building a throwaway object.
But they are still calling it NullPointerException, not NullReferenceException. I guess this change is too invasive
Only a little: it would be a backwards incompatible change affecting pretty much all production java code ever written, requiring everyone on the planet spend time and money on uplifting all their code for what is really just a cosmetic change =)