To expand on this for @causi: The largest thing is the earth's atmosphere, because changes in the ionosphere (and a lesser degree, the troposphere) cause the signals to get refracted differently from moment to moment. Part of the data coming from the constellation includes general ionospheric correction data (and WAAS can provide data that covers smaller (but still kinda large) areas, which allows the receiver to adjust for those refractions to some degree, but there's only so much it can do -- conditions change quickly, and in areas smaller than the correction data can accommodate, so there's still variance that can't be completely removed -- it's not a lot, but at one foot per nanosecond of delay, it can still make a significant difference.

At least, that was generally the case until recently. There is actually a way to almost completely remove the effects of both the ionosphere and troposphere: Those things affect signals differently if you have multiple signals at different frequencies. And, as it happens, GPS does actually have multiple frequencies (L1 and L2, and now L5 as well), and for a fairly long time there have been receivers that could listen to signals from the same satellite but different frequencies, and based on the delay difference between those signals, know exactly what the atmosphere was doing at that exact moment, and dial out the influence of the atmosphere almost completely. You can start getting pretty durned precise once that's not a factor.

The problem is that multi-frequency receivers used to be expensive. Like "started at $10k for the cheap stuff" expensive, even within the last decade. Only in the last few years have inexpensive (under $100) chips become available for doing multi-frequency GNSS. And those can get down into the "under 1 meter in realtime" range trivially, and better than that for a fixed-location station. Phones are getting these now, so things should start getting more accurate, though not that much more accurate.

Multipath is also a big problem in "the urban jungle", but chips are getting better at discriminating, and unless you're just utterly surrounded by skyscrapers, usually isn't too big a deal.

Orbital calculations are also another cause of loss of precision -- the orbits are calculated pretty precisely, but for various reasons the ephemeris data sent down from the satellites doesn't actually represent exact orbital data, but represent data that's "good enough" over the couple of hours the ephemeris data is valid. This can be worked around with patience -- there are ground stations around the world with exactly surveyed locations, which monitor the satellites and calculate the exact orbital paths the satellites actually took, and publish that data (though it takes several weeks to get the "final" data). A typical surveying technique is to record several hours of data from an antenna at a survey location, and then when the precision orbit data is published, post-process that recorded data to remove both orbital and ionospheric effects. This can get you down into the sub-cm range, with enough care.

And then there's also a range of other factors, like solid earth tides, which cause the land masses of the earth to rise and fall by up to a meter(!!!) over time, and when you're trying to figure out exactly where a given point in space is on this big rock ball, that matters!

But, yeah, pretty much it's atmosphere, unaddressed multipath, and orbit precision that makes the difference, and the above is how those are usually dealt with.

</ramble>