> The Data Science team didn’t go to the lengths necessary to support their models. That was their failing. However, the business is ultimately responsible for betting the farm without demanding a real world track record to support those models.
"Well, sure, them folks were (wrong | lying), but its really your fault for listening to them! It's not like making that call was their actual job or anything, ... oh."
I'd say the lesson here is that horseshit is still horseshit even after it has been extensively processed by the finest machines and learning we can buy.
Just imagine being the poor sod there who got a model to say "Bail the fak OUT of this market" 3 months ago: think anyone would have listened?
Maybe they would have listened if the model successfully used historical data to predict when people should have bailed in the past, in addition to now. That is, if it was a properly validated model. I am not an expert, but this seems like a reasonable way to see if a model works.
One of the harder lessons of life is that occasionally people get wedged into ideological positions where they want comforting lies that confirm their existing beliefs. These situations often are very dangerous for companies, because it creates incredible pressure on lower level people to produce dubious models, and on middle managers to accept dubiously supported models.
As far as who is culpable for those situations … it’s not clear to me.
This post strikes me as empty bluster. It basically says "the data scientists didn't consider the possibility that the model might fail to predict the real world." Which is like looking at the Boeing 737 MAX debacle and saying "they didn't consider the possibility that the plane might not be stable". This is most likely both untrue and lacking insight. What almost certainly happened was they validated their model extensively, but overlooked the angle relevant to this use case. Or overexcited executives failed to heed the caveats of the nerds.
> When I advise Data Scientists to understand Topology and Differential Geometry, this is why.
Oh really? I've taken grad-level courses in those fields. It's nice general education but has nothing to do with this situation. Knowing about the Lucas critique from economics would be much more relevant.
My pet theory -- completely unvalidated but at least potentially insightful -- is that they failed to put in adequate margin for adverse selection: homeowners whose homes are worth less than they seem on paper were much more likely to take the Zillow offer. Adverse selection turns prediction variance, which you might naively expect to "average out" when using a well-calibrated model at scale, into a bias towards underperformance.
Another way to put it: asymmetric information. Be very careful betting against homeowners who probably know much more than you do.
I totally agree, this is more of a case of simple over-confidence and exessive risk taking than lack of mathematical understanding. Regardless of the quality of the model you are using there are still countless other factors that may impact your prediction quality. To me, the only way to mitigate this is to take it slow and plan ways for failure correction. Zillow is an example of greed and impatiance winning over common sense and risk-mitigation.
If anything this could have been because of _too much_ higher level statistics and math, as those kinds of things can often shroud underlying problems.
> What almost certainly happened was they validated their model extensively, but overlooked the angle relevant to this use case.
To me, this is a systemic organizational problem in ML, rather than a model problem. Or essentially, pre-Agile development all over again.
Stop me if this sequence sounds familiar: (1) attempt to capture all the details in a nuanced domain that's foreign to you, from people who are experts but not expert developers, (2) develop code from that spec, (3) deliver code to the experts as a black box that they can't opine on, (4) cross fingers and run in production.
Which isn't to say this is the way everyone does it, but is to say it will be the way the majority does it, just as the majority of development is still waterfall under the hood.
But the mismatch is a mismatch in understanding, not in correctness. The experts can't offer meaningful feedback to the technological black box, and developers don't have enough time to spec a lifetime of subject matter experience.
To run a market making business I have to know how to hedge my net position and manage my cost of carry.
I find it crazy hard to believe that they just went long a few billion in houses just based on a model of price and failed to hedge vs. the secular market and didn’t model time to sale. But if you believe the popular press maybe that’s what they did.
I don’t know of anything exchange traded or terribly liquid, but if I called Goldman and said I wanted a one or two billion dollar swap on the Case-Schiller Home Price index I’m sure they could work up a quote for me.
options on anything that's correlated with it should do it. (basically if "RE is up" so should builders, suppliers, big corporate lenders, mortgage processors -- so if they want to hedge against RE going down they could have bought options that pay when those go down)
You are right about this anti-pattern, but it's not the way most ML models are operated - especially in trading operations. The following principles are well known:
- enlist domain expertise
- validate incrementally
- expect adverse selection
- expect failures to generalise
Many, many production ML teams operate in this framework. The fact that Zillow either cut corners or hit a bigger snag suggests there is a more interesting story we haven't heard yet, not that they should have consulted OP or a few hn readers.
Trading models are a very specific and exceptional (in multiple senses) subset of ML use.
A given business probably doesn't employ more than a few people of that class and rigorousness: not least, because those people have the option of making a lot more money at a trading firm.
So yes, all businesses should do all those things. And yes, the best run places do. But it feels like there's going to be a decade or two where employee supply & expertise lags corporate demand, with bad results.
If you don't have the minimum skills on the team, you're going to fail in ways that you don't see coming, the unknown unknowns. To spend billions when a former Zillow employee says that they have no finance people on the team, is just, drastically, wow. I would much sooner believe he is mistaken then believe that is what actually happened.
I think John cook has hit on a brilliant marketing strategy : write mostly about basic, upper-division probability and statistical concepts (e.g., the tail behavior of the gamma distribution, or a list of conjugate distributions), interspersed with the occasional non-controversial polemic (“don’t invert that matrix”). To the uninformed—including his potential clients—he looks like some sort of genius, and for the rest of us, it’s a great reference.
I think they failed to consider adverse selection. For instance, if I'm selling my home, I will check Zillow instant offer and try the traditional market. The Zillow offer will be my floor plus some value for convenience. So they will likely be overbidding.
The other thing is that running a website and being a data aggregator is an entirely different business than speculating on houses. Matt Levine pointed this out last week [0], but its not making markets if the turnover takes 6 months.
> Knowing about the Lucas critique from economics would be much more relevant
“The Market for Lemons: Quality Uncertainty and the Market Mechanism” by George Akerlof is also a nice related resource [1]
> widely-cited[1] 1970 paper by economist George Akerlof which examines how the quality of goods traded in a market can degrade in the presence of information asymmetry between buyers and sellers, leaving only "lemons" behind.
I'm a quant trader and I agree. There's a big difference between being able to predict what will happen, and having a sensible strategy for how to behave.
One notable place this happens is backtesting. Particularly for passive strategies, you don't actually know from historical data what would have happened if you had been in the market at a given time. Would your prices move the market? Maybe. Would you get filled disproportionately when it's bad to get filled? Fair chance of it. So don't act like your presence would not be felt.
Another angle is risk management. You may have high confidence in your model, but you should have a plan for what to do if things don't go as expected. It seems crazy to me that it could get so far out of hand that a quarter of the company gets sacked. Why not set a budget and dip your toes? Then you can watch out for the adverse selection issues as you grow the portfolio and make adjustments.
Finally, it's very normal for quant strategies to look good when they're first proposed, then look a bit worse when traded in real life, and perhaps then degrade over time. Another reason to take it easy when sizing the bets.
You're right about Lucas critique. It's fundamental that models affect the thing they're modelling.
The big difference here is that the counterparty is deciding which orders get filled. Zillow could be right 99% of the time, but that 1% of the time where they're overpaying are the transactions that for sure will go through.
And indeed if you go to /r/realestate anecdotally that is what happened. People would get offers from Zillow/Opendoor etc and talk to agents to see what they would list it at. If Zillow was higher than what the agents said they went with Zillow. If not they'd list it and sell it the old fashion way.
Yeah, it's remarkable. You'd think that "I'll buy your house immediately" should be at a worse price than the guy who insists on the old school inspection and due diligence.
In fact it might not be a bad strategy to simply start with that... see if there's desperate people needing the money fast, and use your corporate heft to both sweep up the deal (brand) and patiently dump it back into the ordinary market, pocketing the spread as a liquidity premium.
One thing I remember my first boss saying is that it's hard to get a good trade on. Attractive deals have several interested parties. If you're getting a lot of trades, they might not be good ones.
Zillow's ambitions were bigger than that. They wanted to get enough inventory so that they could capture the 6% of the transaction going to agents. Plus a chunk of the miscellaneous fees that go to Mortgage brokers, inspectors, appraisers, and so on. For that they needed volume and the way to get volume in real estate is pay more than other people.
I find it hard to believe that everyone there didn't realize that they were burning money. I saw it personally when looking to buy a house a couple years ago. There were a couple Zillow flips out there where the numbers obviously didn't add up. Whether or not they truly believed that it would work eventually or if it was just a way towards higher compensation/promotions/stock price or whatever is anyone's guess. But I suspect pretty much everyone involved from the person banging out ML models up to the CEO knew what was going on.
One thing I thought about is alternative scenarios. If the house market had rocketed up for whatever reason, Zillow could have made a pile of cash without having a sensible strategy.
Then the articles would be talking about what geniuses they were and ML trampling traditional real estate.
They would double down and when the market calmed, all the same issues would reappear. Perhaps that's actually what happened, because the house market has been buoyant recently, and they seem to have had the iBuyer business for a few years?
I'm a former quant trader as well and this article rang true to me.
I've certainly found myself in situations where I developed a model that performed so well in its specific domain that executives wanted to roll it out to a a much broader domain despite my advice.
It very much is a sign of immaturity and lack of discipline from leadership. It's too easy to buy into the hype thinking you are being supportive of AI in your org but not realize that truly being supporting an AI strategy involves a lot of nuanced skepticism.
Oh man, your comment about the need for nuanced skepticism reminded me of the co-CEO of Waymo who publicly claimed he's willing to trust his kids' lives to a driverless car.
"A" driverless car or Waymo's driverless cars? From the outside looking in, it appears Waymo's approach has a lot more self-restraint and is a lot less cavalier about safety compared to the competition.
What if the model said there were only $100m in homes that were viable, but executives wanted $1 billion in purchases? They could have had an alright model, but the volume wouldn't sustain the purchasing quantities leadership wanted. So, they relax the model to accept lower quality results. It's another form of executives attempting to squeeze blood from a stone. It just doesn't work, but it's a common strategy. They'll ask simple questions like "Why can't you just do X?", you advise them not to, and they ignore you.
> overexcited executives failed to heed the caveats of the nerd
Yes, the nerds are so well-known to consider the risks of what they create. If only they’d thought of the risk that someone could use their work in the real world, even though it came with specific instructions not to do that. Maybe they wouldn’t create possibly-destructive ML models then.
Alas, it remains a mystery what a private company might do after paying a lot of money for something. Now where was I? Ah, right! Predicting criminality from baby photos! I love how DARPA sponsors basic research with absolutely no expectation of ever using it. Although I‘d better name it README.MD, just to make sure they do not forget to read it.
Technology has dual uses. Kitchen knives can be used to kill people, so should they never have been invented?
You can take this line of reasoning very, very far. Some people think we never should have left our hunter-gatherer lifestyle. To these people, the agricultural revolution was a disaster and things have only gotten worse from there.
It's a tremendously ignorant viewpoint IMO, but it's where you land if you believe technology with the potential for bad use shouldn't exist in general.
Yes, at every startup I was in, the stats nerds were in fact well-known to tell the nontechnical people and even the devs that what they were doing wasn't going to work, or rather that we didn't have good evidence that it would work. This is such a well-known feature of stats nerds that it's a meme.
As someone who doesn't work at Zillow but follows the space a little closer, don't buy into the superficial explanation/meme going around that they failed because of ML.
My advice would be to read the earnings call transcripts over the last couple of years. What Zillow has communicated in the last earnings call is fairly contradictory to what they've been saying in previous calls.
Some bullet points on my understanding of what went wrong:
-They were paying way too much for too many homes. I believe this is ultimately due to internal incentives and ambitious goal setting. IIRC, they mentioned they had to alter their model output to be able to give themselves the permission to keep buying at high volumes. They lacked fiscal discipline here.
-The holding times on these homes were too long, which increased their costs. My impression is that this was due to lack of operational experience with renovating and selling homes. Zillow was new to this, and they were trying to scale up very quickly, independent of whether they were efficient.
-The core business model of Zillow historically has been as an advertising site/lead gen site. This business model is not in their DNA and their operational expertise. The negative margin pressure from this business in tougher times can really stress the overall business at times.
Look at the other iBuyers in the space - they are still in the space. They just aren't as naive as Zillow was. Zillow's behavior here is really confusing and speaks to poor operations/strategy/discipline. Housing prices have been mostly been only going up in this market and Zillow was losing their shirt.
Well, their strategy of blaming the nerds will at least ensure that they never have the opportunity to bet big on ML again. No one competent will want to work for them now.
Zillow has always paid low compared to the market. They never got the “top nerds” and this was really a CEO problem. I would never bet on that company on any decently complex technology like predictive models. It took them years to make their model decent, something that a machine learning tutorial does better - just a linear regression on square feet and location. They were literally comparing my house to ones on the other side of the city - comparing a downtown property with outskirts.
To be honest linear regression on square feet and location is what potential (human) buyers would mentally use, therefore it's a very, very accurate prediction of the price.
In my area at least Zillow valuations really suck outside of subdivisions. They don’t do some fancy ML, just apply a factor with a cap to tax assessment based on comparable sales and listings.
It mis-values houses near me by 20-40%. Other tools are much more accurate.
Their CEO also seems to have fundamentally misunderstood what he was doing. He described their activity as market making. That’s wrong.
Market makers attempt to buy and sell instantaneously without changing the asset. Zillow had no intention of simultaneous purchase and sale. They also modified the asset. They used a market making model glued to a real estate speculating one to run a house-flipping operation at scale and from a distance.
Instead of focussing on sellers who would sell at a discount for immediacy, they paid over market. There was no effort put into pre-selling or hedging. (The latter would have been so possible as a security against their entire portfolio.)
A market making model would actually be interesting. Setup a potential seller with a buyer beforehand. Then when the seller buys the new home both transactions occur together and you get 2 sales instead of one
> is that they failed to put in adequate margin for adverse selection. Adverse selection turns prediction variance, which you might naively expect to "average out" when using a well-calibrated model at scale, into a bias towards underperformance.
Presumably, they trusted that their model does a more accurate assessment of price and future potential than a person themselves. If you had an excellent model, why _not_ would it balance out?
My theory is that most people would consult various sources and professionals-who-assess-homes (whatever they are called) and this the likelihood of Zillow having a better assessment than the average person is just low.
It might and it might not. Many things have rolled down the pike that in normal markets would cause reevaluation and correction downwards. But so far they’ve all been pushed past.
My professors used to say “when the last stalwart holdout is sold on the premise of the irrational factors holding the market up the correction is near.” (Or something like that.)
I seem to recall it being Enron filing bankruptcy that really kicked off the Great Recession.
Oh further, anytime you have your market appraiser becoming a market participant their valuation becomes highly unreliable. This is another thing that’s just not done if you want to have a reliable indicator of an assets value and sell it knowing whether you’re getting a deal you’ll be comfortable with long term or not.
It’s really not clear to me if we’re in a bubble. A lot of these houses sold to investors are aiming to generate rent, not be resold at a higher price. Zillow appears to be relatively alone among institutional investors in aiming to flip homes and not rent them.
If corporate land lords are extracting enough rent to justify the higher than normal prices, is that really a bubble? It’s socially pernicious, yes, but that doesn’t sound like an asset bubble to me.
Or, more simply, this business only works if prices are going up, and home prices aren't going up. Zillow saw the house price bust coming, and got out.
1. The business decision to go into the 'flipping homes' business.
2. The capabilities of the ML/AI to help optimize that business decision.
The post does a nice job outlining that #2 will have great variance across businesses, use cases, time, etc.
The biz decision is maybe more interesting and instructive. A wrong biz decision can doom the ML/AI out of the blocks - the ML/AI is often a tool to do a job, but there is a business decision to do the job to begin with.
Is it even possible to predict things whose predictions influence the events that are supposed to be predicted?
Suppose you did have a perfect way to appraise houses by some arbitrary criteria. If the houses are "perfectly" appraised, that would change the values of the houses themselves, no?
e.g. magical system predicts 3b/2bath house should cost 500,000. People know the magical system's price is the perfect one. someone who has more money to burn can acknowledge this and still pay even more money, knowing most will not want to "overpay" for a house. house sells for 550,000. prediction is wrong in the end anyway.
schrodinger's predictor - it's accurate right until it's observed.
Nitpicking here, but it is my understanding that there isn't even a consensus about the existence (or not) of an "ideal price" in economics. Just to make it clear, I'm not an economics major nor do I work in the field, this opinion is just based on my amateurish readings.
Predictions like this work on aggregates, not individual cases. There are also sellers who are in a hurry to move and will settle for $450,000. Magical system is wrong in both cases but right in the aggregate.
They don't need to predict the actual sale price of every given property perfectly to make money. They need to be able to say that with 99% probability, we will find at least one offer for this house >= 500,000. The errant buyer with loads of cash is free to come in and offer more. As long as they turn a sufficient profit at the 500,000 price point they should be set. In competitive markets, for people familiar with local dynamics, this is very much possible in a short-term time window. The question is doing this at scale in a fully automated way for a much longer time horizon. Zillow tried and failed.
There was a post on Blind a few days ago from somebody who works in Zillow, stating that management disregarded models and decided to go instead with "intuition"
Workplace chat app, you use your work email to sign in and chatter anonymously with your coworkers and others in the industry. You are ID'ed only by the company you work for.
That post tracks with some of my experiences working in forecasting. Senior leadership loves the DS team until they are told something may not work how they want or reaching their quarterly KPI (which is often tied to their bonus) is infeasible/a bad idea.
This is a generalization, and has happened a couple times in my career. I feel lucky now to be at an organization where this is no longer the case.
It's difficult reading articles without any Headings!
Data mining is common in finance. Finance PhDs are constantly identifying new "factors" (e.g. value, momentum, price/book) based on historical data that don't pan out as time goes on. See here for some academic research: https://www.researchaffiliates.com/publications/articles/710...
Robert Shiller has does extensive research on how psychology impacts real estate prices. Monetary policy isn't standard right now. Machine learning isn't always a good tool for making predictions. Look at Shiller's interviews and surveys for predicting home prices. He doesn't use machine learning or advanced analytics.
Personal example. I bought a condo in the University District in Seattle back in 2017. It's a 92-year-old, 478 sq ft condo. I paid $279,000 (yikes, I know)...Zillow's estimate? $500,000.
I have no idea how they were so far off. Even right now:
Keep in mind that Zillow's HQ is also in Seattle. While we may be protective of the data scientists on HN, this is one case where I agree, there's something wrong with their models. The "similar recently sold" condos that Zillow shows are indeed $400K, but those are at most 50 years old and more square footage. I think a simple logistic regression of square footage and age would give a better prediction than whatever model they are using.
From personal experience, two major contributors to the Zestimate are:
1. The town/city’s assessment of the home’s value for tax purposes
2. Recent sale prices of homes nearby, mainly as the per square foot price.
Another thing I don't understand. How do you develop a model with VOLUMES of real world data to train it on, and not just check how well it predicted pricing every month, both in aggregate (trending prices) and on individual listings that sold that month? This seems like a very small, closed loop to me. You could tweak the heck out of that algorithm in short order based on WHAT ACTUALLY HAPPENED.
> what makes you think they don't check the model on WHAT ACTUALLY HAPPENED?
As I said, it seems like a pretty tight feedback loop to check, and then tweak the model. So it's confusing to me how they could be far enough off to ruin the business plan.
I'll put it back to you: What am I missing? How could they revise the model based on many months of real-world data, and yet get it wrong enough to nearly destroy the business?
Good question. I don’t know. There are a lot of possible interacting causes. Let’s itemize a few.
So there is not checking. This seems unlikely given, as you point out, it’s data right at hand and, as I point out, they publish checks. In general, I’m suspicious of WOW PEOPLE ARE STUPID explanations.
There is checking and mistranslating model accuracy into business outcomes. The model is 90% accurate. Bid for 1000 houses. Maybe mainly people in the wrong by overpricing segment take you up on it. See the vast literature on adverse selection and winner’s curse.
There is tweaking the model because you want the a certain outcome.
> Over the last few weeks, it’s become clear that Zillow made a lot of mistakes. It tweaked its algorithm to be more aggressive with offers, winning bidding wars just as the market was starting to cool and overpaying for properties [1]
The CEO has commented that the volatility of the prediction was a problem. He mentioned that the unit economics swung from over plus 5% to minus 7%. So the model might be a accurate on average but too noisy for their purpose. [2]
Point is it’s not at all obvious that the problem here was accuracy on predictions of house prices.
They became part of the feedback in their model. Real estate markets are high value with limited inventory, if you add market maker (zilllow) they set the floor. They had too much influence on the price, and didn’t just provide liquidity. Additionally these transactions take time to settle, and Zillow tried to improve their profit buy improving adding additional risk by increasing holding time.
Finally im not sure how they hedged the risk. How do you short houses? If you consider mortgages like you holding 20% to protect the bank from losing, the variability can be difficult up to 20%, and leaves you with a hell of a downside.
My neighborhood is a mix of single family homes, condos, and apartments. I live in a single family home. Zillow regularly shows nearby condos as comps - well the condos have always sold for less than the single family homes. The condos also have a high HOA that does not show up in the price. Zillow does not know that there are different kinds of buyers for condos vs single family homes, even when they are close by and a similar size.
My neighbor's house is the mirror image of my own, same size, same layout, same size lot, right next door. Zillow shows my home currently at $226K more than the one next door that is the same thing. Zillow's Zestimate of my home is more than a recent appraisal - which is odd because until 2021 Zillow had a really low Zestimate for my home, like the house next door. No idea why Zillow decided on a much higher value recently for me but not for my neighbor.
And I'll have to say that lenders do consider the Zestimate as part of a loan, especially in a case where needing an appraisal might not be needed - if it on the bubble then the Zestimate might be the difference of whether you get an appraisal or not.
Right. I read their current actions as the result of a liquidity crunch. Something forced them to realize a 25% reduction in labor and other cuts. They also completely ended this home buying program. They’re hoping they’ve cut enough that when they sell their properties they have enough funds to keep them afloat. If not, chapter 11 filings are not far behind.
Just some quick figures. They purchased 15k homes in Q2 alone. Debt on 15,000 homes with average purchase price of 350k (as a ballpark figure) comes to 5.25B.
There’s a lot of information we don’t have but them laying off huge percentages of staff is a scary indicator. Edit-also note their market cap is about 17B.
This might be where google got that figure. Again, as I state above, I was using rough figures to get an idea of the magnitude of the funds they were devoting to their flipping/analysis unit. It looks like they were doing thousands of units instead of tens of thousands with discrepancies among the sources I'm finding in google:
What's not changed is the fact they were leveraging a significant portion of their company's market cap into this segment and it resulted, so far, in them laying off a quarter of their labor and completely terminating the program - amongst other things. This is not good news for their business and prospects of continuing to be an ongoing concern.
I think part of it has to do with black box models being easier to build incrementally. I’ve seen many teams start off with 3-5 features, which gets you to some threshold for launch (say 90% precision and recall) and get badly burnt on the other 10% post-launch
I think (hope?!) that the Zestimate and the model predictions in their iBuying business are completely separate. The Zestimate is essentially a top-of-funnel strategy to get home owners and potential buyers into their normal business platform.
For my home, the Zestimate is about 8% higher than Redfin's.
One thing that I feel this article misses is the notion of domain expertise. Zillow has entered the house flipping market that has been already populated by human real estate agents and investors, who already do their work of pricing houses and striking deals quite well. To compete, a successful entrant needs to excel in at least one thing and be not much worse in all the others. In practice, that is a tall order.
TL;DR: not taking into account adverse selection - other people's prices & trades aren't random/Gaussian, but adversarial (they're trying to profit at your loss) and better (they probably have better information than you)
Behind every order is a thinking, breathing shark. A shark with mouths to feed and a mortgage to pay. A shark who's likely smarter than you. Definitely more ruthless.
You're swimming in shark-infested waters, treating apex predators like specks of geometric Brownian dust
> "The CEO points directly to the failure of their Data Science team as the root cause. Their investors aren’t buying it and I don’t either. Zillow’s Data Science team is excellent. The business knew the risks and ignored expert advice. The initial rollout should have been far more limited and cautious.
> The Data Science team didn’t go to the lengths necessary to support their models. That was their failing. However, the business is ultimately responsible for betting the farm without demanding a real world track record to support those models."
I'm confused. If the "team" was "excellent" (and he says this multiple times), why didn't they do the extra leg work? It feels like he has a dog in this hunt. Did they use him for external consulting or something?
This author doesn’t have a clue about anything. After the first paragraph, this might as well have been written by GPT-3. Sounds like a consultant trying to cash in on the news of the week to me.
I’d really like to see the author write an explanation as to why. I completed by PhD in “AI” (as it was then called) in 2001. Very little DNN work then, but I haven’t a clue where the connection lies.
> Did they consider that they just hired the people who would tell them what they wanted to hear?
Management everywhere thinks they are getting good info from those below them when everywhere I have worked, they get a selective job preserving subset.
This reflects something I say often as a data scientist: the best data science implementations augment humans.
Domain understanding is key. Integration of the domain into ML models are a key part of generalizability -- far beyond train/validate/test, which are subject to data collection biases and temporal trends.
Zillow had a high, general equilibrium impact on a large market. We saw microcosms of this behavior in the "good old days" of Amazon book-selling when competitors would mark up a books price to one penny more than another, resulting in outrageously priced books valued far below the offering price. No behavioral governors were in place, review was inadequate, and they become part of Data Science lore we tell ourselves as our single threaded scripts running Python payloads warm our homes and melt our s'mores whilst our GPU weeps.
Data Science as a field is distinct from ML in that it uses the tools of ML to augment and enhance digital services. Despite the exuberance of boot camps to market prediction and neural nets as the end-all-be-all solution for hard business problems, value in data science adoption isn't solely in prediction, but in validation. Causal modeling and inference are key in this space, but also simply quantitative understanding of statistics and algorithms. As an example, you need someone in the room who understands multi-armed bandit sometimes. But, more importantly, you'll always need someone in the room who understands how multi-armed bandit would improve a business' value proposition.
Sounds more like classic large organization things.
I bet there were some employees who saw the estimates were untrustable and the entire thing was heading towards failure, but good luck changing the course of a ship controlled by hundreds of people.
Most data scientists are not smart. They are basically pushing a few buttons.
This is true even for “Ivy” trained PhDs. There are may be a handful of people who can do truly profound work today. The rest are only capable of replication ie mouse clicks here and there.
Further, they should be telling people a model is only as good as the data it has as input. If the data only sees upward movement in valuations then it will only predict upwards. A model can’t tell you anything about a situation where the underlying factors are different from its data. Same thing happened with mortgage backed securities in 2007/2008. The math was very complex and still couldn’t see around a corner.
The bottom line is that the domain here is under the umbrella of the social sciences. The phenomena the models describe is historical, and how closely they will match what happens from that point on relies on social stability. In the grand scheme of things, social stability is ephemeral. It isn't rocket science.
I'm not making a joke. Newton published his models in the late 17th century. The models hold no better or worse today than they did then. If you had rockets in 1687, they would work the same way then that they do now. But, social science is nothing like hard science. I worry on some fundamental level, even people who will claim that they "of course" know this in fact forget this. There is hubris here.
I agree with the OP that "it's put up or shut up time for Data Science teams. When the business depends on our work for revenue growth, success is expected. Failure is fatal." Zillow's CEO blamed the Data Science team for the failure of a mission-critical business unit and fired 25% of all employees. The stock lost a third of its market cap in a few days. When revenue growth depends on an ML/AI system, failure is indeed fatal.
However, I disagree with the OP's diagnosis.
The initiative failed because to a close approximation the only homeowners who would routinely accept Zillow's offers were those whose homes could not fetch anywhere near as much via traditional sales channels. Everyone else would rather sell their home via conventional listing. Zillow gave homeowners a free put option: Whenever Zillow's offer was above market, the homeowner would take it; otherwise the homeowner could freely ignore it. The incentives were strongly stacked against Zillow from day one.
Zillow unwittingly launched what I would describe as a seller's market for lemons. Naturally, Zillow ended up buying... a lot of lemons.
I think your basic point is good -- a free put option is always going to be a problem.
However, can you unpack "Everyone else would rather sell their home..."? At the core there are two factors at play in this decision. One is financial; anyone with the luxury of time will compare a realtor's comps with an ibuyer's offer. The second is certainty; an ibuyer offer is guaranteed, and can be executed without having to show a house over the course of 1 to X months.
It seems to me that certainty has value to some sellers in some situations. Right now, the certainty value of the traditional route is pretty good because the market is on fire. That is not always the case everywhere.
It depends mainly on how much more money the homeowners believe they can fetch by listing the home versus selling it right away on the spot.
For example, if you're the homeowner and you think you can get, say, a few extra $100K's by listing your home instead of selling it right away on the spot, you would probably take your changes and list it. Even if you're under pressure to sell quickly, you might still try listing the home for a couple of months to see what happens. Only if you think your home is a lemon will you consider selling to Zillow.
That's why Zillow ended up buying all those lemons.
> Only if you think your home is a lemon will you consider selling to Zillow.
What if Zillow just makes an offer that’s high enough that you think it’s unlikely you’d do better if you went through the hassle of looking for other buyers?
Lets say P(h) is the price that a home fetches after being listed, O(h) is the price that an owner expects to get from selling the home, and Z(h) is the price that Zillow expects to get if they sell the home. An owner will sell to Zillow when O(h) < Z(h). They will not sell when O(h) > Z(h). Zillow's outcome would then be (P(h) - Z(h)) * (Z(h) >= O(h)).
The extremely important thing to note is that this is an asymmetric information game. Therefore O(h) < Z(h) is going to be self-selected such that O(h) < Z(h) because P(h) is closer to O(h) then Z(h) is to P(h). If Z(h) was actually able to be a reliable and better estimate of O(h) then it would mean that P(h) wasn't determined by information available to O(h) that was not also available to P(h). The market would already offer this value to O(h). So O(h) value is going to be reflective of Z(h). Its only if the non-public information changes the value that O(h) != Z(h).
Like the previous posts said, the advantages of selling to Zillow are price certainty and immediacy, so your model isn't quite right. But I agree with the adverse selection problem you highlighted.
Even if Zillow does, they still take a 12% fee. When I was selling my house a year ago in Texas, Zillow's offer is both lower than comp in the area AND has a hidden 12% fee. Did you know they also charge YOU the repair fee?
I had a chance to talk to an inspector who looked at my place who has previously worked with Zillow to inspect houses that they were buying. The gist is that Zillow are getting the worst houses (the ones with trash everywhere, burnt carpet, feces on the wall). For them to get a "normal" house is very rare. Nobody with a normal house (and better) under normal condition (and better) will sell to Zillow. You can get a 1%+3% fee full service (pro pix, pro staging, pro cleaning, ...) any day of the week and still sell your house in mere weeks. I was not even in Austin. I guess if you don't care about making tens of thousands in a month (or hundreds of thousands in nicer cities), then you have bigger things to worry about.
Zillow's offer was already higher than OfferPad and other iBuyer's services. After my experience on the sell side with Zillow, I am incredibly wary of ALL iBuyers on the buy side.
Exactly. And to add to what you say, adversarial problems in data science are much harder than the regular ones which data scientists deal with and the hubris that I often see with data scientists won't cut it here. You need a different mindset to tackle these problems and they must be handled incrementally with the real world in the loop.
Even CarMax and Carvana with commodity products in the traditional lemon market have to work around this problem by adding an inspection to confirm the data- and they have a large spread between their buying price and their selling price.
> large spread between their buying price and their selling price
This is the market I was amazed Zillow didn't get into: the "WE BUY UR HOUSE 4 CASH" market.
I'm guessing the volume is too low (few people sell their house for stupid low offers) and advertising costs are too high (those people probably aren't obsessively checking Zillow for their home's value every day).
But still, it seems like easy money, that someone far less knowledgeable and talented than them is currently scooping up, that would better insulate them against market turns.
And don't forget flipping a car is a lot faster and easier than flipping a house at much lower capital. This makes each cycle of trial and error a lot cheaper and a lot more frequent.
Also there is something to be said for a market where the customer brings the product to your location for inspection, and whose product is considerable faster to inspect more rigorously than a house.
Also the products within that market are much more uniform and share several unifying characteristic across even different makes and model so its easier to learn a good and accurate pricing function.
That gives them a lot more cycles to improve their algorithms (both human and machine based).
Plus when CarMax were starting out they likely did so with a bevy of folks already experienced and profitable at buying and selling used cars at a profit who had the systematic knowledge of how to do so in their heads.
It then was simply a matter of building up customer goodwill by focusing on another metric customers cared about, taking a perhaps slight loss to avoid the risk of being considerably ripped off by a savvy local dealer and saving oneself the anxiety induced by high pressure sales tactics.
Having said that I think Zillow could still be quite successful in this endeavor however from my experience they didn't deploy their trading algorithm in a prudent way, they massively over-traded an untested by the real world model, they trusted their paper trading (backtesting) way to much, and they didnt focus on the nuts and bolts of building a successful market making system: making actual money over and over at small size and ramping up size as your empirically demonstrated pnl grows.
Also putting trading aside they didn't have to be a flipper, they would probably naturally be more fit as an exchange+ or intermediary that provides exchange type backing and reduces the friction in buyers finding sellers, taking on risk by only making an offer when they have a bona fide buyer for fixed or % fee, but glossing over all the hassle of two people who dont know each other making a large trade. Offering services like if you dont like the house you just bought through us we will help your sell it at a cost proportional to renting no questions asked.
Carmax and Carvana also provide a service. They bring a bunch of cars to one place that you can peruse and buy. That's a better experience than trying to connect with a bunch of randos on craigslist.
Zillow hasn't gotten to the point where they provide that, but theoretically they could. You could drive around with a Zillow agent and see their 50 homes for sale a lot quicker than coordinating with 50 random sellers.
The broader point is... Zillow's alpha should not be valuation, for all the reasons you listed.
It should be geospatial trends.
There's a fixed margin on any given home, caused by the inefficiency of real estate markets. But many actors (the seller, other agents) have an inkling of what that margin and the numbers on each side of the sale are.
What they do not have as good of a bead on, and Zillow is best placed to know, is high vs low neighborhood boundaries, and how they're moving, across time.
It seems like there would be more (and more reliable) margin in betting on the boundary movement, that the seller isn't aware of yet, than the absolute value of a particular home.
And yes, they are presumably capturing this in their price difference, but it seems like it should be a core foundation.
> Zillow gave homeowners a free put option: Whenever Zillow's offer was above market, the homeowner would take it; otherwise the homeowner could freely ignore it.
Sorry for the possibly dumb question, but doesn’t this just describe buying things in any financial market? Your offer will only be accepted in cases where nobody else is willing to pay more.
> Sorry for the possibly dumb question, but doesn’t this just describe buying things in any financial market? Your offer will only be accepted in cases where nobody else is willing to pay more.
This isn't a financial (securities and derivatives) market, nor is that behavior specific to financial markets. OTOH, in financial markets, you are dealing with masses of interchangeable items (e.g., every share of IBM is the same); whereas with real estate every individual item is different. In financial markets, you typically have (some) other participants order data for the same asset as a pricing signal.
It's a lot easier, when you are willing to give an offer on any item in the market, to be way off in real estate than financial markets without knowing it till you try to flip the assset.
In a financial market (e.g: stocks) 1 share owned an individual investor is just as good as 1 share owned by an institutional investor: they have identical representation. The financialization of housing glosses over the fact the some houses will have foundational issues, need new roofs in a year, etc but you can't see that when you treat all houses as if they were shares, which seems like Zillow did, or by the nature of the asset class, sellers with low-quality product dumped for high prices.
Maybe Zillow went wrong because they deployed too much capital too quickly. With the opportunity to be selective, they could have found homes with 'good bones' that were under-priced. But if the executives mandated X billions of purchases in Y months, you move downstream to progressively less good deals, due to your own insatiable demand, when housing stock is already extremely low. That's not the AI/ML team's fault. That's you simply taking what you can get.
Yeah, it’s not correct to call it a put option. A put option has a time component where you are guaranteed a contract price while the market price fluctuates.
Zillow is an example just bad bidding (due to bad valuation model)
In a commodity market (all goods are fungible) pricing is easy to come by. For idiosyncratic goods it’s harder to get someone to make you an offer. The traditional way of doing price discovery in those cases was an auction. Zillow was willing to give a price upfront with no commitment, even in the sense of face time with a charismatic salesperson. That’s valuable for the seller and dangerous for the buyer.
On an average they may not have. Three separate million dollar houses get offered 900k, 900k and 1.2M. The point is that the guy offered 1.2m knows he will not get the 1.2 anywhere else and takes the offer.
The individual house is overvalued but you don't find out until after, let's say, 7000 of those sales.
Predicting the future is not a use case for machine learning. And failure of financial models happens all the time - Long Term Captial Management is probably the best example of a big one. This has nothing to do with ML/Data Science, other than maybe some people thought it could do something it cant.
Also, the article reads a bit too much like a stream of conscious sprinkled with buzzwords and causal references that the author knows better than Zillow.
I hate to use this language on HN - but this post is a crock of shit. This is just an opportunistic post by the author that cleverly evades defining anything in solid terms, providing even one concrete example, and is a pot of handwavey BS. It started off encouragingly, and two paragraphs in, the smoke alarms in my head were blaring. The author doesn’t even have a clue (or for some mysterious reason didn’t mention) about what went wrong for Zillow.
Zillow’s massive screw up was that their key business assumption was a self fulfilling prophecy. If a model learns that “price to pay today is price to pay yesterday in this area * 1.05”, and Zillow is buying multiple homes in a neighborhood spread over time, you can see that the price is headed to the moon (and the buyer who puts their faith in the model to the gutter) in no time. In my parents’ town, they offered a neighbor a price, and came back 3 months later with an offer $70k higher. I have a strong suspicion what was supposed to be a price prediction model ended up being a price discovery agent due to the lack of a “visited” set. Oops!
I agree with your characterization. I've been in meetings with people like this that will throw out irrelevant stuff (topology, distributions of distributions!?) as if it's an insight, and there is a certain kind of "leader" that loves that stuff, to the frustration of everyone who actually has some background in the area. I assume this is where he's coming from
This is not data science, its speculative trading. So, a data science approach without the massive paranoia and appropriate hedges that must accompany any trading endeavor, is a problem.
But I wonder of the future of mass corporate home buying, which I would assume Airbnb would do like Amazon, create a huge platform and then enter that market to compete with your customers.
This is BS. The issue had little todo with AI, they could have used professional real estate valuers and had the same problem.
This was caused by a faulty business model - houses aren’t shares or bonds that can be flipped instantly. The feedback for their pricing model wasn’t known for three to six months after buying the houses, by which time they’d lost billions.
186 comments
[ 3.2 ms ] story [ 247 ms ] thread"Well, sure, them folks were (wrong | lying), but its really your fault for listening to them! It's not like making that call was their actual job or anything, ... oh."
I'd say the lesson here is that horseshit is still horseshit even after it has been extensively processed by the finest machines and learning we can buy.
Just imagine being the poor sod there who got a model to say "Bail the fak OUT of this market" 3 months ago: think anyone would have listened?
As far as who is culpable for those situations … it’s not clear to me.
> When I advise Data Scientists to understand Topology and Differential Geometry, this is why.
Oh really? I've taken grad-level courses in those fields. It's nice general education but has nothing to do with this situation. Knowing about the Lucas critique from economics would be much more relevant.
My pet theory -- completely unvalidated but at least potentially insightful -- is that they failed to put in adequate margin for adverse selection: homeowners whose homes are worth less than they seem on paper were much more likely to take the Zillow offer. Adverse selection turns prediction variance, which you might naively expect to "average out" when using a well-calibrated model at scale, into a bias towards underperformance.
Another way to put it: asymmetric information. Be very careful betting against homeowners who probably know much more than you do.
Or read this comment, which puts it better than I did: https://news.ycombinator.com/item?id=29140104
If anything this could have been because of _too much_ higher level statistics and math, as those kinds of things can often shroud underlying problems.
To me, this is a systemic organizational problem in ML, rather than a model problem. Or essentially, pre-Agile development all over again.
Stop me if this sequence sounds familiar: (1) attempt to capture all the details in a nuanced domain that's foreign to you, from people who are experts but not expert developers, (2) develop code from that spec, (3) deliver code to the experts as a black box that they can't opine on, (4) cross fingers and run in production.
Which isn't to say this is the way everyone does it, but is to say it will be the way the majority does it, just as the majority of development is still waterfall under the hood.
But the mismatch is a mismatch in understanding, not in correctness. The experts can't offer meaningful feedback to the technological black box, and developers don't have enough time to spec a lifetime of subject matter experience.
Finding alpha in financial markets is just tough (real assets or not), and real estate also needs a lot of capital, which makes "waiting it out" tough
To run a market making business I have to know how to hedge my net position and manage my cost of carry.
I find it crazy hard to believe that they just went long a few billion in houses just based on a model of price and failed to hedge vs. the secular market and didn’t model time to sale. But if you believe the popular press maybe that’s what they did.
- enlist domain expertise
- validate incrementally
- expect adverse selection
- expect failures to generalise
Many, many production ML teams operate in this framework. The fact that Zillow either cut corners or hit a bigger snag suggests there is a more interesting story we haven't heard yet, not that they should have consulted OP or a few hn readers.
A given business probably doesn't employ more than a few people of that class and rigorousness: not least, because those people have the option of making a lot more money at a trading firm.
So yes, all businesses should do all those things. And yes, the best run places do. But it feels like there's going to be a decade or two where employee supply & expertise lags corporate demand, with bad results.
If you don't have the minimum skills on the team, you're going to fail in ways that you don't see coming, the unknown unknowns. To spend billions when a former Zillow employee says that they have no finance people on the team, is just, drastically, wow. I would much sooner believe he is mistaken then believe that is what actually happened.
[0]: https://xkcd.com/1831
That's because it is. A common characteristic of almost all blogs run by consultants. The only exception that jumps to mind is John Cook's [0].
0.https://www.johndcook.com/blog/
The other thing is that running a website and being a data aggregator is an entirely different business than speculating on houses. Matt Levine pointed this out last week [0], but its not making markets if the turnover takes 6 months.
[0] https://www.bloomberg.com/opinion/articles/2021-11-03/zillow...
“The Market for Lemons: Quality Uncertainty and the Market Mechanism” by George Akerlof is also a nice related resource [1]
> widely-cited[1] 1970 paper by economist George Akerlof which examines how the quality of goods traded in a market can degrade in the presence of information asymmetry between buyers and sellers, leaving only "lemons" behind.
[1] https://wikipedia.org/wiki/The_Market_for_Lemons
One notable place this happens is backtesting. Particularly for passive strategies, you don't actually know from historical data what would have happened if you had been in the market at a given time. Would your prices move the market? Maybe. Would you get filled disproportionately when it's bad to get filled? Fair chance of it. So don't act like your presence would not be felt.
Another angle is risk management. You may have high confidence in your model, but you should have a plan for what to do if things don't go as expected. It seems crazy to me that it could get so far out of hand that a quarter of the company gets sacked. Why not set a budget and dip your toes? Then you can watch out for the adverse selection issues as you grow the portfolio and make adjustments.
Finally, it's very normal for quant strategies to look good when they're first proposed, then look a bit worse when traded in real life, and perhaps then degrade over time. Another reason to take it easy when sizing the bets.
You're right about Lucas critique. It's fundamental that models affect the thing they're modelling.
And indeed if you go to /r/realestate anecdotally that is what happened. People would get offers from Zillow/Opendoor etc and talk to agents to see what they would list it at. If Zillow was higher than what the agents said they went with Zillow. If not they'd list it and sell it the old fashion way.
In fact it might not be a bad strategy to simply start with that... see if there's desperate people needing the money fast, and use your corporate heft to both sweep up the deal (brand) and patiently dump it back into the ordinary market, pocketing the spread as a liquidity premium.
One thing I remember my first boss saying is that it's hard to get a good trade on. Attractive deals have several interested parties. If you're getting a lot of trades, they might not be good ones.
I find it hard to believe that everyone there didn't realize that they were burning money. I saw it personally when looking to buy a house a couple years ago. There were a couple Zillow flips out there where the numbers obviously didn't add up. Whether or not they truly believed that it would work eventually or if it was just a way towards higher compensation/promotions/stock price or whatever is anyone's guess. But I suspect pretty much everyone involved from the person banging out ML models up to the CEO knew what was going on.
Then the articles would be talking about what geniuses they were and ML trampling traditional real estate.
They would double down and when the market calmed, all the same issues would reappear. Perhaps that's actually what happened, because the house market has been buoyant recently, and they seem to have had the iBuyer business for a few years?
I've certainly found myself in situations where I developed a model that performed so well in its specific domain that executives wanted to roll it out to a a much broader domain despite my advice.
It very much is a sign of immaturity and lack of discipline from leadership. It's too easy to buy into the hype thinking you are being supportive of AI in your org but not realize that truly being supporting an AI strategy involves a lot of nuanced skepticism.
So are lots of families in Chandler, Arizona
Yes, the nerds are so well-known to consider the risks of what they create. If only they’d thought of the risk that someone could use their work in the real world, even though it came with specific instructions not to do that. Maybe they wouldn’t create possibly-destructive ML models then.
Alas, it remains a mystery what a private company might do after paying a lot of money for something. Now where was I? Ah, right! Predicting criminality from baby photos! I love how DARPA sponsors basic research with absolutely no expectation of ever using it. Although I‘d better name it README.MD, just to make sure they do not forget to read it.
You can take this line of reasoning very, very far. Some people think we never should have left our hunter-gatherer lifestyle. To these people, the agricultural revolution was a disaster and things have only gotten worse from there.
It's a tremendously ignorant viewpoint IMO, but it's where you land if you believe technology with the potential for bad use shouldn't exist in general.
My advice would be to read the earnings call transcripts over the last couple of years. What Zillow has communicated in the last earnings call is fairly contradictory to what they've been saying in previous calls.
Some bullet points on my understanding of what went wrong:
-They were paying way too much for too many homes. I believe this is ultimately due to internal incentives and ambitious goal setting. IIRC, they mentioned they had to alter their model output to be able to give themselves the permission to keep buying at high volumes. They lacked fiscal discipline here.
-The holding times on these homes were too long, which increased their costs. My impression is that this was due to lack of operational experience with renovating and selling homes. Zillow was new to this, and they were trying to scale up very quickly, independent of whether they were efficient.
-The core business model of Zillow historically has been as an advertising site/lead gen site. This business model is not in their DNA and their operational expertise. The negative margin pressure from this business in tougher times can really stress the overall business at times.
Look at the other iBuyers in the space - they are still in the space. They just aren't as naive as Zillow was. Zillow's behavior here is really confusing and speaks to poor operations/strategy/discipline. Housing prices have been mostly been only going up in this market and Zillow was losing their shirt.
It mis-values houses near me by 20-40%. Other tools are much more accurate.
Market makers attempt to buy and sell instantaneously without changing the asset. Zillow had no intention of simultaneous purchase and sale. They also modified the asset. They used a market making model glued to a real estate speculating one to run a house-flipping operation at scale and from a distance.
Instead of focussing on sellers who would sell at a discount for immediacy, they paid over market. There was no effort put into pre-selling or hedging. (The latter would have been so possible as a security against their entire portfolio.)
Presumably, they trusted that their model does a more accurate assessment of price and future potential than a person themselves. If you had an excellent model, why _not_ would it balance out?
My theory is that most people would consult various sources and professionals-who-assess-homes (whatever they are called) and this the likelihood of Zillow having a better assessment than the average person is just low.
Makes nominal decisions with at best somewhat nominal ability.
My professors used to say “when the last stalwart holdout is sold on the premise of the irrational factors holding the market up the correction is near.” (Or something like that.)
I seem to recall it being Enron filing bankruptcy that really kicked off the Great Recession.
Oh further, anytime you have your market appraiser becoming a market participant their valuation becomes highly unreliable. This is another thing that’s just not done if you want to have a reliable indicator of an assets value and sell it knowing whether you’re getting a deal you’ll be comfortable with long term or not.
If corporate land lords are extracting enough rent to justify the higher than normal prices, is that really a bubble? It’s socially pernicious, yes, but that doesn’t sound like an asset bubble to me.
Blame the government; Zillow had nothing to do with the housing price blowup.
1. The business decision to go into the 'flipping homes' business.
2. The capabilities of the ML/AI to help optimize that business decision.
The post does a nice job outlining that #2 will have great variance across businesses, use cases, time, etc.
The biz decision is maybe more interesting and instructive. A wrong biz decision can doom the ML/AI out of the blocks - the ML/AI is often a tool to do a job, but there is a business decision to do the job to begin with.
Suppose you did have a perfect way to appraise houses by some arbitrary criteria. If the houses are "perfectly" appraised, that would change the values of the houses themselves, no?
e.g. magical system predicts 3b/2bath house should cost 500,000. People know the magical system's price is the perfect one. someone who has more money to burn can acknowledge this and still pay even more money, knowing most will not want to "overpay" for a house. house sells for 550,000. prediction is wrong in the end anyway.
schrodinger's predictor - it's accurate right until it's observed.
Cf. also the “market impact” in HFT, for a practical example.
Workplace chat app, you use your work email to sign in and chatter anonymously with your coworkers and others in the industry. You are ID'ed only by the company you work for.
This is a generalization, and has happened a couple times in my career. I feel lucky now to be at an organization where this is no longer the case.
Data mining is common in finance. Finance PhDs are constantly identifying new "factors" (e.g. value, momentum, price/book) based on historical data that don't pan out as time goes on. See here for some academic research: https://www.researchaffiliates.com/publications/articles/710...
Robert Shiller has does extensive research on how psychology impacts real estate prices. Monetary policy isn't standard right now. Machine learning isn't always a good tool for making predictions. Look at Shiller's interviews and surveys for predicting home prices. He doesn't use machine learning or advanced analytics.
I have no idea how they were so far off. Even right now:
Zillow estimate: $479,100 - https://www.zillow.com/homedetails/905-NE-43rd-St-APT-212-Se...
Redfin: $307,337 - https://www.redfin.com/WA/Seattle/901-NE-43rd-St-98105/unit-...
Keep in mind that Zillow's HQ is also in Seattle. While we may be protective of the data scientists on HN, this is one case where I agree, there's something wrong with their models. The "similar recently sold" condos that Zillow shows are indeed $400K, but those are at most 50 years old and more square footage. I think a simple logistic regression of square footage and age would give a better prediction than whatever model they are using.
https://www.zillow.com/z/zestimate/
Based on this I suspect they do check the model based on WHAT ACTUALLY HAPPENED.
what makes you think they don't check the model on WHAT ACTUALLY HAPPENED?
As I said, it seems like a pretty tight feedback loop to check, and then tweak the model. So it's confusing to me how they could be far enough off to ruin the business plan.
I'll put it back to you: What am I missing? How could they revise the model based on many months of real-world data, and yet get it wrong enough to nearly destroy the business?
So there is not checking. This seems unlikely given, as you point out, it’s data right at hand and, as I point out, they publish checks. In general, I’m suspicious of WOW PEOPLE ARE STUPID explanations.
There is checking and mistranslating model accuracy into business outcomes. The model is 90% accurate. Bid for 1000 houses. Maybe mainly people in the wrong by overpricing segment take you up on it. See the vast literature on adverse selection and winner’s curse.
There is tweaking the model because you want the a certain outcome.
> Over the last few weeks, it’s become clear that Zillow made a lot of mistakes. It tweaked its algorithm to be more aggressive with offers, winning bidding wars just as the market was starting to cool and overpaying for properties [1]
The CEO has commented that the volatility of the prediction was a problem. He mentioned that the unit economics swung from over plus 5% to minus 7%. So the model might be a accurate on average but too noisy for their purpose. [2]
Point is it’s not at all obvious that the problem here was accuracy on predictions of house prices.
[1] https://www.curbed.com/2021/11/zillows-i-buying-algorithm-ho...
[2] https://www.google.com/amp/s/seekingalpha.com/amp/article/44...
Finally im not sure how they hedged the risk. How do you short houses? If you consider mortgages like you holding 20% to protect the bank from losing, the variability can be difficult up to 20%, and leaves you with a hell of a downside.
My neighbor's house is the mirror image of my own, same size, same layout, same size lot, right next door. Zillow shows my home currently at $226K more than the one next door that is the same thing. Zillow's Zestimate of my home is more than a recent appraisal - which is odd because until 2021 Zillow had a really low Zestimate for my home, like the house next door. No idea why Zillow decided on a much higher value recently for me but not for my neighbor.
And I'll have to say that lenders do consider the Zestimate as part of a loan, especially in a case where needing an appraisal might not be needed - if it on the bubble then the Zestimate might be the difference of whether you get an appraisal or not.
Look at their balance sheet, they have over $3B in cash. They will be staying afloat comfortably.
There’s a lot of information we don’t have but them laying off huge percentages of staff is a scary indicator. Edit-also note their market cap is about 17B.
Zillow says they bought 3,805 homes in Q2: https://s24.q4cdn.com/723050407/files/doc_financials/2021/q2...
https://www.zillow.com/research/zillow-ibuyer-report-q2-2021...
What's not changed is the fact they were leveraging a significant portion of their company's market cap into this segment and it resulted, so far, in them laying off a quarter of their labor and completely terminating the program - amongst other things. This is not good news for their business and prospects of continuing to be an ongoing concern.
For my home, the Zestimate is about 8% higher than Redfin's.
https://mobile.twitter.com/0xdoug/status/1456032851477028870
TL;DR: not taking into account adverse selection - other people's prices & trades aren't random/Gaussian, but adversarial (they're trying to profit at your loss) and better (they probably have better information than you)
Behind every order is a thinking, breathing shark. A shark with mouths to feed and a mortgage to pay. A shark who's likely smarter than you. Definitely more ruthless.
You're swimming in shark-infested waters, treating apex predators like specks of geometric Brownian dust
> The Data Science team didn’t go to the lengths necessary to support their models. That was their failing. However, the business is ultimately responsible for betting the farm without demanding a real world track record to support those models."
I'm confused. If the "team" was "excellent" (and he says this multiple times), why didn't they do the extra leg work? It feels like he has a dog in this hunt. Did they use him for external consulting or something?
It would have been fortune cookies, it would have been the same.
> When I advise Data Scientists to understand Topology and Differential Geometry, this is why.
This is ridiculous, it has strictly nothing to do with that.
The author thinks he’s smarter than everyone else.
Management everywhere thinks they are getting good info from those below them when everywhere I have worked, they get a selective job preserving subset.
Domain understanding is key. Integration of the domain into ML models are a key part of generalizability -- far beyond train/validate/test, which are subject to data collection biases and temporal trends.
Zillow had a high, general equilibrium impact on a large market. We saw microcosms of this behavior in the "good old days" of Amazon book-selling when competitors would mark up a books price to one penny more than another, resulting in outrageously priced books valued far below the offering price. No behavioral governors were in place, review was inadequate, and they become part of Data Science lore we tell ourselves as our single threaded scripts running Python payloads warm our homes and melt our s'mores whilst our GPU weeps.
Data Science as a field is distinct from ML in that it uses the tools of ML to augment and enhance digital services. Despite the exuberance of boot camps to market prediction and neural nets as the end-all-be-all solution for hard business problems, value in data science adoption isn't solely in prediction, but in validation. Causal modeling and inference are key in this space, but also simply quantitative understanding of statistics and algorithms. As an example, you need someone in the room who understands multi-armed bandit sometimes. But, more importantly, you'll always need someone in the room who understands how multi-armed bandit would improve a business' value proposition.
I bet there were some employees who saw the estimates were untrustable and the entire thing was heading towards failure, but good luck changing the course of a ship controlled by hundreds of people.
This is true even for “Ivy” trained PhDs. There are may be a handful of people who can do truly profound work today. The rest are only capable of replication ie mouse clicks here and there.
I'm not making a joke. Newton published his models in the late 17th century. The models hold no better or worse today than they did then. If you had rockets in 1687, they would work the same way then that they do now. But, social science is nothing like hard science. I worry on some fundamental level, even people who will claim that they "of course" know this in fact forget this. There is hubris here.
However, I disagree with the OP's diagnosis.
The initiative failed because to a close approximation the only homeowners who would routinely accept Zillow's offers were those whose homes could not fetch anywhere near as much via traditional sales channels. Everyone else would rather sell their home via conventional listing. Zillow gave homeowners a free put option: Whenever Zillow's offer was above market, the homeowner would take it; otherwise the homeowner could freely ignore it. The incentives were strongly stacked against Zillow from day one.
Zillow unwittingly launched what I would describe as a seller's market for lemons. Naturally, Zillow ended up buying... a lot of lemons.
However, can you unpack "Everyone else would rather sell their home..."? At the core there are two factors at play in this decision. One is financial; anyone with the luxury of time will compare a realtor's comps with an ibuyer's offer. The second is certainty; an ibuyer offer is guaranteed, and can be executed without having to show a house over the course of 1 to X months.
It seems to me that certainty has value to some sellers in some situations. Right now, the certainty value of the traditional route is pretty good because the market is on fire. That is not always the case everywhere.
For example, if you're the homeowner and you think you can get, say, a few extra $100K's by listing your home instead of selling it right away on the spot, you would probably take your changes and list it. Even if you're under pressure to sell quickly, you might still try listing the home for a couple of months to see what happens. Only if you think your home is a lemon will you consider selling to Zillow.
That's why Zillow ended up buying all those lemons.
What if Zillow just makes an offer that’s high enough that you think it’s unlikely you’d do better if you went through the hassle of looking for other buyers?
The extremely important thing to note is that this is an asymmetric information game. Therefore O(h) < Z(h) is going to be self-selected such that O(h) < Z(h) because P(h) is closer to O(h) then Z(h) is to P(h). If Z(h) was actually able to be a reliable and better estimate of O(h) then it would mean that P(h) wasn't determined by information available to O(h) that was not also available to P(h). The market would already offer this value to O(h). So O(h) value is going to be reflective of Z(h). Its only if the non-public information changes the value that O(h) != Z(h).
I had a chance to talk to an inspector who looked at my place who has previously worked with Zillow to inspect houses that they were buying. The gist is that Zillow are getting the worst houses (the ones with trash everywhere, burnt carpet, feces on the wall). For them to get a "normal" house is very rare. Nobody with a normal house (and better) under normal condition (and better) will sell to Zillow. You can get a 1%+3% fee full service (pro pix, pro staging, pro cleaning, ...) any day of the week and still sell your house in mere weeks. I was not even in Austin. I guess if you don't care about making tens of thousands in a month (or hundreds of thousands in nicer cities), then you have bigger things to worry about.
Zillow's offer was already higher than OfferPad and other iBuyer's services. After my experience on the sell side with Zillow, I am incredibly wary of ALL iBuyers on the buy side.
Even CarMax and Carvana with commodity products in the traditional lemon market have to work around this problem by adding an inspection to confirm the data- and they have a large spread between their buying price and their selling price.
This is the market I was amazed Zillow didn't get into: the "WE BUY UR HOUSE 4 CASH" market.
I'm guessing the volume is too low (few people sell their house for stupid low offers) and advertising costs are too high (those people probably aren't obsessively checking Zillow for their home's value every day).
But still, it seems like easy money, that someone far less knowledgeable and talented than them is currently scooping up, that would better insulate them against market turns.
They could have bootstrapped by offering a product to those cash4home operations.
Also there is something to be said for a market where the customer brings the product to your location for inspection, and whose product is considerable faster to inspect more rigorously than a house.
Also the products within that market are much more uniform and share several unifying characteristic across even different makes and model so its easier to learn a good and accurate pricing function.
That gives them a lot more cycles to improve their algorithms (both human and machine based).
Plus when CarMax were starting out they likely did so with a bevy of folks already experienced and profitable at buying and selling used cars at a profit who had the systematic knowledge of how to do so in their heads.
It then was simply a matter of building up customer goodwill by focusing on another metric customers cared about, taking a perhaps slight loss to avoid the risk of being considerably ripped off by a savvy local dealer and saving oneself the anxiety induced by high pressure sales tactics.
Having said that I think Zillow could still be quite successful in this endeavor however from my experience they didn't deploy their trading algorithm in a prudent way, they massively over-traded an untested by the real world model, they trusted their paper trading (backtesting) way to much, and they didnt focus on the nuts and bolts of building a successful market making system: making actual money over and over at small size and ramping up size as your empirically demonstrated pnl grows.
Also putting trading aside they didn't have to be a flipper, they would probably naturally be more fit as an exchange+ or intermediary that provides exchange type backing and reduces the friction in buyers finding sellers, taking on risk by only making an offer when they have a bona fide buyer for fixed or % fee, but glossing over all the hassle of two people who dont know each other making a large trade. Offering services like if you dont like the house you just bought through us we will help your sell it at a cost proportional to renting no questions asked.
Zillow hasn't gotten to the point where they provide that, but theoretically they could. You could drive around with a Zillow agent and see their 50 homes for sale a lot quicker than coordinating with 50 random sellers.
It should be geospatial trends.
There's a fixed margin on any given home, caused by the inefficiency of real estate markets. But many actors (the seller, other agents) have an inkling of what that margin and the numbers on each side of the sale are.
What they do not have as good of a bead on, and Zillow is best placed to know, is high vs low neighborhood boundaries, and how they're moving, across time.
It seems like there would be more (and more reliable) margin in betting on the boundary movement, that the seller isn't aware of yet, than the absolute value of a particular home.
And yes, they are presumably capturing this in their price difference, but it seems like it should be a core foundation.
Sorry for the possibly dumb question, but doesn’t this just describe buying things in any financial market? Your offer will only be accepted in cases where nobody else is willing to pay more.
This isn't a financial (securities and derivatives) market, nor is that behavior specific to financial markets. OTOH, in financial markets, you are dealing with masses of interchangeable items (e.g., every share of IBM is the same); whereas with real estate every individual item is different. In financial markets, you typically have (some) other participants order data for the same asset as a pricing signal.
It's a lot easier, when you are willing to give an offer on any item in the market, to be way off in real estate than financial markets without knowing it till you try to flip the assset.
Maybe Zillow went wrong because they deployed too much capital too quickly. With the opportunity to be selective, they could have found homes with 'good bones' that were under-priced. But if the executives mandated X billions of purchases in Y months, you move downstream to progressively less good deals, due to your own insatiable demand, when housing stock is already extremely low. That's not the AI/ML team's fault. That's you simply taking what you can get.
Zillow is an example just bad bidding (due to bad valuation model)
Isn't this just another way of saying the models overstated the value of the homes?
The individual house is overvalued but you don't find out until after, let's say, 7000 of those sales.
This isn’t why Zillow started Offers and the zestimate did not play a big part of it. Some same people but those were different teams.
How is this different? They were playing roulette anyway...
Also, the article reads a bit too much like a stream of conscious sprinkled with buzzwords and causal references that the author knows better than Zillow.
Zillow’s massive screw up was that their key business assumption was a self fulfilling prophecy. If a model learns that “price to pay today is price to pay yesterday in this area * 1.05”, and Zillow is buying multiple homes in a neighborhood spread over time, you can see that the price is headed to the moon (and the buyer who puts their faith in the model to the gutter) in no time. In my parents’ town, they offered a neighbor a price, and came back 3 months later with an offer $70k higher. I have a strong suspicion what was supposed to be a price prediction model ended up being a price discovery agent due to the lack of a “visited” set. Oops!
But I wonder of the future of mass corporate home buying, which I would assume Airbnb would do like Amazon, create a huge platform and then enter that market to compete with your customers.
This was caused by a faulty business model - houses aren’t shares or bonds that can be flipped instantly. The feedback for their pricing model wasn’t known for three to six months after buying the houses, by which time they’d lost billions.