Most ML solutions in most businesses have either questionable business value or debatable ROI. This lends itself to many products which are shipped quickly and then semi abandoned. In many cases the ML that's shipped barely outperforms the heuristic solution if at all.
Further, most of the teams tasked with maintaining or improving these systems have little practical interest in the production service - it's much easier and rewarding to bemoan a slower portion of the business while publishing research than improve the production service. It's not uncommon to encounter ML teams who claim multiple year's worth of innovations are backlogged against engineering. I once encountered a 20 person team who owned the core bidding algorithm for a large ad-tech startup which hadn't made a change in 3 years. The business believed that they were making regular releases.
What this all says is that we’re in a strange valley of ML maturity where enough people know about it to try it out, but few are able to navigate how to make a sustainable investment.
This resonates with me. I think that a product ML team can only be successful in the long run if its members can pivot between modeling work and engineering work as the business needs change. There is an entire class of communication and prioritization problems that evaporate when the same people who build the models are pushing them to production.
3 comments
[ 2.5 ms ] story [ 18.2 ms ] threadMost ML solutions in most businesses have either questionable business value or debatable ROI. This lends itself to many products which are shipped quickly and then semi abandoned. In many cases the ML that's shipped barely outperforms the heuristic solution if at all.
Further, most of the teams tasked with maintaining or improving these systems have little practical interest in the production service - it's much easier and rewarding to bemoan a slower portion of the business while publishing research than improve the production service. It's not uncommon to encounter ML teams who claim multiple year's worth of innovations are backlogged against engineering. I once encountered a 20 person team who owned the core bidding algorithm for a large ad-tech startup which hadn't made a change in 3 years. The business believed that they were making regular releases.
What this all says is that we’re in a strange valley of ML maturity where enough people know about it to try it out, but few are able to navigate how to make a sustainable investment.