LLMs should be for general/high level work. For example, an LLM with the right harness could fetch and use that CNN, or train it's own CNN or other model to use for this task.
The harness in my experience makes a huge different too. Out of curiosity I tried having Gemini chat (with Argon) try to solve a level of this new puzzle game I'm linking (Order of the Sinking Star). It just got in an infinite loop and crashed. Then I try Antigravity with Flash 3.8 and it solved it easily. I saw the harness looked at other sokoban puzzles, coded some stuff, tried to replicate the level, etc. All things that a pure Gemini Chat doesn't really do.
LLMs are great for the high level harness, but if you want to count letters for example, it will still rely on "tools" like a piece of code to do it more reliably. Similarly in the above example, you wouldn't have the LLM directly try to analyze the images, but instead fetch and use the best tool for the job. But that can only be as good as the harness/tools it has access to.
The problem is that they are 'everything boxes'. They're the first application that can actually lay claim to being a universal computing aid. What then happens is that people will start using their newfound hammer and before you know it everything becomes a nail, even if for a particular problem there is a better solution you can't beat 'nothing to invest, nothing to learn, get an answer in 10 seconds flat' from a convenience perspective. If I look at the things people use LLMs for I wonder why they take such an unsuitable tool, and then I marvel at the fact that they actually get more-or-less to where they would have gotten with the proper tools, and most likely in less time.
Slide rules displaced tables, calculators displaced slide rules and calculators themselves were displaced for the most part (they are still in use, in education but not nearly as much as they used to be outside of that) by the PC.
LLMs do that sort of thing for just about every information processing aid.
This does make sense, however in view of the recent maths results and since there are tens of thousands of training images for skin cancer available on the public internet I was expecting the LLMs to perform a lot better than they did
This isn't really surprising. LLMs are intended to be generally intelligent. They are spiky on the things frontier labs tend to benchmark against (software/math) but are not intended or expected to be great at everything. The general intelligence unlocks new workflows, it does not perfect existing ones.
That being said, I'm sure a frontier LLM could make some improvements to your CNN.
This doesn't seem that surprising to me? One is tasked with a simple very narrow task. The other has a corpus including billions of other images and pieces of information. It'd seem to me that instinctively one would outperform the other, regardless of age.
This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.
Just use the LLM to make a thing that's actually good at what you need it to do.
it's precisely because you can tune them to do that off a strong base
and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)
Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply
I have been incredibly impressed by LLMs for coding and data analysis and was expecting the LLMs to perform better so I was surprised by the results especially since this is an old model and nowhere near state of the art for skin cancer diagnosis.
A transformer trained only on cancer images could probably do better but the old model is trained on a well curated data set to do one thing. A decision model can decide on a much broader universe of things, only a sliver of which is cancer (and probably there's a smaller cancer dataset too)
> why people are excited about Jev-style decision models.
I think its a combination of being new to classification models and not using in anger.
LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.
My understanding of the statement is trying to use it in production for the first time. Once you start to actually use decision models beyond something simple you start to see the limitations.
For this particular case, it's probably worth it because a dermatologist visit is $200. If I were the NHS or Kaiser, I'd partner with other integrated insurance and service providers to collect as much training data as possible and fund classifier development.
Training classifiers is shockingly easy nowadays with LLMs. This is probably the biggest single piece of bespoke software I ship that I didn’t used to.
We have in the works a classifier for if a prospective customer will be long term profitable or not. This changes everything.
> We have in the works a classifier for if a prospective customer will be long term profitable or not
I'm curious, why do you use a classifier for long-term profitability? Wouldn't predicting expected lifetime profit with a forecasting model be more meaningful than just predicting whether a customer will be profitable or not?
That's awful for the misclassified customers who will not get the attention they need. Not to mention that people talk and if you treat some customers badly, the others (the ones from the top of your pile) might reconsider working with you. That doesn't automatically make this a bad idea, but I would tread very carefully.
> Just use the LLM to make a thing that's actually good at what you need it to do.
This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.
The frustrating part about an LLM is at a fundamental level it's designed to do the wrong thing, it's designed to provide an answer for things it can't do, e.g. one argument I've been in recently at work is someone using it to summarise numerical data. They've a raw data spreadsheet they're putting into ChatGPT and asking it to produce a report in a particular style. So the LLM is doing its job, predicting the tokens to emit, and not actually doing mathematical analysis of the underlying data.
I had a team proudly show me how they were feeding their dashboard data into an LLM to predict growth needs, instead of doing basic linear regression or any number of pre-existing basic statistical analysis. The part that depressed me the most on that was it was a bunch of engineers that did it, the people you'd most hope would understand what an LLM is/does.
For numerical work, there’s no reason to assume the computation is happening in the LLM. All frontier models could handily write a Python script to do linear regression then report the result, so the quality of the responses here depends entirely on the prompt + harness.
Definitely. I’ve found Cursor + Grok to be the best budget models for this type of tool use, although I’ve yet to try out the latest Anthropic ones.
You can go “here’s a spreadsheet of some data, do a thing to it” and it’ll go through, have it plan out the type of calculations you want to do, and it’ll write you a program to parse your sheet, calculate what you want, then output an result sheet.
The key to good results with minimal effort seems to be the planning modes to me, combined with the /goal command. A little bit of validation before implementation goes a long way.
+1. I noticed ChatGPT now pulls out a calculator or writes a script instead of hallucinating numbers. Initially it was terrible, but recently it seems pretty good... that said, if you're doing serious work with this you should probably double check. And if you're doing repeated work, obviously get it to write a script or formula instead of doing 1-offs.
They’re very much trained to do this, in fact. Even Step 3.5 Flash, a medium sized open model from way earlier this year is trained to use python to calculate answers on the fly :)
It makes sense to use an LLM to write the queries when exploring data, but if you are building a dashboard that’s going to be used repeatedly, you should save the queries so they can be validated and improved over time. The LLM shouldn’t be rewriting a new query every time you open the dashboard.
These days I wouldn't be surprised if the models are smart enough to get the analysis correct.
However sometimes I think that AI use shows a revealed preference around things. In this case, the actual data just ... maybe never mattered to them at all? People wanted a shiny chart and they got a shiny chart.
I'm building a little side business with LLMs doing 90%+ of the work, and this includes numerical analysis of revenue to figure out where to direct effort.
Today the problem you're mentioning is easily avoided if you are aware by ensuring you direct the agents to build tools for anything deterministic.
Every week "we" (myself and Claude) start by analysing revenue from the previous week. Claude will do some manual proposals but they will be based on analysis done by a deterministic script. It will then run workflows that are increasingly calling deterministic scripts - every week another few decision points the agent used to do (and that I had to then manually vet) gets replaced by deterministic scripts.
It is what allows what is a very low margin opportunity to work - it is scaling with the total amount of manual labour and total token use remaining near constant. It's an opportunity that I won't go into much detail about because the margins are thin and the moat narrow and it's basically only viable because of this approach of setting a tiny budget where ~1/3 of it or so is dedicated to systematically eliminating manual and LLM work from the process.
Even so, I expect the opportunity to actually extract profit from it is going to be short-lived - the moat will eventually evaporate if LLMs keep getting cheaper and better in favour of people "just" asking an LLM to do the whole thing end to end.
So on one hand I agree with you - there are lots of things that'd be better automated with scripts, and where there might well always continue to be efficiencies you can extract by turning things into deterministic scripts instead. On the other hand, LLMs are already so cheap for a lot of things that it increasingly doesn't matter, and good enough that where it does matter they will increasingly also just write tools themselves if you ask them and sometimes - and likely increasingly so over time - without even being asked.
Yeah like the other say it depends on the harness. "into chatgpt" could mean many things. It's likely not predicting the next tokens to emit but rather thinking "okay the data is in format x i need to map it to y, and also this is a good fit for statistical model z" and then writing and executing a python script that runs whatever statistical model on the inputs and then produces some other output file that it then uses to produce a report.
That being said, the correct way would be to just use the LLM to produce the script, validate it themselves, and then use that script next time instead of an LLM to rewrite the same script over and over every time you need to produce a report.
I just ran an experiment, I took the banking77 dataset, took the labels out, and got an LLM (Opus 5) to label them, then trained a logistic regression classifier (Jeffy http://github.com/nicobrenner/jeffy) on the resulting set
Got the same accuracy and f1 from the Opus-label dataset than the original one (I realize that Opus might have banking77 in its training data, so might need to account for that somehow)
But the point is: it's way cheaper to run the LLM model over your data to label once, and then use a cheap classifier, than using the LLM directly for classification
> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture
There’s lots of applications where it’s not being deployed at scale. Also lots of cases where someone who knows they need something classified does not know what a classifier is.
Don’t those decision models mainly route to other models? My understanding is that they wouldn’t handle a request but forward it along with some kind of semantic capability.
> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture
Agreed, and that’s the premise behind Jeffy, open source, bundles pretrained task-specific models that run (and train) locally on CPU: https://jeffyclassify.com/
At the same time, I’ve been working with some people implementing it, and there’s definitely value in having a more general model like Jev, or an LLM, for initial exploration and data labeling, to then use that data and train the task-specific model. It’s a whole workflow
Also, the market/pain insight of Typesafe is super interesting. They realized that a big chunk of the market was using LLMs as classifiers, and they built a solution around that (very much to your point, a task-specific model, rather than a more general LLM), and it worked! (Also they have the ability to generate a lot of attention, given the founder being ex-OpenAI and having raised quite a bit of money too), so it’s not just a technical/engineering thing, it’s a business/market/need/discoverability issue as well
it's also the unfortunate practice of trying to use the new shiny thing everywhere and anywhere to signal that you're keeping with the times. It's funny how i've seen newer devs brush off things as 'traditional ml'
Which is nuts because one thing LLMs are actually pretty good up is generating "traditional ml" harnesses. You just have to describe the problem you are trying to solve and ask the LLM for algorithm suggestions.
It's also nuts because "traditional ml" generally requires a fraction of the resources of an LLM. If you need to auto label, tag, categorize, it's never been easier to setup that sort of system in any project.
Hardly a surprise when the curriculum is vastly different between institutions with cohort going through an accelerated boot camp and others being told low level computing is not useful. There are resources for people that are curious and want to broaden their knowledge, but it has to come from the individual.
> For any application at scale it'd be way more accurate to have a dedicated classifier
"at scale" or the above use case of "specialized domain"
Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.
System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.
>LLMs drive the cost of making those models down to near zero.
The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models. You could easily spend ten times as much mental energy and effort doing the paperwork and revisiting it annually as you do in the care and feeding of the model itself.
If a canned product meets the requirement, there's a lot to be said for using it. Even if it costs a bit more per decision, the all-in costs are WAY lower.
The only real usecase I can see for jev style like models are processing long lists of items. i.e server logs or financial transactions or sensor data in robotics.
Jev's launch was the best AI era marketing campaign I've seen so far. It was executed and timed so perfectly that Jev became synonymous with decision models overnight.
I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else.
The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.
I think you have a very narrow and specific experience with people claiming to be "devs".
I don't know of any devs in real life who would loudly and publicly get excited about anything unless it was their own completed and working project to show off.
The tools are a footnote. It's precisely because it's so hard to get a project that far along that they would be familiar with many more tools than they would ever speak about.
I think you're coming at this from a very narrow and specific experience with a few people you know.
Jev really went from the announcement to being discussed all over overnight. We're discussing how popular things are in general, not among your narrow and specific group of people you work with.
>many people were only familiar with LLMs as the generic AI tool for everything
That’s exactly how the biggest vendors have been marketing them, it’s no wonder non-experts think of them that way. I’m not an expert myself but have a passing familiarity with theory as a SWE and grad student who did some research for machine translation and currently am in a computing-adjacent field. I use LLMs heavily for work and have never come across any coder-oriented tutorial, guide, etc. encouraging use of LLMs as decision models specifically, even if they are being used that way “beneath the hood.” Jev’s announcement seemed sort of new functionality to me but even in the first post about it on HN (where I heard about it), iirc people were pointing out that LLMs do the same thing, at least to some extent.
Jev's marketing can't be that good because as a dev I was thoroughly unimpressed by it. Isn't it just structured output? If you embed LLMs in your applications, you were almost certainly using structured outputs already. Maybe they did additional training on top of some open Chinese model, but almost always this makes things worse not better. There's no way Jev is better than a 5-5 series Claude model at any kind of classification tasks.
It's like the old adage: "that company is a feature, not a product" - Jev is just a system prompt, not a product.
I think Jev tapped into a real desire of many people. A dream of (a kind of) Oracle. Where you ask the question as messily as you like, but you get as an output a decision that a plank of wood (or a piece of code) could follow literally.
Doesn't matter it's wrong significant fraction of the time and you have no clue what went wrong because the output is yes/no.
Classifier models actually make it think about how the input should be formatted and pre-processed and that kinda breaks the dream.
What we've been trying Jev for is interpreting things like short product descriptions provided by customers, and determine which of the natural language categories fits best.
The list of categories are provided by the government and can change several times a year, and the product descriptions are almost random (think the various titles of AliExpress listings of essentially the same product).
What kind of dedicated classifier would be better? Genuinely curious.
A classifier requires a ton of labeled data to be processed for it to be good. Jev and likes are zero shot classifiers where none of the data is required.
Imo both are useful for different tasks. I am not going to use jev based models for compliance or any tasks where being wrong can bankrupt my business. For something trivial like sentiment analysis it’s a god sent.
I've found that while this technically true, if you're building a classifier of any importance, you're going to need to monitor it and grade the efficacy of any changes, which will require labeled data anyways.
This kind of approach works for one-shot "let's just see if it works" type stuff, but anything that I would stand behind needs at least some level of labeled data and curated dataset.
Prompted LLM classifiers are good for when training data is expensive and/or you don’t expect a high inference volume. It’s basically choosing lower model development costs in exchange for higher inference costs, which can be good for TCO, time to market, or both. Heck, I’ve worked on some applications that weren’t even feasible before LLM-prompted classification because of the cost to acquire enough data to train a supervised model.
Cost/benefit tips in favor of supervised machine learning when training data is cheap and/or plentiful, or when you expect to be able to amortize model development costs across a whole lot of inference.
Classifiers can require a lot of examples to train correctly. You can take a decision model's outputs and apply platt scaling to arrive at a good decision boundary a lot faster.
Eventually, yes, it makes sense to move to a pure classifier model as it's probably a lot cheaper. But then you need to learn how to serve your own classifier model correctly. Compute costs are being driven up by demand to serve models and unless you're careful it's easy to use your GPUs less effectively than a company specializing in serving a model.
There's a great bit of shortform video content floating around that shows a stammering younger guy in a welding mask:
> Do you have two pieces of metal that need to become one?
> But if it became two pieces again it wouldn't be a big safety concern?
> If that's you call Bob's OK welding!
That's Jev, and ~all zero-shot classification-by-LLM. It's universally OK. And that's great, for the tranche of problems where OK is good enough. But anywhere OK is not good enough, one would be advised to use or build something somewhat more robust.
I think there is a parallel between engineering systems (specialized) and commodity cloud hardware (general purpose).
While no doubt, you can get better performance using a specialized hardware … there becomes a breaking point when the commodity / general purpose is good enough while allowing you lots more flexibility as well as typically being cheaper that is become hard to justify the specialized solution.
People like me who are excited about Jev-style models aren't going to use them to diagnose cancer. We are going to use them to make fairly straightforward decisions very quickly (300 ms or less) and very cheaply ($0.042 per M input tokens - output tokens aren't billed). Even the lower end frontier models will take much longer and be much more expensive, so these decision models are perfect for certain things.
If you know the exact domain you're working in, then a trained model could potentially do better, but its not a guarantee. For example, I fine-tuned Muse Glimmer on a specific domain and then tested it against Mercury Decide - which I've found does better than Jev - and they were even (and no fine-tuning was done on Mercury). But Mercury Decide was was faster and cheaper. The P95 for latency was about 5.6x lower. Decision models are very, very fast.
Yes, of course, with time, attention, higher maintenance costs, and slower time to respond to requirements changes you can build a specialized system that performs better than an off the shelf general purpose solution. But you cost money. And you can quit and go work somewhere else, stranding the project. And the data might be garbage you can’t train on without substantial effort. And training takes time, which has opportunity costs that are often substantial. And the requirements may change tomorrow with no notice, invalidating all of your work.
I don’t want to pay someone to maintain a first rate classifier when I can buy a pretty good classifier as a service. I don’t want to pay someone to maintain and build server infrastructure when I can buy that as a service.
I feel like I’m being gaslit on HN when a solution seems obviously valuable to me because it eliminates or modifies constraints, and people dismiss that as if managing constraints wasn’t most of high level engineering work.
I tuned out of HN for a few years, came back in the middle of the AI renaissance, and I am constantly raising my eyebrows and hearing this Upton Sinclair quote in my head:
“It is difficult to get a man to understand something when his salary depends on his not understanding it.”
I think there are a lot of practical applications where something that works few-shot and doesn't require training is quite appealing if the cost of inference is low. That's the sweet spot for Jev and similar models. If you have time and an eval set to train it, clearly a "real" special-purpose classifier should have much better performance (and for most applications cheaper cost/faster inference), but Jev and friends are way better than using a full-fat transformer LLM for a classification task.
As an example, I've been using winnow running locally as a decision model in openchamber to select between llms for doing various tasks. I don't have an eval set of different coding and other tasks that I could use to train a proper classifier. This I just download off huggingface and it works out of the box and has cut my token spend to a fraction of what it was. So it's good enough to be beneficial.
The LLM is like the swiss army knife in these situations. It can KINDA do it all, but not very well in a lot of cases. Better to have specialized tools in most cases. And you're right, the LLM is pretty good for figuring out or building out the right tool for the job.
It seems to me that hype cycle of jev made it harder to understand rather than easier.
Universal classifier just seems nonsensical. It seems clear that it can't have accurate probability distributions on any possible question you might ask it. Moreover, giving only "maybe" answers to everything adds a layer of equivocation that makes it harder to catch it in a clear mistake.
That said, I've encountered plenty of fuzzy judgment questions now (ie is this case more like example X or example Y?) that reasoning models struggle with. I could see how, in lieu of a dedicated classifier, a model like jev, ostensibly trained to guesstimate the relative likelihood of natural language hypotheses, may do well.
But it still feels wrong to call it a classifier in the traditional sense, let alone a universal classifier.
> Just use the LLM to make a thing that's actually good at what you need it to do.
Thinking about this every time someone shares their setup and there's LLMs doing simple periodical things directly instead of having the LLM write a deterministic Python script that will never try to hack your sandbox lol
The appeal is that you can incorporate more context in a decision problem without knowing in advance what that context might look like.
Take the problem of determining if two people in a genealogy database represent the same person. It's a decision problem. There will be some data in structured formats you can count on, but which may be missing, or may even be wrong. There will be lots of fuzzy data (born in Newtown yes, but which Newtown?) and unstructured data (notes, etc.)
Now, do you want to try to assemble a dataset to train your own classifier on that, or do you want to throw a general purpose LLM-rooted decision model on it?
Agreed. I did do image processing on skin and have to say it is just very very difficult to get 100% correct. In my opinion, even if some systems do outperform some doctors, it can only work as diagnostic support.
Anomalies are very hard to detect on certain skin tones and creating data sets from image material is quite a lot of work, even if you have access to large clinics specialised on dermatology with diagnosed and undiagnosed material.
Not seeing LLM to perform very well here and hallucination would be a major issue (not at all a solved problem, even frontier models do hallucinate/match the most probable path. Perhaps a problem that cannot be solved without arbitrary rules.)
And I guess some SVM still outperform CNN, just that CNN provide a pretty good learning environment that can improve upon ingesting more data. Although there is a risk associated with that as well.
Still, if we had well documented data material, many algorithms and AIs would perform well. The work is getting the data in a dependable state.
One thing that is exciting about Jev-style decision models is that it makes it cheap for things where they can do zero-shot of things where it's not worth your time building a curated dataset.
Bonus: after using them for a while, you get the beginnings of a training set if you change your mind.
You can do that with LLMs too - it's just a cost and latency question.
E.g. I use an LLM to label my e-mail. I do it because it's low risk enough that I don't care about it being incredibly precise - it does not automatically act on anything, it just labels. For me for my individual e-mail its cheap enough and fast enough to do with a cheap LLM, but if I wanted to scale it up, Jev or similar would probably be better.
A dedicated classifier would almost certainly provide better results if I invested the time in curating a dataset, but the threshold where that is worth it for me given the precision I get from a cheap LLM is pretty high.
So there's a space for both. Use Jev-like models (or LLMs) when the effort to curate a dataset to train a dedicated classifier is too high, and switch to a dedicated classifier when/if you need to scale something to a point where the economics justifies the effort or when you already have the dataset anyway.
> Just use the LLM to make a thing that's actually good at what you need it to do.
More generally, continue to build systems. More complex, more robust, systems.
An LLM is just one of many tools available to integrate into a system, but it’s not going to replace every system: trivially, any system that requires physical manipulation of reality can’t be replaced by an LLM. I’m not going to eat my morning cereal using an LLM.
There are tradeoffs (cost, accuracy, all sorts of metrics) that play into which tool you use for which part of the system. There will be choices that are always more cost effective:
Part of the excitement to me is when tech simply becomes part of the zeitgeist. It can be simultaneously true that there's nothing new under the sun and that simply getting the knowledge of some useful tech out to the broader population helps everyone by getting many more productive minds involved in improving the processes.
I still fail to see what jev does. ALL llms produce a probability for all tokens. All jev did was stop there. Its function could be replicating by adding one "break;" statement in the right place in any other LLM.
Sure, but for some (if not many) business applications Jev style classification is probably good enough that people don't really see the value in spinning up their own model right now.
Which CNN? A fine-tuned version of Inception v3, fine-tuned on 68,011 dermatology images
Which LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which version
Obviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.
Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).
LLMs shouldn't be treated as if they were purpose-built for specific cases just because they generally output a plausible-looking answer, and this is not something that was discovered only recently
> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.
>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.
This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.
You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
This is a good point and was an issue for skin cancer AI in the early days but has largely been solved now - newer models use ensembles of vision transformers and much greater number of training images and have been proven accurate in clinical trials. See for example https://skin-analytics.com/ (I am not linked in any way)
Jesus Christ the techbros are stumbling upon the fundamentals of fields they’re unfamiliar with and treating them as completely novel discoveries. No wonder they’re so high off their own farts in delusions of grandeur.
I developed many models from scratch deployed in various fields.
When you want to automatically detect a pathology from a give scan, you want it to be model that you own, train, use, and update, and not a generic vision capable LLM.
This is NOT what benchmaxxing is. You want the model to be accurate in terms of your cared metric- depending on the problem.
What would be an unbenchmaxxed model here? A general image classifier like Inception v3, without the fine-tuning?
The 200 test images here are 'wild'. Fine-tuning in-domain is the whole point. This is what model training is.
A skin cancer diagnosis tool doesn't also need to generalise to dog breeds to be valid, and will almost always perform worse if it does.
Training a task-specific small model is just good developer practice.
That's just good developer practice, not "benchmaxxing".
Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks that were supposed to be proxies for general information.
If you train a small model on a specific task that's not "benchmaxxing" because there's no deception involved. It's just training a model for a task.
It is benchmaxxing because the test distribution in these kind of image dataset generally matches the training much more than real world image. It used to be the biggest issue in real AI in most companies that the test data and real data is either different or risk getting drifted apart.
So it is not solving a task, it is solving the same test set. You can easily see these in simple OCR datasets like mnist. Draw a number yourself. It would be much less accurate than the 99% that good architecture claims.
Ok first of all. benchmaxxing, is not a well defined term. It's a word made up by the twitter/reddit crowd to describe overfitting.
That is not overfitting, that is training for a specific task. The measure for overfitting is how well it works on previously unseen data, hence the 200 test images removed from training and validation. That set serves as a second order of validation for the researchers to check themselves they have not overfit the model to match only the training data.
I found SkinGPT-4 when I was looking into this article. From the nature paper "among the 150 cases, a significant percentage of SkinGPT-4’s diagnoses (80.63%) were evaluated as correct or relevant by board-certified dermatologists." Which does outperform the CNN in this article.
I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.
The CNN architecture used here is definitely outdated. I used it as a baseline to compare to the LLMs and because the model is small enough to run using WebGPU in a phone browser for demonstration. Modern approaches use ensembles of vision transformers and have proven accurate in clinical trials
It seems one is more specialized for the purpose and compare it against a general purpose LLM instead of a specalized/focused LLM?
This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).
I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison
here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?
It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.
General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.
Here's a good heuristic for predicting the ability of a modern frontier LLM at any non-automatable task: Is it hugely overrepresented in their pre-training dataset (e.g. code), and were they specifically post-trained on it using reinforcement learning (e.g. coding)? If the answer to one or both is yes, you should expect them to be very good at it. Otherwise, you should expect them to be, at best, as good as an average human who's read a few books on the subject and is good at bullshitting.
I think this is true of any trained anything. It's the fish climbing a tree scenario. The teens on Ellen getting laughed at for not knowing how to fold a paper map.
There's infinite front end code to train on. Virtually infinite python and shell scripts too. As I've worked my way away from such domains, the LLM outputs have gotten worse. It just makes sense.
It's cool that there's an in-browser skin classifier...but it says "It is not a state-of-the-art model". The obvious question is, is there a free in browser state-of-the-art model?
Would be difficult to fit in a browser as state of the art requires multiple vision transformers. Also it would not be sensible to provide access to models without human oversight as at this point in time all models have the potential to miss skin cancer.
As people have mentioned, it isn't that surprising that a specially trained model beats a general purpose model.
However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?
This is like all those “I fine tuned a model for <task> that beats <sota model>” stories. It’s normally possible with sufficient data to build train/test splits and fine tune a narrow model in a way that it outperforms general models, it’s a given that’s it’s possible with the right task definition and data.
The problem is the narrow models are always super fragile. You change the lighting, zoom level, skin tone, or give it a picture of a cat (or more dangerously a misaligned image) instead of a skin lesion, and it breaks completely.
None of this is new, it’s why narrow classifiers peaked almost 10 years ago and why large general transformer models took over. If you have a task where the data distribution is always identical and has no long tail or outliers then inception or resnet or whatever is great. Problem is these are vanishingly rare.
> I compared to a 10-year old CNN and to my surprise found that this was superior.
I'm surprised he's surprised.
> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.
Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
I think the reason I am surprised is that they are exceptionally good at interpreting images generally and there are tens of thousands of labeled skin cancer images on the public internet so I had assumed the general image interpretation abilities would transfer
The specialized medical diagnosis models are trained on that same data though, and also use neural networks. The more noteworthy story will be when they overtake specialized models.
Not only do they train on that data but they train ONLY on that data and don’t have to generalize to other domains. And the architectures themselves, including hyperparameters, loss function, etc are chosen and tuned specifically for the quirks of this domain. So this result doesn’t surprise me either.
Having actually worked with teams doing medical vision models for cancer detection, I can tell you that the datasets we used were not public, and they came from studies and private hospital partnerships that LLM companies don't have access to.
Are transformers better at all use cases? Or only a specific subset? This sounds like a classification problem to me, which neural networks have always been pretty good at.
It's mostly a scale thing. Transformers scale better when you have a lot of training data. (Not saying CNNs don't scale, but it's way more difficult to tune the hyper parameters).
For smaller training data sizes, transformers don't have a clear advantage over CNNs. CNNs are still the way to go if you have problems that are not too complex, and especially constraints regarding inference costs. Think of a small edge device / sensor that counts apples in your apple picking machine.
What we sometimes see is that a backbone is trained using transformers, then a smaller CNN is destilled from that (DinoV3 for example).
such as helping you create a network that's good at playing go
This is really what’s happening right now… LLMs are excellent computer users. Which makes one of the frontiers “amazing things computers always could have done, but we lacked the programmers to do”
216 comments
[ 3.8 ms ] story [ 67.5 ms ] threadThe harness in my experience makes a huge different too. Out of curiosity I tried having Gemini chat (with Argon) try to solve a level of this new puzzle game I'm linking (Order of the Sinking Star). It just got in an infinite loop and crashed. Then I try Antigravity with Flash 3.8 and it solved it easily. I saw the harness looked at other sokoban puzzles, coded some stuff, tried to replicate the level, etc. All things that a pure Gemini Chat doesn't really do.
LLMs are great for the high level harness, but if you want to count letters for example, it will still rely on "tools" like a piece of code to do it more reliably. Similarly in the above example, you wouldn't have the LLM directly try to analyze the images, but instead fetch and use the best tool for the job. But that can only be as good as the harness/tools it has access to.
Slide rules displaced tables, calculators displaced slide rules and calculators themselves were displaced for the most part (they are still in use, in education but not nearly as much as they used to be outside of that) by the PC.
LLMs do that sort of thing for just about every information processing aid.
Doesn't matter the domain or the task, the answer from some researchers is "LLM" always.
That being said, I'm sure a frontier LLM could make some improvements to your CNN.
It's comparing a fine-tuned CNN to general purpose models on a tiny dataset.
Just use the LLM to make a thing that's actually good at what you need it to do.
and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)
Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply
I think its a combination of being new to classification models and not using in anger.
LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.
Is this a typo? are you saying that this is so new people haven't struggled with Jev enough yet? very interesting
> it starts to break down over 20 classifications
Yes, I recently ran a benchmark on exactly that: https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
At around 4 classes already LLM-based classifiers (Jev/Laya/GLiNER et al) start falling behind
My understanding of the statement is trying to use it in production for the first time. Once you start to actually use decision models beyond something simple you start to see the limitations.
An off-the-shelf API has way less friction, if the results are good enough why bother?
People vastly underrate the patience of people to learn and setup a classifier. Jev is one API call. Training a good classifier is hard.
With enough marketing you can sell anything. Also a decent way to make a product is to do the hard part for someone else.
I've been training classifiers for ten years, it used to be hard. Now it's hard to do really but it's trivial to do well enough.
100% this is a very good insight
We have in the works a classifier for if a prospective customer will be long term profitable or not. This changes everything.
I'm curious, why do you use a classifier for long-term profitability? Wouldn't predicting expected lifetime profit with a forecasting model be more meaningful than just predicting whether a customer will be profitable or not?
Eventually we hope to identify ones who are going to be lifetime duds.
I don’t have a deep understanding of this topic, but wouldn’t you also need a good amount of relevant data to train on?
This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.
I had a team proudly show me how they were feeding their dashboard data into an LLM to predict growth needs, instead of doing basic linear regression or any number of pre-existing basic statistical analysis. The part that depressed me the most on that was it was a bunch of engineers that did it, the people you'd most hope would understand what an LLM is/does.
You can go “here’s a spreadsheet of some data, do a thing to it” and it’ll go through, have it plan out the type of calculations you want to do, and it’ll write you a program to parse your sheet, calculate what you want, then output an result sheet.
The key to good results with minimal effort seems to be the planning modes to me, combined with the /goal command. A little bit of validation before implementation goes a long way.
I'm mostly starting to have some success telling TPMs and engineers to ask chatgpt to give them appropriate code that generates more accurate answers.
However sometimes I think that AI use shows a revealed preference around things. In this case, the actual data just ... maybe never mattered to them at all? People wanted a shiny chart and they got a shiny chart.
Today the problem you're mentioning is easily avoided if you are aware by ensuring you direct the agents to build tools for anything deterministic.
Every week "we" (myself and Claude) start by analysing revenue from the previous week. Claude will do some manual proposals but they will be based on analysis done by a deterministic script. It will then run workflows that are increasingly calling deterministic scripts - every week another few decision points the agent used to do (and that I had to then manually vet) gets replaced by deterministic scripts.
It is what allows what is a very low margin opportunity to work - it is scaling with the total amount of manual labour and total token use remaining near constant. It's an opportunity that I won't go into much detail about because the margins are thin and the moat narrow and it's basically only viable because of this approach of setting a tiny budget where ~1/3 of it or so is dedicated to systematically eliminating manual and LLM work from the process.
Even so, I expect the opportunity to actually extract profit from it is going to be short-lived - the moat will eventually evaporate if LLMs keep getting cheaper and better in favour of people "just" asking an LLM to do the whole thing end to end.
So on one hand I agree with you - there are lots of things that'd be better automated with scripts, and where there might well always continue to be efficiencies you can extract by turning things into deterministic scripts instead. On the other hand, LLMs are already so cheap for a lot of things that it increasingly doesn't matter, and good enough that where it does matter they will increasingly also just write tools themselves if you ask them and sometimes - and likely increasingly so over time - without even being asked.
That being said, the correct way would be to just use the LLM to produce the script, validate it themselves, and then use that script next time instead of an LLM to rewrite the same script over and over every time you need to produce a report.
subnanosecond response times
model fits on CPU
I agree that a good model will outperform an LLM, but the LLM is so much faster/easier to build. That’s the usual tradeoff.
Got the same accuracy and f1 from the Opus-label dataset than the original one (I realize that Opus might have banking77 in its training data, so might need to account for that somehow)
But the point is: it's way cheaper to run the LLM model over your data to label once, and then use a cheap classifier, than using the LLM directly for classification
There’s lots of applications where it’s not being deployed at scale. Also lots of cases where someone who knows they need something classified does not know what a classifier is.
Agreed, and that’s the premise behind Jeffy, open source, bundles pretrained task-specific models that run (and train) locally on CPU: https://jeffyclassify.com/
At the same time, I’ve been working with some people implementing it, and there’s definitely value in having a more general model like Jev, or an LLM, for initial exploration and data labeling, to then use that data and train the task-specific model. It’s a whole workflow
Also, the market/pain insight of Typesafe is super interesting. They realized that a big chunk of the market was using LLMs as classifiers, and they built a solution around that (very much to your point, a task-specific model, rather than a more general LLM), and it worked! (Also they have the ability to generate a lot of attention, given the founder being ex-OpenAI and having raised quite a bit of money too), so it’s not just a technical/engineering thing, it’s a business/market/need/discoverability issue as well
Unfortunately LLMs are magic to them.
When you lack a first principles understanding of how something really works, you will bolt on hacks and duct tape on the wrong approach.
It's also nuts because "traditional ml" generally requires a fraction of the resources of an LLM. If you need to auto label, tag, categorize, it's never been easier to setup that sort of system in any project.
"at scale" or the above use case of "specialized domain"
Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.
System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.
The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models. You could easily spend ten times as much mental energy and effort doing the paperwork and revisiting it annually as you do in the care and feeding of the model itself.
If a canned product meets the requirement, there's a lot to be said for using it. Even if it costs a bit more per decision, the all-in costs are WAY lower.
Have you considered putting an LLM on the task of completing the bureaucratic gauntlet
I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else.
The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.
I don't know of any devs in real life who would loudly and publicly get excited about anything unless it was their own completed and working project to show off.
The tools are a footnote. It's precisely because it's so hard to get a project that far along that they would be familiar with many more tools than they would ever speak about.
I think you're coming at this from a very narrow and specific experience with a few people you know.
Jev really went from the announcement to being discussed all over overnight. We're discussing how popular things are in general, not among your narrow and specific group of people you work with.
I think it's clear we're at the tail end of this cycle. I'm very glad to see it.
That’s exactly how the biggest vendors have been marketing them, it’s no wonder non-experts think of them that way. I’m not an expert myself but have a passing familiarity with theory as a SWE and grad student who did some research for machine translation and currently am in a computing-adjacent field. I use LLMs heavily for work and have never come across any coder-oriented tutorial, guide, etc. encouraging use of LLMs as decision models specifically, even if they are being used that way “beneath the hood.” Jev’s announcement seemed sort of new functionality to me but even in the first post about it on HN (where I heard about it), iirc people were pointing out that LLMs do the same thing, at least to some extent.
It's like the old adage: "that company is a feature, not a product" - Jev is just a system prompt, not a product.
Doesn't matter it's wrong significant fraction of the time and you have no clue what went wrong because the output is yes/no.
Classifier models actually make it think about how the input should be formatted and pre-processed and that kinda breaks the dream.
The list of categories are provided by the government and can change several times a year, and the product descriptions are almost random (think the various titles of AliExpress listings of essentially the same product).
What kind of dedicated classifier would be better? Genuinely curious.
This kind of approach works for one-shot "let's just see if it works" type stuff, but anything that I would stand behind needs at least some level of labeled data and curated dataset.
Cost/benefit tips in favor of supervised machine learning when training data is cheap and/or plentiful, or when you expect to be able to amortize model development costs across a whole lot of inference.
Eventually, yes, it makes sense to move to a pure classifier model as it's probably a lot cheaper. But then you need to learn how to serve your own classifier model correctly. Compute costs are being driven up by demand to serve models and unless you're careful it's easy to use your GPUs less effectively than a company specializing in serving a model.
While no doubt, you can get better performance using a specialized hardware … there becomes a breaking point when the commodity / general purpose is good enough while allowing you lots more flexibility as well as typically being cheaper that is become hard to justify the specialized solution.
If you really evaluate the best options, it’s very likely that a Jev-style decision model won’t give the best results.
I don’t want to pay someone to maintain a first rate classifier when I can buy a pretty good classifier as a service. I don’t want to pay someone to maintain and build server infrastructure when I can buy that as a service.
I feel like I’m being gaslit on HN when a solution seems obviously valuable to me because it eliminates or modifies constraints, and people dismiss that as if managing constraints wasn’t most of high level engineering work.
I tuned out of HN for a few years, came back in the middle of the AI renaissance, and I am constantly raising my eyebrows and hearing this Upton Sinclair quote in my head:
“It is difficult to get a man to understand something when his salary depends on his not understanding it.”
Yes, this.
As an example, I've been using winnow running locally as a decision model in openchamber to select between llms for doing various tasks. I don't have an eval set of different coding and other tasks that I could use to train a proper classifier. This I just download off huggingface and it works out of the box and has cut my token spend to a fraction of what it was. So it's good enough to be beneficial.
Universal classifier just seems nonsensical. It seems clear that it can't have accurate probability distributions on any possible question you might ask it. Moreover, giving only "maybe" answers to everything adds a layer of equivocation that makes it harder to catch it in a clear mistake.
That said, I've encountered plenty of fuzzy judgment questions now (ie is this case more like example X or example Y?) that reasoning models struggle with. I could see how, in lieu of a dedicated classifier, a model like jev, ostensibly trained to guesstimate the relative likelihood of natural language hypotheses, may do well.
But it still feels wrong to call it a classifier in the traditional sense, let alone a universal classifier.
Thinking about this every time someone shares their setup and there's LLMs doing simple periodical things directly instead of having the LLM write a deterministic Python script that will never try to hack your sandbox lol
Take the problem of determining if two people in a genealogy database represent the same person. It's a decision problem. There will be some data in structured formats you can count on, but which may be missing, or may even be wrong. There will be lots of fuzzy data (born in Newtown yes, but which Newtown?) and unstructured data (notes, etc.)
Now, do you want to try to assemble a dataset to train your own classifier on that, or do you want to throw a general purpose LLM-rooted decision model on it?
Anomalies are very hard to detect on certain skin tones and creating data sets from image material is quite a lot of work, even if you have access to large clinics specialised on dermatology with diagnosed and undiagnosed material.
Not seeing LLM to perform very well here and hallucination would be a major issue (not at all a solved problem, even frontier models do hallucinate/match the most probable path. Perhaps a problem that cannot be solved without arbitrary rules.)
And I guess some SVM still outperform CNN, just that CNN provide a pretty good learning environment that can improve upon ingesting more data. Although there is a risk associated with that as well.
Still, if we had well documented data material, many algorithms and AIs would perform well. The work is getting the data in a dependable state.
Bonus: after using them for a while, you get the beginnings of a training set if you change your mind.
You can do that with LLMs too - it's just a cost and latency question.
E.g. I use an LLM to label my e-mail. I do it because it's low risk enough that I don't care about it being incredibly precise - it does not automatically act on anything, it just labels. For me for my individual e-mail its cheap enough and fast enough to do with a cheap LLM, but if I wanted to scale it up, Jev or similar would probably be better.
A dedicated classifier would almost certainly provide better results if I invested the time in curating a dataset, but the threshold where that is worth it for me given the precision I get from a cheap LLM is pretty high.
So there's a space for both. Use Jev-like models (or LLMs) when the effort to curate a dataset to train a dedicated classifier is too high, and switch to a dedicated classifier when/if you need to scale something to a point where the economics justifies the effort or when you already have the dataset anyway.
More generally, continue to build systems. More complex, more robust, systems.
An LLM is just one of many tools available to integrate into a system, but it’s not going to replace every system: trivially, any system that requires physical manipulation of reality can’t be replaced by an LLM. I’m not going to eat my morning cereal using an LLM.
There are tradeoffs (cost, accuracy, all sorts of metrics) that play into which tool you use for which part of the system. There will be choices that are always more cost effective:
Which LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which version
Obviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.
Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).
>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.
This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.
You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
I believe the normal term is "gaming a benchmark test".
Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.
The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.
Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.
Jesus Christ the techbros are stumbling upon the fundamentals of fields they’re unfamiliar with and treating them as completely novel discoveries. No wonder they’re so high off their own farts in delusions of grandeur.
I developed many models from scratch deployed in various fields.
When you want to automatically detect a pathology from a give scan, you want it to be model that you own, train, use, and update, and not a generic vision capable LLM.
This is NOT what benchmaxxing is. You want the model to be accurate in terms of your cared metric- depending on the problem.
The 200 test images here are 'wild'. Fine-tuning in-domain is the whole point. This is what model training is. A skin cancer diagnosis tool doesn't also need to generalise to dog breeds to be valid, and will almost always perform worse if it does.
That's just good developer practice, not "benchmaxxing".
Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks that were supposed to be proxies for general information.
If you train a small model on a specific task that's not "benchmaxxing" because there's no deception involved. It's just training a model for a task.
So it is not solving a task, it is solving the same test set. You can easily see these in simple OCR datasets like mnist. Draw a number yourself. It would be much less accurate than the 99% that good architecture claims.
That is not overfitting, that is training for a specific task. The measure for overfitting is how well it works on previously unseen data, hence the 200 test images removed from training and validation. That set serves as a second order of validation for the researchers to check themselves they have not overfit the model to match only the training data.
I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.
Llms have their applications. Other models still do.
This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).
I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?
It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.
General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.
There's infinite front end code to train on. Virtually infinite python and shell scripts too. As I've worked my way away from such domains, the LLM outputs have gotten worse. It just makes sense.
However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?
The problem is the narrow models are always super fragile. You change the lighting, zoom level, skin tone, or give it a picture of a cat (or more dangerously a misaligned image) instead of a skin lesion, and it breaks completely.
None of this is new, it’s why narrow classifiers peaked almost 10 years ago and why large general transformer models took over. If you have a task where the data distribution is always identical and has no long tail or outliers then inception or resnet or whatever is great. Problem is these are vanishingly rare.
I'm surprised he's surprised.
> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.
Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
For smaller training data sizes, transformers don't have a clear advantage over CNNs. CNNs are still the way to go if you have problems that are not too complex, and especially constraints regarding inference costs. Think of a small edge device / sensor that counts apples in your apple picking machine.
What we sometimes see is that a backbone is trained using transformers, then a smaller CNN is destilled from that (DinoV3 for example).
This is really what’s happening right now… LLMs are excellent computer users. Which makes one of the frontiers “amazing things computers always could have done, but we lacked the programmers to do”
The bitter lesson tells us that general _approaches_ outperform domain-specific understanding.
That doesn't mean a general model will outperform a domain specific one.
https://en.wikipedia.org/wiki/Bitter_lesson
Anyone wanna take a bet that the general purpose model will beat these specific models in a year or two?