It makes me smile when runners use "X is a marathon, not a sprint" to hint at an effort that accumulates over time and an optimal use of energy.
I do it too because it's a common expression, and a marathon is of course longer than a sprint, but both have in common that properly raced, they are absolutely brutal efforts that leave you without a single additional drop at the end. The effort length and instantaneous power output changes, of course. Maybe "it's a marathon build, not the race" would be more precise at the loss of nearly all its expressive power (but with a lot more pedanticism points) :-p .
You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can.
Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want.
Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I know and hand it to a machine, I'd do that, but it impossible.
People aren't books or hashmaps. If you want to build something, you need to use the tools, not teach the tools to use you.
[edit: I'm trying to figure out if there's something to be done about this. Email me if you want to chat -- tr at tern dot sh]
> You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can.
> Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want.
First you have to have it, and if you think this is a tasteful solution, then you didn't.
This problem predates AI. If we could externalize such a fickle thing such as good taste it wouldn't be such a valuable skill. And God have people tried. Golden ratios, style guides, naming rules, linters, formatters, templates, margin ratios, color palettes, and on and on and on. And yet here we are.
We can quantize some of the basics, and make a not half bad style guide, but we'll never be able to fully actualize a set of rules to match what humans find generally tasteful. Its too contextual and a moving target.
Gets into the Hard Problem of Consciousness vs AGI which is a discussion that needs to happen in AI.
Subjective "taste" and "feel" are experiences one has, rather than language one predicts out. Language is only produced to report on the experience, like "Wow, that's an ugly couch".
A vision model doesn't model how it experiences or feels (internally) about the image, just objective information about features of the image itself (external).
There are layers to aesthetics - part of it is functionality, utility, the environment vs your needs, but a big part of your style is directly related to your personality, memories, experiences, and how you physically fit with it. It's not correct/incorrect, it's optimizing for the entire circumstance, internal and external.
It can be hard to find the words to explain why an aesthetic works, or feels right (or wrong). What's even more important is when another person agrees. When you can have cohorts, trends, cliques, and hype.
AI can't do any of these inter/intra social activities, and so, like other acts of creation it can never operate at the cutting edge the way a human mind can. But with better and better vision models paired with good language models, synthetic subjectivity will do the job soon enough for most intents and purposes.
We can encode taste -- generative AI depends on it. Ask people to compare two examples and pick the one with better taste. You can even ask them to rate multiple subjective criteria at once. Use that to learn a scoring function based on the rating labels, and raw features. Now you can write tests.
Language count is a decent notoriety signal though pretty coarse. The OP/author should take a look at QRank: https://qrank.toolforge.org/
> QRank is a ranking signal for Wikidata entities. It gets computed by aggregating page view statistics for Wikipedia, Wikitravel, Wikibooks, Wikispecies and other Wikimedia projects
I wrote about this a few months back. Rick Rubin is famous for this. I do think it is something that can be trained though, it just needs a lot more context. Taste builds over time through lots of unit tests, through lots of content writing, through an accumulation of product decisions. It’s hard to put it in the individual spec, but it can be teased out of 100 project specs. And when you get to that scale the AI starts to do it pretty well.
I think another important question is can you distill taste? (another comment uses the phrase "externalize", which might mean something similar).
I think people have been trying for the written word, with some degree of success (anti-slop skills). I have been trying for visuals, and it's pretty meh. It's easy to get a multimodal LLM to follow a style guide, but a style guide doesn't capture everything that accounts for taste. And anything that is dynamic (not a screenshot test) seems really hard or really expensive.
This has been my experience, as well, but it’s a really big support. It just needs adult supervision. I can’t understand how vibe-coded apps, actually work.
As far as “taste,” goes, I test my stuff constantly, checking for even minor “friction points,” sometimes, refactoring back to design, in order to resolve issues that many folks would ship. I’m pretty anal, and want my work to be the best experience possible.
I can’t see any LLM coming close to being able to evaluate the user experience, like I can.
Unrelated to code, but along the same lines. I've been keeping track of the Reckless Ben case to fuel my unhealthy indignation, and we just had a like-for-like comparison between a human and an LLM.
Human: well-scoped argument that does just enough to get the job done with minimal risk.
AI: Extremely clever and correct legal argument that almost any lawyer would have said not to file (at least as written). It tries to burn the world and seriously risks pissing off the judge.
> Overall the evaluation of success was one of the most challenging parts of the project. As a developer, I’m used to building features that either work or don’t and there is often an objective way to measure how well a feature performs. For messy real world data it was hard to evaluate how good or bad the pipeline was. Furthermore, it was easy to start optimising for a specific parameter or route and find later that this work led to severe degradations in other areas.
> Verification becomes hard to reason about because there is no ground truth for points of interest, there are no red/green unit tests for taste. I’m sure these are familiar challenges to data scientists and that there are frameworks and evals for working on them. This will require more iteration and manual overrides. Hopefully with feedback and collaboration from the community. But for now I’ve shipped V1…
I suspect LLMs may be able to help us quantify our taste because they can keep track of so many data points all at once, where we have to lossily abstract these details away.
I am quite confident I could take a series of photos of various designs and classify them as "tacky" or not, and train a neural network to recognize tackiness.
46 comments
[ 3.6 ms ] story [ 94.6 ms ] threadYou absolutely can unit test for taste, just put an agent into loop, and write into prompt what you like. Then do scoring...
Iceland is really bad example, it basically has one populated site (capital) and circular road that goes around the island.
I do it too because it's a common expression, and a marathon is of course longer than a sprint, but both have in common that properly raced, they are absolutely brutal efforts that leave you without a single additional drop at the end. The effort length and instantaneous power output changes, of course. Maybe "it's a marathon build, not the race" would be more precise at the loss of nearly all its expressive power (but with a lot more pedanticism points) :-p .
Nice project !
Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want.
Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I know and hand it to a machine, I'd do that, but it impossible.
People aren't books or hashmaps. If you want to build something, you need to use the tools, not teach the tools to use you.
[edit: I'm trying to figure out if there's something to be done about this. Email me if you want to chat -- tr at tern dot sh]
> Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want.
First you have to have it, and if you think this is a tasteful solution, then you didn't.
We can quantize some of the basics, and make a not half bad style guide, but we'll never be able to fully actualize a set of rules to match what humans find generally tasteful. Its too contextual and a moving target.
Subjective "taste" and "feel" are experiences one has, rather than language one predicts out. Language is only produced to report on the experience, like "Wow, that's an ugly couch".
A vision model doesn't model how it experiences or feels (internally) about the image, just objective information about features of the image itself (external).
There are layers to aesthetics - part of it is functionality, utility, the environment vs your needs, but a big part of your style is directly related to your personality, memories, experiences, and how you physically fit with it. It's not correct/incorrect, it's optimizing for the entire circumstance, internal and external.
It can be hard to find the words to explain why an aesthetic works, or feels right (or wrong). What's even more important is when another person agrees. When you can have cohorts, trends, cliques, and hype.
AI can't do any of these inter/intra social activities, and so, like other acts of creation it can never operate at the cutting edge the way a human mind can. But with better and better vision models paired with good language models, synthetic subjectivity will do the job soon enough for most intents and purposes.
Working, useful, delightful, in that order. Testing can make things more likely to work, that's it.
> QRank is a ranking signal for Wikidata entities. It gets computed by aggregating page view statistics for Wikipedia, Wikitravel, Wikibooks, Wikispecies and other Wikimedia projects
from https://github.com/brawer/wikidata-qrank/blob/main/doc/desig...
I wrote about this a few months back. Rick Rubin is famous for this. I do think it is something that can be trained though, it just needs a lot more context. Taste builds over time through lots of unit tests, through lots of content writing, through an accumulation of product decisions. It’s hard to put it in the individual spec, but it can be teased out of 100 project specs. And when you get to that scale the AI starts to do it pretty well.
There is a reason conference talks are always about plain algorithms and data structures.
I think people have been trying for the written word, with some degree of success (anti-slop skills). I have been trying for visuals, and it's pretty meh. It's easy to get a multimodal LLM to follow a style guide, but a style guide doesn't capture everything that accounts for taste. And anything that is dynamic (not a screenshot test) seems really hard or really expensive.
This has been my experience, as well, but it’s a really big support. It just needs adult supervision. I can’t understand how vibe-coded apps, actually work.
As far as “taste,” goes, I test my stuff constantly, checking for even minor “friction points,” sometimes, refactoring back to design, in order to resolve issues that many folks would ship. I’m pretty anal, and want my work to be the best experience possible.
I can’t see any LLM coming close to being able to evaluate the user experience, like I can.
Human: well-scoped argument that does just enough to get the job done with minimal risk.
AI: Extremely clever and correct legal argument that almost any lawyer would have said not to file (at least as written). It tries to burn the world and seriously risks pissing off the judge.
https://www.youtube.com/watch?v=YRXJnKP6Tu0
> Verification becomes hard to reason about because there is no ground truth for points of interest, there are no red/green unit tests for taste. I’m sure these are familiar challenges to data scientists and that there are frameworks and evals for working on them. This will require more iteration and manual overrides. Hopefully with feedback and collaboration from the community. But for now I’ve shipped V1…
I suspect LLMs may be able to help us quantify our taste because they can keep track of so many data points all at once, where we have to lossily abstract these details away.