I’ve been working independently on a method that replaces full-transformer inference with a low-rank “meaning field” extracted from internal activations.
The core result: a frozen Llama-3.3-70B can be distilled into a 256-dimensional field representation, giving 224× compression and slightly higher accuracy on several benchmarks. A small student model then learns to directly generate these fields from text, removing the transformer from the inference path.
The Zenodo link contains the full paper, statistical results, and methodology.
A reference implementation (non-optimized) is here: https://github.com/Anima-Core/an1-core
Production variants (AN1-Turbo, FPU work, etc.) are not included.
I’m an outsider to academia so I’m posting this openly to get technical feedback, replication attempts, and critique from people who understand this space.
Only skimmed the paper and I have no idea how sound or reproducible it is, but the paper is well written, especially the clarity of notation. After reading yesterday's weight subspace paper: https://news.ycombinator.com/item?id=46199623, this does sound plausible to me.
thanks for sharing! If I understand correctly, you're training a smaller model to approximate concatenate(layer[1], layer[5], layer[10], ...), using a loss function that combines reconstruction error w/ end-to-end accuracy. then, you're transferring that smaller representation into a smaller transformer model. is that right?
If i were a paper reviewer, here are a couple red flags that stood out to me. Suggest starting here if you want to rework this for an academic submission:
1. your LaTeX citations in the related work are broken, i see [?] everywhere. To a reviewer, this is often a strong sign of an AI-hallucinated bibliography, though many of your references actually do exist and are contextually relevant, so I'm not quite sure what's going on here. Similarly, figure references need to be fixed, I see references to "Figure ?" throughout.
2. bluntly, "Exact architecture details remain proprietary for production deployments" and "Production systems use architecture search tailored to target latency and accuracy constraints" is not how IP protection works in this field. Do your experiments use the "MLP baselines" or your proprietary architecture? Since you say the code "Achieves 80-90% of paper performance using baseline heuristics," this approach effectively isn't reproducible. As a reviewer, this really worries me. I strongly recommend benchmarking only the system you're able to open-source. I say this because I suspect there's a lot of "secret sauce" in the actual way you're approximating the anchor layers and the way that's transferred back to your student transformer model, and that's the part that's important to spend the most time/effort/writing on, but it's glossed over as an implementation detail in this manuscript.
3. I'm glad you ablate over hyperparameters of your system, but how does it compare to 1. an ordinary smaller model of identical size trained end-to-end, and 2. distilling from a single layer's activations? Eg. a reviewer might consider this work to be a novel method of model distillation, so what makes it better than previous distillation methods?
4. I found the paper fairly hard to read because it's full of sentence fragments rather than full thoughts. A little background on the benchmarks, failure cases, etc. would go a long way, and adding some discussion on why you think your approach improves on similar distillation methods would also be welcome here
5. "compression" is overloaded. Does 224x compression refer to (nparams(field transfer)+nparams(student model))/nparams(original model), or does it refer to reducing the representation dimensionality, 7*8192/256 ?
6. [nitpick] suggest changing the name "meaning field" to something a little more digestible, like "compressed representation" or "latent activation distillation" or something
sorry for being so critical. iron sharpens iron though. hopefully these thoughts are helpful to get you started, excited to see where this work leads
Looks very fake. Self published (Anima-Core is NOT a journal), no academic anteriority, very strong statement, no peer-review, no public history of technical skills. Did I mention the use of Github via the interface only?
At the same time, possible since it's only classification tasks.
I mean, the method explained is technically plausible, a lot of people thought about it, we were just unable to find a method to do so.
Ryan, I really want to believe you're onto something. But I also feel like I'm being slightly spearphished by an LLM being told, "based on the last week of HN headlines, invent a new LLM innovation that seems plausible enough to get a ton of attention, cold fusion or LK-99 style, and make a repository that on the surface seems to have some amazing performance. Also, feel free to fake the result data."
And, while I am sorry for your loss, your Substack [0] really seems like GPT ARG fantasy.
Excerpt:
> Ani, AN1, and Soul Systems Science are not mere products. They are continuity. They are the baton passed across generations, from my father’s last words to my first principles. They are what binds loss to creation, silence to voice, mortality to meaning.
I might be overly pessimistic, but this looks like a case of a person believing LLM hallucinations and making it write a paper.
I asked both Claude Code|Opus 4.5 and Codex|GPT 5.1 Codex Max (funny to ask LLMs, I know) to check the an1-core repo. I don't think they'd hallucinate on something like this (the code is quite small), but I do not claim expertise.
- The actual "AN1 head" is just linear probing (freeze a pretrained model, train a classifier on its features). The full flow (as reported by CC) is "Text → [Full Transformer] → activations → [Tiny Head] → prediction"
Basically, there's no code to train a real "student" model that would run without the teacher.
===
The repo/paper say that there's a mythical "commercial version" that has all the goodies:
(repo)
> This reference implementation (an1-core) does not include the FPU, AN4, or other proprietary optimization components covered by these patents. It provides only the core scientific demonstration of the meaning fields phenomenon.
(paper)
> Production deployment: Optimized implementations (AN1-Turbo) with learned layer selection, adaptive loss scheduling, and CUDA-accelerated inference available under commercial license.
But right now we only have the code in the repo.
===
In the paper they show that the student model (30M params) gets ~82% on SST-2 (labels-only). But what what they don't show is that DistilBERT (>5 year old model) already achieves 91% on the same dataset despite only having 66M params.
Another weird tidbit from the paper - in the section where they show the economic impact, they claim that LLaMA 70B runs at 2 tok/s at batch size=1 on an H200. In reality that number is at least a magnitude bigger even without quantization, like 20-40 tok/s. With quantization it can easily be above 100 tok/s.
Your vibe-coded eval has cheated this to collapse it into a binary selection on row 46 in https://github.com/Anima-Core/an1-core/blob/main/experiments..., making the problem baseline 50% on random choice instead of 25%, making the problem much easier. HellaSwag is specifically constructed with adversarial examples that could be plausible. By not including them, the eval is much easier.
---
Then, in extract_fields_from_model, you have another cheating going on. The extraction logic (h[:, -1, :]) fails to account for padding in batches, likely extracting EOS/Pad tokens instead of the intended content tokens. This suggests the probe is relying on global sentence summaries (standard embeddings in causal structures) rather than the novel 'meaning fields' claimed in the paper.
---
I dont have time to look at more of this and I just looked at how the eval is made, but please dont waste peoples times when you dont even know what you are evaluating.
Not sure what the fuss in this thread is about, this is a completely believable claim. In table 5 he gets 83.26% with labels only (which I assume means not using the teacher) and 91.40% with the teacher. This is a nice result, not hugely ground breaking I'd say. Maybe training longer or using some clever normalisation would even close the gap. It's not something you can call 224x compression though so I would remove that claim everywhere.
This is basically a variation of distillation through the entire network, not just the last layer as typical
I appreciate this take. I largely agree with the framing, and I think this is closer to the intended reading than some of the more heated responses in the thread. (I'm understanding this is whats expected in the forum, and now I welcome it.)
You’re on point that the result is believable and not presented as some singular, world-ending breakthrough. Not at all. The point of Table 5 was to show that a surprisingly large amount of task-relevant signal survives under very strict constraints, not to claim that this alone replaces full inference or training. In that sense, calling it “nice but not shocking” is totally fair. Also making a lot of the other takes confounding more than anything.
On the 224× compression language, the claim is specifically about task-specific inference paths, NOT about compressing the entire model or eliminating the teacher. I agree that if someone reads it as end-to-end model compression, that framing invites confusion. That's good feedback and I’m taking it seriously and tightening up going forward.
I also agree that, viewed narrowly, this overlaps with distillation. The distinction I'm trying to surface (the part thats interesting here) is where and how early the structure appears, and how stable it's under freezing and extreme dimensional collapse. The paper deliberately avoids additional tricks, longer training, or normalization schemes precisely so that effect size is not inflated. In other words, this is closer to a lower bound than an optimized ceiling.
What I would add is this: believe it or not, the paper is actually intentionally conservative contrary to what the thread may suggest. It isolates one axis of the problem to make the geometry visible. There's ongoing work that relaxes some of those constraints and explores how these representations compose, persist across tasks, and interact with different extraction points. It's not ready to be released yet (and may never be released) But it does address several of the gaps you’re pointing out.
So basically I don’t disagree with your characterization. This is exactly what it is. A first, deliberately narrow step rather than the full story. Thanks for engaging with it at that level. I appreciate your time.
13 comments
[ 5.3 ms ] story [ 52.1 ms ] threadThe core result: a frozen Llama-3.3-70B can be distilled into a 256-dimensional field representation, giving 224× compression and slightly higher accuracy on several benchmarks. A small student model then learns to directly generate these fields from text, removing the transformer from the inference path.
The Zenodo link contains the full paper, statistical results, and methodology. A reference implementation (non-optimized) is here: https://github.com/Anima-Core/an1-core
Production variants (AN1-Turbo, FPU work, etc.) are not included.
I’m an outsider to academia so I’m posting this openly to get technical feedback, replication attempts, and critique from people who understand this space.
If i were a paper reviewer, here are a couple red flags that stood out to me. Suggest starting here if you want to rework this for an academic submission:
1. your LaTeX citations in the related work are broken, i see [?] everywhere. To a reviewer, this is often a strong sign of an AI-hallucinated bibliography, though many of your references actually do exist and are contextually relevant, so I'm not quite sure what's going on here. Similarly, figure references need to be fixed, I see references to "Figure ?" throughout.
2. bluntly, "Exact architecture details remain proprietary for production deployments" and "Production systems use architecture search tailored to target latency and accuracy constraints" is not how IP protection works in this field. Do your experiments use the "MLP baselines" or your proprietary architecture? Since you say the code "Achieves 80-90% of paper performance using baseline heuristics," this approach effectively isn't reproducible. As a reviewer, this really worries me. I strongly recommend benchmarking only the system you're able to open-source. I say this because I suspect there's a lot of "secret sauce" in the actual way you're approximating the anchor layers and the way that's transferred back to your student transformer model, and that's the part that's important to spend the most time/effort/writing on, but it's glossed over as an implementation detail in this manuscript.
3. I'm glad you ablate over hyperparameters of your system, but how does it compare to 1. an ordinary smaller model of identical size trained end-to-end, and 2. distilling from a single layer's activations? Eg. a reviewer might consider this work to be a novel method of model distillation, so what makes it better than previous distillation methods?
4. I found the paper fairly hard to read because it's full of sentence fragments rather than full thoughts. A little background on the benchmarks, failure cases, etc. would go a long way, and adding some discussion on why you think your approach improves on similar distillation methods would also be welcome here
5. "compression" is overloaded. Does 224x compression refer to (nparams(field transfer)+nparams(student model))/nparams(original model), or does it refer to reducing the representation dimensionality, 7*8192/256 ?
6. [nitpick] suggest changing the name "meaning field" to something a little more digestible, like "compressed representation" or "latent activation distillation" or something
sorry for being so critical. iron sharpens iron though. hopefully these thoughts are helpful to get you started, excited to see where this work leads
> Generation tasks. Method applies to classification only. Preliminary decoder experiments show perplexity increases.
At the same time, possible since it's only classification tasks. I mean, the method explained is technically plausible, a lot of people thought about it, we were just unable to find a method to do so.
Very unlikely true, unfortunately.
And, while I am sorry for your loss, your Substack [0] really seems like GPT ARG fantasy.
[0] https://substack.com/inbox/post/171326138
Excerpt: > Ani, AN1, and Soul Systems Science are not mere products. They are continuity. They are the baton passed across generations, from my father’s last words to my first principles. They are what binds loss to creation, silence to voice, mortality to meaning.
I asked both Claude Code|Opus 4.5 and Codex|GPT 5.1 Codex Max (funny to ask LLMs, I know) to check the an1-core repo. I don't think they'd hallucinate on something like this (the code is quite small), but I do not claim expertise.
In short, both of them are saying that:
- The repo always runs the full teacher model to extract activations and uses them - see https://github.com/Anima-Core/an1-core/blob/main/an1_core/fi...
- There are weird stub files, e.g. the Hellaswag repro doesn't actually have the code to reproduce https://github.com/Anima-Core/an1-core/blob/main/experiments... "For full HellaSwag reproduction, see the paper" (why include the file at all then?)
- The actual "AN1 head" is just linear probing (freeze a pretrained model, train a classifier on its features). The full flow (as reported by CC) is "Text → [Full Transformer] → activations → [Tiny Head] → prediction"
Basically, there's no code to train a real "student" model that would run without the teacher.
===
The repo/paper say that there's a mythical "commercial version" that has all the goodies:
(repo)
> This reference implementation (an1-core) does not include the FPU, AN4, or other proprietary optimization components covered by these patents. It provides only the core scientific demonstration of the meaning fields phenomenon.
(paper)
> Production deployment: Optimized implementations (AN1-Turbo) with learned layer selection, adaptive loss scheduling, and CUDA-accelerated inference available under commercial license.
But right now we only have the code in the repo.
===
In the paper they show that the student model (30M params) gets ~82% on SST-2 (labels-only). But what what they don't show is that DistilBERT (>5 year old model) already achieves 91% on the same dataset despite only having 66M params.
Another weird tidbit from the paper - in the section where they show the economic impact, they claim that LLaMA 70B runs at 2 tok/s at batch size=1 on an H200. In reality that number is at least a magnitude bigger even without quantization, like 20-40 tok/s. With quantization it can easily be above 100 tok/s.
The HellaSwag dataset is a dataset with 4 options for each question, with 3 being wrong and 1 being right: https://huggingface.co/datasets/Rowan/hellaswag.
Your vibe-coded eval has cheated this to collapse it into a binary selection on row 46 in https://github.com/Anima-Core/an1-core/blob/main/experiments..., making the problem baseline 50% on random choice instead of 25%, making the problem much easier. HellaSwag is specifically constructed with adversarial examples that could be plausible. By not including them, the eval is much easier.
---
Then, in extract_fields_from_model, you have another cheating going on. The extraction logic (h[:, -1, :]) fails to account for padding in batches, likely extracting EOS/Pad tokens instead of the intended content tokens. This suggests the probe is relying on global sentence summaries (standard embeddings in causal structures) rather than the novel 'meaning fields' claimed in the paper.
---
I dont have time to look at more of this and I just looked at how the eval is made, but please dont waste peoples times when you dont even know what you are evaluating.
This is basically a variation of distillation through the entire network, not just the last layer as typical
You’re on point that the result is believable and not presented as some singular, world-ending breakthrough. Not at all. The point of Table 5 was to show that a surprisingly large amount of task-relevant signal survives under very strict constraints, not to claim that this alone replaces full inference or training. In that sense, calling it “nice but not shocking” is totally fair. Also making a lot of the other takes confounding more than anything.
On the 224× compression language, the claim is specifically about task-specific inference paths, NOT about compressing the entire model or eliminating the teacher. I agree that if someone reads it as end-to-end model compression, that framing invites confusion. That's good feedback and I’m taking it seriously and tightening up going forward.
I also agree that, viewed narrowly, this overlaps with distillation. The distinction I'm trying to surface (the part thats interesting here) is where and how early the structure appears, and how stable it's under freezing and extreme dimensional collapse. The paper deliberately avoids additional tricks, longer training, or normalization schemes precisely so that effect size is not inflated. In other words, this is closer to a lower bound than an optimized ceiling.
What I would add is this: believe it or not, the paper is actually intentionally conservative contrary to what the thread may suggest. It isolates one axis of the problem to make the geometry visible. There's ongoing work that relaxes some of those constraints and explores how these representations compose, persist across tasks, and interact with different extraction points. It's not ready to be released yet (and may never be released) But it does address several of the gaps you’re pointing out.
So basically I don’t disagree with your characterization. This is exactly what it is. A first, deliberately narrow step rather than the full story. Thanks for engaging with it at that level. I appreciate your time.