LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics (arxiv.org)

This Yann LeCun lecture is a nice summary of the conceptual model behind JEPA (+ why he isn't a fan of autoregressive LLMs): https://www.youtube.com/watch?v=yUmDRxV0krg

jepa shows little promise over traditional objectives in my own experiments

> using imagenet-1k for pretraining

Lecun still can't show JEPA competitive at scale with autoregressive LLM.

More optimistic signal it’s very early innings in the architectural side of AI, with many more orders of magnitude power-to-intelligence efficiency to come, and less certainty today’s giants’ advantages will be durable.

lolJEPA

I am a bit confused by the benchmark comparison they are doing. The comparison of a domain specific "LeJEPA" on astronomy images against general models, which are not explicitly fine-tuned on astronomy images seems misleading to me.

Does anybody understand why that benchmark might still be reasonable?

I'm a bit confused about the geometry. I'm not sure if the result ends up being like an fuzzy hypersphere or more like an "spiky hyperstar".