HN Explorer

Ask HN: Has the LLM/transformer architecture hit its limit?

2 points by ianbutler ↗ HN
Have we hit the limit for performance increases on the current architecture of LLMs?

I’ve heard some amount of agreement among professionals that yes we are, and with things like papers showing Chain of Thought isn’t a silver bullet it calls into question how valuable models like o1 are it slightly tilts my thinking as well.

What seems to be the consensus here?

7 comments

[ 8.6 ms ] story [ 29.7 ms ] thread
ianbutler
Maybe better expressed as are we in a tail end of an optimization phase where we’ll see long tail improvements but nothing generational
cranberryturkey
i think o1 isworse than o4 for coding.
razodactyl
This is a great point for what I was trying to explain about where we currently stand with these models - we seem to be hardware constrained in relation to the trade off between rote learning, full memorisation of information and the model's ability to use what it knows and generalise to achieve an outcome.

Your experience aligns with what I've seen: 4o and o1 are better at filling gaps from each other where there should be an upward trend of intelligence instead of having to jump around as we do currently.

So the real question is: are these models forgetting their skills, do we need to make them larger? Does distillation and generalisation work?

So many questions that might end up as LLMs being unable to escape the problem of catastrophic forgetfulness.

razodactyl
I think there's still a lot of room "relatively" to move around but my current opinion is that hardware isn't where we would want it to ideally be to have next level LLMs everywhere.

We've seen the trend of distilling models at what seems to be the cost of more nuanced ability to iterate and achieve correct results.

I'm very convinced LLMs can go much further than we've achieved so far but I'm very welcoming of newer techniques that will improve accuracy / efficiency and adaptability.

ianbutler
So you see it more of a matter of time until the next leap, depending on how long it takes for hardware bottlenecks to be addressed? I'm assuming we're talking on at least the order of years from your perspective? Based on how long it would take for new chips or some type of training asic.
meiraleal
Hopefully. Then we can start to develop solid software on top of it.