The doom video is also in the article itself (headline: "Doom").
I suppose this is the same video as the one from the parent comment, but I don't know for sure - I don't have a twitter account and the above link doesn't work for me.
But when their system is given the instruction "do not fire, simply dodge" - it doesn't "simply dodge", it actually gets close to the fleshy pink demon rather than keeping its distance. Or am I misunderstanding?
I'm not sure the authors realize this is way more than "just a cool demo": if this holds up, it's going to be huge for game QA work.
Instrument your game to output properties of entities near the player and the output is the various control inputs - moment to moment gameplay gets solved. Maybe augment with a tick-by-tick controlled stepping mode if particularly twitchy - an LLM can take care of the higher level reasoning then.
Signed up for the beta! :) would love to put this through some real-world shootouts against traditional LLMs to see where this type of model really excels.
I’m guessing it might be able to replace maybe 40-70% of LLM calls for a given pipeline depending on the business task, cutting the API costs on those calls by an order of magnitude.
It could be used for coding if you gave it an AST.
If you work at TypeSafe please try this.
Side note: This is probably how LLMs would perform with better encoders and next-latent prediction, so eventually those will beat this architecture out. Still amazing though.
I've implemented tree-sitter in pi before, and while it works, I have no real proof it saves me tokens, or is more accurate. I think a better implementation is a model that's trained for AST's, not just "use tool, see what happens".
I'd love to do research on this when I have the time.
I saw the CEO reply elsewhere in the comments to some other question. Maybe he can shed some light on it. My gut feeling is that this is non-trivial and they did not get this to work (yet?), otherwise I can’t come up with a good reason as to why they would not demo that as I assume half of the crowd here (myself included) would line up as customers.
the hard part for coding is actually state engineering (e.g. getting your dependencies in context) - we haven't even tried it yet (because my philosophy is we should automate the easy tasks before the hard and we've been working on getting the model smart on the former)
we do think there's a lot of potential though and do want coding themed releases soon
I could see Jev being great at finding key symbols in codebase before a code generation/code review task. I sent you guys an email (to hello@) about using Jev in Code Review for www.ellipsis.dev.
It's a bit hastily put together, but I made a dspy fork where you can add a decorator to automatically use TypeSafe where possible on Signatures. It shows a fair bit of what actual, hands on usage looks like.
Super intrigued by this - large scale automation using LLMs is quite annoying due to deprecation cycles of models from frontier labs and cost of running your own being prohibitive when you have a blend of them.
I would love for things like this to be accessible via hubs like open router or AWS bedrock. It's hard to justify adding new model vendors directly with all the heightened concerns about privacy and security, but if bold new capabilities are added to a centralized already-vendor like AWS, technical people can adopt them without going through a whole compliance/purchasing/vendor review process. And an extra middleman tax is well worth it when the cost savings of the model itself can be one-two orders of magnitude.
I think the trouble is that Typesafe APIs don't fit into the normal OpenAI-style API that every other regular LLM provider users. You're not just providing unstructured text and getting unstructured text back. It would take a different request and response format than every other model on Open Router. Though you could shoe-horn it in some way, it'd be hacky.
But agreed it'd be very useful to see it deployed on other hubs, and it seems worth it to provide the bespoke API format. Perhaps Typesafe's API will end up becoming the standard for a new type of structured model, the way OpenAI's API did.
This sounds good but so far all claims just sound like marketing terms. I'd love to see real proof. e.g. "RLCD" and "parallel sampling" have nothing to back it up.
also "70-500ms vs 3-329 seconds" are apples-to-oranges unless the LLM baseline is doing comparable work (e.g., long chain-of-thought). If Jev is skipping generation entirely for a narrow structured task, of course it's faster.
Nonetheless i want this to be true, so I'm looking forward to Jev
> If Jev is skipping generation entirely for a narrow structured task, of course it's faster
I think this is reasonable if people are actually using LLMs to solve this type of narrow structured task, which they are. The evidence is that every LLM provider has some method of forcing the output to conform to a json schema in their documentation.
> I really have to say that I like their manifesto
Their manifesto: "you only build on top of it if it's trustworthy." - the irony of this while putting out the most misleading, dishonest marketing campaign I've seen in months for their first public appearance doesn't exactly scream "trustworthy" to me.
Why did they pick the name System One? It's not really explained what "System One tasks" and "System One shaped queries" are. Things that need a fast response?
Does this imply it's a very small model? I couldn't find anything about the model itself.
If you zoom in (especially on the large title), you'll see that the text is a semi-transparent gray with a black internal outline. It seems like all the typography is SVG-rendered. Actually insane. I've never seen this before. Not even the most vibeslopped websites have that.
> We deliberately chose not to publish performance against public benchmarks. In fact, we plan to only have one-off evals when we make product updates.
lol, I bet they would publish them if their score on those benchmarks were good.
I could see this being fantastic for classification tasks. Last year I shifted from using LLMs for bulk data classification tasks to generating embeddings and categorizing based on cosine similarity. It saved a ton of costs and time, but wasn't as accurate as LLMs. This seems like it can give me Terra-level classification ability with the cost/speed I need.
Or a partially completed song, asking for the next note. I’m not sure if you’re joking, but using it for space constrained next token generation within a constrained grammar sounds like a really neat use case.
It looks like a specialized encoder-only(-ish) transformer with scalar and ordinal output heads. Acausal in effect, maybe? Probably not even autoregressive?
I'd use this as a tool an LLM can use for specialized tasks. It's not AI in itself.
375 comments
[ 0.21 ms ] story [ 118 ms ] threadThe doom demo is quite cool
https://x.com/completeskeptic/status/2099925687465570372
I suppose this is the same video as the one from the parent comment, but I don't know for sure - I don't have a twitter account and the above link doesn't work for me.
I can see the individual tweets in the browser while not signed in though.
It's in the parent article under a section named "Doom" in case that asset URL ever changes.
Instrument your game to output properties of entities near the player and the output is the various control inputs - moment to moment gameplay gets solved. Maybe augment with a tick-by-tick controlled stepping mode if particularly twitchy - an LLM can take care of the higher level reasoning then.
I’m guessing it might be able to replace maybe 40-70% of LLM calls for a given pipeline depending on the business task, cutting the API costs on those calls by an order of magnitude.
Yes, that’s the kind of attitude I want to see in these model releases
https://x.com/CompleteSkeptic/status/2099925682726002904
Reminds me of this: https://www.reddit.com/r/ITcrowd/comments/tg05j1/i_cant_beli...
If you work at TypeSafe please try this.
Side note: This is probably how LLMs would perform with better encoders and next-latent prediction, so eventually those will beat this architecture out. Still amazing though.
I'd love to do research on this when I have the time.
we do think there's a lot of potential though and do want coding themed releases soon
youd ask it to pick a location on the ast to add something from the grammar?
i dont see how this stays confined well enough? make a new output space every time?
I think we'll see a bunch of different architectures over the next five years.
https://github.com/typesafeainate/dspy-typesafeify
Thanks for putting this together. I'm surprised the cost saving is so little though. I expected much more based on the post.
I guess you can choose your provider still? But isn't the point that the lowest bidder is doing inference?
But agreed it'd be very useful to see it deployed on other hubs, and it seems worth it to provide the bespoke API format. Perhaps Typesafe's API will end up becoming the standard for a new type of structured model, the way OpenAI's API did.
also "70-500ms vs 3-329 seconds" are apples-to-oranges unless the LLM baseline is doing comparable work (e.g., long chain-of-thought). If Jev is skipping generation entirely for a narrow structured task, of course it's faster.
Nonetheless i want this to be true, so I'm looking forward to Jev
I think this is reasonable if people are actually using LLMs to solve this type of narrow structured task, which they are. The evidence is that every LLM provider has some method of forcing the output to conform to a json schema in their documentation.
Their manifesto: "you only build on top of it if it's trustworthy." - the irony of this while putting out the most misleading, dishonest marketing campaign I've seen in months for their first public appearance doesn't exactly scream "trustworthy" to me.
> [them] Output tokens: FREE (too cheap to meter).
I'm very confused by this.
edit: looks like a framer export where there is a text stroke being applied :|
Does this imply it's a very small model? I couldn't find anything about the model itself.
lol, I bet they would publish them if their score on those benchmarks were good.
it is a structured data model, but technically not a language model (it doesn't generate language)
this is complex, but generating text is highly complicated and requires mode dropping to make long cohesive text
I'd use this as a tool an LLM can use for specialized tasks. It's not AI in itself.