Show HN: Fructose – LLM calls as strongly typed functions (github.com)

Seems like a great feature (and honestly allows us to do smarter things for strictly structured generation). I'm curious, what's your main motivation for local llms vs hosted APIs?

as with marvin, you can just swap the base url and use any of the oss proxy libs that clone the openai api (but since they don't do function calling [except for mistral i think], its not as good afaik)

currently don't have pydantic support yet, but we're not too opinionated on that. I know it's seemed to emerge as a standard, and I imagine useful in the context of running fructose in a FastAPI handler, but we led with dataclasses because they're language native and achieve much of the same thing

It's not 100% yet. Route to that: 1. Clientside, retry strategy on failed parse. Not yet implemented, we throw an exception on parse fail right now, but soon to be implemented. Not ideal because of token burn and latency, but the best quick solution. 2. For the custom model we're building, we use strict grammar definitions to bias outputs toward the needed structure (or if there is only one structurally correct token, outright skip the generation of that token and insert it directly).

I've been impressed at how well gpt-4 does with the default prompt template we use. Even better if you enable the chain_of_thought flavor.

Good Lord that sounds terrifying - I do in general prefer my type definitions to be at least approximately deterministic.

Many of our early users have said this as well. I don't want this to turn into an abstraction monstrosity: the more unadulterated the prompt, the better. We're looking to outlines as inspiration for doing this logic as part of the model vs the client. Thanks for the links!

Still learning about the landscape so can't give informed opinions. LMQL is a new one for me, will check it out.

What we're mostly going for is composability vs abstraction. What's the smallest nugget of lift we can do for you, to make it feel natural to implement what you want? In this case it's treating the calls as functions and leaning on native python features like functions, docstrings, and types, so you can still use the python language like closures to do the weird things you need.

This is all handwavy, put on my wizard language design hat, so take it with a grain of salt. We're just trying things out.

What is TGI? Is that the huggingface Text Generation Inference project on GitHub?

Why do you have Guidance in caps?

https://github.com/guidance-ai/guidance

or ...

https://huggingface.co/docs/text-generation-inference/concep...

or ... ?

A quick glance through these, they don't seem yet to leverage json_object on OpenAI with the word JSON in the prompt, which works wonders with the 0125 models.

eventually, but priority goes toward finding an abstraction that feels right. We're very likely to break this package API, still v0. Sticking with openai till we have more confidence in the foundation being correct.

Currently, quite comparable and obviously Instructor is more mature and feature rich. They're going the "patch the openAI client" approach which makes code written still use openAI SDK patterns which is pretty smart. Jason seems like he knows what he's doing.

We're trying to make it more of a language feature with the decorated functions. Plus exploring the hosted formatting model direction.

How do you feel this compares? Do you think there's any gaps in current tools worth working on?

Plus one on this question! Seems very similar.

yeah I had a moment working with fructose where I realized "oh this is more like functional programming than I expected"

Is the F# POC open source? Link?

Yeah, functional programming and pure functions seem perfect for generative programming. Granted, I think they're perfect in general, but as long as human programmers are still stuck in the world of object-orientation (the modern sense), then they're going to be wasting the time in the LLM feedback loop. The LLM should be be able to write a unit of code however it wants in a way that is as self-contained as possible. Since an LLM can, in theory, quickly "understand" code that most software engineers would object to, then we should get out of the way of LLMs rather than expect them to be like we are.

This project seems pretty different to what you've proposed. Fructose looks like it's "just" asking an LLM to evaluate a function (written in English text), and then jamming whatever comes out back into your type system.

Being able to sometimes answer a given question is perhaps a first step to writing code that can answer that question reliably, but it's a long way from an LLM that does the former to one that does the latter.

The field that has been doing this is called Program Synthesis. Here’s an example survey:

https://www.microsoft.com/en-us/research/publication/program...

I’ve wanted to see the traditional techniques combined with modern ML to sort of drive the search and generation process. Then, we’d still have the advantages of both formal specifications and classic AI (esp traceability). While looking for a synthesis link, I stumbled onto one paper trying to mix the two approaches:

https://ojs.aaai.org/index.php/AAAI/article/download/5048/49...

Oh yeah, you just reminded me of this cool talk I saw at Strange Loop a while back. Not about the AI parts but re: program synthesis in Haskell:

"Type-Driven Program Synthesis" by Nadia Polikarpova https://www.youtube.com/watch?v=HnOix9TFy1A

Links to more projects and papers by Prof. Polikarpova: https://cseweb.ucsd.edu/~npolikarpova/

I think this is one of the main projects she discusses in the talk: https://github.com/nadia-polikarpova/synquid

EDIT: meant to mention this too, which I think has been around a bit longer, not that I've ever used it in production: https://ucsd-progsys.github.io/liquidhaskell/

> Write a type declaration for a function with an expressive type system, e.g. refinement types. Then use LLMs + SAT/SMT to generate provably correct code.

This is how I use copilot currently, so I might not be following on what part of this is 'future' facing or relevant to this Fructose project?

Not being contrarian, I thought this was an interesting point but as I thought about it more I realized, "wait, they're describing what I already do".

Oddly similar to summoning demons.

answered in different thread. tldr: not that different for now. we're likely to do some serverside optimizations, esp. given our gpu inference history.

yeah this seems to be pretty much the same interface as `fn` from marvin, except w/o pydantic (see https://github.com/PrefectHQ/marvin?tab=readme-ov-file#-buil...)

Thanks for asking, and I'd agree. I'd give the same answer as the folks asking about instructor: we built this in a week and are sharing it early, this package API happens to have landed on what Marvin is doing, we're likely to change over time, especially leaning toward running our own models as part of it.

Thanks!

Yeah the pyright doesn't like the annotated return type not being honored by the empty stub function. I wonder if there's a way to trick it.

For your suggestion, the decorator would still be required to overload the function execution with the remote call, otherwise you'd just be calling the function body, but we have considered special wrapper return types to help play better with pyright (and also give programmatic access to debug details of the call), but that'd add bloat to the package and subtract from the more native python feel we're aiming for.

  @ai()
  def stub() -> int:
      """docstring"""
      ...

  # (use ... instead of "pass" in the function body)

Wouldn't the correct way be with the use of ellipsis, as is used in type stubs?

  @ai()
  def describe(animals: list[str]) -> str:
    """ Given a list of animals, use one word that'd describe them all. """

    ...

Technically a function body needs at least one statement. A docstring is just an expression statement (a string), so a function definition with just a docstring is synctatically valid Python. I've seen people say multiline string literals are Python's version of multiline comments, but that's really just convention; it's a noop expression statement. Same as doing

    def foo():
        4

Which is also an expression statement as a function body, and also does nothing. Contrast to actually using a comment as a function body; comments aren't statements (nor expressions, so they can't be used as an expression statement):

    def foo():
        # this doesn't work

> IndentationError: expected an indented block after function definition on line 1

Of course, this doesn't really matter at all, and I get that it feels strange. I've just been thinking about grammars and syntax lately, and it's been interesting to now have the vocabulary and mental model to understand these unintuitive things :)

So you're saying they should ensure compatibility with all LLMs on Day 0 so you can avoid a personal "grating" feeling. It's called an MVP.

True, Grammar has been around with Llama.cpp since the early days IIRC. Microsoft had Guidance as well & now TGI supports it as well. End game for Langchain/Llamaindex if just guarantee or structure adherence was the only reason to use & beg the LLM for usable output.

absolutely! My DMs are open at @erikdunteman on twitter, or erik at banana dot dev

it seems this is in the context of "extraction" where all of the data is already present in the input text, and all that's needed is the reformatting. This is something we've been wrestling with (even today): is the role of fructose and/or our eventual formatting model to provide both intelligence + formatting (such as generating novel data, on the fly, constrained to structure), or just formatting (anything-to-json). Not sure what the answer is and not expecting one, we're just realizing the clear split between needs of users running extraction vs more generative/creative tasks.

OpenAI lets you force valid JSON now

https://platform.openai.com/docs/guides/text-generation/json...

we don't need to limit ourselves to context-free, either. its possible to enforce scope as well, and even force per-token type correctness, as least for somewhat syntactically well-behaved languages that use local type inference.