Show HN: MonkeyPatch – Cheap, fast and predictable LLM functions in Python (github.com)

Great questions! The tests act as few-shot examples for the LLMs, which has been shown to guide the style and accuracy of model outputs and improve performance quite well. For instance we’ve seen accuracy go from <70% to 93%+ vs without including the tests. The hallucinations are still an inherent risk with LLMs, especially with long-form context, but adding more diverse and well aligned examples as tests does reduce the hallucination risk and align the outputs with user intent. In terms of performance management and monitoring, QA for LLMs is a difficult process to get right and we’re looking into ways how to a) make it easy for users to test out different function descriptions and tests on their own datasets to gauge performance and b) introduce ways how to seamlessly carry out continuous monitoring of function outputs with low effort. Still WIP but will keep you posted!

The type constraints are indeed enforced but not by the tests but by the type-hints you give to the patched functions. The constraints and enforced structure are followed, there is also a repair feedback loop in place if the original LLM output is invalid for the types you've declared. Tests are more to align the model how to act for different inputs. Hope this makes it more clear!

Thanks! I find the enforced typed outputs and structured object creation from unstructured inputs very useful, for instance we created a use-case around creating structured support-ticket objects that could be processed in downstream applications without worries of anything breaking or bugs

Cheers!

Thanks! A big part of MonkeyPatch, which Langchain or OpenAI are lacking, is the model distillation aspect, which can reduce costs up to 10x and latency up to 6x in some of the tests we've been running. This means the more you use MonkeyPatch the cheaper the function calls get, which is beneficial for high usage applications with lots of calls

Great to know! We're working on extending MonkeyPatch to typescript, the work-in-progress repository can be found here https://github.com/monkeypatch/monkey-patch.ts

We will keep you posted on when it'll be ready for trying out!

I built a similar library for Typescript: https://github.com/jumploops/magic

Please note: it requires the use of ttypescript or ts-patch, as Typescript transformers aren’t supported by default!

Heya, no worries - I’m glad to share it.

MonkeyPatch ensures reliability through what we call ‘test-driven alignment’, in which the tests that reference the patched functions are guaranteed to pass. The more align ‘tests’ you create, the more rigorous a contract that the functions have to fulfil.

The other way to increase consistency is using more constrained type annotations (i.e using pydantic field annotations), which is a similar concept to MarvinAI and Magentic.

Hey! There are 2 main similarities to Marvin, namely: (a) functions that act as APIs to the LLM backend, and (b) type coercion to ensure that the responses fit into the data model of your application.

However, there are a couple of big additions to Marvin as well, namely: Test Driven Alignment - by using ‘assert’ statements that declare the behavior of a patched function, we create a contract that makes invocations much more predictable, which makes it possible to use these functions in production settings.Automatic distillation - a combination of the function contract defined in function type signature and the alignment tests means we can automatically swap out bigger models for smaller ones. This saves up to 80% of the latency, and 90% of the cost of running these functions (check the benchmarks) Check out the readme, as there is more detail on these points there!

(As I understand it) monkey-patch means to modify code in runtime. I thought the naming was relevant because by adding ‘@monkey.patch’ to an unimplemented function, this library gives it an implementation at runtime.

Native support for OS LLMs is on the roadmap - the main challenge is to figure out how to manage the knowledge distillation for local models. It’s a top priority (along with typescript support), so check back in a few weeks?

Right now any ‘plug-in’ model needs to conform to the OpenAI API.

(P.S Carthago delenda est)

Monkeypatch (as I understand it) means to modify code at runtime. This library modifies functions at runtime to use an LLM as an execution target - I thought it was an apt (if admittedly cheeky) name! Appreciate the critique regardless.

Check out the ‘function_modeler’. Currently it’s OpenAI only, but local models are on the immediate roadmap.

https://github.com/monkeypatch/monkeypatch.py/blob/master/sr...

Thanks a lot. I’d really appreciate any feedback you have on the design!

The python package (and repo) is called ‘monkeypatch.py’ for the avoidance of confusion.

The big one is a Typescript implementation. Other than that, the plan is to support other models (e.g Llama) that can be fine-tuned.

Finally, other persistence layers like S3 and Redis, to support running on execution targets (like AWS Lambda and CloudFlare workers) that don’t have persistent storage.

I think it could be really interesting to support Vercel more tightly too. We currently support Vercel with Python, but I think Typescript + Redis would really enable serverless AI functions - which is where I think this project should go!

PyMonkeyPatch? MonkeyPatch.py?

I would quite like a short and distinctive name!

LLMonkeyPatch is one of the best suggestions here, only adds two letters that fence nicely the scope of the monkey patching, and looks playful.

Other suggestions, like PyMonkeyPatch, leave the reader to guess what is being monkey patched.

The IDEs shouldn't complain if the function has a docstring (which all the MP functions should have as that's the instruction that is executed) and the @patch decorator, atleast the ones we have tried it with have liked the syntax in that sense so far. But adding a "pass" is also permissible if the IDE does complain

Do any other names jump out at you as preferable?

Good to know, we'll make it more clear in the docs! To answer regarding these 2 areas,

1) The data for finetuning currently is saved on disk for low latency reading and writing. Both test statements and datapoints from the function execution are saved to the dataset. We also are aware that saving to disk is not the best option and limits many use-cases so we're currently working on creating persistence layers to allow communication with S3 / Redis / Cloudflare as the external data storage.

2) Currently starting the fine-tuning job happens after the dataset has at least 200 datapoints from GPT-4 executions and align statements. Once the finetuning is completed, the execution model for the function is automatically switched to the finetuned GPT 3.5 turbo model. Whenever the finetuned model breaks the constraints, the teacher (GPT4) is called upon to fix the datapoint and this datapoint will be saved back to the dataset for future iterative finetuning and improvements. We are also working on adding in ways for the user to include a "test-set" which could be used to evaluate if the finetuned model achieves the required performance before switching it as the primary executor of the function

Hope this makes it more clear, if you have any additional questions, let me know!