Launch HN: Openlayer (YC S21) – Testing and Evaluation for AI
The complexity and black-box nature of AI/ML have made rigorous testing a lot harder than it is in most software development. Consequently, AI development involves a lot of head-scratching and often feels like walking in the dark. Developers need reliable insights into how and why their models fail. We're here to simplify this for both common and long-tail failure scenarios.
Consider a scenario in which your model is working smoothly. What happens when there's a sudden shift in user behavior? This unexpected change can disrupt the model's performance, leading to unreliable outputs. Our platform offers a solution: by continuously monitoring for sudden data variations, we can detect these shifts promptly. That's not all though – we’ve created a broad set of rigorous tests that your model, or agent, must pass. These tests are designed to challenge and verify the model's resilience against such unforeseen changes, ensuring its reliability under diverse conditions.
We support seamlessly switching between (1) development mode, which lets you test, version, and compare your models before you deploy them to production, and (2) monitoring mode, which lets you run tests live in production and receive alerts when things go sideways.
Say you're using an LLM for RAG and want to make sure the output is always relevant to the question. You can set up hallucination tests, and we'll buzz you when the average score dips below your comfort zone.
Or imagine you're managing a fraud prediction model and are losing sleep over false negatives. Openlayer offers a two-step solution. First, it helps pinpoint why the model misses certain fraudulent data points using debugging tools such as explainability. Second, it enables converting these identified cases into targeted tests. This allows you to deep dive into tackling specific incidents, like fraud within a segment of US merchants. By following this process, you can understand your model's behavior and refine it to capture future fraudulent cases more effectively.
The MLOps landscape is currently fragmented. We’ve seen countless data and ML teams glue together a ton of bespoke and third-party tools to meet basic needs: one for experiment tracking, another for monitoring, and another for CI automation and version control. With LLMOps now thrown into the mix, it can feel like you need yet another set of entirely new tools.
We don’t think you should, so we're building Openlayer to condense and simplify AI evaluation. It’s a collaborative platform that solves long-standing ML problems like the ones above, while tackling the new crop of challenges presented by Generative AI and foundation models (e.g. prompt versioning, quality control). We address these problems in a single, consistent way that doesn't require you to learn a new approach. We’ve spent a lot of time ensuring our evaluation methodology remains robust even as the boundaries of AI continue to be redrawn.
We're stoked to bring Openlayer to the HN community and are keen to hear your thoughts, experiences, and insights on building trust into AI systems.
32 comments
[ 0.31 ms ] story [ 70.6 ms ] thread1. OpenLayer does not say metrics or monitoring to me
2. OpenLLMetry builds on OpenTelemetry, which it very much reminds me of as a name. It's also a much easier add-on to our existing stack. I don't want to have to log into some company's website to view metrics for a single part of my stack when trying to understand why things are not working as expected.
3. OpenLLMetry is open core, which is what devs desire. Who is really using closed source software in this space now (the logmon space, not ai, though both are largely chasing after open dreams)
Openlayer’s also intended to be used on non-LLM use cases. Here are a few other ways we’re different:
1. Support for other ML task types
2. Includes a development mode for versioning and experimentation
3. Native slack and email alerts (openllmetry might integrate with other platforms that do that, but not sure)
4. Collaboration is deeply embedded into the product
https://www.traceloop.com/
openllmetry is focus on engineers, who wants to use this as more of a piping solution and it sits on top of opentelemtry. While opentelemetry is a popular solution. It is just applying a solution to a new problem.
OpenLayers to me is thinking from the ML/AI problems from ground up and while serving the data scientists and probably prompt engineers.
I do wonder if non-ai talent pivoting to AI is wise or defendable. Do you want to use a service where the creators don't really understand the tech and are not much more than a wrapper around an API? Is that really a defensible posture in a competitive market?
We will see. I'm sure some will hire talent and have the data to do something special.
These past few months, however, we’ve prioritized building out features for testing and monitoring LLMs.
LLMs certainly have their unique challenges, but the evaluation problem in general is not new, and much of what we’ve built historically is very much applicable to this new crop of ML use cases!
*ok, there is a gallery project, but something like this I would expect to be the open source variety of startups. I very much expect something like this to be open core.
On open-core — we’ve been considering open-sourcing the engine that evaluates your models. Will have more on this soon!
We’re definitely prioritizing increasing transparency, and we appreciate your feedback about it!
https://openlayers.org/
Integrity tests tackle data quality issues (e.g. no PII in input data, no duplicate rows, schema checks on specific fields).
Consistency tests help ensure your fine-tuning & validation datasets are well constructed in relation to one another (e.g. don’t have overlap, are sized correctly), and your production data doesn’t drift from your reference data.
Performance tests are focused on your model outputs, and measure common metrics for each task (e.g. accuracy, F1, PR for classification) as well as custom metrics designed to be evaluated by an LLM (e.g. “make sure these outputs don’t contain profanity”). You can apply these metrics to specific subpopulations of your data by setting filters on your input fields.
Re: adding your own evals — yes, you can! The evals are not statically defined — they are flexible structures that allow you to customize them to your needs.
Re: importing evaluations from other libraries — this is something we’re adding more support for. We’ve just added an integration with Great Expectations, and can add an integration with OpenAI’s evals if that is something the community is interested in.
If you want to unlock explainability for your tabular classification or regression, or text classification models, you can upload the actual model binary. We support a bunch of frameworks out-of-the-box, but you can use any architecture through our custom upload.
More info:
https://docs.openlayer.com/documentation/how-to-guides/uploa...
https://docs.openlayer.com/documentation/how-to-guides/write...
https://openlayers.org/
It has an API with class names like "Observable", and there are frequent discussions on inputs and performance. It's gonna make searching for one or the other really hard...
We offer more features around error and subpopulation analysis, versioning, running evals during development, and collaboration. Through what (I believe) is a more clean and simple DevEx and UI!
re: Lilac, there’s some intersect w/r/t dataset exploration, but we have more evals than the ones they offer. More than data quality, we give insights into data drift and model performance and let you set up expectations and get alerts on whether they fail during development and production. + distinct in some of the ways described above
We’re really happy to see more tools and platforms in this space. Definitely a big uptick since we started 3 years ago, w the advent of gen ai this is all top of mind (and deservedly so).