Show HN: Pipelex – Declarative language for repeatable AI workflows (github.com)
Why this instead of yet another workflow builder?
- Declarative, not glue code: you state what to do; the runtime figures out how. - Agent-first: each step carries natural-language context (purpose, inputs/outputs with meaning) so LLMs can follow, audit, and optimize. Our MCP server enables agents to run pipelines but also to build new pipelines on demand. - Open standard under MIT: language spec, runtime, API server, editor extensions, MCP server, n8n node. - Composable: pipes can call other pipes, created by you or shared in the community.
Why a domain-specific language?
- We need context, meaning and nuances preserved in a structured syntax that both humans and LLMs can understand - We need determinism, control, and reproducibility that pure prompts can't deliver - Bonus: editors, diffs, semantic coloring, easy sharing, search & replace, version control, linters…
How we got there:
Initially, we just wanted to solve every use-case with LLMs but kept rebuilding the same agentic patterns across different projects. So we challenged ourselves to keep the code generic and separate from use-case specifics, which meant modeling workflows from the relevant knowledge and know-how.
Unlike existing code/no-code frameworks for AI workflows, our abstraction layer doesn't wrap APIs, it transcribes business logic into a structured, unambiguous script executable by software and AI. Hence the "declarative" aspect: the script says what should be done, not how to do it. It's like a Dockerfile or SQL for AI workflows.
Additionally, we wanted the language to be LLM-friendly. Classic programming languages hide logic and context in variable names, functions, and comments: all invisible to the interpreter. In Pipelex, these elements are explicitly stated in natural language, giving AI full visibility: it's all logic and context, with minimal syntax.
Then, we didn't want to write Pipelex scripts ourselves so we dogfooded: we built a Pipelex workflow that writes Pipelex workflows. It's in the MCP and CLI: "pipelex build pipe '…'" runs a multi-step, structured generation flow that produces a validated workflow ready to execute with "pipelex run". Then you can iterate on it yourself or with any coding agent.
What’s included: Python library, FastAPI and Docker, MCP server, n8n node, VS Code extension.
What we’d like from you
1. Build a workflow: did the language work for you or against you? 2. Agent/MCP workflows and n8n node usability. 3. Suggest new kinds of pipes and other AI models we could integrate 4. Looking for OSS contributors to the core library but also to share pipes with the community
Known limitations
- Connectors: Pipelex doesn’t integrate with “your apps”, we focus on the cognitive steps, and you can integrate through code/API or using MCP or n8n - Visualization: we need to generate flow-charts - The pipe builder is still buggy - Run it yourself: we don’t yet provide a hosted Pipelex API, it’s in the works - Cost-tracking: we only track LLM costs, not image generation or OCR costs yet - Caching and reasoning options: not supported yet
Links
- GitHub: https://github.com/Pipelex/pipelex - Cookbook: https://github.com/Pipelex/pipelex-cookbook - Starter: https://github.com/Pipelex/pipelex-starter - VS Code extension: https://github.com/Pipelex...
16 comments
[ 134 ms ] story [ 52.6 ms ] threadWaiting for partnership to propose to our users
The declarative style keeps the workflow detail at a high enough level to iterate super quick - love that. More important to me is that it’s structured and seems like it would be more testable (I see validation in your docs).
Zooming in to the pipe/agent steps I can’t quite see if you can leverage MCP as a client and make tool calls? Can you confirm? If not what’s your solution for working with APIs in the middle of your pipeline?
Also a quick question, declarative workflows won’t solve the fact that LLMs output is always non deterministic, and so we can’t always be guaranteed the output from prior steps will be correct. What tools or techniques are you using/recommending to measure the reliability of the output from prior steps? I’m thinking of how might you measure at a step level to help you prioritise which prompts need refinements or optimisations? Is this a problem you expect to own in Pipex or one to be solved elsewhere?
Great job guys, your approach looks like the right way to solve this problem and add some reliability to this space. Thanks for sharing!
[1]: https://github.com/boundaryml/baml
I'm wondering if this kind of scenario (essentially it's just the input document is too big) is possible to be handled in Pipelex. In my understanding DSL is good for it's high-level abstraction and easy to understand, but lacks flexibility and the power is restricted. How can the users of Pipelex iterate the pipelines to fulfill the complex need when the business logic became complex inevitably?
Regarding large docs, I know what you mean and we've been there: at one point we considered focusing on that feature, build a kind of agentic system to master large docs. At the time in 2023-2024, everyone was relying on vector store RAG and embeddings to solve that problem but we always thought that solution wouldn't bring enough reliability. We wanted the LLM to actually read the whole docs, in order to know what's in there and where. The idea was to read and "take notes" according to different aspects of what it's reading, and synthesize hierarchically from bottom to top. So, that part of the work could be done once and the structured "notes" could be exploited for each use case, pretty efficiently thanks to the zoom-in/out provided by the architecture. We went pretty far in building that solution, which we called "Pyramid RAG" internally, and the aim was to build AI workflows of top of that. But at some point, the building of the pyramid became pretty complex, and we realized we needed AI workflows to build the pyramid itself, and we needed to to do it right. That's when we decided to focus on what would become Pipelex. Now, in our new setting, solving the "large docs" problem is a large problem we can break down into a collection of Pipelex workflows. So it's part of our plan to provide those building blocks, and we hope to involve the community as much as possible, to cover all the possible applications.
I think a solution to this problem as a runnable example would be a nice showcase of what is achievable with pipelex.
See: https://docs.boundaryml.com/home
As an example, I have a use case where I'm processing millions of documents in a fan-out/fan-in pipeline that uses multiple LLMs to analyze and condense information. This mostly isn't CPU-bound, but it does consume a lot of RAM.
I'm currently using Ray to split the workload across a cluster. Ray has autoscaling, so it's very good for this kind of thing.