Show HN: Orbital – Dynamically unifying APIs and data with no glue code (github.com)
I'm excited to share Orbital, a new approach for unifying APIs and Data sources!
Rather than relying on glue code to bridge endpoints, Orbital leverages annotations in schemas & API specs to build the integration dynamically.
The traditional method of crafting glue code often becomes repetitive and burdensome to maintain in the long run.
With Orbital, developers embed tags to their existing API specs (OAP, Protobuf, etc), indicating where data can be sourced, and publish these specs to Orbital (which runs self-hosted).
Consumers query these tags with our TaxiQL language, and Orbital generates the integration on the fly. That could be merging multiple APIs, blending API and database queries, or enriching event streams to craft custom message payloads.
It feels a lot like writing GraphQL, but there's no resolvers to maintain, and producers are free to use a variety of API Spec languages.
The beauty of using tags over field names is the adaptability. As API developers update and publish their specs, the integration remains seamless and automatically adjusts.
Under the hood, the tags (and associated query language) are actually Taxi - an OSS meta-language and toolchain we build (and have shared previously). Orbital is a query engine that executes TaxiQL queries, generating the integration.
We've been working on this for a while, and have a number of production deployments. We recently made the move to make the source available on Github under a mix of Apache 2 (Open Core) and BuSL.
In a nutshell, Orbital excels in Data Composition (covering APIs, DBs), crafting Bespoke Event Streams, and streamlining ELT workloads into databases.
Excited to hear your thoughts and feedback!
16 comments
[ 3.7 ms ] story [ 53.1 ms ] threadYep, we have (early) SDK's for Typescript and the JVM, to submit the query directly.
Also, you can publish queries as standalone APIs, in a BFF-style pattern.
A growing pain for data integrations is data validations. The schema can be correct but the data itself may have incorrect values. I noticed you had a section about the structure and semantics of the data here (https://orbitalhq.com/docs/background/intro-to-semantic-inte...) but don't see where you could validate records using it. How do you handle these scenarios?
We've spiked it in the past, and had some nice designs that attached validation rules semantically to the scalar types (which are very sharable), and models (which are more system-specific).
The early designs felt nice (albeit early), but the deal we were working on didn't eventuate, so we didn't push it through.
I think Validations become really interesting in both evaluation flows (ie., ingesting in a pipeline or through a mutating API), as well as quality scoring for data-at-rest.
I'm excited to explore these in more detail, given a use-case.
There's a video of an early version of it in action here: https://www.loom.com/share/a78c70f0e2bd42a28b7cb0a5d175c3aa
I think what's nice about using LLM's to build Taxi queries (which is what's actually happening here) is that we're not having to push things like API Specs and DB Schemas to the LLM. Instead, we train it on the Taxi types that are available, get the LLM to build a Taxi query, and let Orbital generate the integration.
It means a lot less information is pushed out.
Still quite early, but it's really powerful.
I'm wondering whether GPT-4 can be used for the same task? Only that instead of chatting you will provide a common schema for the integrations and let GPT-4 figure out how to make the requests and fill out the fields, etc.
There are no resolvers to maintain in those either but you get "standard" GQL instead of TaxiQL which I assume has much smaller ecosystem. What are the strengths of TaxiQL?
At a language-to-language feature level, there's some core differences:
GraphQL-without-resolvers still relies on imperative joins defined somewhere. AFAIK, Wundergraph pushes this to the query layer. GraphQL Mesh pushes this to .yaml files. This means that as the sources change, there's centralized config needs to be updated in lock-step, or things break. This means the API producer is now obligated to work closely with consumers to release changes.
TaxiQL infers join logic by introspecting the schemas. This is especially powerful where joins require hops across multiple sources (eg., Id resolution where services use different ID schemes). Because type metadata is embedded at the producer schemas, producers are free to make changes and the query layer adapts.
GraphQL promotes a single graph that everyone cherry-picks from. TaxiQL lets query authors define their own data contract, and own data shapes. This moves transformation of code into the query layer, which we think is a better place for it.
GraphQL doesn't allow extensibility in the language. (We've seen others hack around this with framework-specific annotations, but that's always felt quite awkward). TaxiQL supports extensibility, adding custom functionality into the query layer, and ships with a pretty rich stdlib for things like avg(), first(), last(), etc.
I for one don't care for "logic" in license files <https://github.com/orbitalapi/orbital/blob/7b167ed37c31cf866...> since it makes it very hard to reason about the license usage without, as in this case, viewing every single file to look for license-traps
0: https://en.wikipedia.org/wiki/Monad_(Gnosticism)
1: https://cyclonedx.org/
2: https://stackoverflow.com/ a / 1732454
I've partially tried to address this with the FAQ[0]. Maybe we could move the codebase around, so there's community/ and commerical/ top-level dirs, and associated licenses under there. (Naming TBD).
[0]https://github.com/orbitalapi/orbital/blob/develop/LICENSE_F...
In software eng, the one true metric for delivering is how fast can you iterate on a piece of software. We already have constructs to deal with code duplication (classes, etc).
eg:
Which allows you to query for either "All film reviews", or just a subset.We've found that actually covers about 80% of use cases.
However, some users have asked for more control in the query langugae, which we're working on improving the TaxiQL query syntax for this right now.
Here's what it looks like: