Very cool. Honestly, I’ve been using a mostly vanilla Airbyte. There are already enough connectors that I usually don’t need to write a new one.
Pretty thrilled with how fast these guys have been going honestly. Only picked up their stuff a few months ago and there are already more and better connectors today than there were back then.
I’ve been slowly moving over my error prone sync scripts to my Airbyte instance. Life is already better.
Data engineers, data scientist & analytics engineers. People that be users of Airbyte but also can be contributors.
For data engineers, the objectives is to help them empower other roles to be data fluent and have the freedom to move data without bugging data engineers
How suited is Airbyte for multi-tennant use-cases?
I might have a project coming up to build a semi-analytics solution where you would obviously want to allow the users to easily connect to their data sources, and Airbyte looks like it might be a good fit for that (for now my rough idea was to build some plumbing around Singer taps). Are you aware of any users that use it for such purposes and are there any specific pitfalls too look out for there?
We do support separate workspaces but we don't have auth yet. However because there is an API available, you can always build the account segmentation and permissionning at your app level and only interact with Airbyte using the API.
Been waiting for some practical uses since I read https://news.ycombinator.com/item?id=26650493. I forgot to ask back then if someone had experience with Apache Camel, Spring Integration, or similar and could compare it to the crop of SaaS offerings like Fivetran (that Airbyte itself is taking on by providing an OSS offering).
Same, I was expecting some 3D printing, maybe some experimentation about how much spacing is needed between pins for insulation and how large the contact surfaces should be etc.
That's what I was expecting as well. Guessing that "connector" an industry term that I'm just not aware of. Seems like some pre-made adapter patterns for ETL's from browsing the docs.
This looks great. I’d love to throw up an instance locally and play with it at work, but that likely wouldn’t happen for a long time (local gov -> slow processes).
That being said, I’ve got a fun project in mind which I may find a use case for this. I’ll give it a shot.
It's a module which extracts data from a source (an API, database, etc..) or pushes data to a destination (warehouse, blob storage, API, etc..). They're ubiquitous in data engineering & analytics workflows where users often need to consolidate their data in one database or warehouse to build dashboards, run analyses, etc..
I already responded to a different comment in this thread, but you seem to be knowledgable on the subject. Is this a tool to assist with ETL's between otherwise incompatible sources/destinations?
Indeed! You could use the CDK to build the data integration pipelines that your ETL solution doesn't support.
Airbyte comes also with ~70 connectors. So if we already support the destination you want, you can only build the source connector that we might not support as well. Our goal is support 200 connectors by the end of the year and address the long tail of integrations in 2022 :).
32 comments
[ 3.5 ms ] story [ 81.4 ms ] threadPretty thrilled with how fast these guys have been going honestly. Only picked up their stuff a few months ago and there are already more and better connectors today than there were back then.
I’ve been slowly moving over my error prone sync scripts to my Airbyte instance. Life is already better.
For data engineers, the objectives is to help them empower other roles to be data fluent and have the freedom to move data without bugging data engineers
I might have a project coming up to build a semi-analytics solution where you would obviously want to allow the users to easily connect to their data sources, and Airbyte looks like it might be a good fit for that (for now my rough idea was to build some plumbing around Singer taps). Are you aware of any users that use it for such purposes and are there any specific pitfalls too look out for there?
I fired up an EC2 instance, docker-compose'd up, added the initial few syncs and then let people just add their own.
Mostly without trouble since then, though I'm running an old version as a result of the fire-and-forget.
(edit: also, text me on Keybase or email me since I don't get notifications for responses here)
I was excited.
That being said, I’ve got a fun project in mind which I may find a use case for this. I’ll give it a shot.
Airbyte comes also with ~70 connectors. So if we already support the destination you want, you can only build the source connector that we might not support as well. Our goal is support 200 connectors by the end of the year and address the long tail of integrations in 2022 :).
We built something similar for on-premise data sources, in Clojure. https://basil.net