Singer is an open-source standard for writing scripts that move data between databases, web APIs, files, queues, and just about anything else you can think of. Lots of companies build ETL scripts to move their data, and there's a huge amount of rework that happens from company to company. We believe that developers should spend less time moving their data and more time using it.
We're open sourcing 12 of our integrations (with more to come) so that they can be used in other applications, and we're excited to see what the community builds. Let me know if I can answer any questions.
There are a couple reasons why we included schemas in the spec:
- JSON doesn't have a robust set of data types, and specifically lacks a datetime/timestamp type. With a schema, Taps can, for example, denote fields in the JSON that contain datetimes represented as strings, and then targets can convert those to proper datetimes and handle them accordingly.
- Dealing with un-structured or flexibly-structured data is hard. Requiring a schema forces a Tap author to think about the structure of the data up front. By validating each data point against a schema, the Tap author should be able to more quickly identify nuances in the data set - like missing fields, nullable fields, mixed-type fields, etc - and either decide to clean them out of the data (if appropriate), or provide the right schema to inform downstream applications about them. Identifying and handling these problems requires an understanding of the source data set, so it is best done as close to the data source as possible.
However, since Singer is built around piping data between applications, your suggestion - to code something that sits between taps and targets - makes perfect sense. The whole "flow" would look like:
The only thing I'd add from Chris's blog post is that in the workflow we tend to see is that most of the transformations tend to be done after loading into the destination. For example, in Redshift the transformations could be defined in SQL or Python UDFs.
We'll definitely be adding targets over time, and BigQuery is a likely one.
It's not as much a matter of open-sourcing the targets within Stitch because our functionality around data warehouse loading includes a lot of things like multi-tenancy, monitoring, and alerting that are necessary for running a SaaS service but not as relevant for a single organization's internal use.
In the meantime, our hope with the CSV target was that it would be a helpful intermediate step for getting data almost anywhere. Here's Google's documentation for loading CSV files into BigQuery: https://cloud.google.com/bigquery/loading-data#loading_csv_f...
The longer answer is that it may take us a while to get to 100% open source, but that's the direction we're moving. All of our new integration development will be open source and be part of the Singer project. Our original integrations were written in a different framework and couldn't be run independently of Stitch, and it's a nontrivial amount of work to convert them to the Singer format.
We included several of our existing integrations as part of this launch, and we'll definitely be adding more of them as well as new integrations.
The most similar thing I've found is Apache Beam[1] which allows you to run ETL code on multiple engines but doesn't necessarily come with off the shelf integrations for commercial products. As far as "how ETL is done today" it's definitely going through changes Airflow[2] is a popular open source tool that can run ETL pipelines but it's more of a task runner so again you're writing your own extraction and loading logic for things off the shelf CRMs and Accounting tools. I'm excited to not have to write extraction scripts for third party tools that I'm sure other people have also written. Very cool project.
We are using Apache NiFi[1] to handle a lot of our ETL use cases.
We have HTTP endpoints set up to receive data from our ERP's accounting system to send data to Concur and to update customers' Lawson punchout ordering systems with shipment information. The 'E' is an HTTP post with an XML payload. The 'T' consists of using the payload to query other databases to build the 'L' payload, and the 'L' is an HTTP post to the consumer's endpoints.
Further, we have NiFi handling HL7 messages inside hospitals. The 'T' is the real winner, here. NiFi has built in transformer for HL7 messages which makes them a breeze with which to work.
EDIT: I wanted to add that we've also used the EventHub processor to connect NiFi to Azure's services as well, and it has been rock solid for us.
NiFi's data provenance, flowfile/attribute system, back pressure settings, auto-queueing, retrying capabilities all make it very reliable and robust.
Looks very cool and it definitely addresses a problem that not only many organizations have but also many individuals have. I know personally I have struggled with moving my data out of various services many times before.
The question I would have to you is this: if the API (or however a tap extracts information) for a given service changes, who is responsible for updating the tap, and is there any way to verify that a tap will extract the data correctly and that said tap isn't assuming an old version of the API?
(Sorry if I'm misunderstanding the way that Singer or any of its associated services work)
That's a great question. For any Singer integrations that we include in the Stitch product, we (Stitch, Inc.) will ensure they stay up to date with the most current version of the API.
That may not be the case with 100% of the community built/ maintained integrations, but our goal with the Singer Slack community and list serve is to connect people using these taps to so that they can validate which are working well and which need additional work.
This has been attempted before by many companies and many standards. No standard is robust enough and no company strong enough to get everyone to use it. I really really hope is gains adoption though!
Thanks nwellinghoff! I'm sure that there will be cases where Singer is not the best tool for the job, but hopefully it will still make a lot of engineers' jobs easier.
I'd love to know which previous standards you're referring to, could you elaborate?
22 comments
[ 3.5 ms ] story [ 76.0 ms ] threadSinger is an open-source standard for writing scripts that move data between databases, web APIs, files, queues, and just about anything else you can think of. Lots of companies build ETL scripts to move their data, and there's a huge amount of rework that happens from company to company. We believe that developers should spend less time moving their data and more time using it.
We're open sourcing 12 of our integrations (with more to come) so that they can be used in other applications, and we're excited to see what the community builds. Let me know if I can answer any questions.
I read through https://github.com/singer-io/getting-started/blob/master/SPE..., but I'm trying to better understand why they're necessary.
- JSON doesn't have a robust set of data types, and specifically lacks a datetime/timestamp type. With a schema, Taps can, for example, denote fields in the JSON that contain datetimes represented as strings, and then targets can convert those to proper datetimes and handle them accordingly.
- Dealing with un-structured or flexibly-structured data is hard. Requiring a schema forces a Tap author to think about the structure of the data up front. By validating each data point against a schema, the Tap author should be able to more quickly identify nuances in the data set - like missing fields, nullable fields, mixed-type fields, etc - and either decide to clean them out of the data (if appropriate), or provide the right schema to inform downstream applications about them. Identifying and handling these problems requires an understanding of the source data set, so it is best done as close to the data source as possible.
Should they code something, to be plugged between taps and targets?
Do you intend to include such transforms in your solution?
However, since Singer is built around piping data between applications, your suggestion - to code something that sits between taps and targets - makes perfect sense. The whole "flow" would look like:
$ tap-mydatasource | do-aggregations | target-mytarget
We'd be eager to hear from anyone who tries this approach!
It's not as much a matter of open-sourcing the targets within Stitch because our functionality around data warehouse loading includes a lot of things like multi-tenancy, monitoring, and alerting that are necessary for running a SaaS service but not as relevant for a single organization's internal use.
In the meantime, our hope with the CSV target was that it would be a helpful intermediate step for getting data almost anywhere. Here's Google's documentation for loading CSV files into BigQuery: https://cloud.google.com/bigquery/loading-data#loading_csv_f...
The longer answer is that it may take us a while to get to 100% open source, but that's the direction we're moving. All of our new integration development will be open source and be part of the Singer project. Our original integrations were written in a different framework and couldn't be run independently of Stitch, and it's a nontrivial amount of work to convert them to the Singer format.
We included several of our existing integrations as part of this launch, and we'll definitely be adding more of them as well as new integrations.
ETL definitely existed before Singer and Stitch, however, and two of the most well known enterprise-focused products are Informatica and Talend.
1. https://beam.apache.org/
2. https://airflow.incubator.apache.org/
We have HTTP endpoints set up to receive data from our ERP's accounting system to send data to Concur and to update customers' Lawson punchout ordering systems with shipment information. The 'E' is an HTTP post with an XML payload. The 'T' consists of using the payload to query other databases to build the 'L' payload, and the 'L' is an HTTP post to the consumer's endpoints.
Further, we have NiFi handling HL7 messages inside hospitals. The 'T' is the real winner, here. NiFi has built in transformer for HL7 messages which makes them a breeze with which to work.
EDIT: I wanted to add that we've also used the EventHub processor to connect NiFi to Azure's services as well, and it has been rock solid for us.
NiFi's data provenance, flowfile/attribute system, back pressure settings, auto-queueing, retrying capabilities all make it very reliable and robust.
1. https://nifi.apache.org/
The question I would have to you is this: if the API (or however a tap extracts information) for a given service changes, who is responsible for updating the tap, and is there any way to verify that a tap will extract the data correctly and that said tap isn't assuming an old version of the API?
(Sorry if I'm misunderstanding the way that Singer or any of its associated services work)
That may not be the case with 100% of the community built/ maintained integrations, but our goal with the Singer Slack community and list serve is to connect people using these taps to so that they can validate which are working well and which need additional work.
I'd love to know which previous standards you're referring to, could you elaborate?
https://en.m.wikipedia.org/wiki/Microsoft_BizTalk_Server