HN Explorer

Best way to build a integration solution for a SaaS platform

10 points by aschema_123 ↗ HN
We're building integrations for our customers to extract data from legacy on-premise or cloud-hosted, but still old sources (think medical records, HR record servers, etc), centalize it, display it, etc.

These tools each have convoluted HTTP REST / SOAP APIs

How would you go about architecting a solution that allows a customer to self-serve connect to their sources and our product would then ingest and cache the data to be served to them in (sometimes) synthesized form?

We're looking at ETL tools etc. but any thoughts come to mind here?

3 comments

[ 1.6 ms ] story [ 201 ms ] thread
pqyzwbq
Maybe you can check:

  - https://github.com/turbot/steampipe (which will query the sources on demand): Use SQL to instantly query your cloud services (AWS, Azure, GCP and more). Open source CLI. No DB required.

  - https://github.com/cloudquery/cloudquery (which will retrieve data in batch, and save them into another database for query)
creidHyyphen
Hey! My startup company (https://hyyphen.io) is building a workflow automation platform that addresses this need.

Not looking to cold sell my product - but I'd definitely be keen to chat with you and learn more about your use case(s). If there's a fit, great, and if not it'd still be valuable just to understand what you're looking to accomplish.

If you're interested, shoot me an e-mail - creid@hyyphen.io

ensemblehq
The general principles of ETL/ELT I assume would apply: identify the source, determine the requirements (I.e. frequency, volume, formats, etc.), determine the transformations, and ingest. You’ll also need to think about how often you need to ingest and the sync patterns between your source and your target.

Other considerations include security for PII, HIPAA (given you’re dealing with medical records).