HN Explorer

Ask HN: Tools to call REST API and export to CSV?

4 points by gabrielsroka ↗ HN
I've written my own in JavaScript, Python, PowerShell, and I've used Microsoft Power BI/Power Query, but they are slow, don't work well, require custom code, etc.

I need to:

1. call REST APIs (with Authorization HTTP header) and get JSON (arrays of objects), also parent-child endpoints (parent has IDs of child objects) -- nested loops

2. paginate using the Link HTTP response header, observe rate limits (HTTP response headers, to avoid 429 errors) and concurrency limits

3. (nice to have) select certain fields from the JSON object; also, data can be "L-shaped" (not every record has the same fields)

4. export to CSV

Open source, cross-platform would be good. Needs to be easy for not-very-technical end users to use.

6 comments

[ 3.8 ms ] story [ 26.5 ms ] thread
nhayfield
datatables.net is a good modular javascript library that handles pagination and child tables and export to csv/excel. you will still need to write some code for formatting the api calls into a useable format.
gabrielsroka
Thanks for the tip.

I gave it a try, and got something really basic to work. It took me several hours (part of it was my fault, part of it was the documentation and its defaults that expect the data in a particular format, and the error messages that weren't very helpful).

It's a complex tool, and it does a lot, but not what I need.

nhayfield
There are no silver bullets here. It is always alot of work.
gabrielsroka
I agree. I've already written 1,000s of lines of code for this. I was trying to a) see if the problem had already been solved, b) see if it was worthwhile for me to continue trying to solve it.

The MS Power Query / BI tools come really close, but not quite.

As I like to say, they do the first 80%, but it's that last 80% that you still need :)

majkinetor
What was wrong with powershell ? You could make it faster if you run using jobs/runspeaces and/or distribute on several machines. Should be dead easy.
gabrielsroka
Nothing was wrong with it. It works fine, but I need, basically, a GUI or something simple that non-programmers/non-technical people can use.

I guess I need to wrap my code with something easy. I was trying to avoid reinventing the wheel, but, so far, I haven't found a wheel.