HN Explorer

Show HN: API Parrot – Automatically Reverse Engineer HTTP APIs (apiparrot.com)

456 points by pvarghav ↗ HN
When automating business processes at work, I found it difficult and time-consuming to reverse engineer business systems' APIs. I often had to manually reverse engineer APIs using developer tools or settle for less optimal technologies such as Robotic Process Automation (RPA).

Often, the issue is that it can be hard to resolve all the cookies, access tokens, and other elements required to successfully execute the requests. Manually trying to resolve these dependencies using developer tools is especially challenging with multiple requests where data is stored in JavaScript objects or HTML elements.

To try to solve this issue, I built a tool called API Parrot that automatically identifies the data correlations between requests and builds a graphical representation of the flow to give users a better understanding. To streamline the process, I also included functionality to record requests, define your own inputs and outputs, and export the entire flow—or parts of it—as JavaScript code.

The application is Electron-based and currently compiled for Windows and Linux. Please try it out and give feedback!

Online Tutorial: A simple example of reverse engineering the USPS API is available at https://docs.apiparrot.com/docs/category/tutorial---reverse-...

121 comments

[ 3.4 ms ] story [ 189 ms ] thread
vhayda
Nice! It needs some refinement and a macOS version.
tommiegannert
Could you give some examples of what refinement you think it needs?
davide_v
Nice, I was looking for something like this. I tried it on Ubuntu but after clicking Capture requests > Launch Chrome, nothing happens.
ashenke
Yep same problem
[deleted]
(comment deleted)
chompin
Same issue, would prefer the option to use any browser also. Chrome is not my cup of tea
7357
Looks like it wants to run chrome using `start chrome` which is AFAIK a Windows-only command.
pvarghav
Thank you for pointing this out. I've addressed the issue, and it should now be fixed in version 0.2.1, which is available for download on the website. Please update to the latest version, and let me know if you encounter any more problems.
4k1l
This problem persist on version 0.2.3
pvarghav
On Linux, the application currently uses the command: ```bash google-chrome --proxy-server=<proxy URL> --ignore-certificate-errors-spki-list=<CA fingerprint> ``` to launch Chrome.

For some users, this command doesn't seem to work properly. I'm working on adding a feature that will allow you to manually set the command used to launch Chrome if the default one isn't working on your system.

In the meantime, you can manually launch Chrome with the arguments shown above. Just replace `<proxy URL>` and `<CA fingerprint>` with the appropriate values for your setup.

brushfoot
Impressive project. I was curious how it discovers data relationships and was going to check the repo, but it looks like there's no code, only issues and releases. Is that right?

Which leads me to...

- Is this closed source?

- Does it cost money?

- How does it discover data relationships?

skeptrune
It's entertaining that Github has become such a common place to find information that even closed source projects put something up there
pvarghav
Thanks for your interest!

- Is this closed source?

Currently, the code is not open source, but I might open-source parts of it in the future.

- Does it cost money?

The software is free to use. If there is demand, I might create a "pro" version for businesses in the future. However, I intend to always have a free version available for individuals.

- How does it discover data relationships?

I've discussed how it discovers data relationships in the documentation here: https://docs.apiparrot.com/docs/tutorial-extras/exchange-mod....

In short, the tool breaks down the data in the requests and responses into smaller parts by identifying their formats. For example, `["foo", "bar"]` would be recognized as a JSON array and broken down into the elements `"foo"` and `"bar"`. By applying this method recursively, you build a tree-like structure of the data.

If an exact match is found between data in a response from a previous request and data in a subsequent request, a correlation is detected.

Please feel free to ask if you have any more questions!

bjt12345
If this can save me time at work, I'd be happy to throw some money at it.

My bosses OTOH...let's just say, there's no penalty within companies for pointy haired bosses not making decisions to purchase something like this and ignoring staff.

It's a false economy but I'm tired of it and just purchase what I can afford.

gtirloni
Is there a ToS/License somewhere?
enricotal
Fantastic Tool ... Mac version is paramount
yellow_lead
Hi, it seems youve spelled reverse wrong

> API Parrot is the tool specifically designed to reverese engineer the HTTP APIs of any website.

pvarghav
Thanks for pointing this out!

It should now be fixed.

yawndex
Any current plans for a macOS release?
pvarghav
Yes, I plan to release a macOS version of API Parrot. Unfortunately, I currently don't own a Mac, and since building macOS applications requires one, this has delayed the release. I'm actively exploring solutions, such as accessing a Mac environment remotely or acquiring the necessary hardware.
rrr_oh_man
That sounds like a gofundme situation :)
MK2k
Try a Hackintosh, e.g. as a virtual machine on your Windows or Linux host.
devops000
Feedback: add a newsletter form to get notification when you will release the MacOSX version
pvarghav
Thank you for your suggestion!

I've added a newsletter sign-up form at the bottom of the webpage: https://apiparrot.com/#newsletter

Feel free to subscribe to receive notifications when we release the MacOSX version.

setheron
Very sad half the comments are asking for MacOS app. The rise of development on MacOS for server development when the final target is Linux will cause long term harm to the newer generation of engineers
yoavmmn
Nowadays everything runs on docker anyway
setheron
You'd never see a Windows developer work in MacOS or a iOS developer work in Linux but Linux developers (server side) routinely work in MacOS

Unnecessary abstraction

quesera
Counter-argument: it could be risky to dev on and deploy to a single monoculture.

But empirically, I've been developing on macOS (etc) and Linux (often simultaneously), and deploying to Linux (Debian, RHEL/AL), Solaris (etc), and FreeBSD ... for more than 20 years.

Aside from package management tooling differences, package naming, and package content splits (e.g. pkg vs pkg-dev) -- all of which are equally inconsistent between Linux distros -- I cannot recall a single issue caused by this heterogeneity.

prophesi
> iOS developer work in Linux

I dream of the day Apple releases official docker images. Building for iOS is the only reason I have to touch a Mac.

cellwebb
All SNES games should have been developed in Mario Paint or it was an unnecessary abstraction
luismedel
Define "unnecessary", please.

> [...] or a iOS developer work in Linux

In the past I did a lot of successful work on iOS apps from a Windows system, thanks to Xamarin and a mac sitting on a shelf, acting as the remote system.

Also, please, remember what "cross compilation" mean.

cdaringe
Not sad at all! Mac has excellent hardware, excellent reliability, excellent day to day performance. Im not a fanboy, but it won for (IMHO) clear and obvious reasons. Of course folks want a mac app. No comment on the “harm” bit.
bearjaws
It is always amazing to me people who will chastise people for using Macs.

It is by far the most robust hardware and 15 years later Windows laptops may finally be catching up.

My first programming job was LAMP so I had a Linux desktop and loved it. Later I got a new job that gave us laptops, but they were quite beefy.

I had a Dell laptop with an Nvidia GPU and an Intel iGPU... After updating my OS my gpu was the only way to use my laptop, which made the battery die in under an hour and of course it was much hotter.

I tried numerous driver installs, proprietary, open source, reinstall OS, different OS... Nothing got it working again on a newer version of the Linux kernel.

Went to the Apple Store bought a MBP and have never had an issue since. Not one dead laptop, in 10 years, I plug in my USB C dock and go.

2 years later, what happened to one of my coworkers? Same exact thing. He spent 3 days trying to fix it and basically had a workaround that crashed occasionally.

I get paid to produce working software not configure my OS, and people wonder why Macs are so popular?

rorroe53
Macbooks have been nice since M1 era, but the Intel Macbooks between years 2013-2020 were hardly robust. My partner's 2014 MBP Retina's screen plastic film started peeling off, which was a known design flaw of those models. Later the ones with butterfly keyboard were notoriously unreliable, with keys getting stuck.

Personally I haven't had much trouble with Linux on modern Thinkpads. Very little to configure manually, as long as you pick the right distro. Even a Dell laptop at work with Linux isn't causing me much OS-related issues, although battery life sucks.

Klonoar
Well, no. The 2015 MBP is a well known workhorse that stretched many people professionally up to the M1. I would absolutely agree that the 2016-2020 Intel MacBooks were rough though.
MarcelOlsz
I switched to old ass thinkpads like the x120e which I absolutely love, and ssh into a server much more powerful to do my work.
alp1n3_eth
Every company can have its issues, I think it's more about how many issues there are and what the company did to address it.

My 2013 4GB RAM MacBook Air is still running great, and is used for browsing in my household. Currently writing this on an M1 Air that is phenomenal as well.

acksin
Now install Linux on it.
rafram
Most people scraping sites aren’t writing anything low-level enough to care about the particular flavor of Unix-like OS it runs on.
F7F7F7
I’d argue that there’s no correlation at all between the two.
victorbjorklund
Why? I mostly code on Mac and deploy on Linux (or FreeBSD). Never really encountered a situation where programming a web app on Mac has caused issues when deploying to the server.
UltraSane
What about issues with CPU architecture?
lionkor
When you write web code you should never have to worry about that. Actually, if you write any user space code, except drivers, you shouldn't have to worry about that. If you have to worry about it, reconsider your tooling very seriously
rmbyrro
i've had issues with aws lambda and compiled ai models, because lambda varies hardware and cpu architecture from one container to another

i can imagine this happening if a team has a myriad of hardware/os flavors and different server setups.

Sardtok
Yes, when you write web apps in x86 assembly, it gets tricky.
rob
I'm still on OS/2 Warp.
smcnally
for x86 or PowerPC?
daemonhorn
Talking about rabbit-holes. I used to have prototype OS/2 PowerPC 64-bit hardware from IBM before they killed the project. I should have kept that early EFI-based system. When the EFI boot sequence would panic, you would get an error message of "Danger Will Robinson".
UltraSane
man OS/2 Warp on PowerPC should be really secure because no one is writing malware for that combination!
SparkyMcUnicorn
AWS Graviton is ARM.

My experience is that having a team with mixed platforms has helped reduce deployment woes, with the rare platform-specific bugs getting worked out beforehand.

kaid
Nowadays there are automated pipelines that build artifacts for different archs and platforms; this shouldn't be a particular issue.
criddell
Or maybe some of the newer generation will take time to update Linux to be more competitive with macOS for developers. Could be a long term win for Linux fans.
Sardtok
Linux is good for development, but Apple hardware is pretty damned nice.

Now if Framework laptops were available in Norway, I'd probably rather have that, even if they're not as powerful.

Also, depending on where you work, there might be restrictions in the choice of platform. Usually limited to Mac or Windows.

frafra
I asked Framework that repeatedly, but no progress. I think they might be violating EU Regulation 2018/302, which is rather common, mostly due to ignorance. The problem is that it is rather hard to enforce such regulation to non-EU/EEA companies. You can still send your wishes to support@frame.work.
[deleted]
(comment deleted)
frafra
Update: you can buy from Norway now, but you need to get it shipped to a different country. You need to select a different country and then chose a billing address different from the shipping one. The message that the website displays on not being able to order from Norway is misleading, and it looks like no email to Norwegian customers has been sent with respect to this possibility. Not perfect, but they got better.

https://knowledgebase.frame.work/en_us/what-countries-and-re...

forty
I agree, people don't realize the value of not depending on a single company to do their work. We can see this problem even more with LLM code generators.
chuckadams
How about a real-world example of the harm you're clutching your pearls over?

Besides, most devs doing web development on Macs are also using Docker, which is always Linux.

Merad
Really? In the modern .Net world (originally .Net Core) it's very common for devs to use Windows machines to write code whose CI pipelines and deployed environments are all Linux. I've seen a handful of issues with things like path separators and file system case sensitivity, but we're talking about 3 or 4 minor problems in 6-7 years that I've been using it.
neonsunset
Hey, some of us have moved to Macs by now :)

(also yes, people keep asking "what about linux" and think it's bad when you say there is literally nothing extra to consider in 95% of situations, sigh)

Merad
I'm actually going to switch to Mac as a pilot for our team at some point this year! I don't expect any issues, I already use Rider and have done plenty of .Net stuff on my personal machine which is a M3 MBP. Really IMO the only question marks will be around using Parallels when we need to occasionally work on a legacy .Net Framework app.
gempir
And the unreasonable hostility towards macOS will have zero affect because in the end the best product wins.

Did the rise of Windows cause long term harm to past generation of engineers? I doubt it since now Windows, which had a gigantic market share, still was forced to implement Linux "compatibility" for developers.

There are three popular operating systems for the modern developer and it's not unreasonable to ask for a build for all of them when presenting a project to a developer focused community.

sverhagen
The rise of a MacOS sort of monoculture certainly affected those developers that were still on Windows. It drove me off of Windows, a system that I otherwise appreciated just fine. I never cared much for MacOS, though, so I went to Linux, but there I'm also constantly feeling the pain of so many developers being on MacOS, as there's so many incompatibilities between the two. So, in the end I guess I prefer things that run everywhere, which this Parrot thing may be in reach of, it being Electron? In that sense I guess I support the ask for a MacOS version. But boy, could the MacOS crowd just stop throwing their weight around?

Edit: Examples:

* Tools that are only available on MacOS (remember the days when tools were only available on Windows)

* I write a BASH script which then doesn't work for the MacOS coworkers

* Tools that are supposedly platform- independent have Linux-specific errors that get no love because their developers don't care about Linux

alp1n3_eth
I think it's a little much to expect most small outfits to user test on every single Linux distro though. There are tons of programs that may run fine on something like stock Ubuntu, but bring it to another flavor and all of a sudden nothing works.
colesantiago
This is the easy part.

One of the issues with these tools is that more and more websites now employ multiple aggressive CAPTCHAs, fingerprints, device check, etc, rendering tools like API Parrot almost useless.

m00dy
can it reverse websocket-protocols ? If so, how does it do binary decoding etc ?
pvarghav
Currently only HTTP requests are supported. I might add support for websockets later, however that is a harder problem to solve due to the binary encoding etc.
moon82
looks amazing! thanks for sharing, will give it a shot in a short while. Btw, how do you keep yourself motivated on working on free projects? Obviosly it takes a lot of effort and no one is paying for that.
pvarghav
Thank you!

Working on this side project has been both fun and rewarding. I've learned a lot throughout the process, which keeps me motivated even without immediate financial gain. I have plenty of ideas on how to improve the software in various ways. Some of these enhancements could become part of a "pro" version tailored for businesses. My long-term ambition is to turn this into a full-fledged product, which would enable me to dedicate more time to its development.

pkkkzip
interesting but not sure what the value add here is, it gives you a graph flow of all the API requests being made? and then the goal is to replay them?

aren't there github libraries that do this already?

sumedh
> aren't there github libraries that do this already?

which ones?

rynn
How does it compare to mitmmitmproxy2swagger?

https://github.com/alufers/mitmproxy2swagger

1a527dd5
The first and immediate difference for me is the ability to recall the name. I can recall Postman/Insomina fine, and now for API Parrot. I'm never going to be able to recall mitmproxy2swagger.

Unfortunately, names matter.

itsafarqueue
Thanks 1a527dd5.
1a527dd5
Ha! Nicely played. That was out of purely laziness. I don't like using one handle across sites, so I take the first 8 chars of (New-Guid).ToString() and then dump it in my password manager.
SparkyMcUnicorn
I often forget the name of things, sometimes even the big ones. GitHub search is one of the primary ways I rediscover them. "reverse-engineer API" returns mitmproxy2swagger as the third result, and this is how I found it last time I needed it.

It is a bit frustrating when a project on GitHub doesn't have good tags or searchable keywords, making it harder to find.

yoavm
As someone who uses mitmproxy and swagger quite often, I actually think the name isn't so bad. I haven't even looked at the readme but I already know what it does, how to run it and what output to expect.
faizshah
This might be more useful than the OP. This thing lets you translate HAR to Swagger…

My usual process is Dev tools -> Copy as CURL -> delete unnecessary headers -> translates to requests in python (these days I just use ChatGPT) -> wrap in python sdk for managing auth etc.

The OP’s correlation features are really nice though.

notcrazylol
Probably a dumb question but if my web app uses graphql, how would I go about with the swagger generation?(since its just one endpoint)

Thank you for sharing this though, I was looking for a tool like this :)

rat87
Why would you use swagger/openapi? My understanding is that graphql has its own schema system that's supposed to be returned by the server when the client requests it.

https://graphql.org/learn/schema/

teichman
Integru has been really great for us. Curious how you think about differentiation?
sidgarimella
Love this. I’ve worked on a few projects in RPA prior and I’m losing faith in selectors. I think either direct data access like this or AI based CV are the automation arms of the future.
remoquete
Looks very interesting. Does it produce an OpenAPI file? That'd help immensely in documenting APIs that lack specifications.
1a527dd5
This is pretty cool, I ran it against one of a largest customer sites and it was very interesting to see how the page all interconnects. I'm pretty sure it can be used to spot architecture/performance problems.
ozim
If only there would be something with schema like XML that people would use for the APIs ;) You could generate diagrams from WSDL and even generate client code from that.

There is also bunch of JSON schema stuff nowadays.

But yeah for a lot of people schema of API contracts feels like too much work and too much hassle.

JSON serialization doesn’t throw errors for new properties quickly added on sending side and receiving side can ignore stuff - well as long as API semantics allow but that’s generally going to be a hassle always even with LLMs somehow autofixing your „schema”.

tveyben
I’m not able to read what the product actually does - I keep getting distracted by the ‘snake’ animation surrounding the content .. not sure what the purpose is ;-)
sebmellen
This is incredible. We’ve spent ages and ages figuring out the weird internals of certain legacy systems that we’ve ended up having to use bots or RPA to integrate with. If you can polish this into a true product, we would pay for it!

Any chance of a Mac version?

pvarghav
Thanks! I'm glad you like the idea—it sounds like you've had the same struggles I've been through.

Good news: the Mac version is now available to download at https://apiparrot.com/#download.

Let me know if you have any feedback!

YaBa
Interesting but... The first website I've tried it (which I'm currently working on due to a change of platform) couldn't find anything other than the main request, and I know for sure there is a POST reguest to the API to get some data (I had a scrapper working, website changed, had to re-do the scrapper again).

I've checked the tutorial, seems that I'm not missing any step, the software simple cannot capture anything if the request is made on the main page, seems to work fine with forms, buttons and "manual" actions.

I can DM you the website plus the expected request that is made, visible with any browser internal debugging tools.