Launch HN: Integuru (YC W24) – Reverse-engineer internal APIs using LLMs (github.com)
Many products need integrations with third-party services, but platforms often lack official APIs. Examples include logistics software, financial services, electronic health records (EHRs), and government websites. To build low-latency integrations, developers must reverse-engineer internal APIs, but this can get complicated. With Integuru, you can have easier access to integrations.
We started as recent college grads trying to make US income tax data accessible. We contacted banks, brokerages, payroll software, and more to request access to their APIs, but none took us seriously. We resorted to building integrations with these systems to extract documents like W-2s and 1099s. We initially used browser automation but ran into two big problems: our integrations (1) weren’t reliable due to changing UIs and (2) had slow execution speeds due to spinning up browsers and waiting for pages to load. We experimented with AI-based automation maintenance, but it didn’t solve slow speeds. So, we concluded that browser automation is useful when execution speed isn’t essential, but reverse engineering is often the only path for performant integrations.
Through reverse-engineering dozens of platforms, we noticed many internal API design patterns that LLMs could decipher. We built an agent to automate the creation of integrations. Today, Integuru can analyze a platform’s internal API designs and build an integration in minutes.
The agent mimics what a human does when reverse-engineering. Say you want to download utility bills from a utility website. You’d first use Integuru to generate a file of network requests and a file of cookies. You pair these two files with a prompt about your desired action—in this case, to download utility bills.
Integuru identifies the final request that downloads utility bills. The request URL might look like this: https://www.example.com/utility-bills?accountId=123&userId=4.... It then identifies parts of the request that depend on other requests. The example URL contains dynamic parts— accountId and userId—that usually are in the response of previous request(s). It then finds other requests whose response contains any of these and adds them to the dependency graph. The newly found request URLs might look like https://www.example.com/getAccountId, https://www.example.com/getUserId, and so on.
This process repeats until the most recently found request doesn’t depend on any other request. Integuru then traverses up the graph, starting from the requests without dependencies while converting each request into a runnable function.
Integuru supports a surprising number of use cases like downloading documents, sending money, creating virtual cards… People already use the agent to build low-latency APIs for platforms like Robinhood, transportation management systems (TMS), and more. However, the agent still has limitations due to current LLM capabilities and long-tail edge cases, but we’ve been giving each platform to the agent for the first try. When the agent does struggle, we find the generated graphs and code still helpful as references for us to complete the work manually.
The agent and all integ...
86 comments
[ 3.7 ms ] story [ 147 ms ] threadI'm a fan.
Love the landing page, please keep it.
Currently you need to trigger the UI actions manually to generate the network requests used by Integuru. But we're planning automate the whole thing by having another agent auto-trigger the UI actions to generate the network requests first, and then have Integuru reverse-engineer the requests.
Though I suppose if you can auto-fix and retry issues within ~1minute or so it could work?
I think it's pretty likely that they just don't look at or test Newpipe when they change their APIs. If the change doesn't break any official clients, it goes through.
With how large Youtube is, I iimagine API changes are not infrequent.
it kept writing scripts, i would paste the output, and it would keep going, until it was able to create its own working discount code on an actual retail website
The only issue with these kinds of things is breaking robots.txt rules and the possibility things will break without notice, and often
The use of unofficial APIs can be legally questionable [1]
[1] https://law.stackexchange.com/questions/93831/legality-of-us...
As the authors of essentially a hacking tool, I would expect at least some legal boilerplate language about not being liable
Regarding the legality aspects — really appreciate you mentioning this — we’ve put a lot of thought into these issues, and it’s something we’re continually working on and refining.
Ultimately, our goal is to allow each developer to make their own informed decision regarding the policies of the platforms that they're working with. There are situations where unofficial APIs can be both legal and beneficial, such as when they're used to access data that the end user rightfully owns and controls.
For our hosted service, we aim to balance serving legitimate data needs with safeguarding against bad actors, and we’re fully aware this can be a tricky line to navigate. What this looks like in reality would be to prioritize use cases where the end-user truly owns the data. But we know this is not always black-and-white, and will come up with the right legal language as you recommended. What does help our case is that many companies are making unofficial APIs for their own purposes, so there are legal precedents that we can refer to.
In the US this is pretty simply covered by Computer Misuse Act and Computer Fraud and Abuse Act, both federal laws
Im not claiming you're liable, just surprised no lawyer pointed this out at YC
If I open the Safeway app and it fetches what is available in a given store without any authentication and everyone sees the same data, that could possibly fall under that exemption.
Computer programmers are not legal experts lol. The law is not a program.
The difference between you accessing it and a computer accessing it makes these things different.
Bits have color and if you don't know what that means, Google that before responding.
If it’s two different things then it’s not the same thing.
Until LLMs become smart enough to emulate a full JS stack, I think we're safe :)
I've found it relatively straight forward to reverse engineer SPA requests however with server side rendered apps, yow would your service handle that?
For example, set context to a particular group and then subsequent navigation depends is filtered on that group even though the filter is not explicit on the client side but due to state stored in the session remotely.
This can also have implications on concurrency for a given session where you need to either create separate sessions or make sure there is some lock on particular parts of server side state.
Would this type of this eventually be possible? Or at least hooks in able for us to add custom code such as session locks
I work at a milk delivery company in the UK (The Modern Milkman). There's this website called findmeamilkman.com and I wanted to scrape all the milk delivery services that serve every UK postcodes to create polygons I can overlay on a map to identify competitors in each region.
I keep getting rate limited by the servers, and there doesn't seem to be any fetch/XHR requests on the network. Instead a SSR request that returns the full HTML.
If you're product could help me solve this by reverse engineering an API that would be amazing
Do you also support internal APIs of mobile applications? If so, how do you deal with AppCheck / PlayIntegrity / Android Key Attestation / Apple App Attest?
We do not directly support APIs for mobile applications; however, if you use MITM software and get all the network requests into a .har file, Integuru should work as expected. We do not handle AppCheck ATM at the moment unfortunately.
I've spent plenty of time trying to dig into the network tab to automate requests to a website without an API. Cool to see the process streamlined with LLMs. Wishing you all the best of luck!
Every task on the web can be reduced down to a series of backend calls, and the key is extracting out the minimal graph that can replicate that task.
Also, just as a friendly suggestion, given what both(?) products seemingly do, this section could use some love other than "we use TLS": https://www.taiki.ai/faq#:~:text=How%20does%20Taiki%20handle... since TLS doesn't care about storing credentials in plain text in a DB, for example
---
p.s. the GitHub organization in your .gitmodules is still pointing to Unofficial-APIs which I actually think you should have kept o/
Regarding the Unofficial-APIs name, it was a really tough decision. We liked the name a lot but just thought it was a bit long. A Real pleasant surprise that you found it :)
Anyway a lot of good points here about legalities she shifting APIs, but I think there are plenty of situations where this is great and none of that applies.
How do you keep up with the maintenance?
From our experience, reverse engineering is still less prone to breakage compared to traditional browser automation. But we definitely want to make integrations even more reliable with maintenance features.
Every X hours test the endpoints and validate the types and field names are consistent... If they change then trigger some kind of alerting mechanism to the user.
[0] - https://httptoolkit.com/
No manual setup or config, just click a button and done.
Avoiding in-depth detection is left as an exercise for the reader, although there are a small set of existing countermeasures in there. In practice, there is definitely a very long tail of further cases of increasing complexity, with diminishing returns on automated solutions, but it turns out in practice you can automate quite a long way down that path and cover most normal cases.
Flutter is the one awkward case here I've found that doesn't fully work. Very interested to see if there are generalizable automated solutions there, or if the recent fork announcements mean the slow death of flutter anyway...
It's time for there to be a legal protection framework for OSS maintainers to stop being bullied with legal threats from Megacorps.
It actually worked decently well but of course there where edge cases and i was thinking of giving it a shot more recently to fill the gaps with LLMs.
Pretty cool to see someone take this idea much further and make it a service and I feel like theres even more potential here, and not just for integrations.