Show HN: Skyvern – Browser automation using LLMs and computer vision (github.com)
We provide a natural-language API to automate repetitive manual workflows that happen within the companies' backoffices. You can check out our code and play with Skyvern here: https://github.com/Skyvern-AI/Skyvern
We talked to hundreds of companies about things they do in the background and found that most of them depend on repetitive manual workflows. The breadth of these workflows surprised us – most companies started off doing things manually, and eventually either hired people to scale the manual work, or wrote scripts using Selenium-like browser automation libraries.
In these conversations, one common point stood out: scaling is a pain either way. Companies relying on hiring struggled to adjust team sizes with fluctuating demand. Companies using Selenium and similar tools had a different problem: it can take days or even weeks to get a new workflow automated, and then would require ongoing maintenance any time the underlying websites changed because their XPath based interaction logic suddenly became invalid.
We felt like there was a way to get the best of both worlds with LLMs. We could use LLMs to reason through a website’s layout, while preserving the advantage of traditional browser automations allowing it to scale alongside demand. This led us to build Skyvern with a few core functionalities:
1. Skyvern can operate on websites it’s never seen before by connecting visible elements with the natural language instructions provided to us. We use a blend of computer vision and DOM parsing to identify a set of possible actions on a website, and multi-modal LLMs to map the natural language instructions to the available actions on the page.
2. Skyvern is resistant to website layout changes, as it doesn’t depend on any predetermined XPaths or other selectors. If a layout ever changes, we can leverage the methodology in #1 to complete the user-specified goal.
3. Skyvern accepts a blob of information when navigating workflows—basically just a json blob of whatever information you want to put, and then we use LLMs to map that to information on the screen. For example: if you're generating a quote from Geico, they commonly ask “Were you eligible to drive at 21?”. The answer could be inferred from the driver receiving their license in 2012, and having a birth date of 1996.
The above strategy adapts well to a number of use cases that Skyvern is helping companies with today: (1) Automating materials procurement by searching for, adding to cart, and transacting products through vendor websites that don’t have APIs; (2) Registering accounts, filing forms, and searching for information on government websites (ex: registering franchise tax information for Delaware C-corps); (3) Generating insurance quotes by completing multi-step dynamic forms on insurance websites; (4) Automating the job application process by mapping user-specified information (such as a Resume) to a job posting.
And here are some use-cases we’re actively looking to expand into: (1) Automating post-checkup data entry with patient data inside medical EHR systems (ie submitting billing codes, adding notes, etc), an (2) Doing customer research ahead of discovery calls by analyzing landing pages and other metadata about a specific business.
We’re still very early a...
148 comments
[ 2.1 ms ] story [ 222 ms ] threadYou're right that this kind of escalation is inevitable
a. From a business POV, we don't onboard any types of use-cases that we think go against the spirit of a good free web. I've had people ask if they could use our product to create Reddit voting or spamming rings and we didn't entertain it
b. From an open source POV, we prefer technologies like these be open source so website owners and other businesses can know what can happen, and decide how to approach it. Tools like selenium have existed for a long time -- largely to the benefit of the world!
If you charge enough, the spammers become valuable customers. Of course they tend to leave before that point, but you don't really care if they leave or stay; you make money either way.
Value for value.
I mod a 1 million+ Facebook group and they can’t even prevent someone from making 200 posts in a minute with the word “crypto” in it. The word list will flag it, but the spam filter won’t.
Reddit constantly has people messaging you in chat about “opportunities.”
Email is a disaster.
My personal blog has over 100,000 spam comments sitting in the filter so at least they were caught, but processing them is impossible.
> I mod a 1 million+ Facebook group and they can’t even prevent someone from making 200 posts in a minute with the word “crypto” in it.
Could you possibly charge a nickel's worth of bitcoin to approve a post?
Manually accept new accounts on your service. That's what I do for my Fediverse server, and I never have to deal with spam on my local timeline :). Does it scale? No. Does everything need to scale? Also no.
Does it matter to you? Yes.
Will you admit it? No.
But yes, these are all decisions we need to make. That manually accepting is some serious dedication. Do you have kids?
> Will you admit it? No.
Are you trying to telling me my opinion? Because no, it does not matter to me. Your account would not be accepted because I don't know you.
There's no reason for somebody to create a website, pay for resources, and hope for some sort of revenue if their visitors are mostly AI.
So why bother creating a UI? Instead it would make more sense to close the website and offer the same information as a paid API service.
Any sort of website that needs to validate human visitors will be plastered with DRM. Rendering these web browsing LLMs useless. And good riddance as well.
Using an LLM to browse the internet feels like a huge waste of resources.
Instead it would make more sense to have a wikipedia-like for AIs to crawl via embeddings.
This kind of pattern makes it so you can serve both users and agents with a single interface
So what? I bot protect my site, redirecting the AI to a minimalistic part that most likely expects some sort of value given?
People will just breach this trust, like OP and abuse tools like Selenium (as they always have) to imitate being a human.
Perhaps a similar thing as robots.txt is in order (agents.txt?)
And yeah, that sucks for content farms but putting up content and getting nothing in return is already how ad blockers work and it hasn't destroyed the them. I seriously doubt that AI traffic will even put a dent 1/1000th of the traffic loss of Google snippets.
I still have fear that the real internet has already split from what I see and I was left behind.
You can easily copy sample curl requests through our UI. Feel free to check out the quickstart on our GitHub and let us know if you have any questions.
Any idea on pricing/business model?
I'd love to chat about your use-case. Happy to follow-up over email (suchintan@skyvern.com) or over a quick call (https://meetings.hubspot.com/suchintan)
What's interesting here is that large companies like UI Path charge thousands of dollars to build a single robot for companies.. I wonder if that large up-front expense will still be necessary in this new world
We have a lot of tooling in place now so most things take minutes. The harder step is getting the data in the client's infrastructure
Also, an idea is to offer a "record" and "replay" mode. Let the LLM run through the instructions, find the selectors, record and save them. Then you can run through again without using the LLM, replaying the interaction log, until the workflow breaks, then re-generate the "interaction log" or whatever.
Re: cost for execution. This really depends on the page, but currently it costs between 5 cents and 20 cents per page to execute (today).
We have an improvement planned to help it "remember" or "cache" actions it's done in the past so it can just replay them and bring the cost down to near zero.
Re: LLMs it's capable of working with, currently it's only GPT-4V. I'll get this updated soon!
Would it be usable for test automation? Would API allow to create asserts?
For example, you could instruct it to go to hackernews and terminate if you don't see a comment from giamma by passing in this payload:
{ "url": "https://news.ycombinator.com", "navigation_goal": "goal is met if you see a post from giamma. Terminate if you don't" }
Might be great for pen testing.
Can you share some data on costs and scalability?
At Kadoa, we're working on fully automating unstructured data ETL from websites, PDFs, etc. We quickly realized that doing this for a few data sources with low complexity is one thing, doing it for thousands of sources daily in a reliable, scalable, and cost-efficient way is a whole different beast.
Using LLMs for every data extraction would be way too expensive and very slow. Instead, we use LLMs to generate the scraper and data transformation code and subsequently adapt it to website changes, which is highly efficient.
We're trying our best not to move into the web scraping space -- we're focusing on automating uncreative, boring, tedious tasks.
We've seen a lot of success going after form-filling on government websites, which would usually be very boring, but happens to work pretty well for us
this seems like this could be used for abuse. the CAPTCHAs are specifically designed to stop botting on 3rd party websites.
or this will just be another cat and mouse game where the next level of CAPTCHAs get more annoying and invasive to verify we are human
You could get some token from the website. It could include encrypted service name and policies, like rate limit, that the authority should enforce. The client passes the token to the eId authority. The authority signs it and adds timestamp, but no user info. Client gives token to the service. Something like that. This is a bad top of mind example.
I think we'll need to rely a lot more on eID in the future. I think it can be done in a good way but then it needs to be thought through before it gets adopted. And we have to be able to trust the eId institutes.
Anti spams are about detecting whether activities are spam.
Binding an identity, is the naive mechanism that makes us think spam wouldn't happen. All it does is say ok we know it's pug35372 that teared the linens apart.
We can put all measures to authenticate users, won't makes them not potentially bots running havoc right after a manual authentication.
There are even farms, manually created accounts by gig seekers who would fill forms, email and phone number verification for less than a dollar.
The service could send ban or lockout requests to the eId authority so that a misbehaving real life user could be locked out from the service even though the service doesn't know who they are (irl).
I would guess it could even be designed so that the authority doesn't know which services a given user has been banned from either. And all the service would need to know is "This user has violated policy X at <timestamp>".
We didn't open source this functionality on purpose, and are very very specific about what use-cases we onboard that require it.
That being said, we've gotten to learn a lot more about browser fingerprinting and captcha solving and it's a really interesting space.
If you're curious about it, check out this blog post: https://antoinevastel.com/bot detection/2018/01/17/detect-chrome-headless-v2.html
There are quite a few services that will solve them in a few seconds, costing less than a dollar per 1000 solved tokens for most common CAPTCHA's (e.g ReCAPTCHA v2 and v3).
I recently had to deal with an attacker doing credit card testing that was using one of these services.
Related, I came across this last week, bypassing ReCAPTCHA with Selenium/Python/OpenAI Whisper API:
https://www.youtube.com/watch?v=-TMNh64ubyM
humans have been training google's ai models for a decade or more each and every time they answered a captcha
at any rate, if someone wants to abuse your site, captcha, and even cloudflare won't help you
> the next level of CAPTCHAs get more annoying and invasive to verify we are human
like the solving puzzles ones? Or more advanced object identification, like selecting the correct orientation? Training more advanced AI now
I have a feeling that this tech will become a commodity and will probably be built-in into the OS or Browser.
Props for open-sourcing though!
One big decision we made was to focus on browser automations (instead of computer automation like Adept or OpenAdapt). The reason for this was that we wanted to leverage the information available inside of a DOM to improve the quality of our agent's actions. We found that relying on image-only analysis with X,Y coordinate interactions wasn't able to offer high enough reliability for production workflows
FULL FUCKING STOP.
[We talk about AI alignment. THIS is an aligment issue]
Do you understand billing code fraud?
If you supply this function - you will *eliminate ANY AND ALL human accountability* unless you have ALSO built a fully auditable provenance from DR <-ehr-whatever-> codes.
Codes ARE why the US health system is BS.
Here - if you want to be altruistic - then you will take it upon the fact that CODES are one of the most F'd up aspects of costing.
Codes = [medical service provided]
so code = 50 = checkup = [$50 <--- WHO THE HECK KNOWS]
So lets say I am Big Hospital. "No, we will only allow $25 for code 50" - and so they get that deal.
I am single clinic so they have to charge $50
Build a dashboard for what the large medical groups can negotiate per code, vs how a small hospital or clinic group gets per code.
Only automate it if you can literally show a dash of all providers and groups and what they can charge per code.
Infact - code pricing is a medical stock market.
(each hospital group negotiates between the price they will pay per code, how much lobbying is a factor and all these other factors...
what we really need an LLM for is to literally map out all the BS in the Code negotiations btwn groups, pharma, insurance, lobbying, kickbacks, political)
Thats the medical holy grail.
[EDIT: Just to show how passionate I am on this issue - here are some SOURCE:
I have designed and built & commissioned out 11+ hospitals.
Built the first iphone app for medical.. it was rejected by YC (hl-7 nurse comm system on iTouch devices) (2006?)
opensourced that app to OpenVista.
Brother was joint chiefs dr / head of va
worked with building medical apps and blocked by every EHR...
Zuckerbergs name is on top of some of the things I built at SFGH before he got there...(and ECH mtn vw)
Ive seen way beyond the kimono
We've talked to a few companies now that would use a product like Skyvern to just automate billing information gathering to make sure patients don't get screwed in the billing process
Are you open to chatting? I'd love to pick your brain about what's behind the kimono
suchintan@skyvern.com or https://meetings.hubspot.com/suchintan
I'm always fascinated by how far you can get with heuristics in certain situations. Check out the code here -- https://github.com/Skyvern-AI/skyvern/blob/d0935755963b017ed...
https://m.youtube.com/watch?v=IXRkmqEYGZA
> self-operating-computer This is quite different than https://github.com/OthersideAI/self-operating-computer
Self-operating-computer uses pixel mapping to control your computer. This is a very good approach, but it's extremely unreliable. GPT-4V frequently hallucinates pixel outputs, causing it to miss interactions, or enter fail-loops
>The approach by AI Jason
AI Jason is using image-only methods to interact with the browser. This is a great first step, but this approach tends to be rife with hallucinations or errors. We do dom parsing in addition to image anaylsis to help GPT-4V correlate information in the image to the interactable elements within the DOM. This dramatically boosts its ability to perform the same task over and over again reliably (which proved impossible with the image-only approach)
interesting concept for problem solving though. congrats!
1. What's behind a select option? You don't know until you click it, which means you need another iteration. This sucks. 2. How do you consistently correlate things in the images to actual actions (ie upload a file to a file input, click on a button, insert a date into a date)? Having the additional HTML Tag information dramatically improves the action selection process (click vs upload vs type)
I played with the Geico example, and it seems to do a good job on the happy path there. But I tried another one where it struggled... I want to get me car rental prices from https://www.costcotravel.com/. I gave it airport + time of pickup and dropoff, but it struggled to hit the "rental car" tab. It got caught up on hitting the Rental Car button at the top, which brings up a popup that it doesn't seem to read.
When I put in https://www.costcotravel.com/Rental-Cars, it entered JFK into the pickup location, but then failed to click the popup.
Thanks for the feedback re: costcotravel.com Skyvern definitely does NOT have 100% coverage of the web. This is one of the reasons we were excited to open source -- so we could learn about more websites where it doesn't work as expected
I've filed an issue for this case here: https://github.com/Skyvern-AI/skyvern/issues/69
Not saying I wouldn't pay that for some use cases, but it would limit me.
One idea: making scrapers is a big pain. But once they are setup, they are cheap and fast to run... this is always going to be slower. What I'd love to see is a way to generate scrapers quickly. So you wouldn't be returning information from the New York City property registry... instead, you'd return Python code that I can use to scrape it in the future.
edit: This is likely because it was struggling, so it had to make extra calls. What would be nice is a simple feature where you can input the maximum number of calls / tokens to use on the entire call. Or even better, do some math and put in a dollar cap. i.e., go fill out the Geico forms for me and don't spend more than $1.00 doing it.
1. You can set a "max steps" limit when you run it locally https://github.com/Skyvern-AI/skyvern/blob/d0935755963b017ed...
We also spit out the cost for each step within the visualizer. Click on any task > Steps > there's a column that's dedicated to how much things cost to run
https://github.com/Skyvern-AI/skyvern/issues/70
2. We have a roadmap item to "cache" or "memorize" specific tasks, so you pay the cost once, and then just run it over and over again. We're going to get to it soon!!
Addressing complex website interactions is a key advantage of this approach. For instance, in the process of generating an auto insurance quote, the sequence of questions and their specifics can vary greatly depending on prior responses. A simple example is the choice of a foreign versus a California driver's license. Selecting a foreign license triggers additional queries about the country of issuance and expiry date, illustrating the complexity and branching nature of such web interactions.
However, we recognize the concerns about cost and are actively working on strategies to reduce it: - Optimizing the context provided to the LLM - Implementing caching mechanisms for certain repeated actions and only use LLMs when there's a problem - Anticipating advancements in LLM efficiency and cost-effectiveness, with the hope of eventually finetuning our own models for greater efficiency
1) Using the LLM to find elements/selectors in HTML
2) Use LLMs to fill out logical/likely/meaningful answers to things
I highly recommend you decouple these 2 efforts. While you gave a good example of "insurance quote step by step webapp", the vast majority of web scraping efforts are much more mundane.
Additionally, even in this instance, the selector brain/intelligence brain don't need to be coupled.
For example:
Selector brain: "Find/click the button for foreign drivers license." Selector brain: "Find the country of origin field." Selector brain: "Find the expiry date field."
LLM-intelligence brain: "Use values from prompt to fill out the country of origin and expiry date fields."
Not-LLM intelligence brain: Inputs values from a JSON object of documentSelector=>value.
We've been approaching it a little bit differently. We think larger more capable models would actually immediately improve the performance of Skyvern. For example, if you run it with LLaVa, the performance significantly degrades, likely because of the coupling
But since we use GPT-4V, and it's rumoured to be a MoE model, I wonder if there's implicit decoupling going on.
I'm gonna spend some more time thinking about this
I don't want to use vision and LLMs for every page. I just want to use vision and LLMs to figure out what elements need to be clicked once. Or maybe every time the site changes the frontend.
Thank you for this feedback
It would save all that long time spent going manually thought every page and figuring out which mistake we did, when that input string doesn't go into that input field or the button on the modal window is not clicked.
Change the UI? Recompile with the AI.
* browser extension that lets you record a few actions * describing what you want to do with text * a url with one or two lines of desired JSON to extract
No, that's something completely different than what bravura is talking about, which is why he made a comment to say explicitly that he still thinks you're missing the point.
From your roadmap:
> Prompt Caching - Introduce a caching layer to the LLM calls to dramatically reduce the cost of running Skyvern (memorize past actions and repeat them!)
Adding a caching layer is not what they're asking for. They want to periodically use Skyvern to generate automation code, which they could then deploy themselves in their testing/CI setup. Eventually their target website may make breaking UI changes, then you use Skyvern to generate new automation code. Rinse and repeat. This has nothing to do with an internal caching layer within your service.
For example, if Skyvern was asked to log-in to a website and do a search for product X, the generated action plan would include: 1. Click the log in button 2. Click "sign in with email" 3. Input the email address retrieved from source X 4. Input the password retrieved from source Y 5. Click log in 6. Click on the search bar 7. Input the search term from source Z 8. Click Search
Now, if the layout changed and suddenly the log-in button had a different XPath, you have two options: 1. Re-generate the entire action plan (or sub-action plan) 2. Re-generate the specific component that broke and assume everything else in the action plan still works
...and you've hit the nail on the head in terms of our design philosophy: use LLMs to generate useful logic, then run that logic without needing to call an LLM/Agent.
With that said, we don't support browser automation. Skyvern is very neat, it reminds me of VimGPT[1], but with a more robust planning implementation.
[0] https://magicloops.dev
[1] https://github.com/ishan0102/vimGPT
We tried approaches like VimGPT before but found the rate of hallucinations to be a bit too high to be used in production. The sweet spot definitely seems to be to combine the magic of Dom parsing AND vision
We're going to definitely work on logic generation and execution, but we're taking it a bit more carefully. Many of the workflows we automate have changing workflow steps (ie I've never seen the exact same Geico flow twice), but this certainly isn't true for all workflows
Bravo, I would pay for this one, or hopefully run it on my GPU - it would be so fast to even just shove out your selectors (xpath, css, dealer's choice) for point-by-point update after you had done an initial code gen, or perhaps it could just diff and update chunks of code for you!
My local code model can already do the diff update stuff in nvim, but being able to pass it a URL and have it slam in all of the pertinent crawling code, wow.
https://news.ycombinator.com/item?id=39698546
LaVague is all about generating selenium code to interact with a specific page, and do it step-by-step
Skyvern is all about taking a simple instruction and converting it to a series of LLM-driven actions. It's meant to be more autonomous ("tell Skyvern what to do")
Try this at your own risk.. any reasonable website would result in extraordinarily high input token costs
We spend quite a bit of our time building a layer between the HTML and the LLM call to distill important pieces of information down to actions the LLM can take.. better weighing cost vs output. We're still not at 100% coverage.
Just created this: https://github.com/Skyvern-AI/skyvern/issues/72
Aside from that cool project!
Feel free to email me at suchintan@skyvern.com -- I can let you know when the self-serve UI is live
We're currently testing dom instead of vision.
Here's a prompt example to try out
{ "url": "https://news.ycombinator.com", "navigation_goal": "goal is met if you see a post from basiep2. Terminate if you don't" }
I should add that this is a particularly grim prospect from a software engineering perspective. It makes me imagine a future where no one bothers exposing a stable API to anything, so the only way to interact with other people's code is using an AI middle-man.
Would be kind of handy to have a “pull all my relevant tax info documents from these sites and zip them up” automation but I only do that once a year.
I’m probably being unimaginative. Anybody have any interesting use cases?
Anyone have
Now imagine it from the accountant POV, where they have the same use-case for hundreds of clients
This is where we've seen something like Skyvern really shine. It's targeting industries and companies that are doing rote work at a significant scale
We're prioritizing on cloude 3, as its performance seems to be good. That said, please join our discord and bring more thoughts/requests to us. code contribution is also more than welcome