HN Explorer

Launch HN: Cyberdesk (YC S25) – Automate Windows legacy desktop apps

73 points by mahmoud-almadi ↗ HN
Hi HN, We’re Mahmoud and Alan, building Cyberdesk (https://www.cyberdesk.io/), a deterministic computer use agent for automating Windows desktop applications. Developers use us to automate repetitive tasks in legacy software in healthcare, accounting, construction, and more, by executing clicks and keystrokes directly into the desktop.

Here’s a couple demos of Cyberdesk’s computer use agent:

A fast file import automation into a legacy desktop app: https://youtu.be/H_lRzrCCN0E

Working on a monster of a Windows monolith called OpenDental (showcases agent learning process as well): https://youtu.be/nXiJDebOJD0.

Filing a W-2 tax form: https://youtu.be/6VNEzHdc8mc

Many industries are stuck with legacy Windows desktop applications, with staff plagued by repetitive tasks that are incredibly time consuming. Vendors offering automations for these end up writing brittle Robotic Process Automation (RPA) scripts or hiring off-shore teams for manual task execution. RPA often breaks due to inevitable UI changes or unexpected popups like a Windows update or a random in-app notification. Off-shore teams are often unreliable and costlier than software, plus they’re not always an option for regulated industries.

I previously built RPA scripts impacting 20K+ employees at a Fortune 100 company where I experienced first hand RPA’s brittleness and inflexibility. It was obvious to me that this was a bandaid solution to an unsolved problem. Alan was building a computer use agent for his previous startup and realized its huge potential to automate a ton of manual computer tasks across many industries, so we started working on Cyberdesk.

Computer use models can struggle with abstract, long-horizon tasks, but they excel at making context-aware decisions on a screen-by-screen basis, so they’re a good fit for automating these desktop apps.

The key to reliability is crafting prompts that are highly specific and well thought out. Much like with ChatGPT, vague or ambiguous prompts won’t get you the results you want. This is especially true in computer use because the model is processing nearly an entire desktop screen’s worth of extra visual information; without precise instructions, it doesn’t know which details to focus on or how to act.

Unlike RPA, Cyberdesk’s agents don’t blindly replay clicks. They read the screen state before every action and self-correct when flows drift (pop-ups, latency, UI changes). Unlike off-the-shelf computer use AIs, Cyberdesk runs deterministically in production: the agent primarily follows the steps it has learned and only falls back to reasoning when anomalies occur. Cyberdesk learns workflows from natural-language instructions, capturing nuance and handling dynamic tasks - far beyond what a simple screen recording of a few runs can encode.

This approach is good for both reliability and cost: reliability, because we fall back to a computer use model in unexpected situations; and cost because the computer use models are expensive and we only use them when we need to. Otherwise we leverage faster, more affordable visual LLMs for checking the screen state step-by-step during deterministic runs. Our agents are also equipped with tools like failsafes, data extraction, screen evaluation to handle dynamic and sensitive situations.

How it works: you install our open source driver on any Windows machine (https://github.com/cyberdesk-hq/cyberdriver). It communicates with our backend to receive commands (click, type, scroll, screenshot) and sends back data (screenshots, API responses, etc). You give our computer use agent a detailed natural lang...

23 comments

[ 5.0 ms ] story [ 64.5 ms ] thread
throw03172019
Looks great. For the EMR use cases, do you sign BAAs? Which CUA models are being used? No data retention?
rkagerer
Personally I think this approach is flawed because it runs in the cloud. If it were an agent I could run locally I'd be much more interested.
lazyninja987
do you plan to introduce automation run scheduling? (ex: run these steps once a week)

what if the target machine is not up and running when the scheduler tried to execute the steps?

MortyWaves
Frankly quite insulting to call any Windows app legacy
mattfrommars
Looks great to automate workload for Windows desktop application. I'd love to understand more deeply how your application works, so the set of commands your backend send is click, scroll, screenshot. Does it send command to say type character into an input field? How is it able to pin point a text field from a screenshot? Is LLM reliable to pin point x and y to click on a field?

Also, to have this run in a large scale, Does it become prohibitively expensive to run on daily basis on thousand of custom workflows? I assume this runs on the cloud.

deepdarkforest
Congrats! I think the space is very interesting, I was a founder of a similar windows CUA infra/ RPA agents but pivoted. My thoughts:

1) The funny thing about determinism is how deterministic you should be when to break, its kind of a recursive problem. agents are inherently very tough to guardrail on an action space so big like in CUA. The guys from browser use realized it as well and built workflow-use. Or you could try RL or finetuning per task but is not viable(economically or tech wise) currently.

2) As you know, It's a very client facing/customized solution space You might find this interesting, it reflects my thoughts in the space as well. Tough to scale as a fresh startup unless you really niche down on some specific workflows. https://x.com/erikdunteman/status/1923140514549043413 (he is also building in the deterministic agent space now funnily enough) 3) It actually is annoyingly expensive with Claude if you break caching, which you have to at some point if you feed in every screenshot etc. You mentioned you use multiple models (i guess uitars/omniparser?), but in the comments you said claude?

4) Ultimately the big bet in the RPA space, as again you know, is that the TAM wont shrink a lot due to more and more SAP's, ERP's etc implementing API's. Of course the big money will always be in ancient apps that wont, but then again in that space, uipath and the others have a chokehold. (and their agentic tech is actually surprisingly good when i had a look 3 months ago)

Good luck in any case! I feel like its one of those spaces that we are definitely still a touch too early, but its such a big market that there is plenty of space for a lot of people.

mwcampbell
Have you looked at using accessibility APIs, such as UI Automation on Windows, to augment screenshots and simulated mouse clicks?
MetaWhirledPeas
Can it do assertions? This could be useful for testing old software.
kjellsbells
This is great, but a part of me wonders if our industry isnt putting a bandaid on a problem that we ourselves created.

Consider your typical early-2000s era Windows app. It would expect a mouse, but for power users, keyboard shortcuts would be available for every action, even if clunky. For example, Alt F tab tab tab to get to some input field, enter text, tab Alt R Return.

By about 2015 these were all straightforwardly scriptable with AutoHotkey amd similar tools.

But too late: by 2015 even Windows users were using web apps, where the keyboard bindings are variable or non existent, where the entire UI can change overnight, etc. I see some RPA approaches desperately trying to decode the DOM or match pixel elements. It's wild, as you point out.

I guess what I'm wondering if going after legacy Windows apps is a small TAM already largely solved, whereas the SPA/webapp market is gigantic, growing every day, and woefully, miserably, broken as far as automation is concerned.

mahmoud-almadi
Great point! The issue with hotkeys is a series of hardcoded keyboard shortcuts is likely to fail due to unexpected popups which are quite frequent. Cyberdesk can automate entire workflows with keyboard shortcuts only, and we typically push for that, but where it really shines is its ability to handle anomalous situations and ensure the workflow works every time. As for webapps, Cyberdesk is equipped to automate them just as much as it can automate desktop apps, and it's been performing just as well on web apps as it is on desktop apps. To our agent a webapp and a desktop app are the same thing. just a bunch of shapes and text on the screen. We just emphasize the legacy apps in our messaging because there's a strong need for that. Almost every customer we've talking to has tried RPA but to no avail
rolstenhouse
Congrats on the launch! I've tried building a windows automation script myself but found existing solutions quite cumbersome booking appointments and fetching information on a desktop SMS.

Can I monitor/manage this remotely? I'm not on site with the client and previously tried to manage through AnyDesk but the client often turned off the machine.

Also is there anyway to run this so that it won't interrupt workflows while someone is using the machine? I imagine a solution could just be having the client run an extra computer that's dedicated for this or running after hours on the local machine.

Scheduled a demo

zkxjzmswkwl
You do not need a driver to do what you're doing. It's just slightly easier with a driver.

You can accomplish this from usermode and you wouldn't give potential customers (anyone who plays modern games) a non-starter for your product.

throw03172019
How do you determine what action is to use the “cached” agent to reduce cost/increase speed?
sgtwompwomp
So once you run a workflow (prompt) once, then that trajectory is cached.

Run the workflow again and it’ll run through that cached trajectory as best as it can, falling back to computer use if needed.

throw03172019
Thanks. In the OpenDental example, if the task is to update a different patient, is it falling back to computer use because the “search results” are in different places? I.e search for “John” there may be two results. John and Johnson.

What would happen in this case?

sgtwompwomp
Yup, this is a case where you always want an agent to do that step. So in the prompt you just say “do a focused_action to select the search result with John”, and then the pathfinder agent will cache in it’s memory to delegate that step to a mini computer use agent, just for that particular task.

After the focused action is done, it’ll go right back to deterministic!

iamcreasy
Congratulations on the launch!

How does it know when to stop and ask a human to intervene?

sgtwompwomp
Our agent would have a tool to essentially bring in the human. Not built this yet, but the closest thing we do have is that our agent can declare a task as failed if it determines it can’t proceed (based on your instructions).

More on this soon! How would you imagine this would be useful?