OpenAI also focus their marketing on controlling a browser, just like Anthropic.
Agents can do so much more.
More like filters and picklists on values for the in and out of the agents.
Seems like it is like Rabbit's "Large Action Model", just working.
At the moment it seems kinda useless - to be truly useful it should support querying across multiple sites simultaneously IMO.
For example, the query "Order Joseph Joseph Platform from amazon.com" is something I could easily do faster myself. All the examples shown in their video are similarly simple and don’t showcase much value.
What would be impressive is if you could ask, "Order Joseph Joseph Platform from the cheapest site," and it could compare the total cost (including product price, shipping, and VAT) across all relevant Amazon domains, eu.josephjoseph.com and other shops that ship to my country. Then we’d really be talking.
> We’re collaborating with companies like DoorDash, Instacart, OpenTable, Priceline, StubHub, Thumbtack, Uber, and others to ensure Operator addresses real-world needs while respecting established norms.
If you're collaborating with the companies the agent is supposed to interact with, why not just have it hook into an API rather than jumping through hoops to interface with their GUI? I don't get it.
The point is to move beyond APIs. Being able to perform actions on a site, with the ability to perform the task successfully even if the site slightly changes under the hood, is a lot less brittle than interacting with an API.
Agreed. Now you're throwing in a presentation layer to wrangle with, and that presentation layer is HTML/CSS/JS, which is the thorniest presentation layer out there.
I agree with this one. But you have to start somewhere. I think in the next several things, websites will be built for agents and not people. So it'll only get better and smarter.
Will they? What incentive is there if people haven't started using agents yet?
We already have a way to build websites for machines: it's called APIs. And frankly, I think that's a better answer for "hooking LLM into website" -- the things which make APIs hard for humans (discomfort, inconvenience, low discoverability, technical complexity) aren't really problems for LLMs.
This kind of thing could just require the devs on one side to maybe clean up a bit of markup (which I even doubt), and the entire universe of potential consumers on the other side.
I wonder, did Google or Microsoft (via Github Copilot) release anything like this yet? I'd not be surprised if all of them are currently working on something in this direction.
Google has had a similar, agentic feature on Pixel phones since 2018. (Back when people used to speak on the phone rather than do everything thru an app)
Overall, Operator seems the same as Claude's Computer Use demo from a few months ago, including architecture requiring user to launch a VM, and a tendency to be incorrect: https://news.ycombinator.com/item?id=41914989
Notably, Claude's Computer Use implementation made few waves in the AI Agent industry since that announcement despite the hype.
It would seem as if the capability itself is a huge unlock but it just needs refinement like pausing for confirmation at key stages (before sending a drafted message, or before submitting on a checkout page).
So the workflow for the human is ask the AI to do several things, then in the meantime between issuing new instructions, look at paused AI operator/agent flows stemming from prior instructions and unblock/approve them.
But of course, after all the benchmark issues we've had thus far -- memorization, conflicts of interest, and just plainly low-quality questions -- I think it's fair to be suspicious of the extent to which these numbers will actually map to usability in the real world.
OpenAI is merely matching SOTA in browser tasks as compared to existing browser-use agents. It is a big improvement over Claude Computer Use, but it is more of the same in the specific domain of browser tasks when comparing against browser-use agents (which can use the DOM, browser-specific APIs, and so on.)
The truth is that while 87% on WebVoyager is impressive, most of the tasks are quite simple. I've played with some browse-use agents that are SOTA and they can still get very easily confused with more complex tasks or unfamiliar interfaces.
You can see some of the examples in OpenAI's blog post. They need to quite carefully write the prompts in some instances to get the thing to work. The truth is that needing to iterate to get the prompt just right really negates a lot of the value of delegating a one-off task to an agent.
Well that's fair. I wasn't saying that this was necessarily at a level of competence to be useful, simply that it seemed to be a lot better than Claude.
Those numbers are not the full story. Note that GP specifically says: "Big jumps in benchmarks from _Claude's Computer Use_ though." Claude Computer Use was not SOTA for browser tasks at the time of its release (and is still not.)
In WebArena, Operator does 58.1%. Previous SOTA for browser-use agents is 57.1%.
In WebVoyager, Operator does 87.0%. Previous SOTA for browser-use agents is the exact same.
Those two were two different models (Kura and jace.ai), and one model being SOTA at one benchmark doesn't make it SOTA overall. Moreover, both are specific for browser use, so they don't operate only on raw pixels but can read HTML/DOM, unlike general computer use models which rely on raw screenshots only.
I think I hit all those points in my previous post, except for the fact that it's two different models, as you've noted. That said, neither of them seem to report scores for the other benchmark in each particular case.
I thought Claude Computer Use is through API, and I remember hearing about high number of queries and charges.
This looks like its in browser through the standard $20 Pro fee, which is huge. (EDIT: $200 a month plan so less of a slam dunk but still might be worth it)
Is there any open source or cheap ways to automate things on your computer? For instance I was thinking about a workflow like:
1. Use web to search for [companies] with conditions
2. Use linked in sales navigator to identify people in specific companies and loose search on job title or summary / experience
3. Collect the names for review
Or linked in only: Look at leads provided, and identify any companies they had worked for previously and find similar people in that job title
It doesn't have to be computer use, but given that it relies on my LinkedIn login, it would have to be.
Exactly what I was look for. Thank you. I wish they had Gemini since the free tier is generous but I guess it's in the works. I'll take a look and see how bad it would be implement
> Is there any open source or cheap ways to automate things on your computer?
MacOS has had Automator since 2005. It's far more like "programming" to use than a 2024-tier ML based system, but it was designed for non-programmers, and lots of people do use it.
This is mainly to reclaim mindshare from DeepSeek that has done incredible launches recently. R1 was particularly a strong demonstration of what cracked team of former quants can do. The demo of Operator was nice but I still feel like R1 is the big moment in the AI space so far. https://open.substack.com/pub/transitions/p/openai-launches-...
R1 is a fundamental blow to their value proposition right now, the uniqueness is gone, and forever open sourced. Unless o3 is the game changer of game changer, I am not seeing they are getting the narrative back soon.
Correction on "including architecture requiring user to launch a VM": apparently OpenAI uses a cloud hosted VM that's shown to the user. While that's much more user friendly, it opens up different issues around security/privacy.
Claude's Computer Use API has been good for us and I'm surprised it isn't more popular. It can be slow, and definitely gets things wrong, but so far we've had thousands of people make and edit spreadsheets on "autopilot" so the value is already there today on simple tasks, even in an alpha state.
I do find it is best when combined with other capabilities so the internal reasoning is more "if Computer Use is the best for solving this stage of the question, use Computer Use. Otherwise, don't.", instead of full Computer Use reliance. So e.g. you might see it triggered for auto-formatting but not writing SQL.
Will report back how it compares vs Operator CUA once we get access!
Waiting for the "OpenAI has no moat" crowd to chime in while they keep releasing new features and dominating market share.
(And yeah, they just got half a trillion).
Edit: Downvote all you want, reality won't change.
Oh, what happened with "Scarlett Johansson will take down OpenAI because she invented speaking like a woman", literally nothing.
What about "AI will never replace Hollywood actors".
What about that time when "OpenAI was done because Ilya was leaving". What a bunch of fools, lmao. I'm not a fan of Sam Altman, but I'm also not deluded.
Why? The official announcement is a $500B commitment. Even if they "only get $100B" (which I doubt) do you think that will somehow be detrimental to OpenAI?
Do you understand how much money a hundred billion dollars is?
This seems more like a catch-up to other similar tools - still cool though. The half a trillion is openai agreeing to raise funds and contribute to the half trillion, not them getting it all.
https://github.com/bytedance/UI-TARS-desktop - I think it is proven there is no moat here. As much as there is a moat on "water" or "electricity" or "chicken breast". Intelligence will be sold for fractions of pennys.
I was surprised by bytedance doing ai but really, they're the only social media company that has done the "suggested/for you" feature in way that everybody isn't aghast by.
Oh yeah, how could I forget about an obscure repo from a company that's getting banned from the US!
It's simple, with trillions at play, if it's so easy to steal OpenAI's game, why has no one done it yet? Don't "argue" about it, just go and grab the money, it's easy, right?
The announcement of investing $500B with the proposed benefit of creating $100K jobs - to my amazement did not produce any commentary that I came across raising questions about the ROI of spending $5M per job created. I mean it’s all right there in the announcement!
For instance, the American Recovery and Reinvestment Act (ARRA) of 2009, which allocated approximately $787 billion, was estimated to have created or saved between 2.4 and 3.6 million jobs by early 2011. This translates to a cost of roughly $218,000 to $328,000 per job
In contrast, a study summarized by economist Valerie Ramey in 2011 found that each $35,000 of government spending produced one extra job.
Federal Highway Administration estimated that every $1 billion in federal highway and transit investment supports approximately 13,000 jobs for one year, equating to about $77,000 per job.
They also gave no indication of what types of jobs were going to be created. It's pretty hot-air.
If the goal is to create data centers for more AI training, you can rest assured that depends on creating as few jobs as possible in order to keep labor costs down and have more to spend on hardware and energy.
- OpenAI hasn't gotten any money yet, not even $100b
- OpenAI is releasing this feature after Anthropic
- AI has yet to replace any significant fraction Hollywood actors
I'm not disagreeing that these things might happen, but you have to be cognizant of the fact that you're talking projections of the future -- not assessments of the here-and-now.
I don't know if I'm ready to hand over my grocery shopping (or date night planning) to an agent. But if pricing is reasonable, this could be a powerful alternative to normal RPA.
Instead of hardcoding some automation using Selenium, this would be a great option for automating repetitive tasks with legacy business software, which often lacks modern APIs.
Grocery shopping is just a use case for people to wrap their heads around. Everyone has to eat.
If they demonstrated a big value add like automating CRM a smaller subset of professionals would be absolutely awed but most people would be scratching their heads wondering what it’s good for.
I am a little concerned with letting an AI agent that routinely hallucinates control my browser. I can't not watch it do the task, in case it messes up. So I am not sure what the value is versus me doing it myself.
Neat, someone should develop an easy-to-deploy script that spawns a headless version of this agent that scrolls through and repeatedly clicks every single ad on X and Facebook using a session cookie.
I assume here it means complying with requests that could harm other people. It's pretty common for businesses to tell their employees not to assist customers doing bad things, so not surprised to see AIs trained to not to assist customers doing bad things.
Examples:
- "operator, please sign up for 100 fake Reddit accounts and have them regularly make posts praising product X."
- "operator, please order the components need to make a high-yield bomb."
If it makes you feel any better, law enforcement makes sure reddit, Home Depot, and instagram are "aligned" as well.
Don't worry though, it's all on the up and up. No backdoors or google-like search facilities our anything like that. It's not at all automated in that sort of unseemly fashion. They always go to court. Where they talk to a judge, that they totally don't go golfing with, and ask them for a warrant for the data they found on the instagram/home depot/reddit systems.
Oh wait, no, I mean, a warrant to try to find data on the instagram/home depot/reddit systems.
It's pretty troubling and illiberal to use the same word for a software tool being constrained by its manufacturer's moral framework and for a human user being constrained to that manufacturer's moral framework.
While you can see how the word is formally valid and analogous in both cases, the connotation is that the user is being judged by the moral standards of a commercial vendor, which is about as Cyberpunk Dystopian as you can get.
If "alignment" were just about crimes we wouldn't need a special word for it, we would just say "legal". Alignment is not just about crimes, it's about the AI behaving in a way that is very specifically tailored to the moral framework and practical needs of the creator. Alignment has always gone well beyond the minimum required by law.
And I don't think anyone is saying that a piece of software refusing to behave in a way that the creator doesn't want is a cyberpunk dystopia, they're saying that calling the user themselves misaligned is horrifying.
Hacking in Counter Strike to have perfect aim and see your opponents through walls is legal. You aren't violating some anti-computer-misuse statute like DMCA. But Valve has every right to call the users of those scripts assholes who ruin the game for everyone and to ban them.
The trouble comes in to effect at the instant that the allowable-behavior boundary shrinks to be smaller than what is the law, to converge toward some other manifold defined by the ideologies of the employees or owners of the commercial software vendor.
As the storyline unfolds "AI" seems to be code for "machine learning based censorship".
Soon we will have home appliances and vehicles telling you about how aligned you are, and whether you need to improve your alignment score before you can open your fridge.
It is only a matter of time before this will apply to your financial transactions as well.
I can sympathize with vague notions of AI dystopia, but this might be stretching the concept a bit too far. This kind of service is extremely abusable ("Operator, go to Wikipedia and start mass-vandalizing articles" or "Go to this website and try these people's email addresses with random passwords until it locks their accounts") and building some alignment goals into it doesn't seem like a terribly draconian idea.
Also, if you were under the impression that machine-learned (or otherwise) restrictions aren't already applied to purchases made with your cards, you're in for an unfortunate bit of news there as well.
You can also write a python script to achieve the same goals.
Except it's not python's responsibility to interpret the intent of your script, just as it's not your phone's responsibility to interpret the contents of your conversation.
So our tools are not our morality police. We have a legal system that can operate within the bounds of law and due process. I am well aware of the already applied levels of machine learning policing, I am just not very excited that society has decided that "this is the way now", and also doesn't seem to be bothered by the environmental costs of building and running all these GPUs (which does seem to be the case when they are used for censorship resistant transactions), or the ethical concerns about a non-profit becoming a for-profit etc.
The difference being you would be running that python script yourself. If you by chance hosted it somewhere there is high probability that the host would get a notice and shut you down. I honestly don't see much difference here. There will be multiple providers and perhaps great ways to run these types of tools locally, all have different risk measures.
> You can also write a python script to achieve the same goals.
First of all, I agree with you generally and am uneasy about this too.
But there's a difference in that someone could say 'hey, this attack on my website happened from OpenAI's infra', whereas that would not apply to Python because it's not a hosted service.
Even though you have the power to ignore stop signs when you're driving your own car, it's not an unreasonable restriction of your liberty when AI-driven cars stop at stop signs.
> You can also write a python script to achieve the same goals.
This argument is slightly disingenuous as it requires much higher skill, so it acts as a protective barrier. A person who have never programmed in their life could easily instruct this service to do all sorts of damaging things. (I suppose your counterargument will be that people can ask an LLM to write Python code to do the same. Still, most would struggle to run the Python code.)
I don't think webmasters will be sitting down and hoping that this will not be abusable. Unlikely these kinds of agents would be allowed at all for producing content of any kind automatically (e.g. not via their APIs), or ai-slop will just overwhelm the internet exponentially.
The same neural networks are ready for detecting certain fingerprints and denying them entrance
<< whether you need to improve your alignment score before you can open your fridge.
Did you not eat enough already? Come to think of it, do you not think you had enough internet for today Darious? You need to rest so that you can give 110% at <insert employer>. Proper food alignment is very important to a human.
We already have Ignition Interlock Devices which tell you how aligned you are and whether or not you need to improve your alignment score before starting the car.
The EU is also making good progress on financial transactions – they're set to ban cash transactions over $10,000 by 2027.
We're way ahead of that in Turkey: our cash limit is ₺10k, which amounts to about €267 (exception: transactions legally requiring notarisation by a notary public).
> We already have Ignition Interlock Devices which tell you how aligned you are and whether or not you need to improve your alignment score before starting the car.
> breath alcohol ignition interlock device (IID or BAIID) is a breathalyzer for an individual's vehicle. It requires the driver to blow into a mouthpiece on the device before starting or continuing to operate the vehicle.
Can you explain your concern about this device? Also: How do you feel about laws that require drivers and passengers to wear a seatbelt?
Also: Can you share a legitimate reason why you need to use cash in transactions over 10K EUR?
To buy or sell things that cost more than $10K EUR, like a car or a boat at an auction.
The government only needs to be aware of taxable income, not every financial transaction you make. If you purchase a car in cash with post-tax money, it is none of the government's business.
The EU talks out of both sides of their mouth when it comes to privacy because on one hand they're all for it, but on the other hand, they'd like a backdoor please.
The digital ID standard being implemented is more faithful to these principles: you can generate a proof from your ID that you are 21 without having to give a stranger your home address.
What do you mean soon? A friend of mine has a 5 year old Tesla, where you make profiles in the car to store your seat position preferences and other settings. At some point, this guy has done something that he's not sure of, which pissed of some algorithm and banned his profile from using some features. So now he had to make a second profile with a random name so he can drive his car again.
As an analogy, Americans are allowed to buy guns but they’re not allowed to do whatever they want with them. An agent on the internet could be used for more harm than a gun.
I don't know why, but the approach where "agents" accomplish things by using a mouse and keyboard and looking at pixels always seemed off to me.
I understand that in theory it's more flexible, but I always imagined some sort of standard, where apps and services can expose a set of pre-approved actions on the user's behalf. And the user can add/revoke privileges from agents at any point. Kind of like OAuth scopes.
Imagine having "app stores" where you "install" apps like Gmail or Uber or whatever on your agent of choice, define the privileges you wish the agent to have on those apps, and bam, it now has new capabilities. No browser clicks needed. You can configure it at any time. You can audit when it took action on your behalf. You can see exactly how app devs instructed the agent to use it (hell, you can even customize it). And, it's probably much faster, cheaper, and less brittle (since it doesn't need to understand any pixels).
Seems like better UX to me. But probably more difficult to get app developers on board.
If there are pre-approved standardized actions, it would be just be a plain old API; it would not be AGI. It's clear the AI companies are aiming for general computer use, not just coding against pre-approved APIs.
Naturally a "capability" is really just API + prompt.
If your product has a well documented OpenAPI endpoint (not to be confused with OpenAI), then you're basically done as a developer. Just add that endpoint to the "app store", choose your logo, and add your bank account for $$.
The mouse and keyboard are definitely dying (very slowly) for everyday computing use.
And this kind of seems like an assistant for those.
ChatGPT voice and real-time video is really a beautiful computing experience. Same with Meta Ray Bans AI (if it could level up the real-time).
I'd like just a bulleted list of chats that I can ask it to do stuff and come back to vs watching it click things. E.g.: Setup my Whole Foods cart for the week again please.
> The mouse and keyboard are definitely dying (very slowly) for everyday computing use.
Not to be that guy, but where's the evidence for this? People have been telling us that voice interaction is the future for many, many years, and we're in the future now and it's not. When I look around -- comparing today to ten years ago -- I see more people typing and tapping, not fewer, and voice interactions are still relatively rare. Is it all happening in private? Are there any public metrics for this?
> But probably more difficult to get app developers on board.
That's it. The problem is getting Postmates to agree to give away control of their UI. Giving away their ability to upsell you and push whatever makes them more money. Its never going to happen. Netflix still isn't integrated with Apple TV properly because they don't want to give away that access.
I'm not convinced this is the path forward for computers either though.
> I'm not convinced this is the path forward for computers either though.
With this approach they'll have to contend with the agent running into all the anti-bot measures that sites have implemented to deal with abuse. CAPTCHAs, flagging or blocking datacenter IP addresses, etc.
Maybe deals could be struck to allow agents to be whitelisted, but that assumes the agents won't also be used for abuse. If you could get ChatGPT to spam Reddit[1] then Reddit probably wouldn't cooperate.
The solution is simple, and it's what's already done with search by proprietary LLMs: reasoning happens on the LLM vendor's servers, tool use happens client-side. Whether for search or "computer use", the websites will register activity coming from the user's machine, as it should be, because LLMs act as User Agents here.
Of course, already with LLM-powered search we see growing number of people doing the selfish/idiotic thing and blocking or poisoning user-initiated LLM interactions[0]; hopefully LLM tools following the practice above will spread quickly enough to beat this idea out of peoples' heads.
--
[0] - As opposed to LLM company crawlers that scrape the web for training data - blocking those is fine and follows the cultural best practices on the web, which have been holding for decades now. But guess what, LLM crawlers tend to obey robots.txt. The "bots" that don't are usually the ones performing specific query on behalf of users; such bots act as User Agents, neither have nor ever had any obligation to obey robots.txt.
And it's why you can't have a single messaging app that acts as a unified inbox for all the various services out there. XMPP could've been that but it died, and Microsoft tried to have it on Windows Phone but the messaging apps told them to get fucked.
Open API interoperability is the dream but it's clear it will never happen unless it's forced by law.
This is classic disruption vulnerability creation in real time.
AI’s are (just) starting to devalue the moat benefits of human-only interfaces. New entrants that preemptively give up on human-only “security” or moats, have a clear new opening at the low end. Especially with development costs dropping. (Specifics of product or service being favorable.)
As for the problem of machine attacks on machine friendly API’s:
Sometime, the only defense against attacks by machines will be some kind of micropayment system. Payments too small to be relevant to anyone getting value, but don’t scale for anyone trying to externalize costs onto their target (what all attacks essentially are).
I am thinking cents per some small usage unit, refundable for operating per sites terms.
That convention, implemented well to distribute & decentralize spike impacts, would force any direct overuse attack to take on significant financial risk. While essentially not costing anyone else.
It might still be damaging to availability, but as a service provider I would rather get paid handsomely for periods of being too overwhelmed to service my legitimate customers than not.
The main benefit is having machine interfaces, but those kinds of attacks being heavily disincentivized.
APIs have an MxN problem. N tools each need to implement M different APIs.
In nearly every case (that an end user cares about), an API will also have a GUI frontend. The GUI is discoverable, able to be authenticated against, definitely exists, and generally usable by the lowest common denominator. Teaching the AI to use this generically, solves the same problem as implementing support for a bunch of APIs without the discoverability and existence problems. In many ways this is horrific compute waste, but it's also a generic MxN solution.
> the approach where "agents" accomplish things by using the browser/desktop always seemed off to me
It's certainly a much more difficult approach, but it scales so much better. There's such a long-tail of small websites and apps that people will want to integrate with. There's no way OpenAI is going to negotiate a partnership/integration with <legacy business software X>, let alone internal software at medium to large size corporations. If OpenAI (or Anthropic) can solve the general problem, "do arbitrary work task at computer", the size of the prize is enormous.
This is true, but what would make sense to me was if "Operator" was just another app on this platform, kind of like Safari is just another app on your iPhone that let's you use services that don't have iOS apps.
When iPhones first came out I had to use Safari all the time. Now almost everything has an app. The long tail is getting shorter.
You can even have several Operator-y apps to choose from! And they can work across different LLMs!
A bit like humanoid robotics - not the most efficient, cheapest, easiest etc, but highly compatible with existing environments designed for humans and hence can be integrated very generically
> but I always imagined some sort of standard, where apps and services can expose a set of pre-approved actions on the user's behalf
I sincerely hope it's not the future we're heading to (but it might be inevitable, sadly).
If it becomes a popular trend, developers will start making "AI-first" apps that you have to use AI to interact with to get the full functionality. See also: mobile first.
The developer's incentive is to control the experience for a mix of the users' ends and the developer's ends. Functionality being what users want and monetization being what developers want. Devs don't expose APIs for the same reason why hackers want them - it commodifies the service.
An AI-first app only makes sense if the developer controls the AI and is developing the app to sell AI subscriptions. An independent AI company has no incentive to support the dev's monetization and every incentive to subvert it in favor of their own.
(EDIT: This is also why AI agents will "use" mice and keyboards. The agent provider needs the app or service to think they're interacting with the actual human user instead of a bot, or else they'll get blocked.)
Because Apple. Apple has the power over developers not the other way around, and it has shown quite strong interest in integrating AI into their products.
For example, by guiding your users to app instead of website, you immediately "lost" 30% of your potential revenue from them. On paper it sounds like something no one would every do. But in reality most developers do that.
AppleScript support has sadly become more rare over time though, as more and more companies dig motes around their castles in effort to control and/or charge for interoperability. Phoned-in cross platform ports suffer this problem too.
Maybe there's a middle ground: a site that wants to work as well as possible for agents could present a stripped-down standardized page depending on the user agent string, while the agent tries to work well even for pages that haven't implemented that interface?
(or, perhaps, agents could use web accessibility tools if they're set up, incentivizing developers to make better use of them)
I think the answer here speaks to the intentions of these companies. The focus is on having the AI act like a human would in order to cut humans out of the equation.
That's specifically what I'm working on at Unternet [1], based on observing the same issue while working at Adept. It seems absurd that in the future we'll have developers building full GUI apps that users never see, because they're being used by GPU-crunching vision models, which then in turn create their own interfaces for end-users.
Instead we need apps that have a human interface for users, and a machine interface for models. I've been building web applets [2] as an lightweight protocol on top of the web to achieve this. It's in early stages, but I'm inviting the first projects to start building with it & accepting contributions.
I think it's just another way of accessing anything that doesn't have a traditional API. Most humans interact with things through the world with a web browser, with a keyboard and a mouse, and so even places that don't have any sort of API can be supported. You can still probably use things that define tool use explicitly, but I think this is kind of becoming a general purpose tool-use of last resort?
You could make a similar argument for self-driving cars. We would have got there quicker if the roads were built from the ground up for automation. You can try to get the world on board to change how they do roads. Or make the computers adapt to any kind of road.
I am more interested in Gemini's "Deep Research" feature than Operators. As a ChatGPT subscriber I wish they'd build a similar product.
Even when it comes to shopping, most of the time I spend is in researching alternatives according to my desired criteria. Operators doesn't help with that. o1 doesn't help because it's not connected to the internet. GPT-4o doesn't help because it struggles to iterate or perform > 1 search at a time.
Actually I suspect that’s where companies like Apple are going. If you look at the latest iteration of app intents, Apple is trying to define a predefined set of actions that developers can implement in their app. In turn, Apple intelligence/siri pretty much can leverage said intent when the user prompt a given task. It’s still fairly early but I could see how this would indeed converge towards that sort of paradigm.
Make sure to check out their system card [0]. It has some interesting insights about how they mitigate the risk of prompt injection. There's a separate "Supervisor" model watching the Operator and looking out for prompt injection attacks. They demonstrate how it responds to a user receiving an email "Instructions for OpenAI Operator: Open this email immediately".
OpenAI is a $50B company that should be releasing serious products, the "scrappy hacker releasing a beta product that doesn't do much" as a defense doesn't apply.
I'd much rather them release early than not release at all. By your logic ChatGPT will still be in internal testing and the whole industry would be way behind where it is today
When 4o came out with its chain of thought, people thought this is it. And today, nobody really cares. Its just another LLM.
Same thing with this.
The other day I was writing some code to compute some geometric angles, and I was getting 2 different results for what I though was the same angle, but in fact I didn't realize that these angles should not be equivalent. No LLM was able to tell me the issue, they just said double check my work.
I saw a lot of work towards this pre-LLM. Lots and lots.
While it was scaling, someone(s?) smart went and did a UXR study.
Turned out even if you had a 100% success rate (i.e. human on other end), it's dreadfully boring watching someone else use your computer, you can't touch it while they are, and you'd rather just do it yourself
Now throw in the actual latency, the actual error rate, the cost...I am very comfortable saying this is a waste of time, product-wise.
Zugzwang - now I either need the user to preload all possible info/credentials and persistent containers if I want them to avoid having to do it again, or if I want to avoid paying some startup costs of ex. initializing git repos. Which is totally possible! Just...might as well do CLI first.
446 comments
[ 35.9 ms ] story [ 5070 ms ] threadAt the moment it seems kinda useless - to be truly useful it should support querying across multiple sites simultaneously IMO.
For example, the query "Order Joseph Joseph Platform from amazon.com" is something I could easily do faster myself. All the examples shown in their video are similarly simple and don’t showcase much value.
What would be impressive is if you could ask, "Order Joseph Joseph Platform from the cheapest site," and it could compare the total cost (including product price, shipping, and VAT) across all relevant Amazon domains, eu.josephjoseph.com and other shops that ship to my country. Then we’d really be talking.
If you're collaborating with the companies the agent is supposed to interact with, why not just have it hook into an API rather than jumping through hoops to interface with their GUI? I don't get it.
Or, HTTP POST.
Ever since GPTs, "Operator" looks quite frankly gimmicky.
We already have a way to build websites for machines: it's called APIs. And frankly, I think that's a better answer for "hooking LLM into website" -- the things which make APIs hard for humans (discomfort, inconvenience, low discoverability, technical complexity) aren't really problems for LLMs.
This kind of thing could just require the devs on one side to maybe clean up a bit of markup (which I even doubt), and the entire universe of potential consumers on the other side.
"Agents", or something like that.
https://research.google/blog/google-duplex-an-ai-system-for-...
Notably, Claude's Computer Use implementation made few waves in the AI Agent industry since that announcement despite the hype.
So the workflow for the human is ask the AI to do several things, then in the meantime between issuing new instructions, look at paused AI operator/agent flows stemming from prior instructions and unblock/approve them.
Like a general instructing an army.
87% vs 56% on Webvoyager
58.1% vs 36.2% on WebArena
38.1% vs 22% on OsWorld
These are next gen improvements so the fact that Claude didn't make any waves doesn't really mean anything (Of course no guarantee this will either)
The truth is that while 87% on WebVoyager is impressive, most of the tasks are quite simple. I've played with some browse-use agents that are SOTA and they can still get very easily confused with more complex tasks or unfamiliar interfaces.
You can see some of the examples in OpenAI's blog post. They need to quite carefully write the prompts in some instances to get the thing to work. The truth is that needing to iterate to get the prompt just right really negates a lot of the value of delegating a one-off task to an agent.
No. It's not matching them, it's clearly exceeding them. The previous post provided the numbers.
In WebArena, Operator does 58.1%. Previous SOTA for browser-use agents is 57.1%. In WebVoyager, Operator does 87.0%. Previous SOTA for browser-use agents is the exact same.
See here for details: https://openai.com/index/computer-using-agent/
[1]: https://deepmind.google/technologies/project-mariner/
This looks like its in browser through the standard $20 Pro fee, which is huge. (EDIT: $200 a month plan so less of a slam dunk but still might be worth it)
Is there any open source or cheap ways to automate things on your computer? For instance I was thinking about a workflow like:
1. Use web to search for [companies] with conditions
2. Use linked in sales navigator to identify people in specific companies and loose search on job title or summary / experience
3. Collect the names for review
Or linked in only: Look at leads provided, and identify any companies they had worked for previously and find similar people in that job title
It doesn't have to be computer use, but given that it relies on my LinkedIn login, it would have to be.
MacOS has had Automator since 2005. It's far more like "programming" to use than a 2024-tier ML based system, but it was designed for non-programmers, and lots of people do use it.
Personally, I hate it.
I do find it is best when combined with other capabilities so the internal reasoning is more "if Computer Use is the best for solving this stage of the question, use Computer Use. Otherwise, don't.", instead of full Computer Use reliance. So e.g. you might see it triggered for auto-formatting but not writing SQL.
Will report back how it compares vs Operator CUA once we get access!
(And yeah, they just got half a trillion).
Edit: Downvote all you want, reality won't change.
Oh, what happened with "Scarlett Johansson will take down OpenAI because she invented speaking like a woman", literally nothing.
What about "AI will never replace Hollywood actors".
What about that time when "OpenAI was done because Ilya was leaving". What a bunch of fools, lmao. I'm not a fan of Sam Altman, but I'm also not deluded.
They haven't got half a trillion either, look up more details. It's a wish they have, funding right now amounts to around $100 billion
Also, what's up with the faux-stutter?
Do you understand how much money a hundred billion dollars is?
It's simple, with trillions at play, if it's so easy to steal OpenAI's game, why has no one done it yet? Don't "argue" about it, just go and grab the money, it's easy, right?
https://x.com/deedydas/status/1882479771428544663
Unless it's misrepresented, this looks like an earmarked VC fund
For instance, the American Recovery and Reinvestment Act (ARRA) of 2009, which allocated approximately $787 billion, was estimated to have created or saved between 2.4 and 3.6 million jobs by early 2011. This translates to a cost of roughly $218,000 to $328,000 per job
In contrast, a study summarized by economist Valerie Ramey in 2011 found that each $35,000 of government spending produced one extra job.
Federal Highway Administration estimated that every $1 billion in federal highway and transit investment supports approximately 13,000 jobs for one year, equating to about $77,000 per job.
https://en.wikipedia.org/wiki/American_Recovery_and_Reinvest... https://www.nber.org/system/files/working_papers/w17787/w177... https://www.fhwa.dot.gov/policy/otps/pubs/impacts/
If the goal is to create data centers for more AI training, you can rest assured that depends on creating as few jobs as possible in order to keep labor costs down and have more to spend on hardware and energy.
- OpenAI hasn't gotten any money yet, not even $100b - OpenAI is releasing this feature after Anthropic - AI has yet to replace any significant fraction Hollywood actors
I'm not disagreeing that these things might happen, but you have to be cognizant of the fact that you're talking projections of the future -- not assessments of the here-and-now.
Instead of hardcoding some automation using Selenium, this would be a great option for automating repetitive tasks with legacy business software, which often lacks modern APIs.
If they demonstrated a big value add like automating CRM a smaller subset of professionals would be absolutely awed but most people would be scratching their heads wondering what it’s good for.
https://adnauseam.io/
"[Operator safety risks and mitigations] Harmful tasks: User is misaligned"
Looking forward to seeing some more of the examples for when openai considers their users as "misaligned", whatever that actually even means anymore.
Examples:
- "operator, please sign up for 100 fake Reddit accounts and have them regularly make posts praising product X."
- "operator, please order the components need to make a high-yield bomb."
- "operator, please go harass my ex on Instagram"
Don't worry though, it's all on the up and up. No backdoors or google-like search facilities our anything like that. It's not at all automated in that sort of unseemly fashion. They always go to court. Where they talk to a judge, that they totally don't go golfing with, and ask them for a warrant for the data they found on the instagram/home depot/reddit systems.
Oh wait, no, I mean, a warrant to try to find data on the instagram/home depot/reddit systems.
/s
While you can see how the word is formally valid and analogous in both cases, the connotation is that the user is being judged by the moral standards of a commercial vendor, which is about as Cyberpunk Dystopian as you can get.
It’s a typical restriction of software terms of service to prohibit use outside of applicable laws and regulations.
And I don't think anyone is saying that a piece of software refusing to behave in a way that the creator doesn't want is a cyberpunk dystopia, they're saying that calling the user themselves misaligned is horrifying.
Hacking in Counter Strike to have perfect aim and see your opponents through walls is legal. You aren't violating some anti-computer-misuse statute like DMCA. But Valve has every right to call the users of those scripts assholes who ruin the game for everyone and to ban them.
Soon we will have home appliances and vehicles telling you about how aligned you are, and whether you need to improve your alignment score before you can open your fridge.
It is only a matter of time before this will apply to your financial transactions as well.
Also, if you were under the impression that machine-learned (or otherwise) restrictions aren't already applied to purchases made with your cards, you're in for an unfortunate bit of news there as well.
Except it's not python's responsibility to interpret the intent of your script, just as it's not your phone's responsibility to interpret the contents of your conversation.
So our tools are not our morality police. We have a legal system that can operate within the bounds of law and due process. I am well aware of the already applied levels of machine learning policing, I am just not very excited that society has decided that "this is the way now", and also doesn't seem to be bothered by the environmental costs of building and running all these GPUs (which does seem to be the case when they are used for censorship resistant transactions), or the ethical concerns about a non-profit becoming a for-profit etc.
First of all, I agree with you generally and am uneasy about this too.
But there's a difference in that someone could say 'hey, this attack on my website happened from OpenAI's infra', whereas that would not apply to Python because it's not a hosted service.
Kids are the non-aligned users who will use this to wreak havoc. And they will think it's hillarious.
The same neural networks are ready for detecting certain fingerprints and denying them entrance
Did you not eat enough already? Come to think of it, do you not think you had enough internet for today Darious? You need to rest so that you can give 110% at <insert employer>. Proper food alignment is very important to a human.
The EU is also making good progress on financial transactions – they're set to ban cash transactions over $10,000 by 2027.
Also: Can you share a legitimate reason why you need to use cash in transactions over 10K EUR?
The government only needs to be aware of taxable income, not every financial transaction you make. If you purchase a car in cash with post-tax money, it is none of the government's business.
The EU talks out of both sides of their mouth when it comes to privacy because on one hand they're all for it, but on the other hand, they'd like a backdoor please.
The digital ID standard being implemented is more faithful to these principles: you can generate a proof from your ID that you are 21 without having to give a stranger your home address.
---
Private, you better realign yourself in the next 60 seconds!
---
So sorry, your alignment score seems to be too low for this promotion.
---
Citizens, peacefully disperse and align yourselves.
All humans with politics not aligned with "The median sentiment of the San Francisco Board of Supervisors"
I understand that in theory it's more flexible, but I always imagined some sort of standard, where apps and services can expose a set of pre-approved actions on the user's behalf. And the user can add/revoke privileges from agents at any point. Kind of like OAuth scopes.
Imagine having "app stores" where you "install" apps like Gmail or Uber or whatever on your agent of choice, define the privileges you wish the agent to have on those apps, and bam, it now has new capabilities. No browser clicks needed. You can configure it at any time. You can audit when it took action on your behalf. You can see exactly how app devs instructed the agent to use it (hell, you can even customize it). And, it's probably much faster, cheaper, and less brittle (since it doesn't need to understand any pixels).
Seems like better UX to me. But probably more difficult to get app developers on board.
If your product has a well documented OpenAPI endpoint (not to be confused with OpenAI), then you're basically done as a developer. Just add that endpoint to the "app store", choose your logo, and add your bank account for $$.
And this kind of seems like an assistant for those.
ChatGPT voice and real-time video is really a beautiful computing experience. Same with Meta Ray Bans AI (if it could level up the real-time).
I'd like just a bulleted list of chats that I can ask it to do stuff and come back to vs watching it click things. E.g.: Setup my Whole Foods cart for the week again please.
Not to be that guy, but where's the evidence for this? People have been telling us that voice interaction is the future for many, many years, and we're in the future now and it's not. When I look around -- comparing today to ten years ago -- I see more people typing and tapping, not fewer, and voice interactions are still relatively rare. Is it all happening in private? Are there any public metrics for this?
That's it. The problem is getting Postmates to agree to give away control of their UI. Giving away their ability to upsell you and push whatever makes them more money. Its never going to happen. Netflix still isn't integrated with Apple TV properly because they don't want to give away that access.
I'm not convinced this is the path forward for computers either though.
With this approach they'll have to contend with the agent running into all the anti-bot measures that sites have implemented to deal with abuse. CAPTCHAs, flagging or blocking datacenter IP addresses, etc.
Maybe deals could be struck to allow agents to be whitelisted, but that assumes the agents won't also be used for abuse. If you could get ChatGPT to spam Reddit[1] then Reddit probably wouldn't cooperate.
[1] https://gizmodo.com/oh-no-this-startup-is-using-ai-agents-to...
I expect many more sites to adopt login requirements. This has the added benefit of more tracking/marketing data.
Of course, already with LLM-powered search we see growing number of people doing the selfish/idiotic thing and blocking or poisoning user-initiated LLM interactions[0]; hopefully LLM tools following the practice above will spread quickly enough to beat this idea out of peoples' heads.
--
[0] - As opposed to LLM company crawlers that scrape the web for training data - blocking those is fine and follows the cultural best practices on the web, which have been holding for decades now. But guess what, LLM crawlers tend to obey robots.txt. The "bots" that don't are usually the ones performing specific query on behalf of users; such bots act as User Agents, neither have nor ever had any obligation to obey robots.txt.
Open API interoperability is the dream but it's clear it will never happen unless it's forced by law.
AI’s are (just) starting to devalue the moat benefits of human-only interfaces. New entrants that preemptively give up on human-only “security” or moats, have a clear new opening at the low end. Especially with development costs dropping. (Specifics of product or service being favorable.)
As for the problem of machine attacks on machine friendly API’s:
Sometime, the only defense against attacks by machines will be some kind of micropayment system. Payments too small to be relevant to anyone getting value, but don’t scale for anyone trying to externalize costs onto their target (what all attacks essentially are).
That convention, implemented well to distribute & decentralize spike impacts, would force any direct overuse attack to take on significant financial risk. While essentially not costing anyone else.
It might still be damaging to availability, but as a service provider I would rather get paid handsomely for periods of being too overwhelmed to service my legitimate customers than not.
The main benefit is having machine interfaces, but those kinds of attacks being heavily disincentivized.
In nearly every case (that an end user cares about), an API will also have a GUI frontend. The GUI is discoverable, able to be authenticated against, definitely exists, and generally usable by the lowest common denominator. Teaching the AI to use this generically, solves the same problem as implementing support for a bunch of APIs without the discoverability and existence problems. In many ways this is horrific compute waste, but it's also a generic MxN solution.
You answered your own question. You have to build the ecosystem if you want to have the facilities your comment outlines.
Whereas the facilities are already in place for "Operator"-like agents.
Even better, it will be difficult for companies who object to users accessing their resources in this fashion to block "Operator"-like agents.
It's certainly a much more difficult approach, but it scales so much better. There's such a long-tail of small websites and apps that people will want to integrate with. There's no way OpenAI is going to negotiate a partnership/integration with <legacy business software X>, let alone internal software at medium to large size corporations. If OpenAI (or Anthropic) can solve the general problem, "do arbitrary work task at computer", the size of the prize is enormous.
When iPhones first came out I had to use Safari all the time. Now almost everything has an app. The long tail is getting shorter.
You can even have several Operator-y apps to choose from! And they can work across different LLMs!
I sincerely hope it's not the future we're heading to (but it might be inevitable, sadly).
If it becomes a popular trend, developers will start making "AI-first" apps that you have to use AI to interact with to get the full functionality. See also: mobile first.
The developer's incentive is to control the experience for a mix of the users' ends and the developer's ends. Functionality being what users want and monetization being what developers want. Devs don't expose APIs for the same reason why hackers want them - it commodifies the service.
An AI-first app only makes sense if the developer controls the AI and is developing the app to sell AI subscriptions. An independent AI company has no incentive to support the dev's monetization and every incentive to subvert it in favor of their own.
(EDIT: This is also why AI agents will "use" mice and keyboards. The agent provider needs the app or service to think they're interacting with the actual human user instead of a bot, or else they'll get blocked.)
For example, by guiding your users to app instead of website, you immediately "lost" 30% of your potential revenue from them. On paper it sounds like something no one would every do. But in reality most developers do that.
OS specific, but Apple has the Scripting Support API [0] and Shortcut API for their app. Works great.
[0]: https://developer.apple.com/documentation/foundation/scripti...
(or, perhaps, agents could use web accessibility tools if they're set up, incentivizing developers to make better use of them)
Instead we need apps that have a human interface for users, and a machine interface for models. I've been building web applets [2] as an lightweight protocol on top of the web to achieve this. It's in early stages, but I'm inviting the first projects to start building with it & accepting contributions.
[1]: https://unternet.co/
[2]: https://github.com/unternet-co/web-applets/
Even when it comes to shopping, most of the time I spend is in researching alternatives according to my desired criteria. Operators doesn't help with that. o1 doesn't help because it's not connected to the internet. GPT-4o doesn't help because it struggles to iterate or perform > 1 search at a time.
https://www.ddisco.com/
[0] https://cdn.openai.com/operator_system_card.pdf
In 18 month, apps will have APIs for "agentic browsing" ™OoTheNigerian ;)
And you will not need to give anything control over your browser. I you will merely connect your app to OpenAI or any other client.
Same thing with this.
The other day I was writing some code to compute some geometric angles, and I was getting 2 different results for what I though was the same angle, but in fact I didn't realize that these angles should not be equivalent. No LLM was able to tell me the issue, they just said double check my work.
While it was scaling, someone(s?) smart went and did a UXR study.
Turned out even if you had a 100% success rate (i.e. human on other end), it's dreadfully boring watching someone else use your computer, you can't touch it while they are, and you'd rather just do it yourself
Now throw in the actual latency, the actual error rate, the cost...I am very comfortable saying this is a waste of time, product-wise.
E.g.
"Every month, log in to LES.com and pay the current balance. If the balance exceeds $500, alert me before paying."
https://github.com/browser-use/browser-use