Impressively, it also quickly passed levels 1 (checkbox) and 2 (stop sign) on http://neal.fun/not-a-robot, and got most of the way through level 3 (wiggly text).
My general experience has been that Gemini is pretty bad at tool calling. The recent Gemini 2.5 Flash release actually fixed some of those issues but this one is Gemini 2.5 Pro with no indication about tool calling improvements.
It is actually quite good at following instructions, but I tried clicking on job application links, and since they open in a new window, it couldn't find the new window. I suppose it might be an issue with BrowserBase, or just the way this demo was set up.
I sure hope this is better than pathetically useless. I assume it is to replace the extremely frustrating Gemini for Android. If I have a bluetooth headset and I try "play music on Spotify" it fails about half the time. Even with youtube music. I could not believe it was so bad so I just sat at my desk with the helmet on and tried it over and over. It seems to recognise the speech but simply fails to do anything. Brand new Pixel 10. The old speech recognition system was way dumber but it actually worked.
There are some absolutely atrocious UIs out there for many office workers, who spend hours clicking buttons opening popup after popup clicking repetitively on checkboxes etc. E.g. entering travel costs or somesuch in academia and elsewhere. You have no idea how annoying that type of work is, you pull out your hair. Why don't they make better UIs, you ask? If you ask, you have no idea how bad things are. Because they don't care, there is no communication, it seems fine, the software creators are hard to reach, the software is approved by people who never used it and decide based on gut feel, powerpoints and feature tickmarks. Even big name brands are horrible at this, like SAP.
If such AI tools allow to automate this soulcrushing drudgery, it will be great. I know that you can technically script things Selenium, AutoHotkey whatnot. But you can imagine that it's a nonstarter in a regular office. This kind of tool could make things like that much more efficient. And it's not like it will then obviate the jobs entirely (at least not right away). These offices often have immense backlogs and are understaffed as is.
How big are Gemini 2.5(Pro/Flash/Lite) models in parameter counts, in experts' guesstimation? Is it towards 50B, 500B, or bigger still? Even Flash feels smart enough for vibe coding tasks.
Knowing it's technically possible is one thing, but giving it a short command and seeing it go log in to a site, scroll around, reply to posts, etc. is eerie.
Also it tied me at wordle today, making the same mistake I did on the second to lass guess. Too bad you can't talk to it while it's working.
Not great at Google Sheets. Repeatedly overwrites all previous columns while trying to populate new columns.
> I am back in the Google Sheet. I previously typed "Zip Code" in F1, but it looks like I selected cell A1 and typed "A". I need to correct that first. I'll re-type "Zip Code" in F1 and clear A1. It seems I clicked A1 (y=219, x=72) then F1 (y=219, x=469) and typed "Zip Code", but then maybe clicked A1 again.
I am on https://gemini.browserbase.com/ and just click the use case mentioned on the site "Go to Hacker News and find the most controversial post from today, then read the top 3 comments and summarize the debate."
It did not work, multiple times, just gets stuck after going to Hacker news.
It's a bit funny that I give Google Gemini a task and then it goes on the Google Search site and it gets stuck in the captcha tarpit that's supposed to block unwanted bots. But I guess Google Gemini shouldn't be unwanted for Google. Can't you ask the search team to whitelist the Gemini bot?
Interesting, seems to use 'pure' vision and x/y coords for clicking stuff. Most other browser automation with LLMs I've seen uses the dom/accessibility tree which absolutely churns through context, but is much more 'accurate' at clicking stuff because it can use the exact text/elements in a selector.
Unfortunately it really struggled in the demos for me. It took nearly 18 attempts to click the comment link on the HN demo, each a few pixels off.
Many years ago I was sitting at a red light on a secondary road, where the primary cross road was idle. It seemed like you could solve this using a computer vision camera system that watched the primary road and when it was idle, would expedite the secondary road's green light.
This was long before computer vision was mature enough to do anything like that and I found out that instead, there are magnetic systems that can detect cars passing over - trivial hardware and software - and I concluded that my approach was just far too complicated and expensive.
Similarly, when I look at computers, I typically want the ML/AI system to operate on a structured data that is codified for computer use. But I guess the world is complicated enough and computers got fast enough that having an AI look at a computer screen and move/click a mouse makes sense.
Computer use is the most important AI benchmark to watch if you're trying to forecast labor-market impact. You're right, there are much more effective ways for ML/AI systems to accomplish tasks on the computer. But they all have to be hand-crafted for each task. Solving the general case is more scalable.
My town solved this at night by putting simple light sensors on the traffic lights so as you approach you can flash ur brights at it and it triggers a cycle.
Otherwise the higher traffic road got a permanent green light at nighttime until it saw high beams or magnetic flux from a car reaching the intersection.
It was my first engineering job, calibrating those inductive loops and circuit boards on I-93, just north of Boston's downtown area. Here is the photo from 2006. https://postimg.cc/zbz5JQC0
PEEK controller, 56K modem, Verizon telco lines, rodents - all included in one cabinet
I recently spent some time in a country house far enough from civilization that electric lines don’t reach. The owners could have installed some solar panels, but they opted to keep it electricity-free to disconnect from technology, or at least from electronics. They have multiple decades old ingenious utensils that work without electricity, like a fridge that uses propane, oil lamps, non-electric coffee percolator, etc. and that made me wonder, how many analogous devices stopped getting invented because an electric device is the most obvious way of solving things to our current view.
Detects if you are coming to the intersection and with what speed, and if there is no traffic blocking you automatically cycles the red lights so you don’t have to stop at all.
I don't know the implementation details, but this is common in the county I live in (US). It's been in use for the last 3-5 years. The traffic lights adapt to current traffic patterns in most intersections and speed up the green light for roads that have cars.
91 comments
[ 2.2 ms ] story [ 89.3 ms ] threadObviously much harder with UI vs agent events similar to the below.
https://docs.claude.com/en/docs/claude-code/hooks
https://google.github.io/adk-docs/callbacks/
If such AI tools allow to automate this soulcrushing drudgery, it will be great. I know that you can technically script things Selenium, AutoHotkey whatnot. But you can imagine that it's a nonstarter in a regular office. This kind of tool could make things like that much more efficient. And it's not like it will then obviate the jobs entirely (at least not right away). These offices often have immense backlogs and are understaffed as is.
Knowing it's technically possible is one thing, but giving it a short command and seeing it go log in to a site, scroll around, reply to posts, etc. is eerie.
Also it tied me at wordle today, making the same mistake I did on the second to lass guess. Too bad you can't talk to it while it's working.
> I am back in the Google Sheet. I previously typed "Zip Code" in F1, but it looks like I selected cell A1 and typed "A". I need to correct that first. I'll re-type "Zip Code" in F1 and clear A1. It seems I clicked A1 (y=219, x=72) then F1 (y=219, x=469) and typed "Zip Code", but then maybe clicked A1 again.
It did not work, multiple times, just gets stuck after going to Hacker news.
Unfortunately it really struggled in the demos for me. It took nearly 18 attempts to click the comment link on the HN demo, each a few pixels off.
This was long before computer vision was mature enough to do anything like that and I found out that instead, there are magnetic systems that can detect cars passing over - trivial hardware and software - and I concluded that my approach was just far too complicated and expensive.
Similarly, when I look at computers, I typically want the ML/AI system to operate on a structured data that is codified for computer use. But I guess the world is complicated enough and computers got fast enough that having an AI look at a computer screen and move/click a mouse makes sense.
Motorcyclists would conclude that your approach would actually work.
Otherwise the higher traffic road got a permanent green light at nighttime until it saw high beams or magnetic flux from a car reaching the intersection.
PEEK controller, 56K modem, Verizon telco lines, rodents - all included in one cabinet
Detects if you are coming to the intersection and with what speed, and if there is no traffic blocking you automatically cycles the red lights so you don’t have to stop at all.