Show HN: Gemini Cursor – A Multimodal AI Cursor for Your Desktop (Open Source) (github.com)

20 points by 13point5 1y ago ↗ HN

I built Gemini Cursor, an open-source multimodal AI cursor that guides users through tasks on their desktop by pointing and speaking.

It leverages Gemini 2.0 Flash and Google's live multimodal API to analyze what's on screen and provide real-time assistance.

In this demo, my friend tries to add a payment method to Amazon, and the AI cursor walks them through the entire process with visual cues and spoken instructions.

I've also used it to interpret diagrams from research papers—curious to see what other use cases people find this useful for!

Demo: https://x.com/27upon2/status/1889128655672029582

Repo: https://github.com/13point5/gemini-cursor

2 comments

[ 2.9 ms ] story [ 4.6 ms ] thread

jcmp 1y ago ↗

Wow! Especially smart with the seperate api call for the position of the element

wifipunk 1y ago ↗

This is a surprisingly useful base project for AI presentation handling where you can be hands free pointing out various details or be typing simultaneously.

Hope you keep working on the project. Different cursor settings for size, shape, and color would be nice.