OmniParser V2 – A simple screen parsing tool towards pure vision based GUI agent (github.com)

This is not the intended use but it good working on parsing document layout from image.

Very cool work. Accurate GUI text and element parsing is exactly the kind of input that LLMs need to be effective agents.

The OS has additional information including how different graphics layers are composited, and what accessibility metadata is attached to interface elements. It ought to be useful to exploit this to do better than screenshot parsing.

One ponders the connections with the Recall feature