"ReALM reconstructs the visual layout of a screen using textual representations. This involves parsing on-screen entities and their locations to generate a textual format that captures the screen's content and structure. Apple researchers found that this strategy, combined with specific fine-tuning of language models for reference resolution tasks, significantly outperforms traditional methods, including the capabilities of OpenAI's GPT-4.*
I would have agreed with you more readily a year or two ago. Here’s a quote from the current Sherman lawsuit against Apple:
As Apple’s executives openly acknowledge: “In looking at it with hindsight, I think going forward we need to set a stake in the ground for what features we think are ‘good enough’ for the consumer. I would argue we’re already doing more than what would have been good enough. But we find it very hard to regress our product features YOY [year over year].”
It sounds to me like they think their products are too excellent, and they want them to be worse actually.
Apple needs to go back to the Steve Jobs era where they just released a final product with no rumors. Just "here it is" on stage, and available that day in the store. They need to replace Siri ASAP with a real AI LLM.
Imagine you have a picture book, and each page shows a different scene with various characters and objects. Now, think of a smart robot that can look at these pages and understand what's in them.
Apple's scientists have created a robot called ReALM that does something similar but with computer screens. When ReALM looks at a screen, it doesn't just see an image. Instead, it reads the screen like a book, identifying all the different things on the screen and where they are located.
ReALM then writes down what it sees in a special way, kind of like making a list of everything on the screen and giving each item a specific place. This helps ReALM understand the screen's content and how it's organized.
By doing this and with some extra training, ReALM has become really good at a task called "reference resolution." This means that when you ask ReALM about something specific on the screen, it can quickly find and point out what you're asking for, even better than other smart robots like GPT-4.
In short, ReALM is like a super-smart robot that can read computer screens, make a list of what it sees, and help you find things on the screen faster and better than ever before!
We don't need an auto-generated summary LLM-spam on every post. I'm not saying you do this every post, but if every post got this it would be fairly obnoxious.
I find it to be useful. Would you feel any different if they read the article, thought about it, typed up a summary, edited it a few times, and posted it instead?
That also doesn't really happen right now and I'm not sure a summary is an improvement whoever writes it. You can just run that article through an llm yourself to get a summary - comment threads are useful for discussions with other humans about the content. What's it adding to the conversation? Are you hoping that everyone starts replying to the summary to discuss so it's replies take over to become the discussion for the article? If not, does it just get pushed to the bottom since nobody interacts with it, and then what's the point? It could be nice as a feature but as a comment I'd argue human summaries or AI summaries are pretty much just spam. Because unlike the archive link comments that people post where there's a clear point that it's done and you don't need to post another one - what happens if you don't like the summary? Or if it's controversial and nuanced? Summary wars? I feel even more strongly about that now that I've thought through it - no I wouldn't feel different and would dislike both equally.
14 comments
[ 2.3 ms ] story [ 46.6 ms ] thread"ReALM reconstructs the visual layout of a screen using textual representations. This involves parsing on-screen entities and their locations to generate a textual format that captures the screen's content and structure. Apple researchers found that this strategy, combined with specific fine-tuning of language models for reference resolution tasks, significantly outperforms traditional methods, including the capabilities of OpenAI's GPT-4.*
As Apple’s executives openly acknowledge: “In looking at it with hindsight, I think going forward we need to set a stake in the ground for what features we think are ‘good enough’ for the consumer. I would argue we’re already doing more than what would have been good enough. But we find it very hard to regress our product features YOY [year over year].”
It sounds to me like they think their products are too excellent, and they want them to be worse actually.
Imagine you have a picture book, and each page shows a different scene with various characters and objects. Now, think of a smart robot that can look at these pages and understand what's in them.
Apple's scientists have created a robot called ReALM that does something similar but with computer screens. When ReALM looks at a screen, it doesn't just see an image. Instead, it reads the screen like a book, identifying all the different things on the screen and where they are located.
ReALM then writes down what it sees in a special way, kind of like making a list of everything on the screen and giving each item a specific place. This helps ReALM understand the screen's content and how it's organized.
By doing this and with some extra training, ReALM has become really good at a task called "reference resolution." This means that when you ask ReALM about something specific on the screen, it can quickly find and point out what you're asking for, even better than other smart robots like GPT-4.
In short, ReALM is like a super-smart robot that can read computer screens, make a list of what it sees, and help you find things on the screen faster and better than ever before!
So basically “Johnny 5”.
https://youtu.be/lv9owSY015w?feature=shared