I'm sure the tech is neat. But I get a real sinking feeling from how hard tech giants like "Meta" are trying openly to create dystopia. The "Metaverse" is not aspirational!
> That understanding would let AR glasses tailor content to you and your individual context, like seamlessly blending a digital overlay with your physical space
Horrifying, if it ever takes off in the way that "Meta" wants it to. Take the modern smartphone disease and amplify it a few orders of magnitude.
> While LLMs rely on vast amounts of training data that typically comes from a range of publicly available text sources on the web, no such repository of information yet exists for physical spaces at the scale needed for training an end-to-end model.
Considering the tremendous scale in question, the "yet" is also terrifying. The only way to collect such a dataset would be if some large fraction of people were wearing always-on, network connected cameras. Meta clearly sees this future and wants it; the kind of future where everyone wears a Meta headset, all the time, and has their experience of reality mediated through it - optimized to maximize shareholder value of course, under the watchful eye of The Algorithm. A panopticon AI shoggoth.
Meta is an AI paperclip maximizer, and it is getting uncomfortably close to the "RELEASE THE HYPNODRONES" stage.
Re: that "yet". I noticed that after a recent update on my Quest the OS gave me a pop-up about a new feature, that when enabled would allow the device to remember my play boundaries and restore them automatically rather than requiring they be re-entered every time the system gets confused about where I am. I was a little confused why it didn't just turn that on by default, it seemed like a straight upgrade to the current handling and shouldn't require anything but a bit of local storage. Scrolling down in the pop-up I was informed that enabling the feature would authorize Facebook to scan my surroundings and upload a complete map of them to use for whatever purposes they desired. I guess they legally need consent for that, and tying a feature improvement to data collection is still totally allowed somehow. So I imagine that they are building that real-world dataset presently.
>I guess they legally need consent for that, and tying a feature improvement to data collection is still totally allowed somehow.
As egregious as this is, it still doesn't strike me as nakedly insulting as Google's refusal to let you turn on the (entirely passive!) GPS receiver on your phone without uploading your precise location to their servers, for similar data collection purposes.
I'd love to see this sort of thing be made illegal, but the only regulatory domain with an Overton window even close to that is the EU, and it's such a minefield of potential gray areas that I doubt it would get off the ground even there.
You must have missed the part where they created a synthetic dataset of 100000 environments, trained the network on that, and released it to the public?
Not saying I’m not worried about their plans too, but that is clearly addressed in the text.
* How uncomfortable the headsets are to wear for long periods if time
* How useless they are (what are they solution for?)
* How ridiculous they look
* How expensive they are
* How expensive it is to produce content for
After they solve these issues then I will worry about the dystopia we are in. VR headsets may look cool and futuristic in the movies but in reality they are just screens you wear on your head and that's it.
I'd think that all of them would be solved by time. I'm just saying they are nothing to be concerned with until they are (and we are still far way from that point in time). They are just fancy demos to make a company look attractive to investors.
Much has been written about the dystopia we are already in, but I am specifically concerned about the dystopia they are earnestly building. Meta is aware of these problems and is trying very hard to solve them.
And they won't be useless! That's the worst part. They will provide compelling value, like smartphones, and as such will be impossible to boycott. At first the value will be simple, obvious utility, like route overlays. Later it will be value within the system, much like how the main utility of smartphones now is to run the mandatory apps to participate in aspects of society that previously did not require a smartphone. And of course they will be addictive.
They wish. Naturally wearing screens on your head is not a healthy thing for humans to do for hours at a time. Naturally that tends to cause headaches, eyestrain, nausea etc.
Now as far as making them more useful I suppose *if* they can solve that first problem plus make them match the form of IMAX glasses (preferably smaller). Then yes they could be useful (providing there's good enough applications).
But don't cause the user to get sick feature is not 'a nice to have' you can't expect it to spread like the smartphone if people can't use it for long periods of time.
Will Meta solve that problem? Doubtful. Will they provide good enough applications to make AR worth using once they do? I wouldn't bet on it.
Just because a company has an incentive for something to be successful doesn't mean they can pull off. Just look at metaverse.
Want more data? Create an mobile app to cartograph new rental appartments. I know in other countries you often rent with furniture in it. We in germany for example take everything with us and have to check if/how the furniture fits.
With a system like the one in the video you walk through the old place, and walk through the new place, and output for example 2 Sweet Home 3d plan's.
This was a very important missing piece for mixed/augmented and virtual reality.
The semantic understanding of the objects around you (without upload to a server).
A detailed set of actual meshes of the objects in your living and working spaces.
So much can be built on this key building block, to actually help people with their daily lives, low vision people are only one group that will see massive benefits.
I also think this technique could and should be extended generally into computer vision applications, understanding semantic meaning, volumes and masses via the often highly structured environments around us.
Starting from a principled position of meaning, volumes and meshes and then training to assign gaussian splats to those meshes seems like something clever people could solve a lot of problems with.
Doesn't matter. If the app has to render them then it can still send them to the server. If the app does not render directly it can still infer them by doing collision testing against them. If it can't even check for collisions then there isn't much point in having the data in the first place.
It gets worse. The more you want AR/MR to understand the scene the more fine grained data you need. The camera sees things like: the underwear you left on the floor, the medicines/pills on your table, the posters on your wall, the bank account browser tab still on your computer screen, the adult toys you forgot to put away, your undressed body, your dressed body, your political baseball cap or t-shirt, your naked roommate/spouse/kids who just walked into the room.
I don't see how this can be solved in a trust-less way. In other words, you'll have to trust each and every individual AR/MR app you use, that they aren't spying on you. The platform/device/OS itself can not prevent this and still allow the majority of the AR/MR uses that people imagine.
Companies don't spy on you because they don't like you, they spy on you because they make money out of it. Outlaw making money out of personal data (by outlawing targeted ads, data purchase for insurance, and data brokers) and the incentives to spy on you disappear.
You can't be probusiness and pro-privacy at the same time though, you need to limit businesses in order to protect the people.
Yes, the camera sees everything. But we could avoid teaching the model certain things. For instance, an undressed body and a dressed body could both be taught as a body. Likewise, medicine pills as well as regular mints could just be taught as mints.
While this approach doesn’t address privacy completely, it avoids certain elements of our life that are considered really private.
I think half of the point of using the simulated dataset to train the model was to safeguard the above mentioned kind of privacy (with the other half being lack of real-world dataset).
I think there must be a way of doing this so you just have to trust the OS. The camera feed itself could be essentially private and only pass on locations and types of objects. In this way you could whitelist / blocklist items or categories.
But you would defo have to trust the OS, and in this case Meta…
My issue with this approach is that the synthetic data, and the language used to describe it, necessarily encode a viewpoint - a bias - on the part of the creators. I am firmly behind the need to use language as an intermediary signifier, not just for utility purposes, but because it unlocks the ability to use language as an anchor for arbitrary AR data. My problem with this particular approach is that it requires us all to agree on what things are called, and requires the training dataset to cover everything users might come across.
My feeling is the better approach is to train a generalized segmentation model, derive multi-parameter representations of those segmented ‘things’, derive embeddings from those things for compression purposes, then allow users to ‘tag’ or ‘label’ or ‘describe’ those things as a service to other users. Then we could choose whose language about the world we ‘subscribe’ to, and anyone could attach any digital payload to any real thing/person/place without anyone else’s help or permission.
> My problem with this particular approach is that it requires us all to agree on what things are called
Walk around your home and notice [almost] everything already has a name for it as it comes from some store (unique handmade items are an outlier). Let's just label things we all agree they are called (or give them multiple labels).
I think you overestimate the degree to which your choice of words overlaps with those of others, especially when you try to be more specific than a generic noun by applying modifiers or less-used nouns. What you call a daybed I might call a couch or a sofa or a lounge chair or a chaise or a hassock or a flat couch or a downstairs bed or a family bed … and all this presumes I speak the same flavor of English you do, or that whoever made the data set is familiar with the type of furniture I live around, or how it’s used. The examples are literally limitless, before you even get into the truly subjective things - not just names but opinions about things - that people might want to use to connect digital things to real things. Stuff like ‘larger than average’, ‘wasteful’, ‘beautiful,’ ‘sacred’, ‘durable’, etc etc etc
Believing there is one map from the real world to the world of symbols is solipsism.
31 comments
[ 2.4 ms ] story [ 72.6 ms ] thread> That understanding would let AR glasses tailor content to you and your individual context, like seamlessly blending a digital overlay with your physical space
Horrifying, if it ever takes off in the way that "Meta" wants it to. Take the modern smartphone disease and amplify it a few orders of magnitude.
> While LLMs rely on vast amounts of training data that typically comes from a range of publicly available text sources on the web, no such repository of information yet exists for physical spaces at the scale needed for training an end-to-end model.
Considering the tremendous scale in question, the "yet" is also terrifying. The only way to collect such a dataset would be if some large fraction of people were wearing always-on, network connected cameras. Meta clearly sees this future and wants it; the kind of future where everyone wears a Meta headset, all the time, and has their experience of reality mediated through it - optimized to maximize shareholder value of course, under the watchful eye of The Algorithm. A panopticon AI shoggoth.
Meta is an AI paperclip maximizer, and it is getting uncomfortably close to the "RELEASE THE HYPNODRONES" stage.
That made my day. I do agree, it is creepy but its what theyre pushing.
As egregious as this is, it still doesn't strike me as nakedly insulting as Google's refusal to let you turn on the (entirely passive!) GPS receiver on your phone without uploading your precise location to their servers, for similar data collection purposes.
I'd love to see this sort of thing be made illegal, but the only regulatory domain with an Overton window even close to that is the EU, and it's such a minefield of potential gray areas that I doubt it would get off the ground even there.
You must have missed the part where they created a synthetic dataset of 100000 environments, trained the network on that, and released it to the public?
Not saying I’m not worried about their plans too, but that is clearly addressed in the text.
* How uncomfortable the headsets are to wear for long periods if time
* How useless they are (what are they solution for?)
* How ridiculous they look
* How expensive they are
* How expensive it is to produce content for
After they solve these issues then I will worry about the dystopia we are in. VR headsets may look cool and futuristic in the movies but in reality they are just screens you wear on your head and that's it.
And they won't be useless! That's the worst part. They will provide compelling value, like smartphones, and as such will be impossible to boycott. At first the value will be simple, obvious utility, like route overlays. Later it will be value within the system, much like how the main utility of smartphones now is to run the mandatory apps to participate in aspects of society that previously did not require a smartphone. And of course they will be addictive.
Here is a short sci-fi film depicting what I fear: https://www.youtube.com/watch?v=YJg02ivYzSs
Now as far as making them more useful I suppose *if* they can solve that first problem plus make them match the form of IMAX glasses (preferably smaller). Then yes they could be useful (providing there's good enough applications).
But don't cause the user to get sick feature is not 'a nice to have' you can't expect it to spread like the smartphone if people can't use it for long periods of time.
Will Meta solve that problem? Doubtful. Will they provide good enough applications to make AR worth using once they do? I wouldn't bet on it.
Just because a company has an incentive for something to be successful doesn't mean they can pull off. Just look at metaverse.
Wait, Snowcrash was horror for you? I thought the whole point of post-cyberpunk was to not assume all tech would be dystopian.
With a system like the one in the video you walk through the old place, and walk through the new place, and output for example 2 Sweet Home 3d plan's.
No need to have 3d glasses for that.
The semantic understanding of the objects around you (without upload to a server).
A detailed set of actual meshes of the objects in your living and working spaces.
So much can be built on this key building block, to actually help people with their daily lives, low vision people are only one group that will see massive benefits.
I also think this technique could and should be extended generally into computer vision applications, understanding semantic meaning, volumes and masses via the often highly structured environments around us.
Starting from a principled position of meaning, volumes and meshes and then training to assign gaussian splats to those meshes seems like something clever people could solve a lot of problems with.
> without upload to a server
Doesn't matter. If the app has to render them then it can still send them to the server. If the app does not render directly it can still infer them by doing collision testing against them. If it can't even check for collisions then there isn't much point in having the data in the first place.
It gets worse. The more you want AR/MR to understand the scene the more fine grained data you need. The camera sees things like: the underwear you left on the floor, the medicines/pills on your table, the posters on your wall, the bank account browser tab still on your computer screen, the adult toys you forgot to put away, your undressed body, your dressed body, your political baseball cap or t-shirt, your naked roommate/spouse/kids who just walked into the room.
I don't see how this can be solved in a trust-less way. In other words, you'll have to trust each and every individual AR/MR app you use, that they aren't spying on you. The platform/device/OS itself can not prevent this and still allow the majority of the AR/MR uses that people imagine.
You can't be probusiness and pro-privacy at the same time though, you need to limit businesses in order to protect the people.
While this approach doesn’t address privacy completely, it avoids certain elements of our life that are considered really private.
I think half of the point of using the simulated dataset to train the model was to safeguard the above mentioned kind of privacy (with the other half being lack of real-world dataset).
But you would defo have to trust the OS, and in this case Meta…
I sometimes feel like this may not work without language helping it along
"where are my keys?"
It should be able to respond -
"Here on the dining room table" with an astar path drawn to them in the headset.
It's a stream of tokens that you can almost just reverse search on to find the last instance of "keys" in it (thats a simplification but...).
My feeling is the better approach is to train a generalized segmentation model, derive multi-parameter representations of those segmented ‘things’, derive embeddings from those things for compression purposes, then allow users to ‘tag’ or ‘label’ or ‘describe’ those things as a service to other users. Then we could choose whose language about the world we ‘subscribe’ to, and anyone could attach any digital payload to any real thing/person/place without anyone else’s help or permission.
Walk around your home and notice [almost] everything already has a name for it as it comes from some store (unique handmade items are an outlier). Let's just label things we all agree they are called (or give them multiple labels).
Believing there is one map from the real world to the world of symbols is solipsism.