An interesting concept. I'm (probably) one of the few people that thought the Humane Pin had an interesting idea at its core, it was just executed poorly at the wrong time. A stationary version that leans into ambience solves a lot of the interaction and engineering issues.
I also like that it addresses my gripe with XR interfaces; you aren't actually sharing a digital object in space and it looks weird to everyone else.
Nice. This reminds me how I felt watching the first pre-iPhone multitouch demos.
One small quality-of-life improvement that could be added: have the software timestamp when you say "here" and then (maybe using a short low-FPS video buffer) analyze the corresponding frame of video. This would eliminate the pause when you're pointing at something, waiting for the computer to respond.
Another very small usability improvement might be a "follow where I'm pointing" vision algorithm. This way, you don't have to stretch your arm out so far.
Early multitouch in drafting table and wall-mount form factors (ie similar to those shown) suffered from arm fatigue after extended use, aka Gorilla Arm syndrome.
Perhaps, using vision, a Lightbulb Computer could greatly reduce that fatigue, even in short sessions? That would be revolutionary all by itself...
I think the mission statement here is on to something. Helmets, obtrusive physical appendages etc are unappealing to me. The idea of using extant space or surfaces however, is extremely appealing. And while it might suggest lower quality today, as with most things, it would almost certainly improve over iterations and time.
Holographic computing, I think, will be the future. At least for interfaces.
Author here; I was definitely inspired by my visits to Dynamicland many years ago, which Folk Computer is also inspired by.
One big difference in direction is that Dynamicland + Folk Computer use projector+computer vision as substrate to try and reinvent/rethink the interactive/dynamic medium, and what end-user programming could be (which is very cool).
In that way I am much less ambitious; this exploration is more about how there is an entire class of existing interactions (the ones showed in the piece) that we perform today on screens, that could be much more interesting + collaborative if overlaid in the real world. And I am also terribly uninspired by the eyewear-based computing vision all major companies have pushed lately.
I totally agree. It feels like we’ve crossed a line where tech is overtly hostile to users because everyone is chasing recurring revenue and data mining is the only viable option
Nothing, ideas are free! The value is in the execution. This being a prototype, it's much closer to idea than execution, given its stated bulk and latency.
I made this, fun to see my project on the front page!
The main question I tend to get from hacker-types is "what are the tech specs?". The demos run on a Mac with custom software I built for projection mapping/rendering/etc. (the hand tracking uses Apple's built-in framework, which is very good), a small consumer 4k laser projector, and a basic webcam for vision.
But really, this is primarily a research/design prototype - I think it's much more important to figure out what the user experience should ideally feel like, if the interactions are even plausible & desirable in the first place; and then work backwards to the technology.
Is a basic webcam really able to read text at that angle for the “what is this word” example?
I wasn’t sure if any of this was real and that example looked extremely fake to me because I had a hard time believing the device being able to read text from that distance.
I met @heliographe a few years ago in a HN Tokyo (Slack) meeting. Happy to see you are still designing awesome things after your time at Apple. I look forward to seeing how your project(s) develop. PS: as soon as I saw the kitchen I knew this was Japan, and then I saw the hair and though —“hmm, I feel like I have seen this person before”— and it was you!!
Right. Tiny projectors have been done before.
Samsung had a smartphone with a projector back in 2014, but nobody wanted it.
Some startup called Moviphone has announced a new one for 2027.[1]
The use cases have not been compelling and fighting ambient lighting with a small projector is always tough.
Possible use case: projecting menus, ads, and bills on restaurant tables. With a camera and enough automated layout to work around what's on the table.
It might sell to the people who buy refrigerators with displays on the door.
Could be useful as a cooking training aid, like cooking videos but present where you need them.
The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
It's a good thing that most inventors don't fall for the "something vaguely like this has been tried with inferior technology before, and it sucked" line of reasoning, or our world would look very different :)
Well, if you're a inventor, you look at the prior art to see if it's been done before. I hold six issued patents, two of which were very profitable. I had one patent rejected because I missed relevant prior art from 1948.
Then again, I feel like I know industrial designers who play with building on ideas that came before, reexamining whether those ideas might have a fresh form and a place in the world today.
Who iterate and hone until they have something worth existing in today’s world—often doing do more publicly than corporations’ R&D departments—then let the patent technicalities be the well-paid lawyers’ problem…
Legal counsel at multiple companies have always advised me to not do a patent search before writing up an invention. I guess often the lawyers do the search, and the inventors don’t.
That said, this project isn’t well summarized as merely “tiny projectors”. Many patents have been granted for systems that include parts that are also patented. And many products have good sales without being patented; patents are not the end-all of invention. So I feel like you’ve been much too quick to dismiss.
> The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
gonna hard disagree! you aren't going to have 360° projector in there with wasted light going nowhere. you're going to have 1-4 steerable projectors. the "facade" as a globe bulb means you can place it at any direction you like depending on where the cord needs to go. for ceiling as well you don't want it to have a fixed throw direction. bulb makes perfect sense.
Imagine downvoting John effing Nagle. However my humble opinion is that sometimes revisiting, if not reinventing, the wheel is beneficial in light of accrued epistemic benefits (or for the fun of tinkering with a cutesy creation) . Not for patenting, of course. Besides, the bulb form-factor is not too different from cctv cam ceiling installations.
This is an awesome concept. How do you see this working across several rooms (if you've thought this far)? This would be great for when I have ideas in the shower.
The appeal of the lightbulb form factor is that you can mount it in a portable base to move it around the house; or in a fixed socket for broader room coverage.
It's not mentioned in the piece, but for the latter I don't see any technical obstacle to having several of these in different rooms in the house, or a couple in the same room to augment coverage.
A lot of people have Alexa or Google devices in each of their rooms, this seems like a very logical step. Perhaps 2 generations from what is being currently released based on how things go. Maybe sooner!
Could you do beamforming with this, or would that get too expensive/awkward? At the very least, having more of these guys in the house (if you can get the tech cheap, ofc) would lead to better coverage potentially?
The main issue is probably gonna be that the display sucks compared to actual displays :( but it looks cool though! and the whole VLM thing makes a lot of sense
Its sad that search engines have become useless. There was another UX researcher who did something very much like this and then later got recruited to Google. As I recall he was using the cameras in Wii Nunchucks because they could do SLAM easily. And yet I cannot find his research, sigh.
This is the most impressive tech demo I have seen in years. Spatial computing just became real. I hope, I can buy something like this in a few years. Thanks you for inventing and sharing.
I don't know about these things. To me they just turn everything into a screen.
Sure, it's a little more hands free, and projecting around definitly is nicer than screens. But projecting has a lot of problems too. Shadows, projection surface, ambiant light, heat are difficult to handle, but to me the worst is that it's not persistant. You can't just project 24/7 your family planner on the wall.
I'd really rather have something that prints, or better draws.
You want a recipe => print it, don't beam it.
You want a map of kyoto => tiny robot walks around the page and draws it for you. And you can draw over it if you want.
And it doesn't have to be exclusive, you can do both beam and print. But I think protable printing / drawing has a lot of unexplored potential.
The future of computing is certainly multimodal. What I see with beaming is being both hands-free and prep-free further enhances the spontaneity of life and helps fill in small gaps.
Printing otoh requires hands and prep. That's greater effort. You wouldn't print a recipe if you were just a little uncertain about step 3, you would ask for some kind of just in time modality instead. Voice might be fine for short info but sometimes you'd prefer to have it up and there's beam.
yeah, I don't disagree, though you could think that a robot with a heat-paper printer or pen could go a long way.
But more importantly I ultimately think that one of the unstated goals is to reduce screen time. And setups like these are only one voice command away from beaming youtube videos all around your house.
I don't need the family planner to be on the wall 24/7 though, it can be projected only when I'm at the room. Maybe even only when I'm close to the wall.
If I printed each recipe I followed... gush that would be a lot of paper, and also I bet that paper would get messy very quickly.
For a map, losing the ability to zoom, mark points etc seems like a big loss. Sure you can draw on it (how do you erase your mistakes?), but if everything you draw just ends up lost when you discard the paper, that's not so ideal.
Quite nice. But this imbues the house with a constant presence of the machine. A jinn which is always ready to grant your wish, but never leaves you alone, ever watching and listening. The fact that the jinn takes orders spoken in normal human language amplifies the effect quite a bit. I would have rather not have to feel such presence in my bathroom. Or bedroom. I would still be glad to be able to operate "smart" switches, and even find items. But I would rather explicitly unlock my phone, start an app, and scan the bookshelf, until a book is highlighted.
OTOH this would be a great thing for non-private spaces, that are watched by security cameras 24/7 anyway. A school, a library, a clinic, an office, a workshop, a warehouse, a parking garage, maybe even a museum.
Of course, trust and security are going to be major considerations. Maybe not for everyone though: people voluntarily enable various voice assistants, and even pay to order Amazon Echo.
For me, a lot would depend on who controls the information gathered by the machine. If it’s all purely local and under my control, I doubt I’d even feel it as a “presence” - I already have that today with various cameras in my house, and it doesn’t bother me because I know the data never leaves my local system.
If it’s all being sent to some big tech co, then it becomes extremely different.
Sadly the reality of consumer product development under current economic and legal frameworks is that we’re much more likely to get the latter than the former.
Sadly if it ever gets manufactured it will use cloud and become just a better version of Flock. Even cat in the photo looks with obvious displeasure realizing this.
What does the lightbulb have to do with the projection? Obviously the lightbulb is not the camera... nor the projector (nothing clear is getting through a frosted white shell). Is the lightbulb just a prop and imagined form-factor?
I feel like the demos are disingenuous because they suggest that the lightbulb is responsible for the entire thing.
I think screens are time tested and will have some enduring use for now, but I could see people liking this; I've wanted to try a projector screen out but there was a whole rabbit hole of choices that didn't seem to improve upon screens for me for now (and this takes things in a whole other direction with projection)
I've got something similar to this using cheap ONVIF security cameras and kasa smart bulbs. I have an nvidia card process the capture and for the smart speaker I use just a microphone and speaker hooked up to a normal computer. The architecture is spread across many machines and it's nowhere close to this realtime. Also the video capture is fairly lousy because the lighting and resolution and coverage isn't great. The cameras can change the lighting to try to get a better shot. (well it's qwen with toolcalls)
I also have (I think, haven't counted recently) 12 cameras that are all ballhead so they move around and look for me when I say something to the smart speaker.
But I didn't think of the projector ... there's ~$25 ones ... might be worth buying a few.
Getting this level of response, feedback and accuracy, that's the real magic. The latency is garbage. I'm just using cheap OTS hardware though.
Anyone can do this ... it's not that expensive. Smart bulbs are cheap, wifi cams are cheap, omnidirectional usb microphones are cheap and projectors are cheap too.
still researching them ... some of these look like garbage ... i'll come back to this thread in a few days when i find something decent. I don't want high nits or resolution, just want to be able to send pixels and not have it be noisy and get a response in under 500ms.
95 comments
[ 0.27 ms ] story [ 45.9 ms ] threadOne small quality-of-life improvement that could be added: have the software timestamp when you say "here" and then (maybe using a short low-FPS video buffer) analyze the corresponding frame of video. This would eliminate the pause when you're pointing at something, waiting for the computer to respond.
Quite the impressive demo!
Early multitouch in drafting table and wall-mount form factors (ie similar to those shown) suffered from arm fatigue after extended use, aka Gorilla Arm syndrome.
Perhaps, using vision, a Lightbulb Computer could greatly reduce that fatigue, even in short sessions? That would be revolutionary all by itself...
What they didn't say is that the bulb shown in all the images are a prop.
Holographic computing, I think, will be the future. At least for interfaces.
One big difference in direction is that Dynamicland + Folk Computer use projector+computer vision as substrate to try and reinvent/rethink the interactive/dynamic medium, and what end-user programming could be (which is very cool).
In that way I am much less ambitious; this exploration is more about how there is an entire class of existing interactions (the ones showed in the piece) that we perform today on screens, that could be much more interesting + collaborative if overlaid in the real world. And I am also terribly uninspired by the eyewear-based computing vision all major companies have pushed lately.
Youtube videos - https://www.youtube.com/results?search_query=sixth+sense+tec...
Reminds me of the projection bracelet scam that Captain Disillusion debunked years ago https://youtu.be/Cw7g8ixbomU?t=75
The main question I tend to get from hacker-types is "what are the tech specs?". The demos run on a Mac with custom software I built for projection mapping/rendering/etc. (the hand tracking uses Apple's built-in framework, which is very good), a small consumer 4k laser projector, and a basic webcam for vision.
But really, this is primarily a research/design prototype - I think it's much more important to figure out what the user experience should ideally feel like, if the interactions are even plausible & desirable in the first place; and then work backwards to the technology.
I wasn’t sure if any of this was real and that example looked extremely fake to me because I had a hard time believing the device being able to read text from that distance.
Possible use case: projecting menus, ads, and bills on restaurant tables. With a camera and enough automated layout to work around what's on the table.
It might sell to the people who buy refrigerators with displays on the door. Could be useful as a cooking training aid, like cooking videos but present where you need them.
The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
[1] https://moviphones.com/
Who iterate and hone until they have something worth existing in today’s world—often doing do more publicly than corporations’ R&D departments—then let the patent technicalities be the well-paid lawyers’ problem…
Nothing truly new under the sun
That said, this project isn’t well summarized as merely “tiny projectors”. Many patents have been granted for systems that include parts that are also patented. And many products have good sales without being patented; patents are not the end-all of invention. So I feel like you’ve been much too quick to dismiss.
gonna hard disagree! you aren't going to have 360° projector in there with wasted light going nowhere. you're going to have 1-4 steerable projectors. the "facade" as a globe bulb means you can place it at any direction you like depending on where the cord needs to go. for ceiling as well you don't want it to have a fixed throw direction. bulb makes perfect sense.
You’re not doing anyone any favors by replacing genuine feedback with sycophancy. That’s how you end up with Nobel disease and similar afflictions.
It's not mentioned in the piece, but for the latter I don't see any technical obstacle to having several of these in different rooms in the house, or a couple in the same room to augment coverage.
How many watts does the projector use for how many lumens? I would be curious if these numbers plus size would be a fundamental limit to the tech.
The main issue is probably gonna be that the display sucks compared to actual displays :( but it looks cool though! and the whole VLM thing makes a lot of sense
http://johnnylee.net/projects/wii/
https://www.ideo.com/case-study/designing-the-future-kitchen
Here's hoping the induction cooling/cooking becomes prevalent too!
Sure, it's a little more hands free, and projecting around definitly is nicer than screens. But projecting has a lot of problems too. Shadows, projection surface, ambiant light, heat are difficult to handle, but to me the worst is that it's not persistant. You can't just project 24/7 your family planner on the wall.
I'd really rather have something that prints, or better draws.
You want a recipe => print it, don't beam it.
You want a map of kyoto => tiny robot walks around the page and draws it for you. And you can draw over it if you want.
And it doesn't have to be exclusive, you can do both beam and print. But I think protable printing / drawing has a lot of unexplored potential.
Printing otoh requires hands and prep. That's greater effort. You wouldn't print a recipe if you were just a little uncertain about step 3, you would ask for some kind of just in time modality instead. Voice might be fine for short info but sometimes you'd prefer to have it up and there's beam.
But more importantly I ultimately think that one of the unstated goals is to reduce screen time. And setups like these are only one voice command away from beaming youtube videos all around your house.
If I printed each recipe I followed... gush that would be a lot of paper, and also I bet that paper would get messy very quickly.
For a map, losing the ability to zoom, mark points etc seems like a big loss. Sure you can draw on it (how do you erase your mistakes?), but if everything you draw just ends up lost when you discard the paper, that's not so ideal.
I guess what disturbs me is that I’d rather have one screen in my pocket than tens of screens in my house.
OTOH this would be a great thing for non-private spaces, that are watched by security cameras 24/7 anyway. A school, a library, a clinic, an office, a workshop, a warehouse, a parking garage, maybe even a museum.
Of course, trust and security are going to be major considerations. Maybe not for everyone though: people voluntarily enable various voice assistants, and even pay to order Amazon Echo.
If it’s all being sent to some big tech co, then it becomes extremely different.
Sadly the reality of consumer product development under current economic and legal frameworks is that we’re much more likely to get the latter than the former.
I feel like the demos are disingenuous because they suggest that the lightbulb is responsible for the entire thing.
However, the light bulb concept is limited to indoor usecases - which is not bad - that's where people live most of the time.
The broader goal should be to get rid of dependency on screens, technologies and information. That's when AI can stop enslaving us with HITL etc.
I think screens are time tested and will have some enduring use for now, but I could see people liking this; I've wanted to try a projector screen out but there was a whole rabbit hole of choices that didn't seem to improve upon screens for me for now (and this takes things in a whole other direction with projection)
I also have (I think, haven't counted recently) 12 cameras that are all ballhead so they move around and look for me when I say something to the smart speaker.
But I didn't think of the projector ... there's ~$25 ones ... might be worth buying a few.
Getting this level of response, feedback and accuracy, that's the real magic. The latency is garbage. I'm just using cheap OTS hardware though.
Anyone can do this ... it's not that expensive. Smart bulbs are cheap, wifi cams are cheap, omnidirectional usb microphones are cheap and projectors are cheap too.