> we aim for a computing system that is fully visible and understandable top-to-bottom
I mean even for something that is in theory fully understandable like the linux kernel it is not feasible to actually read the source before using it.
To me this really makes no sense. Even for traditional programming we only have so powerful systems because we use a layered approach. You can look into these layers and understand them but it is totally out of scope for a single human being.
I'm sympathetic, but I do think Realtalk could be improved with some simple object recognition and LLMing.
One of the challenges I found when I played with RealTalk is interoperability. The aim is to use the "spacial layer" to bootstrap people's intuitions on how programs should work, and interact with the world. It's really cool when this works. But key intuitions about how things interact when combined with each other, only work if the objects have been programmed to be compatible. A balloon wants to "pop if it comes into contact with anything sharp". A cactus wants to say "I am sharp". But if someone else has programmed a needle card to say "I am pointy", then it won't interact with the balloon in a satisfying way. Or, to use one of Dynamicland's favorite examples: say I have an interactive chart which shows populations of different countries when I place the "Mexico card" into the filter spot. What do you think should happen if I put a card showing the Mexican flag in that same spot, or some other card which just says the string "Mexico" on it? Wouldn't it be better if their interaction "just works"?
Visual LLMs can aid with this. Even a thin layer which can assign tags or answer binary questions about objects could be used to make programs massively more interoperable.
I've never experienced dynamicaland in person (only seen videos). However, one concern I have about it's demos so far is that they use a projector. So you need a room dark enough to for the projected light and you need to keep your heads, hands, and body out of the way of it.
I've made a lot of progress recently working on my own homebrew version, running it in the browser in order to share it with people. Planning to take some time soon to take another stab at the real (physical) thing.
I'm only just now reading about Dynamicland for the first time, so maybe I'm not understanding something obvious. The text description is not very helpful, as far as I can tell from pictures it's a place where you can move around physical objects and papers to do computer programming type stuff?
Under visibility they say:
>To empower people to understand and have full agency over the systems they are involved in, we aim for a computing system that is fully visible and understandable top-to-bottom — as simple, transparent, trustable, and non-magical as possible
But the programming behind the projector-camera system feels like it would be pretty impenetrable to the average person, right? What is so different about AI?
I’m an artist who’ve always struggled to learn how to code. I can pick up on computer science concepts, but when I try to sit down and write actual code my brain just pretends it doesn’t exist.
Over like 20 years, despite numerous attempts I could never get past few beginner exercises. I viscerally can’t stand the headspace that coding puts me in.
Last night I managed to build a custom CDN to deliver cool fonts to my site a la Google fonts, create a gorgeous site with custom code injected CSS and Java (while grokking most of it), and best part … it was FUN! I have never remotely done anything like that in my entire life, and with ChatGPT’s help I managed to it in like 3 hours. It’s bonkers.
AI is truly what you make of it, and I think it’s an incredible tool that allows you to learn things in a way that fits how your brain works.
I think schools should have curriculum that teaches people how to use AI effectively. It’s truly a force multiplier for creativity.
12 comments
[ 2.3 ms ] story [ 34.9 ms ] threadI mean even for something that is in theory fully understandable like the linux kernel it is not feasible to actually read the source before using it.
To me this really makes no sense. Even for traditional programming we only have so powerful systems because we use a layered approach. You can look into these layers and understand them but it is totally out of scope for a single human being.
One of the challenges I found when I played with RealTalk is interoperability. The aim is to use the "spacial layer" to bootstrap people's intuitions on how programs should work, and interact with the world. It's really cool when this works. But key intuitions about how things interact when combined with each other, only work if the objects have been programmed to be compatible. A balloon wants to "pop if it comes into contact with anything sharp". A cactus wants to say "I am sharp". But if someone else has programmed a needle card to say "I am pointy", then it won't interact with the balloon in a satisfying way. Or, to use one of Dynamicland's favorite examples: say I have an interactive chart which shows populations of different countries when I place the "Mexico card" into the filter spot. What do you think should happen if I put a card showing the Mexican flag in that same spot, or some other card which just says the string "Mexico" on it? Wouldn't it be better if their interaction "just works"?
Visual LLMs can aid with this. Even a thin layer which can assign tags or answer binary questions about objects could be used to make programs massively more interoperable.
It's still at the cool demo level, though. How do you scale this thing?
[1] https://www.youtube.com/watch?v=7wa3nm0qcfM
I've made a lot of progress recently working on my own homebrew version, running it in the browser in order to share it with people. Planning to take some time soon to take another stab at the real (physical) thing.
Progress so far: https://deosjr.github.io/dynamicland/
Under visibility they say:
>To empower people to understand and have full agency over the systems they are involved in, we aim for a computing system that is fully visible and understandable top-to-bottom — as simple, transparent, trustable, and non-magical as possible
But the programming behind the projector-camera system feels like it would be pretty impenetrable to the average person, right? What is so different about AI?
Like, you can write a script that talks to functionality that may or may not exist yet.
Programming by moving pieces of paper around deservedly gets attention, but there's a lot more to it.
I’m an artist who’ve always struggled to learn how to code. I can pick up on computer science concepts, but when I try to sit down and write actual code my brain just pretends it doesn’t exist.
Over like 20 years, despite numerous attempts I could never get past few beginner exercises. I viscerally can’t stand the headspace that coding puts me in.
Last night I managed to build a custom CDN to deliver cool fonts to my site a la Google fonts, create a gorgeous site with custom code injected CSS and Java (while grokking most of it), and best part … it was FUN! I have never remotely done anything like that in my entire life, and with ChatGPT’s help I managed to it in like 3 hours. It’s bonkers.
AI is truly what you make of it, and I think it’s an incredible tool that allows you to learn things in a way that fits how your brain works.
I think schools should have curriculum that teaches people how to use AI effectively. It’s truly a force multiplier for creativity.
Computers haven’t felt this fun for a long time.