Ask HN: Has anyone fully attempted Bret Victor's vision?
I'm much more curious about a programming paradigm that no longer uses text to communicate with computers but instead just directly manipulating data, receiving past, present, and future feedback of how it would change given your manipulations. Or to put it a different way "What if" feedback. "If you did this, the data would change in this way" is visualized across many different dimensions, allowing you to 'feel your way' through feedback where you wish to go.
In other words, you give your computer your input data, and you modify dimensions which allow you to specify what you want the program to do.
To be clear, I'm not searching for specialized interpretations of this "Oh someone did this with typography" or "Oh someone did this with a game" but rather some more generalizable form like "Someone tried to replace Python with an idea like this"
I suppose the nearest thing I can think of is manually modifying the parameters of a neural net but that's perhaps too cumbersome because there are so many. Perhaps if you can put an autoencoder on top of that, and reduce the parameters down to a smaller "meta" set of parameters that you can manipulate which manipulate the population of parameters in the larger neural net?
I'm just really curious if there have been instantiations along these lines (as opposed to code live-running with results on the sides).
I realize this is all quite difficult, may even seem 'impossible' to have some sort of generalizable system that does this for all sorts of programs. I've heard people say it can't be done, and code is the ideal format. I hold that in abeyance, I don't really know, but intrigued to discover those who have a counter perspective to that and have attempted to build something.
Also really curious if you know other similar people to Bret Victor I should check out!
160 comments
[ 0.23 ms ] story [ 202 ms ] threadreminds me of Bush's memex "pathways"
Reactivity has certainly become more popular, and is a standard part of web development now. And ipywidgets are an example of creating manipulatable abstractions in data science.
tbf, he measures the timescale of the Dynamicland project in decades, and marks 2022 as the first time "Dynamicland meets the world" [1]. Seems like he's matching the pace he set for himself, so you really can't say he hasn't had popular success if he hasn't even tried to popularize it yet.
Also tbf, any concept involving "a place where a bunch of people congregate and touch things" has suffered since the pandemic, so I think we can still maintain a "wait-and-see" stance instead of declaring the ideas dead.
[1] https://dynamicland.org
https://lively-web.org
It's trivial to explore visually the space of all possible outputs for a program of length 1 instruction. But how about a million? The output is just gonna be useless noise 99.99999%.
What even is a dimension of a computer program? How many divisions it performs? How many gotos there are?
The entire industry’s understanding of the power of the relational model has been destroyed by SQL, which is the largest foot gun of so many in our industry.
In fact, much more of most business applications (hell, just about all applications) can and would profitably be expressable in First Order Logic. The resulting programs would be simpler, easier to modify, easier to get correct, and other things besides.
Much of what Bret Victor has shown could be expressed in First Order Logic.
Would you still want Turing Complete traditional programming languages? Of course. But both programmers and non-programmers would benefit from being able to express more of programs using the sorts of declarative idioms that Bret Victor has shown and that the relational model provides.
If we had a meta-language like Lisp or ML that had very good capabilities to quickly develop DSLs with very clean restricted semantics for every particular component of a big system, and tooling to automate proofs for said restricted DSLs, software would be more robust and easy to develop.
Alan Kay was pursuing parts of this vision at Viewpoints Research Institute. Racket is also very focused on DSLs. Any others?
[1] Or some other made up, but still very high number.
Prolog is actually Turing complete.
When people say stuff like this, I think of traditional post&beam construction guys complaining about the way that stud framing has destroyed the industry's understanding of the power of "real construction". Whether or not that's true, stud framing has been used to build homes for on the order of a million times more people than post&beam. Perhaps it is true that stud framing obscures a deeper understanding of the nature of wood, joinery, loads and so forth, but sometimes the point is just to build a shit ton of houses, cheaply, efficiently, effectively, knowing that changes in requirements will make most houses largely redundant before (most) stud framed houses are gone.
SQL: shallow, weak, wrong, incomplete, misleading, and how the world gets built.
I was one of the backers of https://www.kickstarter.com/projects/ibdknox/light-table/des... and have wanted to see progress in this domain for a long time.
One thing I would add to the conversation is that one of the most potent ways to move this discussion forward is to create technical demonstrations of how this sort of interface could work, presented as video. It's completely unimportant if the functionality is actually working, so long as you disclose this up front.
The goal is to give people with less imagination and hopefully more technical acumen an opportunity to roll up their sleeves and maybe work on making it real.
whats funny is Bret’s message wasnt actually “you should go make direct manipulation ui’s”. it was “you should have design principles” and direct manipulation happened to be his baby (to the point where he went off to do dynamicland). i have heard he feels most people misunderstand his talk for the superficial wow moments.
I don't think there's any way to get away from this abstraction. At it's lowest level, everything is encoded in binary. All abstractions on top of binary are just interpretations of the underlying stream, text being a relatively simple encoding (ASCII table or UTF8's multi-byte structure). Structure data is similar, just multiple pieces packed into one contiguous space. You will always have to build on top of this fundamental, there is no simpler
That being said, I quite like:
* Datasette - https://datasette.io/ - I have a feeling ou could connect a lot of these instances and truly make something interesting there
* Lightable IDE - https://www.youtube.com/watch?app=desktop&v=H58-n7uldoU
* I have at least 2 more but I can't find them in my favorites
I'm trying to make my own as well. Hardest thing is giving myself enough time to do it, but I'm currently starting to structure my life around it.
“ In his influential essay No Silver Bullet, Fred Brooks makes the case that software is inherently "invisible and unvisualizable", and points out the universal failure of so-called "visual programming" environments. I don't fault Fred Brooks for his mistake -- the visual programming that he's thinking of indeed has little to offer. But that's because it visualizes the wrong thing.
Traditional visual environments visualize the code. They visualize static structure. But that's not what we need to understand. We need to understand what the code is doing.
Visualize data, not code. Dynamic behavior, not static structure.”
http://worrydream.com/#!/LearnableProgramming
Max, PureData and now a new generation of software modular synthesis applications visualize code-that-spews-data-at-other-code. They are used to build highly modifiable (though not dynamic) structures that behave in ways that are often hard to hold in a human mind. They are widely used, much loved, and insanely powerful. They use visualization to add visual memory to the cognitive toolset in ways that textual code does not.
Of course, such tools are unlikely to ever be used build such tools. You don't implement/bootstrap a visual (audio) data flow language using a visual data flow language.
I do hope someone gets nerd sniped by this challenge. Not me, but someone.
Grasshopper is meant to parametrically generate 3D objects but it was so good at splitting and recombining tables of data that were being continuously updated that I started using it for general purpose text manipulation.
the ENIAC was a dataflow machine before it was a stored program machine btw
I guess the most popular dataflow language in the world would be Verilog? But that's not visual. I don't know much about how it works professionally though.
It's a good distinction to draw, that dataflow is separate from the editor interface. I suppose Excel is a visual dataflow editor as well.
They are successful in artist and music contexts because the graphs tend to be simple pipelines at heart. Having dealt with sufficiently complex grasshopper graphs, I disagree that it's good at arbitrary list processing, certainly compared to ordinary list operators and iterators in code.
My conclusion is that a dataflow environment that does allow for lambdas and proper n-way forking would necessarily have to be an effect system in the FP sense. It's a data flow graph that computes its own continuation and which has no fixed or preset topology. It can rewire itself based on the data flowing through it.
Its taken 60 years of improvements to finally build ChatGPT and Stable Diffusion. If we collectively gave up on AI years ago, we might never know that.
We still don't know if VR will ever get good enough for mass market appeal. Facebook (Meta) is investing big. I hope their investment pays off, but we don't know yet! It might not work. But it might! I appreciate the risk their investors are taking.
And I think the same is true for visual dataflow programming. We have no idea until someone builds it out. And if their implementation is bad, we might still not know. Any implementation might just be a bad design.
Anyway, Brett Victor talks about ideas that go way beyond visual dataflow programming. There's a lot of interesting stuff in there, even if a lot of it might never come to pass.
https://futureofcoding.org/ --> https://futureofcoding.org/catalog/
Edit: This comment is a goldmine: https://news.ycombinator.com/item?id=34485994
——-
There are lots of hobbyists, academics, and even companies inspired by Bret Victor’s talks alone.
I know of at least 2 open source experimental programs that were inspired by specific demos:
https://github.com/laszlokorte/reform-swift
http://recursivedrawing.com/
I know there are more too but I can’t find them right now. You could probably find a lot of good stuff just searching GitHub for “Bret Victor”.
There are lots of people in academia experimenting with programming languages and environments. Try searching for papers that cite Bret Victor as well and I’m sure you’ll find plenty.
For a quick glimpse at the academic world without spending hours looking for papers worth reading, I recommend perusing the Strange Loop Conference YouTube channel. There are some interesting experimental programming languages and IDEs out there.
https://m.youtube.com/watch?v=TqISbaJ7qug
Iterating on this in a modern way remains TODO. As John Henry and his counterpart note in that video, the beauty of those tools was that people who didn’t care about code were able to create interactive experiences.
The whole “everyone learn to code or you’ll be poor” thing of the past 10 years has been a huge and unnecessary distraction.
Back to your question, from my naive understanding, smalltalk seems to be an all in one environment. The Glamorous Toolkit [1] seems to be that environment on steroid. I have no useful experience to share though.
https://gtoolkit.com/
Another thing I considered was that, instead of the lisp, you'd have a math-y language and compose everything together. This has the added benefit that it compiles easily to GPU code. So like a chroma key thing would be a function that chooses one buffer over the other based on pixel color. You the compose that with other functions to create a frame. An APL-like language could be amazing here.
I feel like there's _a lot_ of potential here.
- Jupiter notebooks
- Dev Cards
- Storybook
- Dark Lang
- Excel!
- AfterEffects
- "what if" feedback loops (crudely, "live programming")
- direct manipulation (an old idea but beautifully captured in his projects)
- making logic feel more geometric / concrete
- visualizing and manipulating data, especially over time
- humane interfaces (putting "computing" into the world around us, but without AR)
- etc.
Bret Victor is very much Alan Kay's protege and has unfortunately inherited the curse of people cherry-picking particular ideas and missing the bigger picture.
So as others have pointed out, the only person who may be fully attempting Bret Victor's vision is Bret Victor with Dynamicland. You may also be curious to check out Humane [1] which is a hardware startup founded by ex-Apple people. They're rumored to be shipping a projection-based wearable device this year. This device could potentially be a platform for people to experiment more in the direction of Bret Victor's vision.
[1] http://hu.ma.ne
Maybe it's time we lay some of the blame for us idiots just not getting Alan Kay's ideas on Alan Kay. At this point he only has himself to blame if he's spent 50 years trying and failing to communicate his wonderful ideas.
Not everything ended up being adopted in the form he envisioned or with the semantics he proposed, but there have been a lot of right calls and influential designs in his 50+ year career.
Same for Berners-Lee. Sure, he remains influential over the web's incremental progress on W3C, but anything more visionary seems to be a miss: XHTML, semantic web, the Solid project...
Engelbart invented the mouse and the entire idea of pointing at things on the screen to interact with them. The results of that interaction is now what you call "the web" (hypertext).
Though there are a few examples of moving in the right (?) direction, for example styled components (for React, which moves CSS into JS/TS code, there's a VScode plugin for it, and thanks to the tagged templates they can be validated).
You might find this list of Kay's talks interesting:
https://tinlizzie.org/IA/index.php/Talks_by_Alan_Kay
Things got really interesting in 1996 when Alan and Dan released Squeak Smalltalk with Etoys as free and open source with this almost metacircular virtual machine.
In 2008 we had progressed to designing SiliconSqueak, a Smalltalk Wafer Scale Integration, a 10.000 manycore microprocessor with the late bound message passing Squeak Smalltalk as its IDE, operating system and the RoarVM virtual machine with adaptive compilation. We are still working on that, it costs $100K for the mask set that you send to the TSMC chip fab and you get back a 180nm wafer with the 10 billion transistor supercomputer for $600 a piece. Getting funding for mask sets at smaller nodes like $3 million for 28nm or the most advanced 3nm node what costs over 50 million for a million cores is a life's work.
We have not been directly working for Alan Kay, Dan Ingalls or David Ungar but we exchange emails, write scientific papers [2], give lectures [1] and meet in online video sessions [1] with the vibrant Smalltalk community.
When these researchers release the source code like the STEPS project, RoarVM or the Lively Kernel we try to port it to our SiliconSqueak supercomputer prototypes and of course we develop our own Smalltalk of the Future, parallel adaptive compilers, virtual machines and hardware X86 emulators.
So to answer your first question, yes, there are hundreds of lectures and talks on Youtube and we share all this work with the world. Bret Victor's, Dans or Alans lectures are just a small part of that.
The hard part of our research is getting $100K funding together for the 10.000 core supercomputer, a $2000 wafer scale integration (WSI) computer is a little to big an amount for a crowdfunding project.
So I still hope YCombinator will fund me, but they have this silly 'no single founder' restriction. You seem to be a researcher at ETH Zurich, why don't you join me as cofounder?
We make a 3 cent Smalltalk microcontroller (an ALTO on a chip) and a $1 version with 4 MB and gigabit ethernet, with Smalltalk, Etoys and Scratch built in you get a superior Raspberry Pi/Arduino successor that 5 year old children can program because Smalltalk and Etoys where designed with children in mind.
Our Morphle WSI would be a great desktop supercomputer but the real advance would be the $20.000 (retail price) costing 3nm wafer scale integration. More than 40 trillion transistors, a runtime reconfigurable amount of 1 million cores and the full IDE, GUI and OS in 10.000 lines of Smalltalk language, IDE, GUI and OS at exaflops per second. Way more advanced than CUDA on a GPU. I gave a 2 hour talk on that:
[1] https://vimeo.com/731037615
[2] https://scholar.google.nl/citations?user=mWS92YsAAAAJ&hl=en&...
https://scholar.google.nl/citations?hl=en&user=6wa49gkAAAAJ
[3] https://web.archive.org/web/20140501222143/http://www.morphl...
[4] https://youtu.be/id1WShzzMCQ?t=519
https://www.youtube.com/watch?v=jIlzXEaOH1I
how are you going to cool the wafer? what's the TDP? :o
100K sounds very doable for crowdfunding - or maybe you need to find just one eccentric multi millionaire.
Given 45 trillion transistors (45x10e12) times 3 femtojoule (3x10e-15) to switch each transistor at 1 Ghz (10e9) you get 1.000.000 joules/sec = 1 megawatt. These are ball-park numbers, back of the envelope calculations. In reality I make full physics simulations and electrical SPICE simulations of the entire wafer on a supercomputer aka on the wafer scale integration FPGA prototypes and the wafer supercomputer itself.
The EDA (Electronic Design Automation) software tools we write ourselves in Smalltalk and Ometa, and these also need our own supercomputer to run on. Of course the feedback loops are Bret Victor style visualizers [3][2]. Apple Silicon or this small company demonstrate that only with custom EDA tools can you design ultra-low power transistors to prevent our wafer to melt.
The FPGA prototype is a few thousand Cyclone 10 or Polarfire FPGA's with a few terabytes/sec memory bandwidth or a cluster of Mac Studio Ultra's networked together in a Slim Fly network that can double as a neighbourhood solar smart grid [5]. You need a dinosaur egg to design a dinosaur, or is is it the other way around? [6]
A TDP (Thermal Design Power) of 1 megawatt from a 450 mm disk is huge, it will melt the silicon wafer. But then not all transistors are switching all the time and we have the cooling effect of the liquid.
We must power the wafer from a small distance inductively or capacitively, best with AC. So we need AC-DC inverters on the wafer, self-test circuits to make sure we find defects from dust and contamination and isolate those parts and reroute the network on the wafer.
[1]https://vimeo.com/731037615 at 21 minutes
[2] https://youtu.be/V9xCa4RNfCM?t=86
[3] https://youtu.be/oUaOucZRlmE?t=313
[4] https://bit-tech.net/news/tech/cpus/micro-magic-64-bit-risc-...
[5] https://www.researchgate.net/profile/Merik-Voswinkel/publica...
[6] Frightening Ambitious Startup Ideas (dinosaur egg)
https://youtu.be/R9ITLdmfdLI?t=360
http://www.paulgraham.com/ambitious.html
Solar electricity prices can be below 1 cent per kWh [1]. I generate 20kW solar in my own garden and store some in my own custom battery system with our own charger chips. The prototype supercomputer warms my room. I hope to move to my own design off-grid tiny house in a nature reserve in Spain or Arizona to get 2.5 times more energy yield and even lower cost of living and cheaper 10 Gbps internet.
If you only run the computation during daylight and then move the computation with the sun to two wafers in two other timezones when that location has sunlight you keep below 1 cent per kWh. Some supercomputers do this already. In contrast, running 24/7 from batteries raises the cost to almost 2 cents per kWh, still far below bulk electricity prices in datacenters. Batteries turn out to be more expensive than having three solar supercomputers in three time zones. You learn from all this that energy costs dominate the cost of compute hardware, even our cheapest transistors cost. Hence our ultra-low transistors, not just to prevent melting of the wafer but mostly to make cheaper compute (for the cloud).
The wafer scale integration at 180nm costs around $600 per wafer to manufacture, only once it cost $100K to make the mask set, amortised over the $500 wafers you mass produce, this is how you get to $600 for 10000 cores at >1 Ghz.
These $600 wafer supercomputers use less than 100-700 watt with normal use, because not all transistors switch all the time at 1 Ghz. They are asynchronous ultra low power transistors, no global clock wasting 60% of your energy and transistors and you don't touch all SRAM locations all of the time. The larger 3nm wafer scale integrations won't use 1 MW either, just a few kW, less than a Watt per core.
Actually building these supercomputers will cost $100k for 180nm, $3 million at 28nm or around $30 million at 3nm. The FPGA prototypes cost $10 per core, similar to GPU prices. This includes the cost to write the software, the IDE, compilers, etc.
You can run X86 virtual machines unchanged on our 10000 - 1000000 manycore wafer scale integrations at 1 cent per kWh. This is by far the cheapest hyper-scale datacenter compute price ever and may come to outcompete current cloud datacenter which currently consume more than 5% of all the worlds electricity. And by locating our wafer supercomputers in your hot water storage tank at home [6], you'll monetise its waste heat so the compute cost drops below 1 cent per X cores (dependant on the efficiency of your software [5]). Another place you need these ultra low power wafer scale supercomputers is in self driving cars, robots, space vehicles and satellites, you can't put racks of computers there and you need to be frugal with battery storage.
These CMOS wafer scale integration supercomputers are themselves prototypes for carbon solar cells and carbon transistors we will grow from sunlight and CO2 in a decade from now [2]. Then they will cost almost nothing and run on completely free solar energy.
Eventually we will build a Dyson Swarm around our sun and have infinite free compute [3] called Matrioshka Brains [4]. To paraphrase Arthur C. Clarke, if you take these plans to seriously you will go bankrupt. If your children do not take these plans seriously they will go bankrupt.
[1] https://www.researchgate.net/profile/Merik-Voswinkel/publica...
[2] https://web.pa.msu.edu/peopl...
* "What if" feedback loops/direct manipulation
Victor's vision abstractly seems to be trying to predict/explore the consequence of some action in programming, and in specific demonstration seems to be using small widgets to allow easy manipulation of inputs to get an intuitive understanding of outputs. This could be boiled down to different goals: "Allow a program to be more easily tweaked" and "Explore a concept to get intuition of a different viewpoint". The more cynical/pragmatic interpretations for these are "make a GUI for your program" and "use interactive demos when teaching certain topics".
The first interpretation is almost comical, but we can maybe expand this to be "when you make a GUI, think about how your interface is being interpreted intuitively and this can help make your app more usable". This can maybe understood more easily when taken with the fact that Bret Victor helped design the interface for the first iPhone - famously intuitive to use. This also leads to its limitations - only concepts that have another more intuitive viewpoint can be represented. I can add a colour wheel to my WYSIWYG editor rather than hex values, but I can't easily create a GUI that lets me express that I want to validate, strip the whitespace from an email address and put it into lowercase.
The second interpretation leads to explorable explanations, which Victor has made a few of himself [0,1], but I would also cite Nicki Case [2] and unconed [3] as being other good examples. Again, this is only afforded to specific topics that have scope for exploration.
* Making logic feel more geometric/concrete
This can be seen in things like Labview (made in 1986), Apache NiFi (made in 2006) among others, e.g. SAS. In a sense, this has existed in the form of UNIX pipelines and functional programming since the first LISP was made. There is a further point which is "there currently aren't tools like this that are suitable for a non-programming audience", which is what 'Low Code' and 'No Code' is trying to achieve, but unfortunately in practice as soon as you hit a limitation of the framework then you're back to needing an engineer again.
* Human Interfaces
Sort of addressed in 'feedback loops' point above, but the DynamicLand is an interesting demo of what he's trying to get to. I think this speaks more to me with internet of things. I have friends who have set up full smart-home heating systems and can move music between rooms which are all very much seen the same as adjusting a physical thermostat rather than 'programming' or similar.
There is definitely a lot that can be explored here for certain applications, but there probably isn't direct utility in arranging pieces of paper with coloured dots on it in order to set the path of a robot. I can see this in a more consulting/capture sense of presenting certain input parameters in a more physical format, but again this is deviating from the OP's notion that this is a whole programming environment.
[0] http://worrydream.com/LadderOfAbstraction/
[1] http://worrydream.com/KillMath/
[2] https://ncase.me
[3] https://acko.net