Yeah, but what good is a specification of an obscure language that commercial projects won't touch because it's share-alike, and free projects won't touch because it's not-free?
It would need to be a success to warrant alternative implementations, and the license is inadequate for a language to get widespread adoption.
I've been pondering similar ideas as this for a while. I think you are correct by starting with Excel files.
What I've been hammock thinking is a way to bridge the gap between spreadsheets and what we tend to consider code. Why can't we iterate spreadsheets to something that makes it easier to: diff, review, separate logic from data, track in version control, write new functions/macros, re-use, package and distribute.
I think as more people will need to use computers more, that everyone will need to have access to some form of programming skills. I think spreadsheets are a successful mechanism for people to rapidly manipulate data, but they deteriorate quickly with complexity. A tool that doesn't throw out the baby with the bathwater could help people ramp up to programming much easier. If you teach someone to write a function that operates on a medium they already know like a spreadsheet, it is a lot less intimidating than writing a .py file, and downloading python and learning what compile is.
There's a lot of thinking around those lines, specially since the work of Bret Victor on data-rich programmable environments.
Spreadsheet-like IDEs are the most logical direction to meet that vision, specially when you want to attract non-programmers; figuring out the language syntax and the runtime execution model are the hardest parts of learning to program, and spreadsheets minimize both.
I put my bet on code notebooks, of the functional reactive variety. They have the same interaction model of the spreadsheet, intermixing data and code in the same environment, and they allow you to grow a program iteratively with cells acting as reusable objects - a simpler metaphor than OOP, but equally powerful. Data scientists are already making good use of them, and it's easy to think that these notebooks will be getting more and more of the debugging and DevOps functions of traditional IDEs.
// passing values through check
x do {
check >? 0
check even?()
+ 1
}
appear to be in conflict with stated goal of
"It is an interactive medium for data processing by people who don’t want to learn how to program. For example, scientists who want to process data without becoming “data scientists”. Or people who want to build simple custom apps without learning to program."
[This fighting text with text is indeed a CSFi soap opera :)]
I once built a complex spreadsheet for my colleagues. When I had to leave, what I built for them was almost instantly useless because there was too much they needed to understand. That got me thinking about the gulf between spreadsheets and programming languages. There are important differences:
- Spreadsheets don't separate code from data, which is an important aspect of programming. Blending the two is a nice shortcut for spreadsheets, but it ruins maintainability.
- Naming is important for programming, but spreadsheets encourage users to not bother with names and fall back to cell coordinates like "A10" instead. It's a great shortcut, but again, it ruins maintainability.
- Spreadsheet formulas operate on scalar values, while programming languages operate on nested data structures. Data structures are a key to managing complexity, yet the spreadsheet paradigm has only a very limited view of them.
If users are ever going to be able to use a spreadsheet-like interface for serious programming, most of those issues will need to be addressed in a way that doesn't hurt usability. That's a tall order! Users already struggle with function calls in formulas. At some point it's smarter for users to just pay a programmer to figure out how to build it.
> the gulf between spreadsheets and programming languages
There is no gulf. Spreadsheets just are programming languages. You've certainly identified problems or at least quirks of spreadsheets as a 'language', but I think people do a huge disservice to their own ability to understand programming as a field when they try to carve out exclusions for languages and environments on the basis of who uses them. Lots of things no one would ever argue aren't a programming language have all these flaws and more.
When people make a big ass complicated spreadsheet, they are programming. They are doing so whether they're a seasoned professional or an accountant or a lawyer or a homemaker.
Of course people who use spreadsheets are programming; they're building data structures (with columns as object fields), generating values from formulas, and debugging the program's execution when editing cells.
What develops typically don't get is that the paradigm of the programming language and environment matters a lot, triple so for people without formal training in coding. Most people who can code with a spreadsheet would be totally unable to create a very simple shell script, because the paradigm of imperative, syntax-led classic programming languages require building a mental model of a whole Turing machine in your head, which requires years of training. That's the gulf the GP was referring to; not from which people happen to use them, but from how they think about coding.
The functional-reactive paradigm (and spreadsheets in particular) avoid that need, because they're highly composable so you can truly think of each part of the program in isolation, something which side-effects and program flow in imperative programs won't allow.
> That's the gulf the GP was referring to; not from which people happen to use them, but from how they think about coding.
I'm calling out an implicit bias that always comes up in discussions like these and I see lurking behind this sentiment, because these two things are inextricably related, not an accidental correlation.
They think better in this paradigm because its a better suited paradigm to their actual work, and it fits better with things they understand already because of their experiences and knowledge.
When we prioritize the "thought patterns" of professional programmers we forget this, and then we lament the lack of understanding for our field from people outside of it. Our solutions, when we look at it this way (with a stark divide between "programmers" and "non-programmers"), end up demanding that they come to our understanding instead of us to theirs. And that's basically always doomed to fail.
> When we prioritize the "thought patterns" of professional programmers we forget this, and then we lament the lack of understanding for our field from people outside of it. Our solutions, when we look at it this way (with a stark divide between "programmers" and "non-programmers"), end up demanding that they come to our understanding instead of us to theirs. And that's basically always doomed to fail.
I concur, that's a good way to put it. The "way of the programmer" requires an unnatural way of reasoning that takes a lot of effort to master, and requires a lot of short-term memory effort to mentally simulate how the code behaves.
I don't even think this programming style is truly required for professional programmers either; more a by-product of how programming started as a way to feed instructions to mono-processors in the context of mathematics and physics computations on limited hardware.
Symbolic business logic doesn't typically depend so strongly on the specifics of runtime execution, and recently there has been a very active movement to build new programming environments which emphasize the relationship between code and data, rather than between code and intermediate states of computation as in classical programming languages.
You identified the essential qualities of spreadsheets, which many programmers fail to realize how important these are; these qualities are what make spreadsheets so useful to non-programmers. A successful development tool for non-coders should retain these, and explore alternative ways to add all the benefits of IDEs without the burden of classic programming languages.
- Spreadsheets don't separate code from data, and that's the single most important reason why they're easy to use and allow non-developers to build automations. They work like an always-on debugger, so normal users can work like a programmer works when exploring and/or fixing a program's behaviour. Practically no other coding tool for non-developers allow this.
- Naming is important for code reuse, not necessarily for one-shot functions; positional reasoning can work equally well there. Spreadsheets do allow naming cells and accessing them by name when needed; but the option not to, which classic PLs don't allow, lowers the entry barrier. Programmers could also name all their parameters 'a', 'b', 'c'; that they learn to use better names is a matter of discipline, not a limitation of the tool.
- Scalar vs ranges: this one is actually the most limiting part of spreadsheets. They are great for tabular data structures (doh!), but nested structures are tricky. There are research projects to improve this (Microsoft hosts a lot of them on this line, specially around ML and Programming By Example), but in the end you need to overcome the paradigm and apply spreadsheet-like programming concepts to tools allowing hierarchical, nested structures. That lies the future of computation for the masses.
- You can have an always-on debugger while still factoring common code out of varying data. Subtext does this.
- You can give things names after you create them instead of before, and it's not clear that argument names are necessarily either necessary or sufficient for code reuse. Subtext makes it easy to change argument names whenever you want.
- It's reasonable to describe Subtext as a programming-by-example environment, and it does support nesting—in fact, it's rather pervasively nested. An alternative approach is to solve your problems without hierarchical, nested structures. SQL and co-dfns are two interesting directions there. I wrote an overview of this kind of thing five years ago: http://canonical.org/~kragen/memory-models/
It works to solve the same space of problems, so at least it's fair to compare it as an alternative.
As I said elsewhere, it's a pity that its chosen license works directly against widespread adoption. I may take a look at how it's done to gather some implementation ideas, but I would never depend on it for any project that have a chance of growth.
It doesn't work to solve the same space of problems as spreadsheets either, nor is it fair to compare it as an alternative, except in the sense that any programming language is an alternative to spreadsheets. The thing here that's licensed in a not-very-useful manner is a prototype of a prototype, less complete than some previous iterations of Subtext, and in particular without implementation ideas.
The author explicitly says that it "is searching for the missing link between spreadsheets and programming", so I gather form that that JonathanMEdwards actually thinks of this as fulfilling the role of spreadsheets in "build[ing] simple custom apps without learning to program". Other programming languages are not alternatives to spreadsheets in that sense.
I've found Jonathan's research in Subtext inspirational for a long time, though only through watching videos and reading papers. It's too bad it's not under a free license, but hopefully people can get it to run well enough to get some experience with the interaction model he's refined through all these iterations.
Yeah that didn't happen. I stopped this work before I got to reimplementing the UI. I worked out some problems I've been obsessing about forever, but the result is just too complex and alien. Back to the drawing board. Thanks for your encouragement all these years.
Any significant improvement will necessarily be a significant change and therefore alien.
Computers were alien, and the culture projected all its fears onto them: rightists churned out endless movies about slave revolts with computers and robots in the role of slaves, while leftists tried to bomb and burn down computer centers as oppressive instruments of the System.
FORTRAN and COBOL were alien to experienced programmers, who seriously doubted how optimal the output assembly would be.
VisiCalc was alien to Fortran programmers. You're still pushing back against the remnants of that rejection today.
MacOS in 01984 was alien to those of us who'd grown up writing BASIC. It provoked hostility, too!
Pascal was alien to me because it didn't have line numbers.
Online chatting was alien. We called ourselves "geeks".
When we pioneered AJAX and Comet later that year at KnowNow (one of three independent inventions), it was alien, and in fact it took four years before Gmail took AJAX mainstream; Comet took even longer.
So don't worry about being alien. You're shooting for a revolutionary change, not an incremental improvement. Revolutionary changes are difficult under the best of circumstances, but they are impossible to achieve without being alien. Your weirdness budget isn't unlimited but it's necessarily larger than for an incremental change.
I know it would be a headache to achieve it, but it would be pretty useful to have an open-source runnable version of any of the older Subtext prototypes.
I didn't know of the existence of Subtext, and I've felt impressed by the core set of ideas behind it (even if at first I was underwhelmed when first looked at how it was licensed, sorry about that). I've seen recently many other converging to the same region of the design space, but your approach hits all the relevant points on what I would expect on a system for a next-generation style of computing.
I have some ideas on my own on building a new style of computing environment that doesn't feel alien, and I've arrived to a lot of the same conclusions that you describe in the "Notable features" section.
My approach is to focus the environment in content-editing first (i.e. making it feel like a "structured content editor" more than a IDE), and representing code primarily with a metaphor of content transformation over collections (think Google's OpenRefine or MS Power BI), rather than as processes running on an application.
I have a core idea at the base of the structure for this system, which is to reify function applications as a named object in the environment, much like your idea of treating function results as collections of values accessible as raw data. This reified object, which I call a wit for memorability, acts very much like items in your Subtext, except that:
* Each wit has a unique identifier (a 'label') which can be used to annotate data directly through direct manipulation, like giving a name to a cell in a spreadsheet. The user can build collections by applying the same tag to all the values in the collection, manually or supported by a PbE process. The wit represents the data structure containing all those values; but at the same time it may represent the process used to generate them.
* There is no distinction among scalars and collections. I think this is the most unique feature of my system; you may apply a function to a wit (identified by its label), and it gets lazily applied to all its values, thus creating a new collection of transformed values. This is similar to named ranges in a spreadsheet, but it doesn't depend on a positional grid. And single values are treated the same as collections, so there's no need for loop constructs; the user can simply build templates with wits as variables, which then get applied to all their possible values in their input domain. (Microsoft has a similar feature with their research on FlashFill, which was posted here a few days ago[1]).
* Wits may be used as identifiers of attributes, to extract content from records. That content may be previously stored values or computed on the fly from their formula and current inputs (just like in your "Subtext is both a PL and a DB" line). I.e. wits can represent relations among entities, not just code applied over data. This blurs the metaphors of function application and data structure, which I believe is a good thing for easy-to-use computational environments.
* Since code is seen as data collections, you can build repeatable behaviors by changing the input on a sequence of transformations. For example, recorded recent user actions are not seen as a sequence of code steps, but a sequence of intermediate processed states of the transformed collection (with each step being itself represented by a wit, and thus accessible as data). I think you have a similar idea with your "functions seen as expanding to traces", but I envision using the intermediate steps as possible inputs to new functions (facilitating creation of pipes and recursion by demonstration).
I've felt my ideas validated when seeing that you've created a system which has evolved to a very similar design space; your Feedback mechanism is a neat solution to the representation of interaction and user updates, for which I didn't have a concrete solution (other than the abstract idea of representing all state as updates on functional-reactive streams).
Unfortunately my design process involves primarily ...
Interesting. Check out the futureofcoding.org slack. There are some workshops that take early stage ideas, like LIVE and PX. Keep working on your ideas and try to pull them together into a coherent explanation. I'd be happy to take a look and give you some feedback.
Thanks! I guess I'll clean up my notes and post them on futureofcoding.org community through the share-your-work section, to explain all the details and possibilities.
I took inspiration from this [1] classic article by Henry Lieberman, where tags are applied to elements of a composite as a way to edit its structure; and I combined it with the idea of a templating system to process or generate items in bulk (all templating systems do this, my idea is to abstract it into a generalized, always present interaction mechanism).
I liked that book so much I bought it in paper form; also YWIMC. Lieberman's work in particular has always been inspirational. I think he sits down the hall from Jonathan, but I could be misremembering.
This morning I was working on a problem that I was thinking would be ideal for a system like this: I was taking notes in prose about potential blowing agents for waterglass or other geopolymer foams—things that produce gas when the polymer is heated to its softening point. For example, chalk, ammonium chloride, sulfur, aluminum hydroxide, sal mirabilis, saltpeter, or carbon plus silica. Each of these blowing agents produces a certain number of moles of gas per mole of blowing agent, has a certain molar mass, a certain density, a certain cost, and a certain activation temperature, plus certain material-compatibility concerns—you can't disperse hydrated sal mirabilis in an aqueous solution, for example, or chalk in an acid solution, if you want them to be intact in the unblown geopolymer preform. These are "givens" that can be found in a literature search or, in some cases, a search on MercadoLibre.
In the prose document, I would like to be able to tag the gas production per mole, molar mass, density, activation temperature, and material compatibility in such a way that I can render a table of the candidate materials in the document itself, and then sort it by gas production per mass, per volume, or per dollar, and filter it by activation temperature. Gas production per mass is calculated from molar mass and molar gas production; gas production per volume is calculated from gas production per mass and density, and similarly for gas production per dollar.
Is that the kind of thing you're thinking of, or am I imagining wits very differently from what you really have in mind? I look forward to reading your notes! (I'm not in the Slack. I hate Slack. It's ransomware.)
A step beyond would be to type the reaction equations into the prose document and calculate the gas production per mole from that, and use a database of atomic weights to calculate the molar mass from that as well. With a database of the formation entropies and enthalpies of the compounds, I could not only calculate how endothermic the decomposition is (or, rarely, exothermic) but plot an Ellingham diagram as well, which would be pretty useful as a double-check on the sanity of the reported activation temperatures from the littrachaw. Together with combinatorial search functionality, it might also enable me to anticipate problematic side reactions, or how the activation temperatures would shift if the foaming was carried out at 1 MPa.
Some of the quantities I'm calculating on have units attached, such as temperature and density, and some have intervals of uncertainty, such as price, and it would be useful to have the computer track those things instead of the human. Confusing a Gibbs free energy in kJ/mol with kJ/kg can be a problem; similarly gas production per gram or per cubic centimeter.
Some of Jonathan's work on Subtext has focused on variants of or alternatives to test-first programming; in spreadsheets people sometimes do test-first programming (for things like the equilibrium-temperature calculation I mentioned above, for example) and then delete the tests. In a context like the blowing-agent survey, being able to hide them instead would be better.
Finally, ideally, I'd like to be able to transclude at least the givens and possibly the calculation results in other notes: "According to my notes in [0], aluminum hydroxide is stable at atmospheric pressure up to 220°, ..." perhaps eventually refactoring those inputs into a centralized database so that I don't have to remember which note has the boiling point of sulfur in it.
The calculation part would be easy to do in Gnumeric or localc, but those handle the prose, units, and intervals very poorly at best; and they would be hopeless for the equation parsing part and, at best,...
> In the prose document, I would like to be able to tag the gas production per mole, molar mass, density, activation temperature, and material compatibility in such a way that I can render a table of the candidate materials in the document itself, and then sort it by gas production per mass, per volume, or per dollar, and filter it by activation temperature.
Yes, that seems be a very direct application of my wit concept, with one wit for each magnitude, placing each wit as a column of the table and having one row for each material. In the end, a wit is basically like a user-friendly version of a typed pointer in C, with a 'visual' GUI for combining them like an Excel pivot table or a OLAP 'data cube'. You can read the value at the end, or you can iterate it over a collection to process a list of values; the meaning is located in the code that uses it, more than in the object itself, and you can share the reference to the same value in many different places.
Of course there are already several systems where you can define a data structure in code and show it rendered as labels on a document (see idyll[1], or the original Explorable Explanations), but I have the idea that using such system shouldn't require any code for basic or moderately-complex programs:
1) the user should be able to define the data structure through direct interaction, like on a spreadsheet (e.g. Airtable does this fairly well). But the raw data could be extracted from existing documents, like a web scrapper, with the scrapping expressions defined through point-and-click over the actual data. The now extict kimonolabs[2] had a good interaction model for this part. **
2) this structured data could be transferred between applications with a copy/paste-like universal protocol (most current applications require coding against APIs for this),
3) calculations can be defined and re-applied to new data collections through simple formulas and more direct manipulation, possibly with Programming-by-example assistance tools. The people from MS Excel are doing this, but they lack the universal API - everything happens inside Microsoft applications.
For your example, you could store your database of atomic weights as "hidden wits" somewhere else, extract the desired value by naming the label (wit labels are global variables under a particular namespace), and feed its values to the expression where you want to calculate a function application with the data.
> The calculation part would be easy to do in Gnumeric or localc, but those handle the prose, units, and intervals very poorly at best; and they would be hopeless for the equation parsing part and, at best, clumsy for the database queries and Ellingham-diagram plotting.
The good thing of wits is that you could chain them and transfer information between applications, with each app handling what it does well, and returning the data back to the origin; i.e. any workflow that you could do manually with copy-paste. In this sense it's like a Unix pipe, but in a visual environment and transferring structured data (which can be de-composed at the receiving end, extracting one wit at a time). They are also like those awkward connector lines in dataflow visual languages, except that instead of a line you have a label with the wit name at the origin and destination.
Again MS has Power Automate as a tool with these capabilities (I think its way more complex and powerful than the classic Apple Automator), but again Power Automate restricts you to a closed runtime environment of applications that support it, which you glue only through code.
> Is that the kind of thing you're thinking of, or am I imagining wits very differently from what you really have in mind? I look forward to reading your notes! (I'm not in the Slack. I hate Slack. It's ransomware.)
I don't like walled gardens either; I've heard Matrix.org and Element.io are lovely this time of the year... :-P
I'm reading through The Lab Notes from Alexander Obenauer[1], and I'm noticing lots of coincidences. Universal data portability [2] in particular shares the same code insight of reusing all user data in any context where it's needed, and Universal reference containers [3] are the primary use case for my idea of Wit.
With the example interactions in that website you can get very close to what I imagined my system to be (though I would use a "select+tag" metaphor more often than the cumbersome drag&drop).
I still have the idea to allow spreadsheet-like processing on top of those metaphors, but the essence of the system is quite similar to that one; many information systems are converging in that same direction toward a similar model of interaction.
As someone who has been working in this area for a while, for a programing language to be embraced by non-programmers it has to not look like a programing language. That is, no matter how streamlined an simple it feels to us programmers, when you stand two meters back from the screen if it looks like c++ then it won't have any significant uptake.
I scroll through the subtext page and just see code samples that look pretty much like any other language, and that's what the non-programmer target audience will notice.
Strong agree. The syntax in this work was meant to be mostly hidden from the end user. I've found that syntax is still really useful for language design and implementation. I stopped before building the graphical programming environment on top.
Designing a productive graphical programming environment is very challenging. I think it is probably at least as difficult as designing the underlying language mechanics. I'd love to see any experiments you might be able to share.
> I hope not. Open source works best for software used by other programmers, because that is how the contributors get their cred. But the economics of startups conflict with the goal of liberating software development. Perhaps a hybrid form like a Public Benefit Corp or a Coop offers a workable middle way.
... Huh? The question is about open source. The answer seems to be about startups. Was this an editing mixup?
You don't see how the economics of startups interact with the open source ecosystem and how someone's view of that interaction might affect their choice to not make something open source?
I can certainly see how they might be related, but the answer doesn't clarify how---rather it seems completely disconnected in the first and second halves, which is why I'm wondering if it's just an editing error.
I'm intrigued, but need to read through this a few times to get everything.
One thing I don't understand is the dismissal of operator precedence for infix operators. It seems to me that creates an unnecessary impedance mismatch for getting formulae entered into the computer.
Having read some more, and watched the 2018 video, I now understand that ditching operator precedence was a necessary sacrifice to make your interactive environment something that could be implemented in a reasonable timeframe.
Yeah, I've gotten a lot of pushback on that. One approach I've seen is to reject expressions that would violate standard precedence, and require extra parentheses.
like lisp and forth, the moving of things to/from variables is removed from primacy, to good effect. being able to use a result without having to name it is a quite powerful tool.
As with spreadsheets, the absolute or relative location can fill in as a address. Perhaps there should be some relative addressing for code, as you hinted ^ with spaces around it is the previous result.
48 comments
[ 3.0 ms ] story [ 120 ms ] threadhttps://github.com/JonathanMEdwards/subtext10/blob/master/LI...
I believe the lack of a patents grant clause is the only worrisome thing about CC licenses?
It would need to be a success to warrant alternative implementations, and the license is inadequate for a language to get widespread adoption.
What if you add formulas?
I have no idea if Subtext 10 has legs. It kinda reminds me of Fran, so that's a good start (direction).
All I'm saying is that if I was gonna tackle this problem, I'd start with turning Excel files into into source code.
What I've been hammock thinking is a way to bridge the gap between spreadsheets and what we tend to consider code. Why can't we iterate spreadsheets to something that makes it easier to: diff, review, separate logic from data, track in version control, write new functions/macros, re-use, package and distribute.
I think as more people will need to use computers more, that everyone will need to have access to some form of programming skills. I think spreadsheets are a successful mechanism for people to rapidly manipulate data, but they deteriorate quickly with complexity. A tool that doesn't throw out the baby with the bathwater could help people ramp up to programming much easier. If you teach someone to write a function that operates on a medium they already know like a spreadsheet, it is a lot less intimidating than writing a .py file, and downloading python and learning what compile is.
Spreadsheet-like IDEs are the most logical direction to meet that vision, specially when you want to attract non-programmers; figuring out the language syntax and the runtime execution model are the hardest parts of learning to program, and spreadsheets minimize both.
I put my bet on code notebooks, of the functional reactive variety. They have the same interaction model of the spreadsheet, intermixing data and code in the same environment, and they allow you to grow a program iteratively with cells acting as reusable objects - a simpler metaphor than OOP, but equally powerful. Data scientists are already making good use of them, and it's easy to think that these notebooks will be getting more and more of the debugging and DevOps functions of traditional IDEs.
Things like
appear to be in conflict with stated goal of"It is an interactive medium for data processing by people who don’t want to learn how to program. For example, scientists who want to process data without becoming “data scientists”. Or people who want to build simple custom apps without learning to program."
[This fighting text with text is indeed a CSFi soap opera :)]
- Spreadsheets don't separate code from data, which is an important aspect of programming. Blending the two is a nice shortcut for spreadsheets, but it ruins maintainability.
- Naming is important for programming, but spreadsheets encourage users to not bother with names and fall back to cell coordinates like "A10" instead. It's a great shortcut, but again, it ruins maintainability.
- Spreadsheet formulas operate on scalar values, while programming languages operate on nested data structures. Data structures are a key to managing complexity, yet the spreadsheet paradigm has only a very limited view of them.
If users are ever going to be able to use a spreadsheet-like interface for serious programming, most of those issues will need to be addressed in a way that doesn't hurt usability. That's a tall order! Users already struggle with function calls in formulas. At some point it's smarter for users to just pay a programmer to figure out how to build it.
There is no gulf. Spreadsheets just are programming languages. You've certainly identified problems or at least quirks of spreadsheets as a 'language', but I think people do a huge disservice to their own ability to understand programming as a field when they try to carve out exclusions for languages and environments on the basis of who uses them. Lots of things no one would ever argue aren't a programming language have all these flaws and more.
When people make a big ass complicated spreadsheet, they are programming. They are doing so whether they're a seasoned professional or an accountant or a lawyer or a homemaker.
What develops typically don't get is that the paradigm of the programming language and environment matters a lot, triple so for people without formal training in coding. Most people who can code with a spreadsheet would be totally unable to create a very simple shell script, because the paradigm of imperative, syntax-led classic programming languages require building a mental model of a whole Turing machine in your head, which requires years of training. That's the gulf the GP was referring to; not from which people happen to use them, but from how they think about coding.
The functional-reactive paradigm (and spreadsheets in particular) avoid that need, because they're highly composable so you can truly think of each part of the program in isolation, something which side-effects and program flow in imperative programs won't allow.
I'm calling out an implicit bias that always comes up in discussions like these and I see lurking behind this sentiment, because these two things are inextricably related, not an accidental correlation.
They think better in this paradigm because its a better suited paradigm to their actual work, and it fits better with things they understand already because of their experiences and knowledge.
When we prioritize the "thought patterns" of professional programmers we forget this, and then we lament the lack of understanding for our field from people outside of it. Our solutions, when we look at it this way (with a stark divide between "programmers" and "non-programmers"), end up demanding that they come to our understanding instead of us to theirs. And that's basically always doomed to fail.
This is important.
I concur, that's a good way to put it. The "way of the programmer" requires an unnatural way of reasoning that takes a lot of effort to master, and requires a lot of short-term memory effort to mentally simulate how the code behaves.
I don't even think this programming style is truly required for professional programmers either; more a by-product of how programming started as a way to feed instructions to mono-processors in the context of mathematics and physics computations on limited hardware.
Symbolic business logic doesn't typically depend so strongly on the specifics of runtime execution, and recently there has been a very active movement to build new programming environments which emphasize the relationship between code and data, rather than between code and intermediate states of computation as in classical programming languages.
- Spreadsheets don't separate code from data, and that's the single most important reason why they're easy to use and allow non-developers to build automations. They work like an always-on debugger, so normal users can work like a programmer works when exploring and/or fixing a program's behaviour. Practically no other coding tool for non-developers allow this.
- Naming is important for code reuse, not necessarily for one-shot functions; positional reasoning can work equally well there. Spreadsheets do allow naming cells and accessing them by name when needed; but the option not to, which classic PLs don't allow, lowers the entry barrier. Programmers could also name all their parameters 'a', 'b', 'c'; that they learn to use better names is a matter of discipline, not a limitation of the tool.
- Scalar vs ranges: this one is actually the most limiting part of spreadsheets. They are great for tabular data structures (doh!), but nested structures are tricky. There are research projects to improve this (Microsoft hosts a lot of them on this line, specially around ML and Programming By Example), but in the end you need to overcome the paradigm and apply spreadsheet-like programming concepts to tools allowing hierarchical, nested structures. That lies the future of computation for the masses.
- You can give things names after you create them instead of before, and it's not clear that argument names are necessarily either necessary or sufficient for code reuse. Subtext makes it easy to change argument names whenever you want.
- It's reasonable to describe Subtext as a programming-by-example environment, and it does support nesting—in fact, it's rather pervasively nested. An alternative approach is to solve your problems without hierarchical, nested structures. SQL and co-dfns are two interesting directions there. I wrote an overview of this kind of thing five years ago: http://canonical.org/~kragen/memory-models/
As I said elsewhere, it's a pity that its chosen license works directly against widespread adoption. I may take a look at how it's done to gather some implementation ideas, but I would never depend on it for any project that have a chance of growth.
Computers were alien, and the culture projected all its fears onto them: rightists churned out endless movies about slave revolts with computers and robots in the role of slaves, while leftists tried to bomb and burn down computer centers as oppressive instruments of the System.
FORTRAN and COBOL were alien to experienced programmers, who seriously doubted how optimal the output assembly would be.
VisiCalc was alien to Fortran programmers. You're still pushing back against the remnants of that rejection today.
MacOS in 01984 was alien to those of us who'd grown up writing BASIC. It provoked hostility, too!
Pascal was alien to me because it didn't have line numbers.
Online chatting was alien. We called ourselves "geeks".
I still have anti-JS JS from 02000 on http://canonical.org/~kragen/hotlist.html because JS was alien and I hated it.
When we pioneered AJAX and Comet later that year at KnowNow (one of three independent inventions), it was alien, and in fact it took four years before Gmail took AJAX mainstream; Comet took even longer.
So don't worry about being alien. You're shooting for a revolutionary change, not an incremental improvement. Revolutionary changes are difficult under the best of circumstances, but they are impossible to achieve without being alien. Your weirdness budget isn't unlimited but it's necessarily larger than for an incremental change.
I know it would be a headache to achieve it, but it would be pretty useful to have an open-source runnable version of any of the older Subtext prototypes.
I have some ideas on my own on building a new style of computing environment that doesn't feel alien, and I've arrived to a lot of the same conclusions that you describe in the "Notable features" section.
My approach is to focus the environment in content-editing first (i.e. making it feel like a "structured content editor" more than a IDE), and representing code primarily with a metaphor of content transformation over collections (think Google's OpenRefine or MS Power BI), rather than as processes running on an application.
I have a core idea at the base of the structure for this system, which is to reify function applications as a named object in the environment, much like your idea of treating function results as collections of values accessible as raw data. This reified object, which I call a wit for memorability, acts very much like items in your Subtext, except that:
* Each wit has a unique identifier (a 'label') which can be used to annotate data directly through direct manipulation, like giving a name to a cell in a spreadsheet. The user can build collections by applying the same tag to all the values in the collection, manually or supported by a PbE process. The wit represents the data structure containing all those values; but at the same time it may represent the process used to generate them.
* There is no distinction among scalars and collections. I think this is the most unique feature of my system; you may apply a function to a wit (identified by its label), and it gets lazily applied to all its values, thus creating a new collection of transformed values. This is similar to named ranges in a spreadsheet, but it doesn't depend on a positional grid. And single values are treated the same as collections, so there's no need for loop constructs; the user can simply build templates with wits as variables, which then get applied to all their possible values in their input domain. (Microsoft has a similar feature with their research on FlashFill, which was posted here a few days ago[1]).
* Wits may be used as identifiers of attributes, to extract content from records. That content may be previously stored values or computed on the fly from their formula and current inputs (just like in your "Subtext is both a PL and a DB" line). I.e. wits can represent relations among entities, not just code applied over data. This blurs the metaphors of function application and data structure, which I believe is a good thing for easy-to-use computational environments.
* Since code is seen as data collections, you can build repeatable behaviors by changing the input on a sequence of transformations. For example, recorded recent user actions are not seen as a sequence of code steps, but a sequence of intermediate processed states of the transformed collection (with each step being itself represented by a wit, and thus accessible as data). I think you have a similar idea with your "functions seen as expanding to traces", but I envision using the intermediate steps as possible inputs to new functions (facilitating creation of pipes and recursion by demonstration).
I've felt my ideas validated when seeing that you've created a system which has evolved to a very similar design space; your Feedback mechanism is a neat solution to the representation of interaction and user updates, for which I didn't have a concrete solution (other than the abstract idea of representing all state as updates on functional-reactive streams).
Unfortunately my design process involves primarily ...
I took inspiration from this [1] classic article by Henry Lieberman, where tags are applied to elements of a composite as a way to edit its structure; and I combined it with the idea of a templating system to process or generate items in bulk (all templating systems do this, my idea is to abstract it into a generalized, always present interaction mechanism).
[1] http://acypher.com/wwid/Chapters/23ProblemSolving.html Making Programming Accessible to Visual Problem Solvers
I liked that book so much I bought it in paper form; also YWIMC. Lieberman's work in particular has always been inspirational. I think he sits down the hall from Jonathan, but I could be misremembering.
This morning I was working on a problem that I was thinking would be ideal for a system like this: I was taking notes in prose about potential blowing agents for waterglass or other geopolymer foams—things that produce gas when the polymer is heated to its softening point. For example, chalk, ammonium chloride, sulfur, aluminum hydroxide, sal mirabilis, saltpeter, or carbon plus silica. Each of these blowing agents produces a certain number of moles of gas per mole of blowing agent, has a certain molar mass, a certain density, a certain cost, and a certain activation temperature, plus certain material-compatibility concerns—you can't disperse hydrated sal mirabilis in an aqueous solution, for example, or chalk in an acid solution, if you want them to be intact in the unblown geopolymer preform. These are "givens" that can be found in a literature search or, in some cases, a search on MercadoLibre.
In the prose document, I would like to be able to tag the gas production per mole, molar mass, density, activation temperature, and material compatibility in such a way that I can render a table of the candidate materials in the document itself, and then sort it by gas production per mass, per volume, or per dollar, and filter it by activation temperature. Gas production per mass is calculated from molar mass and molar gas production; gas production per volume is calculated from gas production per mass and density, and similarly for gas production per dollar.
Is that the kind of thing you're thinking of, or am I imagining wits very differently from what you really have in mind? I look forward to reading your notes! (I'm not in the Slack. I hate Slack. It's ransomware.)
A step beyond would be to type the reaction equations into the prose document and calculate the gas production per mole from that, and use a database of atomic weights to calculate the molar mass from that as well. With a database of the formation entropies and enthalpies of the compounds, I could not only calculate how endothermic the decomposition is (or, rarely, exothermic) but plot an Ellingham diagram as well, which would be pretty useful as a double-check on the sanity of the reported activation temperatures from the littrachaw. Together with combinatorial search functionality, it might also enable me to anticipate problematic side reactions, or how the activation temperatures would shift if the foaming was carried out at 1 MPa.
Some of the quantities I'm calculating on have units attached, such as temperature and density, and some have intervals of uncertainty, such as price, and it would be useful to have the computer track those things instead of the human. Confusing a Gibbs free energy in kJ/mol with kJ/kg can be a problem; similarly gas production per gram or per cubic centimeter.
Some of Jonathan's work on Subtext has focused on variants of or alternatives to test-first programming; in spreadsheets people sometimes do test-first programming (for things like the equilibrium-temperature calculation I mentioned above, for example) and then delete the tests. In a context like the blowing-agent survey, being able to hide them instead would be better.
Finally, ideally, I'd like to be able to transclude at least the givens and possibly the calculation results in other notes: "According to my notes in [0], aluminum hydroxide is stable at atmospheric pressure up to 220°, ..." perhaps eventually refactoring those inputs into a centralized database so that I don't have to remember which note has the boiling point of sulfur in it.
The calculation part would be easy to do in Gnumeric or localc, but those handle the prose, units, and intervals very poorly at best; and they would be hopeless for the equation parsing part and, at best,...
Yes, that seems be a very direct application of my wit concept, with one wit for each magnitude, placing each wit as a column of the table and having one row for each material. In the end, a wit is basically like a user-friendly version of a typed pointer in C, with a 'visual' GUI for combining them like an Excel pivot table or a OLAP 'data cube'. You can read the value at the end, or you can iterate it over a collection to process a list of values; the meaning is located in the code that uses it, more than in the object itself, and you can share the reference to the same value in many different places.
Of course there are already several systems where you can define a data structure in code and show it rendered as labels on a document (see idyll[1], or the original Explorable Explanations), but I have the idea that using such system shouldn't require any code for basic or moderately-complex programs:
1) the user should be able to define the data structure through direct interaction, like on a spreadsheet (e.g. Airtable does this fairly well). But the raw data could be extracted from existing documents, like a web scrapper, with the scrapping expressions defined through point-and-click over the actual data. The now extict kimonolabs[2] had a good interaction model for this part. **
2) this structured data could be transferred between applications with a copy/paste-like universal protocol (most current applications require coding against APIs for this),
3) calculations can be defined and re-applied to new data collections through simple formulas and more direct manipulation, possibly with Programming-by-example assistance tools. The people from MS Excel are doing this, but they lack the universal API - everything happens inside Microsoft applications.
For your example, you could store your database of atomic weights as "hidden wits" somewhere else, extract the desired value by naming the label (wit labels are global variables under a particular namespace), and feed its values to the expression where you want to calculate a function application with the data.
> The calculation part would be easy to do in Gnumeric or localc, but those handle the prose, units, and intervals very poorly at best; and they would be hopeless for the equation parsing part and, at best, clumsy for the database queries and Ellingham-diagram plotting.
The good thing of wits is that you could chain them and transfer information between applications, with each app handling what it does well, and returning the data back to the origin; i.e. any workflow that you could do manually with copy-paste. In this sense it's like a Unix pipe, but in a visual environment and transferring structured data (which can be de-composed at the receiving end, extracting one wit at a time). They are also like those awkward connector lines in dataflow visual languages, except that instead of a line you have a label with the wit name at the origin and destination.
Again MS has Power Automate as a tool with these capabilities (I think its way more complex and powerful than the classic Apple Automator), but again Power Automate restricts you to a closed runtime environment of applications that support it, which you glue only through code.
> Is that the kind of thing you're thinking of, or am I imagining wits very differently from what you really have in mind? I look forward to reading your notes! (I'm not in the Slack. I hate Slack. It's ransomware.)
I don't like walled gardens either; I've heard Matrix.org and Element.io are lovely this time of the year... :-P
I've bought the domain kragen ↗ I look forward to seeing the interaction model you come up with! Or your notes! TuringTest ↗ I'm reading through The Lab Notes from Alexander Obenauer[1], and I'm noticing lots of coincidences. Universal data portability [2] in particular shares the same code insight of reusing all user data in any context where it's needed, and Universal reference containers [3] are the primary use case for my idea of Wit.
With the example interactions in that website you can get very close to what I imagined my system to be (though I would use a "select+tag" metaphor more often than the cumbersome drag&drop).
I still have the idea to allow spreadsheet-like processing on top of those metaphors, but the essence of the system is quite similar to that one; many information systems are converging in that same direction toward a similar model of interaction.
[1] https://alexanderobenauer.com/labnotes/000/
[2] https://alexanderobenauer.com/labnotes/002/
[3] https://alexanderobenauer.com/labnotes/003/
> I hope not. Open source works best for software used by other programmers, because that is how the contributors get their cred. But the economics of startups conflict with the goal of liberating software development. Perhaps a hybrid form like a Public Benefit Corp or a Coop offers a workable middle way.
... Huh? The question is about open source. The answer seems to be about startups. Was this an editing mixup?
One thing I don't understand is the dismissal of operator precedence for infix operators. It seems to me that creates an unnecessary impedance mismatch for getting formulae entered into the computer.
Thanks for sharing.
As with spreadsheets, the absolute or relative location can fill in as a address. Perhaps there should be some relative addressing for code, as you hinted ^ with spaces around it is the previous result.
But the inspirations differ https://inflex.io/blog/bibliography
to me this is really the desirable core idea