Launch HN: Optic (YC S18) – Automate Routine Programming
A really simple use case that Optic enabled for some of our early users is keeping their backend and frontend in sync. Optic can read the endpoints in a backend and use that to generate the HTTP requests on the frontend. It even maintains that code over time so if the backend is updated, a pull request is generated to update the networking code. You can test that here: https://github.com/acunniffe/optic-networking-code-demo.
We have also had teams configure Optic to:
- Write/Maintain standard tests for their React components
- Migrate to GraphQL by generating wrappers for each endpoint
- Generate/Maintain CRUD routes from models
- Wrap their Tensorflow models in APIs
I left the company and went into research mode for around 18 months. A lot of the automated programming projects do their functional job well, but when it come time to integrating into a developer’s workflow they are littered with tradeoffs. Automation isn’t worth it if it means giving up control, rewriting lots of code, rebuilding your app visually, describing your project in some foreign dsl, or becoming tied to a vendor. Many of these projects haven’t caught on because the tradeoffs of using them have been too high.
When I started working on this problem again I was determined to get the developer experience right. For me this meant making Optic:
- work with your existing code
- plug in to your favorite IDE
- useful throughout the lifetime of your project
GET JSON objects describing different types of code. Powered by a regex like pattern matcher we wrote that walks AST Trees.
PUT new values of the same shape back to update the code. This was the hardest part by far. We ended up training decision trees on different programming languages using the raw code and the resulting AST tree as input. We use these models to regenerate code in the smallest sections possible so formatting and manual changes are preserved.
POST new code into the project. This was actually the easiest. A generator can be reformulated as a Parser + Mutator so we bootstrapped all the generation in Optic by combining the GET / PUT functionality.
This API was really powerful and it became the foundation of Optic. Generating code, doing transformations, syncing projects — all Optic’s major features are built on top of this API. It’s such a solid foundation (you get parsing, generating and the round-trip problem in a box) that I believe a lot of new meta-programming tools will be built on our platform. Our work is open source (https://github.com/opticdev) and there will always be a free-forever version available. We plan to make money from some specialty features aimed ...
48 comments
[ 2.6 ms ] story [ 128 ms ] threadWhat kind of AST are you using? Can you translate code between languages?
Also, what do you think of the Graal/Truffle project?
(BTW I'm getting 404 at https://useoptic.com/docs/what_is_optic/what-is-optic/ )
We can't translate code between languages but we do the next best thing by moving the meaning of that code around. ie backend is Flask and frontend is JS. We can generate JS code that connects to the Flask API automatically.
Graal is pretty cool, and I'm happy to see a big org backing a polygot tool. I'm most interested in Truffle and looking into ways it can make Optic better. Have any ideas?
Thanks for catching the 404! I'll look into it ASAP
Also, a couple of other question that come to mind:
1. Jared Forsyth gave a great talk on avoiding breaking changes as a library author by using code-mods - would Optic be able to wrap up some of the changes it makes to my library code as a code-mod that I could ship with a new version of my code to auto-upgrade all my dependents? That could significantly help with some of the higher-churn libraries (some people are iffy about this though).
2. Have you considered using something closer to lsp for the editor protocol? Emacs support would be nice, and that'd be a good way to jumpstart the implementation.
3. It actually doesn't look too bad to implement the plugin for e.g. even without lsp (https://useoptic.com/docs/authoring/adding-optic-support-for...). Looking at the 'search' event though, why use `///` instead of picking up `cmd-f` (or `C-s` in emacs)? I was a bit confused about the interaction model.
Optic looks pretty interesting, best of luck!
1. We've been exploring migrating one Optic skill to a newer version. So far it seems promising for the cases you'd expect a code-mod to help with, but if you completely change the API and deprecate some stuff it can point out the changes but won't be able to help you -- no magic here.
2. I looked into language servers (lsp) but the implementation was a lot more involved and I found bugs in some of the IDEs that claim to have full lsp support. As the protocol matures and we add more functionality to Optic I expect to see lsp become our main avenue for IDE/Optic interactions.
3. We wanted to make it really natural to ask Optic a question. We thought '///' would be unique and easy to remember, but we're open to changing it. It's nice that '///' also evaluates as a comment too because before we used it sometimes characters in your search would cause the AST parse to fail. We're open to any ideas here -- a lot of users have said it's funky.
Second, your Marvin sub-project, if I understand it correctly, is just awesome. Congratulations.
In re: Truffle, and lang-to-AST conversion in general, I've taken a winding path over the better part of the last twenty-odd years. The closest thing to a serious idea I have to offer is this: Use Lisp as your inter-language AST.
I'm grinning but I'm not quite joking. ;-)
Think it through. What are the pros and cons?
(I can't help but mention, I'm working on a system that may one day "eat" other software and translate it into essentially an AST. The AST is actually just code in a language called Joy that resembles Forth+Lisp. The language is very simple and can be implemented on top of other languages in a few days. So I can (eventually) read in other code to this Joy/AST, clean it up, then run it on whatever substratum makes sense for a given use-case. Joy code is easily refactored, and can be fairly easily converted to logical relations and then you can use e.g. Prolog or miniKanren to search for optimizations. This is sort of "one step beyond" type inference, moving into the more esoteric but useful things where you can do partial evaluation and "super-compilation" (chase down alternate control-flow paths) to improve the abstract code and emit specialized e.g. assembly or LLVM IR or JVM bytecode or whatever.
My motivations are two: I want to teach normal people to program, and I want to contract the total amount of [distinct] software in the world, eliminating incorrect code and collecting, refining, and curating all correct code into a single unified codebase.
The Joy notation is easy enough to teach to normal people. I've created a simple GUI that presents a Joy environment where users can build new commands (Joy code). The system uses type inference to prevent the user from executing commands that won't work with the current system state, and to prevent the construction of new commands that won't work at all. It's impossible to make incorrect code in this (virtual) world.
The Joy language is highly-factorable, enabling a kind of "semantic compression". It's also Categorical (in the sense of Category Theory), meaning that programs in Joy represent abstract computations over categories and can deliver multiple correct programs for the same Joy expression. For example, the type inferencer works by evaluating Joy expressions over a category of stack effects and type variables to generate the stack effects of the expressions. Conal Elliot has a paper "Compiling to Categories" where he's exploring this sort of thing using Haskell, converting it to a "point-free" Joy-like notation and then implementing categories for dataflow diagrams and circuit descriptions and other cool things, to produce those all from the same original Haskell code.
I'm hopeful that I've got something here. :-) I've been sitting in a room in front of a computer for about a year and I'm just at the point where I have to go out and talk to people about it. Egad.
Anyhow, that's part of why I'm so impressed with what you've accomplished (and are no doubt going to accomplish!) I look at what you've done already and I think, "Daaaamn, this dude is running circles around me." Makes me feel sheepish, but I'm stoked for you! You're bringing the fire down from Olympus my Promethean friend, don't doubt it.
If I'm understanding your idea, it sounds like you want to use a compiler to a) collect all the useful/best control flows into some kind of library that you then b) use to generate code when compiling any other language. Like an ouroboros of compilers. Then you want to have some simpler programming interface on top of that which can be expanded into great code?
In my notebook I've been calling Optic an up-piler because it turns raw code from multiple languages into the same higher level abstractions. So on a spectrum of processor instructions -> ideas it's further to the right than the host AST. Sounds kind of similar to what you've been working on for part A.
What confuses me about your idea is how I as a developer would benefit from it? Are you a drop-in compiler that will compile my code into something 'better'? Can you just take some code I wrote and refactor it into a better format that does the same thing as before? Will the way a project compiles change over time? I'm not sure if I want the nails in my log cabin going form 20mm to 30mm overnight. Shouldn't break anything but you never know...
Happy to chat offline sometime aidan@useoptic.com Think we'd have some fun convos
To directly answer your question, I don't think developers would benefit from it. If it works as intended, normal non-developers will be able to use it to build simple programs themselves. I could see this putting a lot of mid-level developers out of work.
The "ouroboros of compilers" is a side-effect. The simpler programming interface came first. I realized that other languages could be translated into the Joy language. Higher level abstractions are captured in Joy by defining new combinators. Ideally, the system would form a global Gödel machine with an "oracle" consisting of the world's best and brightest mathematician/programmers.
We use it for algorithmic trading here at KloudTrader and this is an issue we suffers from. Looking forward to adding Optic to our CI pipeline.
https://KloudTrader.com
Business rules engines would be another really great application. For a lot of IFTTT type of applications, if the rules can be compiled into finite state machines instead of simply interpreted, that would be a great boost for productivity and efficiency!
Absolutely agree there are some meta-programming applications for Optic. It's always been difficult to build graphical tools that generate code -- everyone basically reimplements string concatenation / static analysis over and over again and then they can't handle even the slightest manual change.
Ideally visual tools would tap into Optic API to parse the code, display it, then press user changes back into code. This would allow users to add custom code as well and not break your abstraction. Our source of truth is code so there's no dsl or model saved on your end anymore.
KloudTrader looks great. Let's chat offline about your requirements and I'll make sure to document / open some of the functionality you'll need.
I'm particularly interested in VS Code and perhaps Intellij out of the gate
All the plugins do is forward the state (col/row and filepath) to the local Optic server so they're usually <70loc. If you wanted to build support for another one we wrote a spec that all the plugin authors relied on during their development https://github.com/opticdev/optic-editor-plugin-spec/blob/ma...
We've been working really hard to build tooling around this because the first few versions were hard to learn/use. Would love to hear any feedback you have on the teaching process. We don't think Optic will take off until we get this just right.
https://useoptic.com/docs/authoring/writing-lenses
Other than that, one vertical usecase that would be super appealing would be the ability to automatically generate (and maintain) api servers ontop of AI modules. As in if we had a tensorflow model, can we automatically create an endpoint to serve live traffic? I guess more generally, if we can define the JSON input to a function (any python function doesnt have to be AI model), Optic can write all the boiler plate microservice for us!
If you can put together an example of the kind of code you want Optic to write for each module I'm happy to help you guys create Optic skills around it. There's also information in our docs on how to do this yourself. If you generated the routes with Optic you'd also be able to consume them easily and keep all your code synced as you update the input your models require.
How often do you find your team writing these microservices / tests?
I created the VS Code plugin for Optic: https://github.com/opticdev/optic-vscode-plugin. It was actually my first time writing a text editor plugin, but between the VS Code docs, Optic SDK, and Optic plugin spec, it was surprisingly easy. The information just gets sent through web sockets under the hood, but it was all just basic javascript callbacks and some VS Code specifics for finding ranges, etc.
Optic itself isn't particularly useful for me as a solo developer building small website frontends, but I must say I was a little jealous of those that this is geared towards because even developing the plugin gave me a couple of those "magic" moments. Getting to make a change to just some javascript object's property and then get to keep it in sync was pretty satisfying.
It would also be cool if this could help generate code in the best-practice style defined by third-party libraries. For instance, if the Redux team defined the canonical way to define a selector, it would be handy to be able to have Optic use that information to generate selectors for me. This would take something like create-react-app and break down the best practices into more bite sized chunks.
Really excited about this!
If you haven’t read about our Optic Markdown it’s designed for what you’re describing. A team could specify their conventions and best practices in Markdown and Optic will use the example code in the generators.
Do you see any other areas this kind of tool would be useful? Happy to chat about our roadmap for supporting other languages.
Thanks!
I wonder if the parsing, schemas and the transformations etc. could be a starting point for an advanced neural network AI programmer. Perhaps if you could combine it with an interactive system that converts text to schemas.
For example, say you had a way (a skill?) to convert a schema to a web form. Then the AI would need to handle speech-to-schema-change such as "add a field for Middle Name". Then perhaps there is a skill to go from the web form to a save data request and another to accept the request and another to update the database schema etc.
But the AI might come more in handy for interpreting more complex schemas such as nesting, grouping or workflow etc.
But besides an advanced domain specific AI it seems like this foundation of parsing, schemas and transformations would be a good start for speech-based or other higher level program generation tools. I can envision a web page that generates a CRUD application from mobile front end to Firebase back end and then spits out the code that you can then add more advanced features to.
So it seems a smart strategy to embrace open source because that can allow other developers to add mappings that you can leverage for your own app or program generation tools which you can sell.
To me the interactive web-based CRUD application generator is low hanging fruit but more interesting would be to combine your tech with a Google Duplex style interactive speech based AI programmer front end. Probably only Google and a relatively small number of companies could pull something like that off though.
I'd love to see a 2-way code generator built on Optic for a simple crud builder like you describe. It could let people choose the libraries they want to use from the list and design their schemas, routes and queries at some higher level of abstraction. Unlike the current generator tool you could actually make changes to the high level models and get Optic to update the underlying code for you even after you make changes.
That whole field could be huge and I think Optic's major contribution will be abstracting away all the dirtiness of code gen. People who build this next generation of tools could interact purely with Swagger-like-JSON.
Is this something you'd be interested in collaborating on?
A two-way code generator for CRUD applications is a lot more realistic for a start. I am interested in trying to collaborate on that, although I can't promise how much time I will have because of other obligations.
But it would be fun to give it a start at least. You started describing something, maybe if you want you could elaborate a little bit more about how you think this type of tool would work in a document online somewhere or just in a reply. It would be fun to play around with Optic with that type of goal. I will need to read the docs and experiment some to make sure I understand the system better before I get too far along in coding something. Mainly I use Node and JavaScript these days so theoretically that could work for a web-based CRUD builder.. I guess by talking to the Scala server over REST? I will need to read your docs.
Optic React / JS for all its GUIs. We’re planning to have microeditors that pop up and help you work with specific kinds of code so this might just run from within Optic somehow.
All fun ideas to brainstorm. Let’s take it offline. Email me at aidan@useoptic.com
Excited to chat / collaborate
The objective of the training is bootstrappig a prettier. String template (pattern in the code) for each node type.
So we can learn things like arguments are separated by “, “ but if the total # of chars of all of them is greater than X it’s more common to use “,\n”. Stuff like that.
We had LSTMs for a while but they performed worse than a decision tree with a few key features as input.
My favorite recent episode was futureofcoding.org/episodes/26.
Is there a package manager or common repository for sharing and searching Optics skills?
You can also resolve them locally by putting them in your optic.yml file
You can add a skill and containing as many transformations and lenses as your want.
It’s explained at the bottom of these docs https://useoptic.com/docs/authoring/writing-skills
It’s working here so not sure what could be causing this besides an issue with our CDN
The whole idea is still not fully clear to me. After watching the video, it initially looks like some kind of code templating engine. From further reading, I seem to understand it also links many "instantiated" templates together, so that changing one of them can change others; is that correct?
Also, one thing that is still not clear to me: do I have to build such templates by hand, or does the engine somehow automatically infer / detect new templates from an existing codebase?
We have plans to release on windows and Linux later this year. Currently investigating cross platform kits for the GUI.
The engine should run anywhere a JVM does but I haven’t written any docs on that yet. Is a scriptable version of interest to you?
Your initiation is basically right. Optic is a code generator with templates. What’s special about it is that it uses your existing code as input so there’s no manual specification done. You can also manually update the generated code and it still works with Optic. All this combined allows you to “sync” different parts of your code that are related as you said.
You have to specify the skills by hand or import them from our registry. https://useoptic.com/docs/authoring/writing-skills
The JVM is somewhat allergy inducing, but I think Graal may solve a lot of these issues. Optic is written in Scala because a) it's a great functional language and b) the JVM is super portable. If we can get Graal working nicely in our builds, binary sizes should drop markably and performance should improve.
Glad you are doing this!