Two years back or so I was trying to build a new user interface for the ZynAddSubFX synth and getting the level of customization I wanted while being able to embed the interface cleanly into plugin hosts was difficult. I could have gone the route of producing a series of raster based assets and providing fixed locations for everything, but since I had different devices with rather different resolutions I wanted something to be more flexible than that. After starting with Qt prototypes I borrowed the ideas of Qt's QML and applied them to a ruby based widget system (I would have gone with com
mon-lisp + CLOS, but that's not the best route to entice contributors), a nanovg/OpenGL rendering system, and a open sound control based messaging layer between the GUI and backend. Initially there was a GLPK based layout system, though that was unfortunately scrapped due to the runtime cost.
The end result is MRuby-Zest and it's a system that's reasonably efficient, embeddable, and in my opinion fast to iterate upon designs. The last note is mostly due to the fact that the system can hotload code updating existing widgets in a live GUI. That hotloading combined with the fact that widget code is all ruby (rather than building off an existing C/C++ toolkit) has made working with the framework pretty fun for myself. Overall the project is still pretty young and somewhat tied to the Zyn GUI (named Zyn-Fusion), but I thought some of the design choices and trade-offs might be interesting to the broader HN community.
Outside of the technical choices this work is some of the first open source work that I've seriously tried to get community funding for. After working on the software for a few (~3) months fulltime I was able to put binaries up for sale and I've managed to get ~200 sales for my efforts.
I've seen Crystal a few times in the past, though I haven't used it for a project so far. Looking at their website and issue tracker it looks like it would be rather difficult to be able to hotload code if Crystal was used compared to MRuby (based upon the notes that a Crystal REPL isn't a primary feature as a compiled language). Additionally I can't find too many details (though I'm sure they're there) about the runtime requirements imposed by Crystal applications. One of the nice things about the MRuby implementation is that it makes it easier to combine all parts of your program (C extensions and ruby compiled to VM instructions) into a single binary with minimal external dependencies.
Yep, that was one of the things that I got working early on. It's used a fair bit in the codebase, though it's not necessarily all that apparent if you don't know where to look.
This is one of the features of the overall framework which may be a bit under-tested at the moment. I've fixed a number of bugs in the system, though you certainly can get into odd edge cases when updating bindings during widget creation or destruction.
> Yep, that was one of the things that I got working early on. It's used a fair bit in the codebase, though it's not necessarily all that apparent if you don't know where to look.
hmmm... but what I don't see is stuff like `onColorChanged: doStuff(color)`. Basically, the event system ?
- Embed-ability: Qt can be used to build a plugin interface, but when a plugin host (which may be implemented in some version of Qt) loads multiple different plugins (which can be different versions), then it's possible to see conflicts with the global data that Qt creates. These problems are fairly hard for typical users to report and debug. IIRC this can be worked around with static linkage, though that opens up a separate can of worms with respect to Qt.
- Licensing: For the initial fundraiser the plan was to release binaries of the GUI for a fee with the full source being released at a later date. Doing this was simplified by using custom tools which were almost all under the MIT license.
- Layout: QML tries to specify widget properties in terms of fixed sizes of the leaf nodes and then using bottom-up algorithms to place the parent widgets. The initial system in zest was to use a constraint layout framework to specify all system constraints without hard coding pixel sizes for any widget (other than the top level window as specified by the window manager). Incorporating this different approach did not mesh cleanly with QML's existing abstraction
- Remote Data: QML works great if you're going to hook onto an app driven by Qt classes and the associated signal-slot system. Unfortunately Qt's approach does not work in a 'realtime-safe' manner. That is to say that if it was deeply integrated it could cause faults where the audio processing would be delayed generating audio glitches due to the timing being incorrect. When working with the OSC interface that existed on the target app (ZynAddSubFX), specifying parameter bindings felt rather unnatural and once again, didn't mesh with QML's expectation of how events should flow.
- Javascript: Personally I'm not a fan of javascript, so developing in a different scripting language made the process faster in Ruby.
- QML's C++ API: At the time I started work prototyping the GUI I needed to occasionally work with QML classes from a C++ perspective and sometimes extend/create new classes there. The C++ API was under-documented and the API felt clumsy. This is something that the Qt documentors have likely resolved at this point, but at the time it was a limitation.
- QML's mental model: I have worked with Qt in the past, though not all that extensively. Qt and QML work very well if you stay within their own realm, but once you start going against the grain, you need to understand the internals fairly well. I found it difficult to learn about the internals and as such for a long development process which needed to have fast iterations, it seemed like a better fit to work with code that I was more familiar with even at the cost of needing to write it myself. In the end I think this was mostly justified by the relative small size of the 'framework' vs the size of the widget code within the overall Zyn-Fusion GUI written using mruby-zest.
I know there were a few other issues at the time that I decided it would be more productive to build a framework along the way, though those are the items that come to mind first.
that was a fairly interesting and extensive answer, thanks ! I'm asking because I'm working on an OSC based DAW built with Qt (https://ossia.io) and the route I went is to entirely isolate the UI event loop, the OSC / network event loop and the audio loop each in their own threads / groups of threads. Still have a priority inversion problem to fight here and there but overall I'm fairly satisfied with the result.
> I'm working on an OSC based DAW built with Qt (https://ossia.io)
Ah, cool. I remember siting in on part of the 2017 LAC workshop for i-score (i-score was renamed to ossia score, right?). Once you remove the embedding criteria Qt itself becomes a much more feasible option.
I haven't used score, much, though I should likely play around with it given the amount of OSC work that I've done around zyn.
oh.. then we've met ! I was the one giving the workshop, and had a good time with you people :) (and yes, there was a rename).
By any chance, have you looked into OSCquery (https://github.com/mrRay/OSCQueryProposal) ? It's a protocol we're currently a bunch working on and which is starting to make some inroads even in paid media software ; it greatly facilitates interoperation between multiple OSC-compliant software since it allows multiple software to easily query all the OSC nodes of each other through simple JSON over websocket communication.
This is the first time that I think I've seen the proposal. When I was building https://github.com/fundamental/rtosc I was pretty heavily influenced by the work done within liblo and libmapper. I personally really like their implementation, but like all of the options adoption drives success. With rtosc there is https://github.com/7890/oscdoc support (XML/XLST based) as well as a more standard json schema https://github.com/mruby-zest/osc-bridge/blob/master/schema/... . For the running application the discovery mechanism is in the vein of oscit where there are special discovery paths and everything is over UDP OSC.
One of the limitations for ZynAddSubFX is making discovery a multi-step process, since there is a very large number of OSC paths (if you count out all enumerations it's ~4,000,000). So flat listings don't really work and even if that's resolved the rest of the information needs to be presented in a compact/efficient format for the system to be performant. I'd say the OSCQueryProposal looks like it falls back on HTTP concepts and javascript conventions too heavily and adds more verbosity than I'd personally prefer. Looking over the options that I had considered in the past I'm looking at libmapper again for some possible future ideas and work.
One of the plans down the line is to create a few views as essentially "widget zoos". These will hopefully serve as illustrative examples as well as locations where the automated screenshot mechanisms can be used to collect pictures of common widget types.
Right now if you get the GUI running by building from the mruby-zest-build repo, then you'll be able to start the interface and it will load up MainWindow.qml as well as child widgets.
I meant music / tracks / synth examples ?
I would be ok to follow a tutorial about how to use the interface but having real life examples to tinker with is more fun ;)
At the bottom of http://zynaddsubfx.sf.net/ you'll see some examples of the synth in action from the community. If you're looking for particular walkthrough/tutorial video content there's a few videos linked in a playlist at http://zynaddsubfx.sf.net/support.html . Some work is being done to the manual to both update it and make it more user friendly (i.e. discuss what is available in the program and why you'd want to use it), though the linked pages only have the older docs at the moment.
This looks pretty interesting. I'd like to play around with it. Kudos to you.
Aside: Did you know the FUDI protocol(0) used in the pure data language is dead simple, if you wanted a faster bridge than OSC into/from that language.
Neat. I have seen a few talks about pure data, but they tend to mention transmission of data across the network, but not how they do it. In terms of message communications for audio tools (beyond just MIDI) I've heard OSC discussed a fair bit, LV2 atoms somewhat, and then a good variety of custom protocols used by a single application.
I'm biased towards OSC however since the binary serialization is very easy to manipulate in a realtime context (i.e. no locks, no dynamic memory). In fact I was biased enough that I ended up giving a talk at the linux audio conference about realtime safe OSC http://lac.linuxaudio.org/2018/pages/event/39/ (with some nice performance figures to boot).
It's cool to know what puredata is actually using under the hood.
Oh, thanks for the link. I'm about to read your paper.
I should clarify that when I said FUDI is faster, I was repeating the grumblings I've heard on pd's email list, despite not having benchmarked anything myself. With tcl's "everything is a string" (0) format, I could imagine this being contextual. But I am really curious.
You might find this kinda funny, a piece of pd's documentation regarding the osc objects:
"OSC is a complicated networking protocol (FUDI, as used in netsend/netreceive is simpler and better but less widely used). oscparse and oscformat make no attempt to deal with timetags or aggregates of packets, nor with streaming OSC. Also, no attempt is made here to clearly distinguish between the OSC address (symbols) and the following data, nor between blobs and lists of numbers - it is assumed that you know what types the message should contain. You can alternatively use the OSC objects from mrpeach which have more features than these."
I can't really blame them for their position. OSC has some really cool ideas, but the standards organization that was shaping the protocol ended up dissolving (somewhat) before all the rough edges could be sorted. I personally find it to be a nice serialization format with enough complexity to do what you need without it becoming all consuming.
26 comments
[ 2.9 ms ] story [ 73.6 ms ] threadTwo years back or so I was trying to build a new user interface for the ZynAddSubFX synth and getting the level of customization I wanted while being able to embed the interface cleanly into plugin hosts was difficult. I could have gone the route of producing a series of raster based assets and providing fixed locations for everything, but since I had different devices with rather different resolutions I wanted something to be more flexible than that. After starting with Qt prototypes I borrowed the ideas of Qt's QML and applied them to a ruby based widget system (I would have gone with com mon-lisp + CLOS, but that's not the best route to entice contributors), a nanovg/OpenGL rendering system, and a open sound control based messaging layer between the GUI and backend. Initially there was a GLPK based layout system, though that was unfortunately scrapped due to the runtime cost.
The end result is MRuby-Zest and it's a system that's reasonably efficient, embeddable, and in my opinion fast to iterate upon designs. The last note is mostly due to the fact that the system can hotload code updating existing widgets in a live GUI. That hotloading combined with the fact that widget code is all ruby (rather than building off an existing C/C++ toolkit) has made working with the framework pretty fun for myself. Overall the project is still pretty young and somewhat tied to the Zyn GUI (named Zyn-Fusion), but I thought some of the design choices and trade-offs might be interesting to the broader HN community.
Outside of the technical choices this work is some of the first open source work that I've seriously tried to get community funding for. After working on the software for a few (~3) months fulltime I was able to put binaries up for sale and I've managed to get ~200 sales for my efforts.
Have you thought about bringing in Crystal when it gets more mature?
Congratulations on the sale and good luck with the project.
Did you reimplement property binding ? That's the #1 feature of QML imho and I couldn't find an example in your source.
As mentioned at https://github.com/mruby-zest/mruby-qml-spawn/blob/master/mr... the properties form a dependency graph, so the property bindings should stay up-to-date when you change dependent properties.
This is one of the features of the overall framework which may be a bit under-tested at the moment. I've fixed a number of bugs in the system, though you certainly can get into odd edge cases when updating bindings during widget creation or destruction.
hmmm... but what I don't see is stuff like `onColorChanged: doStuff(color)`. Basically, the event system ?
Elsewhere in the codebase you'll see whenEvent style callbacks https://github.com/mruby-zest/mruby-zest/blob/master/qml/Val...
Most of the events, however, are handled through more traditional methods as that ended up being more comfortable when scaling to a larger program.
- Embed-ability: Qt can be used to build a plugin interface, but when a plugin host (which may be implemented in some version of Qt) loads multiple different plugins (which can be different versions), then it's possible to see conflicts with the global data that Qt creates. These problems are fairly hard for typical users to report and debug. IIRC this can be worked around with static linkage, though that opens up a separate can of worms with respect to Qt.
- Licensing: For the initial fundraiser the plan was to release binaries of the GUI for a fee with the full source being released at a later date. Doing this was simplified by using custom tools which were almost all under the MIT license.
- Layout: QML tries to specify widget properties in terms of fixed sizes of the leaf nodes and then using bottom-up algorithms to place the parent widgets. The initial system in zest was to use a constraint layout framework to specify all system constraints without hard coding pixel sizes for any widget (other than the top level window as specified by the window manager). Incorporating this different approach did not mesh cleanly with QML's existing abstraction
- Remote Data: QML works great if you're going to hook onto an app driven by Qt classes and the associated signal-slot system. Unfortunately Qt's approach does not work in a 'realtime-safe' manner. That is to say that if it was deeply integrated it could cause faults where the audio processing would be delayed generating audio glitches due to the timing being incorrect. When working with the OSC interface that existed on the target app (ZynAddSubFX), specifying parameter bindings felt rather unnatural and once again, didn't mesh with QML's expectation of how events should flow.
- Javascript: Personally I'm not a fan of javascript, so developing in a different scripting language made the process faster in Ruby.
- QML's C++ API: At the time I started work prototyping the GUI I needed to occasionally work with QML classes from a C++ perspective and sometimes extend/create new classes there. The C++ API was under-documented and the API felt clumsy. This is something that the Qt documentors have likely resolved at this point, but at the time it was a limitation.
- QML's mental model: I have worked with Qt in the past, though not all that extensively. Qt and QML work very well if you stay within their own realm, but once you start going against the grain, you need to understand the internals fairly well. I found it difficult to learn about the internals and as such for a long development process which needed to have fast iterations, it seemed like a better fit to work with code that I was more familiar with even at the cost of needing to write it myself. In the end I think this was mostly justified by the relative small size of the 'framework' vs the size of the widget code within the overall Zyn-Fusion GUI written using mruby-zest.
I know there were a few other issues at the time that I decided it would be more productive to build a framework along the way, though those are the items that come to mind first.
Ah, cool. I remember siting in on part of the 2017 LAC workshop for i-score (i-score was renamed to ossia score, right?). Once you remove the embedding criteria Qt itself becomes a much more feasible option.
I haven't used score, much, though I should likely play around with it given the amount of OSC work that I've done around zyn.
By any chance, have you looked into OSCquery (https://github.com/mrRay/OSCQueryProposal) ? It's a protocol we're currently a bunch working on and which is starting to make some inroads even in paid media software ; it greatly facilitates interoperation between multiple OSC-compliant software since it allows multiple software to easily query all the OSC nodes of each other through simple JSON over websocket communication.
One of the limitations for ZynAddSubFX is making discovery a multi-step process, since there is a very large number of OSC paths (if you count out all enumerations it's ~4,000,000). So flat listings don't really work and even if that's resolved the rest of the information needs to be presented in a compact/efficient format for the system to be performant. I'd say the OSCQueryProposal looks like it falls back on HTTP concepts and javascript conventions too heavily and adds more verbosity than I'd personally prefer. Looking over the options that I had considered in the past I'm looking at libmapper again for some possible future ideas and work.
Right now if you get the GUI running by building from the mruby-zest-build repo, then you'll be able to start the interface and it will load up MainWindow.qml as well as child widgets.
Aside: Did you know the FUDI protocol(0) used in the pure data language is dead simple, if you wanted a faster bridge than OSC into/from that language.
(0) - https://en.wikipedia.org/wiki/FUDI
I'm biased towards OSC however since the binary serialization is very easy to manipulate in a realtime context (i.e. no locks, no dynamic memory). In fact I was biased enough that I ended up giving a talk at the linux audio conference about realtime safe OSC http://lac.linuxaudio.org/2018/pages/event/39/ (with some nice performance figures to boot). It's cool to know what puredata is actually using under the hood.
I should clarify that when I said FUDI is faster, I was repeating the grumblings I've heard on pd's email list, despite not having benchmarked anything myself. With tcl's "everything is a string" (0) format, I could imagine this being contextual. But I am really curious.
(0) - https://wiki.tcl.tk/3018
You might find this kinda funny, a piece of pd's documentation regarding the osc objects:
"OSC is a complicated networking protocol (FUDI, as used in netsend/netreceive is simpler and better but less widely used). oscparse and oscformat make no attempt to deal with timetags or aggregates of packets, nor with streaming OSC. Also, no attempt is made here to clearly distinguish between the OSC address (symbols) and the following data, nor between blobs and lists of numbers - it is assumed that you know what types the message should contain. You can alternatively use the OSC objects from mrpeach which have more features than these."
https://media.ccc.de/v/lac2018-39-rtosc_realtime_safe_open_s...