Visual node based programming
a.k.a. dataflow graphs, patchers, visual coding, flow-based programming
June 13, 2020 — February 12, 2022
Graphical dataflow programming languages, a.k.a. node-based visual programming. Probably descended from flow based programming or at least dataflow programming. A specific kind of DSL, with a visual functional graph interface. There is much crossover between these and the new generation of low code development systems.
For the record, I hate programming using a purely visual dataflow, although I do like visualising my programs this way. The patchers that offer only a GUI workflow are arse-backwards for my work process, forcing you to click a million times to do every simple thing. Code representations of programs are prime. If I lose this I am crippled. Purely visual methods seem simple to the sort of people who prefer clicking on 100 buttons instead of tapping 100 keys. These are the same kind of people who will drive their cars until the rims are on the road rather than take half an hour to pump up the tyres.
There is a place for visual programming, IMO. Either
- as a visualisation of the code, or
- with a bijection between code and visual representation so you can use whichever modality is convenient, i.e. so that this enhances the world of the programmer.
Only a few of the systems here make that cut.
This is popular in audio, visual multimedia in systems like VVVV and Quartz Composer, where we generally refer to “patcher” programming styles. Also seen in web api programming systems like nodered.
Possibly the most important application now is the neural net paradigm that treats neural data flow as a dataflow graph.
Graphical dataflow is not the only way we can program visually. The hierarchical tree-like structures of teaching tool scratch or art lisp project fluxus take a different approach. There are surely others; I’m not sure which are useful.
1 Luna
Visual patcher programming is being made rigorous by projects like Luna, which translate general-purpose languages back and forth between equivalent graphical and textual representations. This is AFAICT a bijection approach where both visual and code representations are first-class citizens.
2 Nodes
Nodes is a JavaScript-based 2D canvas for computational thinking. It’s powered by the npm ecosystem and lives on the web. We take inspiration from popular node-based tools but strive to bring the visual interface and textual code closer together while also encouraging patterns that aid the programmer in the prototype and exploratory stage of their process.
They have probably the best explanation of the whole dataflow patching mode, for the outsider:
One common differentiation between visual programming tools is the level of abstraction they operate on. Some tools, like Origami for example, are meant to prototype interactions of user interfaces and not much more. Others — like Pure Data for instance — are meant to be very low-level and let users craft their own environments. Some — like UE4 Blueprints or the Unity Visual Effects Graph — are embedded and part of a larger system (e.g. a game engine environment).
Nodes is more directly comparable to Cable.gl, FlowHub or MaxMSP. It does come with some batteries included but there are fewer preconceptions of what you use it for. However, unlike most of them we adopt a more programmer-first approach where the nodes themselves are generally a lot more high-level and users are encouraged to peek inside, copy and modify, and write their own. While you could drop to a lower level and express language-level semantics like if-checks and for-loops, or math concepts like add or multiply on a node level we generally don’t feel that is a productive way to work and prefer to write that logic in a textual programming language. This might feel conceptually familiar if you have used NoFlo.
3 Noisecraft
NoiseCraft is an open source, visual programming language and platform for sound synthesis and music making, with the goal of creating a community for the open exchange of musical ideas. The design is loosely inspired by PureData, Max/MSP as well as other visual programming languages. NoiseCraft is built on technologies such as the Web Audio and Web MIDI APIs, and runs in a web browser. The plan is for the app to remain free to use and ad-free for the foreseeable future.
maximecb/noisecraft: Browser-based visual programming language and platform for sound synthesis.
Example: April 14th (Aphex Twin) additive sines “vocalised”.
Because it is weird, also noted as an audio software framework.
4 Userland
userland applies dataflow visual programming to several surprising domains:
Userland is an integrated dataflow environment for end-users. It allows users to interact with modules that implement functionality for different domains from a single user interface and combine these modules in creative ways.
The UI works as a series of cells. Each cell can be set to a different mode, and each mode implemented as a separate module. There are currently three different modules:
- spreadsheet - basic spreadsheet-like behaviour, activated by typing formulas starting with an equals sign (e.g. =1+1, =A1 * (3/2))
- shell - posix shell mode, where each cell represents one command and its output, activated by typing shell. As the cell switches to shell-mode, it displays its current directory and allows commands to be entered.
- synth - synthesiser mode, activated by typing commands starting with a tilde (e.g ~triangle 220). Pressing Enter when a synth cell is focused will start/stop the audio wave.
5 Origami
Origami Studio is a user interface design tool that uses visual patcher programming:
Great, slightly backhanded pitch by developer Julius Tarng:
I believe the best way to design something is to prototype it. I often prototype many ideas as quickly as I can, as to assess many different options before making a decision. Fun tools help me move quickly, since the more fun I have, the more ideas I want to try out. It doesn’t matter if the tools are esoteric or niche. For example, Origami Studio (whose noodles cause many to balk) is one of my go-tos when designing products. Even though the learning curve was extremely high and debugging can be a brain-melter, slapping down patches feels viscerally fun! It reminds me of playing the Incredible Machine when I was younger. It doesn’t matter if the contraption was contrived if you put the ball in the box—often, that was the best part. I look for libraries that evoke a similar feeling when programming. It should let me create new pieces easily, and mix them around to create new combinations.
6 Flume
Flume is a framework for creating flow-based “patcher”-style programming languages in javascript. (Does it do Javascript FRP?)
Create your own visual programming language, while guaranteeing that users can’t create invalid logic. Colour-coded ports make this concept easy and intuitive for end-users.
Source code looks abandoned?
7 KNIME
KNIME seems to be a low-code data analytics platform. It looks like this.
That is all I know for now.
8 Cables.gl
cables.gl, mentioned in javascript graphics
cables is your model kit for creating beautiful interactive content. With an easy to navigate interface and results in real time, it allows for fast prototyping and prompt adjustments.
Working with cables is just as easy as creating cable spaghetti:
You are provided with a given set of operators such as mathematical functions, shapes and materials. Connect these to each other using virtual cables to create the scene you have in mind. Easily export your piece of work at any time. Embed it into your website or use it for any kind of creative installation.