noah adler

As I’ve explored more and more audio and visual software, I’ve had my interest piqued by patchbay style graphical languages, such as Pd and Max/MSP. Particularly in the VJ community, softwares such as Salvation and VSXu have made it easy to assemble powerful sound-reactive visuals, and tweak them on the fly using the same interface.

My last attempt to explore this subject dealt with visualizing Python object graphs for making visuals. I consider it a qualified success, but Python is a funny language. The weak typing and late binding make coding extremely flexible and mutable, but unfortunately also leads to a dearth of useful metadata. Contrasting this to .NET, which has metadata oozing out everywhere, I decided to try a similar experiment with that platform.

Objects in .NET tend to store references to other objects, forming a graph in memory. Connected portions of this graph are traversed at various times, such as during serialization, but being able to see a visual depiction of the graph could be really nice for some high level control. The best part is, due to the metadata, the visual depictions can automatically deduce what controls and connections of the underlying objects should be displayed.

2865My prototype uses WinForms, simply because it is convenient. A list displays several .NET types which can be instantiated by dragging them onto the main canvas. In this primitive version, only classes with default constructors can be created, but it’s enough to get a picture of the potential. Objects which implement a custom IModule interface can also have custom WinForms controls added to their visual module.

All the public properties of a particular object are retrieved via reflection and listed on the left edge of the module. These can be connected to other modules, as long as the types are compatible, and these connections will be displayed via arrows. When the connection is made visually, the system calls the property’s setter (again via reflection) with the linked object as the value parameter. A properly designed object should then have a reference to the linked object, which it can internally call for whatever it wishes.

There are a few caveats. Most notably, only reference types will properly ‘connect’ currently, since value types are simply copied. Therefore, for my beginning experiments, I created a class Scalar, which is basically just a referencable holder for a float. Furthermore, Scalar can be subclassed in order to implement IModule in different ways: for instance, a value controlled by an on-screen linear slider, or one controlled by a text box. However, it would still be nicest if regular value types could be used directly. Why is this a problem?

There seem to be two fundamental approaches to module-wire computation systems. The method which I’ve used so far in my systems, in which the module objects store their own data, including connections to other modules which they can call at any time, could be termed a data pull system. The method embraced by Pd and others takes a different route, forcefully pushing the data from into a module, getting the result, and transferring that to the next module. The difference can also be thought of as where the data is kept: in the former, it’s stored in the modules, while the latter it’s (conceptually) in the wires themselves.

The push method clearly obviates the problem with value types mentioned above, as they are constantly being copied to their successors in the signal chain. Unfortunately, just when these copies should be made is another unanswered question. One of my big frustrations with Pd was its inherent update rate tied to the audio frequency. This is great for audio, and solves the question of when to push data for that specific domain, but it essentially amounts to a magic number for any other type of system. If I want to create a visualization system that has nothing to do with audio, why should the audio driver rate matter at all? Figuring out an appropriate time to push value types remains an open problem for me.

2870The WinForms prototype was a fun experiment, and verified .NET’s feasibility to me for this sort of project. I personally hate coding for WinForms, though. It’s convenient for the standard Windows interfaces, but patching connections between modules isn’t such an application. The custom control rendering is hackish, sloppy, and slow, because it doesn’t really fit in with the traditional application model. Additionally, I love Linux and Mono’s support for WinForms, while rudimentary, still leaves plenty to be desired. Its underlying drawing layer, though, known as Cairo, holds much promise. After some hacking around I got a nice little test application rendering some vector graphics onto an OpenGL texture. My plan for the next step is to redevelop the patcher using Cairo to draw the main interface on top of an OpenGL window which can be used render 3d scenes, controlled by the patchbay interface in real time. It should be cool.