A visual language for composing interactions
The Concept
firmly
performer left foot stomp
Elements used
physical concept
actor action effect
elements clarifications
firm
movements
Concept flow diagram
up
/ pitch
performer right hand cello bow
down up / down
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
A visual language for composing interactions
Concept spatial layout
performer
left hand right hand
press
key move move
combination hand joystick
MIDI
key pad joystick
device + buttons
start/stop select shape
sound preset sound
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
A visual language for composing interactions
The physical implementation
Elements
interface
&
hardware software output
physical
objects
Directions of communication
one-way
both ways
Types of connections
wired protocol
wireless protocol
software protocol
Types of streams
audio data stream buffer
video event spectral
Example physical implementation
audio usb
microphone sound card laptop junXion
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
A visual language for composing interactions
Role of input / Input labels
input data stream parameter of process enable/disable trigger
Example of input labels
laptop
Process types, Math & Arithmetic operations
+ - / *
add subtract divide multiply
% 22
modulo number box
Example
foot pedal
*22 volume
sound card
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
A visual language for composing interactions
Conditional
Relational operations
> < >= <=
greater than less than or
greater than less than or equal to equal to
== !=
equal to not equal to
Logical operations
not and or xor
Comparison of data streams
min max
min max
Check whether value changed
Δ?
change in value
Example
accelerometer Δ >10 play sequence speaker
Ableton Live
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
A visual language for composing interactions
Flow control (modal/states)
external factor external factor external factor external factor
inverse
gate switch inverse switch multi switch multi switch
Input to output range
max max
min min
range
Split
375 max
250
125
min
split 0 within range
Change character
max max step size max max max max
min min min min min min
invert quantise segment nonlinear
Constrain
max max max max
min min min min
block clip wrap fold
Buffer
buffer
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
A visual language for composing interactions
Example of flow control
button
toggle
accelerometer sound
gate quantise
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
A visual language for composing interactions
Legend
operator filter
type accumulation
Filter
filter filter filter filter
windowed past input/output reset
filter filter filter filter
Δ
smoothing change change accumulation
Analysis
Amplitude/bandwidth/range analysis
analysis analysis
min max
min in stream max in stream envelope peaks
analysis analysis analysis
onsets spectral power
Example of filtering
filter 1 600
α=0,01
accelerometer sound
0 30
smoothing nonlinear
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
A visual language for composing interactions
Resampling
resample upsample downsample
reset
Time
3x
3s
transition timer delay repeat
Button related processes
+1
toggle gate trigger switch counting
… … 1 3
3s
4 2
multiclick timed multiclick combinations sequence timing
Mixing signals
preset
matrix sensor fusion interpolation implicit methods
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
A visual language for composing interactions
Composing Interactions
an Artist's Guide to Building Expressive Interactive Systems
by Marije Baalman
https://composinginteractions.art
(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
