Synesthesia is a condition where one of the senses is coupled with another, where people report of hearing images, of words having colors or of tasting sounds. The translation between senses has been an ongoing source of inspiration for creators for centuries with responses being instruments such as the color organ where a note triggers a visual experience. Even prior to the invention of electricity, there were have been scores of attempts to create color organs. Some had popups, opened curtains to show colored panels or later with electricity, turned on projected colored lights when a key was struck.
Along with these elaborate inventions, various artists, scientists and thinkers have sought correspondences between the elements of color (hue, saturation and value) and those of sound (pitch, amplitude, and tone color) developing color scales such as the one as shown here.
Read more on the history of visual music machines here:
Painters such as Kandinsky have also been captivated by the translation between the visual and the sonic domains. His paintings have been described as having visual “chords”. Working in the early 1900s he often described his works as compositions, symphonies and harmonies.
Today this exploration continues in the medium of software and sound visualizers have become ubiquitous, shipping with almost every music player and app.
Sound visualization/synthesis works:
- Joshue Ott + Kenneth Kirschner release composition systems as apps that combine visuals and sound.
- Or the other way.. Zack Dunham makes sound from visual patterns in Found Rhythm.
- Amanda Ghassaei makes interactive sound sculpture that are synthesizers that respond to ambient light.
Software Instruments:
Software instruments can also richly tie visual sensation to the auditory or musical. An instrument however is a tool and demands interactivity. It must be able to be played and when manipulated, yield a predictable result.
- The Manual Input Work Station, 2004 by Golan Levin and Zach Lieberman is an instrument that interprets hand gestures and finger movements across the interface of an augmented overhead projector.
- Sonic Wire Sculpture by Amit Pitaru creates sound via drawing in 3D space.
- Keyboard instruments: A classic of this genre of work is the keyboard instrument with Patatap being a particularly refined example. As you try it, concentrate of the transitions and interaction design.
- Drum Circle by Andi Mcc that is comprised of a visual system for composition.
- SUFI PLUG INS is an interdisciplinary project by Jace Clayton and is a free suite of Ableton add-ons that are based on nonwestern notions of sound & a poetic interface. This brief explanation video shows them in action. Download them here.
- TransProse is a project by Hannah Davis. It is the first iteration of a program that finds different emotions throughout text and programmatically creates music with a similar emotional tone.
- PIGS is a performable, improvisable instrument by Amy Alexander. It uses silent percussion to create expressive, improvised cinematic performance. It can be used with performance interfaces including iPads, Leap Motion controller, and silent MIDI drums. Real-time gestures are drawn on the iPads and Leap Motion, and can then replayed with variations by striking the MIDI drums.
- Cory Arcangel, video instruments. Here is Schonenberg composed algorithmically with with cat videos.
For more of these see this playlist.
Other instruments of note:
- Imogen Heap‘s software gloves are a new interface for performance and composition.
- David Rokeby’s Nervous System (1982-1991) is a classic and very early computer vision work where the body is the controller.
- Alison Parish’s writing instruments (warning! these are not sound but take on the principles of musical instruments).
- Todd Anderson’s Chrome Extension, Synth Typer. Uses your interactions with the browser to synthesis beautiful synthy music.
- Rebecca Fiebrink’s Wekinator software tool is a platform for the development of instruments using machine learning techniques. See a video on this here. And here is a tutorial on using wekinator in Processing.
- Here is Phoenix Perry on using Wekinator to make a gesture based instrument also using the kinect.
- Apps by Lullatone
Assignment brief: Software Instrument
Your task is to design a software instrument. The interaction design of your instrument should either be using the mouse or keyboard. Your instrument could trigger sound from files using the Soundfile Player class, or they could be synthesized using the sound oscillator functions. These functions produce different types of sound waves such as sine, saw, square, triangle or pulse waves (note their differing aesthetic qualities). If you are triggering sound files, carefully consider where you source your suite of sounds come from. Will you record them from a group of things – eg. a collection of recordings of people saying/singing/shouting a particular word? Or sounds sourced from water/hitting particular objects/the body? Or will your sounds be cut from a particular film/song/source media? Or will you construct a particular scale such as the western chromatic scale? Or a different scale (there are many).
What visuals will your instrument trigger? They could trigger photos, drawings, animation walk cycles or other visuals.
If you use a suite of files for you sounds and/or images, these must be stored and accessed from an array.
Note the example below on how to deal with triggering events only once from a key stroke (rather than over and over again when the key is pressed. This same technique could also be applied to a mouseclick.
Due week 10, November 2nd in class.
Come ready to play or have your software instrument played in class!
Dealing with triggering sound files just once.
The draw loop of Processing can make it challenging to trigger a sound just once, instead of over and over again. You will need to use a boolean variable to manage this as shown in the template below.
You will need to add this sound file to the data folder of this saved sketch.
//import processing sound library
import processing.sound.*;
//Make an instance of the soundFile class called soundtrack
SoundFile soundtrack;
//Make an instance of the AudioIn and Amplitude classes too!
AudioIn in;
Amplitude amp;
boolean isPlaying=false; //boolean to check if sound is playing
void setup()
{
size(400, 400);
//load in new file to the soundtrack object
soundtrack = new SoundFile(this, "soundtrack.mp3");
// loop sound
soundtrack.loop();
// stop sound to prevent it from playing automatically
soundtrack.stop();
// Create an Input stream which is routed into the Amplitude analyzer
in = new AudioIn(this, 0);
amp = new Amplitude(this);
// start the Audio Input
in.start();
// patch the AudioIn
amp.input(in);
}
void draw(){
background(255);
// draw an ellipse based on current volume level
float vol = amp.analyze();
println(vol);
noStroke();
//visualizer
fill(0,0,255);
ellipse(width/2, height/2, map(vol, 0, 0.5, 0, width), map(vol, 0, 0.5, 0, height));
//draw some play and stop buttons
// play button
fill(0,255,0);
ellipse(25, 25, 50, 50);
// stop button
fill(255, 0, 0);
ellipse(75, 25, 50, 50);
println(isPlaying);
}
void playsound() {
//check our file is not already playing
if (isPlaying == false) {
//play file
soundtrack.play();
//set isPlaying to true as our file is playing
isPlaying=true;
}
}
void stopsound() {
//check file is actually playing
if (isPlaying == true) {
//stop it playing
soundtrack.stop();
//therefore isPlaying is false
isPlaying=false;
}
}
void mousePressed() { //if the mouse is clicked
// and if the mouse is inside the play button
if (dist(mouseX, mouseY, 25, 25)<25) {
playsound();
}
//and if the mouse is inside the stop button
if (dist(mouseX, mouseY, 75, 25)<25) {
stopsound();
}
}