Shaping sound
> Visuals shapes and forms translated into sounds
“The ‘Barcode Band Music Box’ we produced includes barcodes, formed by computer graphics, and the barcode reader. When the graphics of the barcode, which is printed on the music box, are scanned by the reader, the recognized barcode will reproduce its own sound of the musical instrument.
It is an instrument that communicates through the language of music. Therefore, a person can conduct music using only the barcode reader without a musical instrument. The making of the music box was composed of several different teams taking part as sound and video, design of the box, and the design of the graphics inside the box.” Directed by Kang Woon Jin, Designed by Lee Ha Lim, Crafted by Kim Yong Duk, 2012. A video from a live performance can be watched here.
“Colour a Sound is an installation that uses an overhead projector, a camera and a laptop to allow the user to draw whatever they’d like and roll it past a point and have it trigger sounds, player-piano style. Red, green and blue control three different octaves, or different sound sets (I use blue to control a set of 808 drum sounds in the example video). The rolls used to draw could be played forward or backward, and people could play previous compositions made from past users.” Blair Neal, 2010. More info here.
Image and Video to Sound. Sound Generator Test. Sonification of a snowFX video. Jeongho Park, 2015. Source Code.
“This installation plays the keys of a piano based on the movements and shapes of the clouds. A camera pointed at the sky captures video of the clouds. Custom software uses the video of the clouds in real-time to articulate a robotic device that presses the corresponding keys on the piano. The system is set in motion to function as if the clouds are pressing the keys on the piano as they move across the sky and change shape.” David Bowen, 2014. More info here.
The Variophone was invented by Evgeny Sholpo in 1930 at Alexander Shorin’s Central Laboratory of Wire Communication in Leningrad.
Unlike Avraamov, who shot still images of sounds on an animation stand, Sholpo used cardboard disks with circular images of combs with suitably shaped cogs rotating synchronously with a moving filmstrip. The advantages of the Variophone were in its flexible and continuous pitch control and vibrato.
It already incorporated one of the most crucial and necessary devices – a mechanism for the precise and continuous changing of the speed of rotation of the optical disk with the sound wave pattern i.e. a means of controlling the pitch with the possibility of synth. Andrey Smirnov’s “Graphical Sound”
The technological basis of his invention was the method of photo-optic sound recording used in cinematography, which made it possible to obtain a visible image of a sound wave, as well as to realize the opposite goal – synthesizing a sound from an artificially drawn sound wave. The method of Scholpo gives easier access to varieties of timbres. He doesn’t shoot still images of sounds on animation stand, instead using paper disks with a circular images of combs with appropriate shapes of cogs, rotating synchronously with a moving filmstrip. Exclusive benefits of Variophone are in flexible pitch control and vibrato.
Derek Holzer “A brief history of optical synthesis”
Moreon Variophone and other graphical sound synthetics inventions from the beginning of the 20th century in Russia in the book “Sound In Z: Experiments In Sound And Electronic Music In Early 20th Century Russia by Andrey Smirnov.
Andrey Smirnov’s website, Sound in Z exhibition’s catalog, Derek Holzer ‘s website, and Creative Digital Motion’s article.
Phono paper is a mobile application that converts drawings into sound. It is a format of graphical representation of sound based on the ANS synthesiser by Evgeny Murzin.
The sound is drawing on paper, then, in real-time it is analysed throw the camera device and turning into sound. More on project’s page and at Creative Applications’ article.
A sound controller that uses tattoo as a music score.
This is a special instrument that combines human body and robotic system into a single entity that is designed to automate creative process in an attempt to represent the artist and his instrument as a creative hybrid. The device consists of a railing with comfortable hand holders and two parallel, but offset from each other black lines’ sensors that move along the arm using a stepper motor. It is equipped with a 3-dimensional Wii remote controller that uses the OSC protocol in order to give a possibility of additional expression achieved by moving hand in space. VTOL, more info on project’s page.
DYSKOGRAF is a graphic disk reader. Each disc is created by visitors to the installation by way of felt tip pens provided for their use. The mechanism then reads the disk, translating the drawing into a musical sequence. Avoca.
> Sound generated from shapes and other graphic features
In 30-s Russia also great developments and achievements, both technical and artistic, were being made based on the technique of synthesising sound from light.
In those years in Russia there were different trends of work, that grouped several Russian creators working in new techniques to use graphical characteristics of shapes to synthesise sound (from Ornamental Sound to Cutting Paper or Spectral analysis decomposition and re-synthesis).
Collection of ornamental soundtracks. Boris Yankovsky, Moscow 1931.
“While most inventors of electronic musical instruments were developing tools for performers, the majority of methods and instruments based on Graphical Sound techniques were created for composers. Similar to modern computer music techniques, the composer could produce the final synthesised soundtrack without need for any performers or intermediates. Smirnov Andrey, 2011 “Graphical Sound”.
More: Smirnov A., Pchelkina L., 1917-1939. Son Z / Sound in Z. PALAIS / Palais de Tokyo Magazine, Paris, 2008, №7, pp. 66-77 (English and French).
Smirnov A., Pchelkina L., GENERATION Z. Booklet of the Generation Z exhibition. OSA Archivum, Budapest, 2011. (English and Hungarian).
Left: Avraamov’s first graphic scores experiments
Right :Boris Yankovsky’s mathematically calculated waves for sound synthesis
In 1931, Nikolai Voinov, (from Avraamov’s group Multzvik) started to develop his own research on graphical sound. Instead of drawing on the film, Voinov developed a new technique by cutting paper shapes. This paper shapes were used for the sound synthesis and its carefully calculated sizes allowed a new “surprising efficient level of control over the dynamics of sound”. Video documentary explaining Voinov’s techniques.
In Light Music (Lis Rhodes,1975) two projectors placed opposite each other project a sequence of flickering straight lines, where its graphic characteristics (spacing, thickness, colour) determine the sound (frequency, amplitude, tonality, .)
Like Norman Mclaren experiments, Rodes drawn the lines with a pen on the optical strip of the film, Rhodes was in fact composing a music/noise piece by drawing a score of lines. The soundtrack is exactly the projected image.
”What you see, is what you get”. Lis Rhodes
vimeo.com/video/47856075
video from inside of Light Music installation by a visitor – Michael Lewis
In 1931, Nikolai Voinov, (from Avraamov’s group Multzvik) started to develop his own research on graphical sound. Instead of drawing on the film, Voinov developed a new technique by cutting paper shapes. This paper shapes were used for the sound synthesis and its carefully calculated sizes allowed a new “surprising efficient level of control over the dynamics of sound”. Video documentary explaining Voinov’s techniques.
Light music
In Light Music (Lis Rhodes,1975) two projectors placed opposite each other project a sequence of flickering straight lines, where its graphic characteristics (spacing, thickness, colour) determine the sound (frequency, amplitude, tonality, .)
Like Norman Mclaren experiments, Rodes drawn the lines with a pen on the optical strip of the film, Rhodes was in fact composing a music/noise piece by drawing a score of lines. The soundtrack is exactly the projected image.
”What you see, is what you get”. Lis Rhodes
vimeo.com/video/47856075
video from inside of Light Music installation by a visitor – Michael Lewis
youtube.com/embed/ts5uT0Pdj4c
Lis Rhodes Light Music report for Tate modern
In 2012 “Light Music” was exhibited at Tate Modern in London as part of the event series “The Tanks: Art in Action”. In the report video for Tate Modern she talks a previous film she made, in 1971, which was the predecessor for “Light Music”.
The film is “Dresden Dynamo”, it was made without a camera, and the “sound is exactly the image and the image is exactly the sound”. Using the same “drawing on film” technique allowed that the synchrony between image and sound, instead of being added later, was “sync from the moment it was made, exactly”
more info: “Lis Rhodes’s ‘Light Music’ Challenges Perceptions of Film” and “Light Music on Tate Modern“
Light turntable
I always like to match together projects from different times, different technologies, that at the end share common ends. After the Russian pioneers from the 30s and Lis Rhodes in the 70s we make a jump to 2013 to “Light Turntable”.
It is a “Real time experiment with video, audio and RGB led light.”
A record with strips of colours is rotating in a turntable, a camera in the top captures the colours of the strips and that information for the sound manipulation.
vimeo.com/video/61756008
Things Happen.