Engine Box #9

Brian Tuohy
Digital Media and Arts Research Centre
University of Limerick
Castletroy, Limerick, Ireland
0743518@studentmail.ul.ie

Abstract the IR sensor. The cause and effect is presented to the

This paper presents a detailed overview of “Engine Box
#9” – a new instrument for computer music performance.
The instrument takes the form of a wooden box, which is
equipped with an Arduino microcontroller and several
electronic sensors. The musician interacts with an Infrared
Range Finder (IR), Force Sensitive Resistor (FSR) and two
momentary switches to control the manipulative
parameters applied to an audio sample in the MaxMSP
environment.
Keywords: A Box, Arduino, Audio Freeze, Max/MSP,
Phase Vocoder, Pitch Shift, Time Stretch
1. Introduction
This instrument provides the musician with the opportunity
to explore microscopic sound. The intention behind the
sound processes offered is to take a short audio sample and
explore its minute details in order to present a vastly
altered output. This is achieved by implementing extreme
time stretching measures, which allow the musician to
slow the sample playback down to a point where it literally
freezes a frame of the sound without affecting the pitch.
The resulting sound then becomes the new audio source
for the musician to transpose as they wish within a defined
scale.
The design of the instrument was based on several
concepts, which we wished to consider:
• Instrument should offer causality during
performance.
• Should require attention of the musician.
• Should rely on simple interface to place emphasis
on performance technique.
With acoustic instruments, there is an intrinsic
connection between the interface and the sound engine,
e.g. the strings of an acoustic guitar generate sound when
they are vibrated. This is not the case with electronic
instruments – a connection must be established [1]. With
this conceptual connection, there comes a certain
responsibility to inform the audience of how the instrument
is being used to generate and manipulate sound. For this
reason, we decided to use two continuous sensors, which
provide a significant amount of visual evidence as to how
they are being used: an FSR and an IR sensor. The
musician’s left hand controls the level of force applied to
the FSR and the right hand determines the distance from
audience due to the visible movement and effort exerted
[1].
We did not wish to design an instrument that could be
left unattended and still generate sound. If a piano sits on
stage with nobody playing it, it does not create any musical
content [2]. This was a function that we felt needed to be
built into our instrument – the requirement of interaction
and attention in order to produce results. The instrument
will only trigger sounds when the IR sensor detects
presence within its range, when this presence leaves, the
sound stops. The IR sensor, in this sense, has gained
importance equivalent to that of a string on a guitar.
Figure 1. Engine box #9 Interface
An over-complicated interface could easily distract
from the musical matters at hand. For this reason, we
limited the interface to a simple box with four defined
points of interaction; Button 1, Button 2, FSR, IR (see
Figure 1.). With no visible circuitry or components, the
musician can concentrate on developing their playing
technique as opposed to worrying about technical matters.
Our intention was to create an instrument and package it to
be presented to a musician not a technician or programmer.
The technology behind the instrument is inside the box but
the musician is presented with everything they need to
interact with on a flat playing surface beyond which their
attention need not extend. The same approach was taken to
the design of the sound engine within Max/MSP.
2. Technology
The inner workings of this instrument are based on an
Arduino microcontroller. The communication between the
musician and the instrument occurs as a result of several
elements. The sensors take values depending of level of
interaction with the musician. These values are routed to
the Arduino through a breadboard for ease of development.
The values are then sent to the computer via USB and
appropriated in Max/MSP.
Additional electrical components used in the circuit
were:
• Two * Red LEDs
• Two ∗ RGB LEDs
• Two * Capacitors
The Red LEDs were connected to the two momentary
switches and allowed for important visual feedback as to
the state of the instrument. The switches were referred to
throughout the process as ‘Button 1’ and ‘Button 2’ in
respect to their position from left to right on the surface of
the instrument. Button 1 controls the Freezing of a note
and Button 2 acts as a shift control, which activates simple
alternate mappings for the FSR and IR. This will be
discussed in more detail later.
RGB LEDs were used to provide constant feedback
relating to the state of the continous sensors. These LEDs
were used because they are able to reflect constantly
changing values. In this way, the movement of values
throughout a certain range can be shown with changing of
the LED’s colour through stages: off, green, yellow, red.
This is achieved by constantly updating the values sent to
the Red, Green and Blue pins of the LED.
The sensors used allowed for reasonably accurate
readings, which would much more easily afford precise
control. However, two 22pF capacitors were used in order
to stabilize the signal from the power source. These were
added to account for unpredictable peaks in the values read
by the sensors. Adding these across the side-rail of the
breadboard allowed for much more controlled results.
3. Development
3.1 Programming Arduino
Throughout the development process, the Arduino code
had to undergo a number of changes in order to best
achieve the relatively complex results that were required of
it. Reading values from the buttons and sensors was
straight-forward, as outlined in many of the examples of
code provided with the Arduino program. Similarly,
activating the LEDs for the buttons was simply a case of
mapping the values read in by the pins for the buttons to
the outputs of the pins for the LEDs.
Controlling the RGB LEDs required slightly more
research. The LEDs consist of four pins – Red, Green,
Blue and Ground. Each of the colour pins was fed in a
value from an assigned PWM pin on the Arduino. This
would output a value fro 0-255 depending on the reading
from the relevant sensor. This number would then
represent the level of brightness of each particular colour.
The force sensor provided a range of 0-1023. This was
then divided up into three sections which would represent
different steps in the transition process: 0-255 = Green
follows value of sensor; 256 – 511 = Green remains at 255
and Red increases with force; 512 -1023 = Red remains at
255 and Green reduces for every second value of force.
This allowed for smooth transitions between the colours
that were directly representative of the values taken in by
the force sensor.
The IR sensor acted under the same premise but the
values had to be appropriated to match the scale first. The
range of values read by the IR sensor were from 0 – 600.
With no interaction, the sensor was reading a typical base
level that was peaking at approximately 90. In order to
eliminate these unusable values and then expand our
values to fit the desired range of 0-1023, the following
formula was used:
dist = ((dist – 90) * 2.0058))
3.2 Connecting with Max/MSP
Similar re-appropriating measures needed to be taken in
Max/MSP in order to gain more usable values that were
less susceptible to sudden peaks in value. A low-pass filter
was applied to the values received and these values then
underwent a running average process, which would
constantly output the average of the last fifteen values that
were received. This introduced slight latency into the
control process but it was nothing significant enough to
affect how the instrument is used.
4. Performance/ Musical Aesthetic
This instrument contains several references to
characteristics of existing instruments. As regards
performance, there are obvious connotations to a theremin-
like style. The control of pitch by non-contact interaction
affords the user a novel yet familiar means of interaction.
The force sensor provides a certain degree of haptic
feedback simply due to the fact that the musician is
pressing against a hard surface and can judge their effort
themselves. The basic interactions involved in the playing
style are simple but a great deal of practice and attention is
required in order to craft convincing performance
techniques. The shift button (Button 2) takes advantage of
the conceptual rather than mechanical connection between
the sensors and the sound parameters. With the shift button
pressed, the IR sensor changes from controlling values of
the frequency domain to those of the time domain.
Similarly, with the shift button pressed, the Force sensor
takes on new meaning as the control of Loop Offset rather
than Loop Size. Button 1 provides left hand interaction
similar to that of a pianist. Pressing this button will freeze
the sound at the current point and play it constantly as if it
were a sustained background chord. This is similar to how
a piano player may play a chord with the left hand and a
melody with the right hand. Perry Cook referred to the
advantages of this kind of leveraging of pre-existing
techniques as an essential factor in designing new
instruments[3].
Musically, the instrument sits in an area populated by
droning instruments assigned to set pitch intervals such as
an organ. The transposition of the sound occurs in steps,
creating defined notes relative to the original sound. These
steps can be seen as similar to the keys on an organ. The
value taken in by the IR sensor is divided up equally so
that the transposition in semitones ranges from 0 – 12. The
transposition occurs in a descending Aeolian scale, which
gives a slightly Eastern quality to the sound, particularly at
very low playback rates. A transposition of zero is
achieved at the highest possible distance from the IR
sensor and when the hand is as close as possible to the
sensor the sound is transposed down a full octave.
The sound capabilities of an instrument that is based on
a pre-recorded audio sample can scarcely be disputed when
it is explored on a microscopic level with the digital signal
processing capabilities afforded by Max/MSP. A five
second sample provides 221 separate frames, each of
which can be used as a separate starting point for a loop or
as a frozen note in itself. In this way, the sonic capabilities
of the instrument are as rich as the source will afford. For
example, a cello can create any sounds that the string will
permit, which is a huge amount of tonal capability,
however, as with all instruments, including electronic
instruments, there are limitations. The ability to recreate
our instrument by exploring a new sample is like replacing
the string on the cello with any tangible item that can
create perceivable sound.
5. Personal Contribution
5.1 Sound Engine
The author’s main contribution to the instrument was the
realisation of the sound engine within Max/MSP. This
required several different processes and approaches, hence
the satirical title of the instrument. It was within Max/MSP
that the mapping of the sensor inputs to manipulative
parameters took place. The main parameters were
controlled with primary one to mapping; however, some of
the values were also mapped to secondary controls such as
effects processes.
The signal chain of the audio begins with a sample
being recorded to an FFT; this is done upon opening the
Max patch and pressing the ‘Startup’ Button. For our
purposes with the instrument, we used a five second loop
from the beginning of the Thomas Newman song
‘American Beauty’, which consists of a short four note
piano phrase. This sound is played back when the IR
sensor is activated by motion. As opposed to simply
turning the sound on and off, we applied an ADSR
envelope to the output so that the sound has a certain
attack when it is activated. This added to the musical effect
of the instrument. This sound would then be sustained until
the IR sensor was inactive, at which point the release
values of the envelope would be applied to the volume of
the sound.
Once the sound is activated, it goes through a process of
time stretching which will slow the sound down in a range
of [-0.5 to +2]. This playback rate is initially set to 1 and
will be reset back to 1 again when the IR sensor de-
activates. Pressing the shift button and interacting with the
IR sensor control this rate. A feature built into the patch is
the ability to hold values when the musician activates the
shift button. When the control of the IR sensor is changed
from pitch to speed, the current pitch value is held and will
remain at that point until the shift button is depressed and
the IR value either exceeds the previous value or
terminating right hand interaction deactivates the note.
This means that when switching between control functions,
all values can be maintained so there will not be a jump to
unpredictable values. Essentially, Button 2 acts as both a
hold and shift function.
Figure 2. Mapping
Figure 2 gives an example of how the mapping looks.
After the time domain manipulation, the signal is sent to a
pitch shifting section, which will transpose the sound
depending on the position of the IR value within the
defined scale. This is achieved with the use of more FFTs.
This transposed signal is then sent to two more
transposition processes, which create a version of the
sound that is shifted an octave up and another an octave
down. This creates much greater sonic content.
The three sounds are then combined and sent through an
effects process, which includes Overdrive, EQ, Delay,
Reverb and Limiting. Some of these effects are controlled
dynamically by mapping sensor values. Overdrive is
controlled by the rate of change of the IR value. In this
way, the faster the musician changes from one position to
another, the more bite there is to the notes attack. The
middle EQ frequency on a filter is controlled by the force
value, so as the loop size decreases, the emphasis of the
EQ moves down to lower frequencies. When the note is
deactivated by removing the right hand, a delay effect is
activated, adding a tale to the sound. This sound is then
sent to the output.
Independent of this interaction is the freeze button,
which can stop at any point in the sample and freeze a
frame, extended indefinitely. This is achieved with the
same time stretching processes as explained above. The
frozen sound is not transposed or put through and other
octaves, it is simply sent to another effects patch and is
controlled by similar functions to those already mentioned.
6. Discussion & Conclusion
The instrument described is our approach to the control
of audio manipulation using extreme time stretching and
pitch shifting. The results we found were quite musical and
significantly different from their original source. As an
instrument in an orchestra, ours would sit somewhere
towards the back; it is not intended to be a lead instrument.
One idea we tried to address was the avoidance of typical
‘electronic’ sounds in exchange for a more traditional,
expressive instrument similar to the acoustic examples we
are familiar with.
The instrument has addressed some of the problems of
electronic instruments in computer music performance
regarding the definition of performer vs. programmer [2].
One rewarding result from our research was the
implementation of more interesting one to many mappings,
which have been described as much more interesting for
the performer [4].
We found that by complying with the objectives we set
in the introduction, it is possible to create an electronic
instrument that bears much more resemblance to the
traditional instruments we are familiar with, while still
maintaining a huge degree of control over sonic
capabilities. Further work could possibly involve exploring
different mappings and reducing the computational
requirements of the signal processing.
References
[1] B. Bongers, “Electronic musical Instruments: Experiences
of a new Luthier,” Leonardo Music Journal, 2007, pp. 9–
16.
[2] W.A. Schloss, “Using contemporary technology in live
performance: The dilemma of the performer,” Journal of
New Music Research, vol. 32, 2003, pp. 239–242.
[3] P. Cook, “Principles for designing computer music
controllers,” Proceedings of the 2001 conference on New
interfaces for musical expression, 2001, pp. 1–4.
[4] A. Hunt, M. M. Wanderley, and M. Paradis, “The
importance of parameter mapping in electronic instrument
design,” Journal of New Music Research, vol. 32, no. 4, pp.
429–440, 2003.