Handheld Electronic Camera Flash Lamp as a Tangible User-interface for Creating Expressive Visual Art Works

Corey Manders and Steve Mann

Dept. of Electrical and Computer Engineering 10 Kings College Rd, Toronto, Canada

manders@ieee.org and mann@eecg.utoronto.ca ABSTRACT

We propose the use of a hand-held camera ash as a tangible user interface to a photographic lightpainting algorithm/system. Our system includes a method of photographic multiple-exposure lighting using a single camera ash to obtain the results that would otherwise require complicated and more expensive professional multiple-light photographic studio setups. The system is simple in its approach, because it provides a direct metaphor-free interaction between the user, the camera, and the computer. The system mimics traditional multiple exposure photography by computationally combining images linearly to produce visual art works. of the individual lights. On top of this problem, adjusting the lights is largely hit-or-miss. There is little indication to the photographer of what the result will be until the image is taken. Furthermore, the photographer only sees the net result of the lights. Luckily, with current digital cameras, LCD displays on the back of the camera allow the photographer some indication to this net result without having to go through the process of developing and printing to see the image. However, it is our intent to improve this process. We propose a tangible user-interface [1][2][3] based on a modied hand-held electronic ash that remotely controls a tripod-mounted electronic camera. The overall intent of the system is to make the process easier for the the photographer. An application we have developed allows a user to see the contribution of each of the lights independently, and an ability to individually control the brightness of the lights after the photograph has been taken. To provide even greater exibility, our system allows for a retroactive adjustment of lighting.

Categories and Subject Descriptors

H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems; J.5 [Arts and Humanities]: Fine arts

General Terms
Tangible user interface, Lightspace Imaging, Photoquantimetric imaging, photographic lighting techniques, linear image manipulation, system integration, embedded systems.

Keywords
Photoquantigraphic Imaging, photographic lighting, Multiple image photography, Cumulative image photography

1.

INTRODUCTION

Photography is a practice with long-standing traditions. Typically, photographers will aim a camera toward the subject matter and set up lighting to achieve a pleasing image. Lighting a subject in this manner does have several drawbacks. As more and more lighting is used, the photographer is likely to become less aware of the contribution of each

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specic permission and/or a fee. MM06, October 2327, 2006, Santa Barbara, California, USA. Copyright 2006 ACM 1-59593-447-2/06/0010 ...$5.00.

Figure 1: The test button present on most hotshoemounted camera ashes. We use this button as a tangible bit connecting cyberspace (our multiple image composition program) to the physical environment.

1.1

Hand-held ash lamp as a tangible user-interface

Ordinarily, camera ash lamps are mounted to cameras

I.Tx

O.Rx

O.Tx

I.Rx
S

Tx
INBOUND

Rx
OUTBOUND

FLASH

OPENING

Tx
X

OUTBOUND

Source of Illumination

Rx

Figure 2: The ash component of our tangible user interface. Rather than simply triggering the ash, the test button of the ash is connected to a twoway radio. Pressing the test button sends a signal to the camera to open the aperture, and a second signal is sent from the camera to the ash to re.

INBOUND CAMERA SHUTTER WITH SOLENOID

Camera and Base Station

or are on light stands. Often they are used with wireless receivers so that they are actuated when they receive a camera synchronization pulse. In this traditional camera setup, the camera controls the ash. We propose the use of a handheld ash lamp that has a reverse-wireless link, i.e. the ash controls the camera, rather than the other way around. A key component of our system is this remote ash with a tangible bit[1]. Users of cameras and ashes will be familiar with the test button present on virtually all hotshoe mounted camera ashes. This button on a typical ash is shown in gure 1. As one may expect, when the test button is depressed, the ash is actuated and the area where the ash is pointing is illuminated. When a photographer is in a typical studio setting, he must simultaneously deal with multiple lights, arranging their positions and intensities to obtain the desired nal result [4]. It would be far easier if the photographer could move around the subject matter with a handheld ash, pointing it as desired. Each time the photographer pressed the test button, the new source of lighting would be added to the cumulatively lit image in cyberspace. This is the goal of our system. Rather than only producing a burst of light from the ash, our system intercepts the signal from the test button on a standard camera ash. A simplied diagram of the ash component of this process is shown in gure 2. The test button remotely actuates the camera and a sync pulse is sent back to the ash causing it to re (the camera, or base station component of the system is depicted in 3). Thus the ash appears to behave as it does in its unmodied state. From this perspective, the user simply seems to be using the test button to re the ash. However, the modied system turns this button into a tangible bit, a physical interface into our cyberspace program which adds the images together.

Figure 3: The camera/basestation component of the multiple-exposure photographic system. The system receives a signal from the ash actuating the camera. When the aperture is open, a sync pulse is sent back to the remote hand-held ash. were used, both of which are PTP compliant. However, the software and system developed are not restricted to these cameras. Any PTP compliant camera will suce. As prototypes, we have created two ash controlling mechanisms, each of which has its own advantages. The basic system setup in use is shown in gure 4.

2.1

Extending the system

Rather than just controlling the ash, the photographic system is also able to control the exposure time and Fstop of the camera from the remote ash triggering mechanism. Given that there is already a data radio transmitting ash signals to the camera-side base station, additional signals are used to control the cameras Fstop and exposure. Given that the camera used is PTP compliant, controlling these settings using the computer connected to the camera is relatively easy. We have developed a program which uses a bluetooth receiver and translates the signals from the handheld remote ash into the PTP commands necessary. Once the camera has been actuated, and the image acquisition process is complete, the program downloads the image from the camera and uses a lightspace algorithm [6][7] [8] to combine the image using a method of cumulatively adding light. This code (a mac os X application named cement) is available at http://www.eyetap.org/corey/CODE.

2.2

Using the system

2.

BASIC PHOTOGRAPHIC SETUP

As opposed to using a complete photographic studio, our imaging systems uses one ashlamp, a computer, and a PTP compliant camera. PTP is a camera protocol which was standardized by PIMA (the Photographic and Imaging Manufacturers Association)[5] and is available on many current digital cameras. For this paper, a Nikon D70 and D2X

Given that a remote ash has been implemented, the user connects a PTP- compliant camera to a computer running our cement application. The user holds the ash in a position appropriate for the illumination desired. The user then clicks the ash side trigger, causing the camera and the ash to re. The resulting image will appear on the left side of the application as shown in gure 6. For the image presented in gure 6, the software has been set to rst strip the captured colour information from the captured image, the user then selects a color to paint the captured image. The end result is often more dramatic than

Camera field of view

Flas h Lam p

Wearable Camera Remote and Flash Controller

Camera
Wireless Base Station

Portable Computer

Figure 4: The top image shows a simplied version of the experimental setup. A computer is connected to the camera using a standard USB connection and controls the cameras actions using the PTP protocol. A wireless receiver is also connected to the computer (again using a USB connection), and allows the user of the photographic system to control the camera remotely. The system allows the user to move the ash to various locations while completely unattached from the camera and computer. The bottom image shows a multiple image photographer using the photographic setup as described.

Figure 5: The remote camera controllers developed for the imaging system. The user is able to actuate the camera, causing the remote ash to re, simply by pressing a button on the interface (the user is shown pressing this button). Both interfaces use separate buttons to control the Fstop and exposure time of the camera, however one prototype features a wearable component (top), whereas the second prototype (bottom) is a single unit. In the top instance of the interface, the four buttons are in a diamond pattern close to the ash button and an LCD display allows the user to see the current exposure time and Fstop setting of the camera. In the second prototype, the remote ash and triggering mechanism are in one housing.

Figure 6: The captured image as it rst appears in the interface without colour information.

Figure 7: The captured image as it rst appears in the interface without colour information. retaining the colour information and multiplying by the selected color. A less dramatic method may also be used which is closer to that employed by photographers using multiple lights and gels. Specically, the user interface of the cement application allows the user to tint the image with a specied colour when the retain original colour option has been chosen. This mimics very closely the eect of using a colour gel with a studio ash. Using the strip and recolour technique mentioned, the gure presented in gure 6 may then be coloured and added to the cumulative image, as demonstrated in gure 7. Because the image is converted to lightspace [9], it is also possible to multiply through the lightspace representation by a scalar. This has an identical eect to increasing or decreasing the original exposure time of the image within a reasonable limit. Thus, the photographer may retroactively adjust the intensity of the ash. In the cement application interface, this adjustment may be done using a slider on the GUI. This slider may been seen on any gure involving the cement application (for example gures 6 and 7), and is located beside the colour box in the bottom left-hand corner of the GUI. Continuing on with the process, the ash is moved and another picture is taken. Again the image shows up on the left side of the interface. The artist may then colour the image as desired and add it to the cumulative image. This stage of the process is depicted in gure 8.

spired by the test button present on most camera ash units. We also have noticed that children like to press the test button of a ash, to cause the ash to be red without actually taking a picture. Likewise, our system turned out to be easy enough to use that children could produce high-quality images, and new forms of visual art, by simply playing with our version of the electronic ash. This is not to say that a professional, studio-quality image could easily be created by a child, but we simply found that children found the system amusing, and were able to use it to produce interesting results. For this reason, we set up the system in a child-care facility and began using the hand-held (non-wearable, selfcontained) version of the prototype. The action caused interest in the children who began producing multiple-image light paintings. Some examples of the children using the interface are shown in gure 9. The artwork produced is also shown in gure 9. Some features of the system were not used by the children. For example the interface present on the computer in which the current image and the cumulative image were shown seemed to be confusing. Furthermore, even with the colours available to be used for coloring the image being presented as crayons on a computer screen, the interface lacked intuition. Though this was not a problem from experienced photographers, this was a problem with the children. What we believe would help this matter, is to create tangible bits for the crayons. Much the same way Ishii and Ullmer use phicons in their implementation of metaDESK and transBOARD[1]. That is too say, the painted bits which are instantiated in cyberspace as virtual crayons could easily become atoms, a set of phicons recognized by a simple visual system. We consider this a direction for future research. We believe that the tangible bits we created on the controller with the use of the test button, and which was found to be successful, could be extended to tangible bits in the colouring process of the system. For example, using phicons that look like typical crayons, the user would touch the screen with the phicon, colouring the newly acquired image as wanted. Though more testing must be done with experienced photographers (our tests were limited to a few), some outstanding results were achieved. To highlight some of the success using the interface we have included some of the artwork developed as the last page of this document. The addition of controlling the Fstop and exposure time of the camera through the handheld interface were additions made out of the recommendation of the experienced photographers.

4.

LINEARITY AND SUPERPOSITIONAL IMAGING

3.

TESTING THE SYSTEM

As previously stated, our interface metaphor is largely in-

Using our system involves combining multiple dierently illuminated pictures of the same subject matter. To do this, we require a linear superposition of these images. Unfortunately, most image processing is done on pixel values. Often images in this format are termed imagespace[9]. Due to the history of imaging, and a range expanding eect which has traditionally been present on most televisions, pixels present a range compressed representation of the photoquantigraphic information stored as an image. This is easily demonstrated by attempting to linearly scale

Figure 8: The cementing process is depicted. As the user continues to capture component images (usually termed light vectors), each component image appears in a drawer to the side of the cement application. The new vector also appears on the left side of the cement application. The image may then be added to the cumulative image, which is located on the right side of the cement application. The eect of adding the light vector on the left side of the application is apparent in the cumulative image in this case. the content of the image as pixels. For example, if two photographs are taken with exposure times of 1/60th and 1/30th of a second and we double the pixel values of the images taken at 1/60th of a second, the resulting image will be dramatically dierent than the image taken at 1/30th of a second. This eect is demonstrated in gure 10. Aside from the visual evidence shown in gure 10, the subject of pixels being non-linear has been studied to a great extent. For example [10], [11] and [12]. The problem with the compressed range representation of pixels is that our system essentially adds and tonally scales images taken with dierent lighting. To be able to perform this addition, our system rst transforms pixel values into a linear space by estimating the photoquantimetric value (or linear light intensity) of the light sampled to produce the pixel values observed in the range-compressed version of the image. The linear version of the image is commonly referred to as lightspace. These computations are critical to the software which was developed. For a more complete description of this process, refer to [13], [7] and [11]. image, the overall image benets from a technique known in signal processing as signal averaging. Signal averaging in its most simple form may be taking multiple images of the same scene with identical conditions and then averaging the resulting pixel values for each spatial location. Because the noise in a camera generally has a zero-centered mean, the result of averaging pixels in each spatial location will create more accurate pixels with each pixel less prone to noise. Because this happens inherently in our photographic system, it is possible to use a lesser-quality camera, and get results comparable to a very high-end camera using advancing noise reduction algorithms in hardware. Secondly, the system uses one ash but mimics the result of using any number of ashes. The benets of this condition of using one as opposed to many ashes are numerous. Aside from the economical benet, using only one ash reduces the bulk of photographic equipment necessary, and allows the system to be portable. Because of this portability, it is possible to do large outdoor photographic projects such as architectural imaging rather easily. Contrarily, attempting to do architectural imaging with multiple ashes brings about several unwanted conditions. Some of which are ash sequencing, balance of lighting, and the added possibility of equipment failure.

5.

BENEFITS AND SIGNAL PROCESSING CONSIDERATIONS

Without question, the photographic system developed is far more exible in its ability to artistically light a scene. The fact that children were able to use the system reasonably eectively also demonstrates its ease of use. The ability to see the construction of the image as each light is added, and the ability to tune each light is much easier to use than traditional photographic studio lighting and may produce spectacular results. A dramatic example of this is depicted in gure 11. To contrast a traditional image (using one ash) with the developed photographic system, consider gure 12. In terms of signal processing, one must consider the eect of adding cumulative images in the system. Because many images are added together in the process of creating a nal

5.1

Constructing the handheld ash prototypes

Each of the two handheld ash prototypes were constructed in a dierent manner. The rst prototype (shown at the top of gure 5) uses two AVR microcontrollers. One microcontroller is used on the wearable remote side, the other microcontroller communicates to the camera and ash. Connected to the serial lines of the AVRs on both sides of the system are two Radiotronix wi.232 radios. The camera/ash side of the system mimics a USB keyboard. Thus pressing buttons on the wearable remote ash controller map to keyboard strokes on the camera/ash side of the system. Most importantly, the ash test button (which is also mapped to a

Figure 10: The leftmost image was taken with an exposure time of 1/60th of a second whereas the centre image was taken with an exposure time of 1/30th of a second. The rightmost image was created by taking the leftmost image and simply doubling the pixel values (and limiting the pixel values to 255). Many dierences are apparent between the centre image and the rightmost image, showing that pixels do not behave linearly.

Figure 11: An example of how the user is able to tune each component of the nal image. Aside from the ability to retain or strip the original colour information, the user is able to colour or tint an individual image component, adjust it, and then observe how the component contributes to the overall image. The component may then be retained, or subtracted out of the cumulative image without changing the cumulative image.

Figure 12: The left image was taken using traditional photographic setup and only one ash. The right image is a composite of multiple images taken by a user of the system positioning the ash in various locations. The multiple images are then added in a linear domain and then transformed back into non-linear pixel values. Using the remote ash interface with the program developed eases this process tremendously. red button on the wearable device) triggers the camera to be actuated. This is the basis of our tangible bit metaphor. However, to a lesser extent, the Fstop and exposure controls which are on the wearable remote may also be considered tangible bits. Initially, our thought was to use the Radiotronix radios for communication both to and from the ash unit. The latency in the ash signal was measured to be 1/25th of a second. This assumes the microcontrollers are running at 8MHz and the radios are set to their maximum transmission rate. Though this latency is acceptable for many situations, in some cases the latency is too great. In such a case, it is possible to use faster, commercially available ash synchronization radios such as the receivers and transmitters produced by the Pocket Wizard company. In this situation, the initial signal is still transmitted using a switch on the ash. The switch transmits the character to the base station causing the camera to re. However, the signal from the hot shoe to the ash is transmitted using the Pocket Wizard transmitter and receiver pair. This system reduces the latency of the system to 1/1000th of a second, but unfortunately has the added cost of the high-performance radios. To reduce the latency to a greater extent, it is certainly possible to connect the two microcontrollers with two-conductor wire, with the obvious trade-o of reduced portability and the awkwardness of the wire. The second prototype constructed (shown on the bottom of gure 5 was developed in a much simpler manner. Realizing that in one sense the wearable system was a modied long-range keyboard, we modied the internal parts of a wireless Bluetooth keyboard such that when actions such as the ash test button was triggered, the signal for a key being depressed would occur. The key pressed would then remotely be sent to the Bluetooth receiver which would then communicate to the cement program to actuate the camera. A Pocket Wizard connected to the ash and hotshoe of the camera would then synchronize the camera and ash. Though this prototype was easier to construct than the wearable prototype, it was very easy to use (and was certainly easier to construct). This setup was the only one used during the tests with children in a day-care facility who seemed to easily use this version.

6.

CONTINUED RESEARCH AND ONGOING WORK

Initially, we have found that experienced photographers like the control of the lighting supported by the system. However, the number of photographers who have tested the system is still low. We are in the process of introducing the system to another university and a few technical colleges where photography is taught. What has been tested to a greater extent is the acceptance of children to the system. As shown (and we believe evident) in gure 9 is that children appreciate the system as a method of artistic expression. Many children enjoyed using the second handheld remote prototype and were interested in the resulting images that were created. However, with children, some aspects of the program should be modied. Currently, once the user res the ash, causing the camera to be actuated, the new image appears on the left side of the cement application. This image may be manipulated in several ways. For an experienced photographer, the initial nding were that this is advantageous. However, for children this process was confusing. What is easier for children seemed to be setting the program such that the new image was automatically added to the cumulative image. This way they could simply look at the computer running the cement application and see how the cumulative image had changed. Alternatively, we are considering the use of phicon crayons. A modied version of the program would use the test button on the interface to re the ash and the camera. The new image would appear on the screen and would be able to be coloured using the phicon crayons. When the desired colour is found for the new image, this would be added into the cumulative image. Given the success of the tangible bit in the hand-held interface, we believe that the phicon crayons would further improve the system for children developing this new form of visual art. Much the same way Ishii sug-

Figure 9: Examples of children using the handheld camera remote (left side of the page and the top two images on the right side of the page). Also shown is the artwork produced by the children through their own experimentation with the handheld remote (bottom right side of the page).

gests using RFID for a set of cards in the transBOARD system, a simple visual system or RFID could be used with the crayon phicons. Experienced photographers expect imaging systems to work with the utmost reliability. The system developed uses radio technology which in itself can become unreliable due to the distance between the remote ash and the computer/camera base station. This distance can become a concern in such situations as architectural photography, where large distances between the ash and the basestation may be especially effective. The simple act of testing the maximum distance the radio system works reliably still need to be explored. The cement application is an entity in itself which needs to be explored in terms of developing an eective GUI. There is the potential of allowing the user to change an increased number of camera parameters directly from the user interface of the cement program. However, this increase in functionality does complicate the user interface. What has been included in the user interface was considered the minimum in terms of what was necessary for the multiple imaging procedure. The positioning of the images presented on the interface as well as the camera controls is a target for user study which still needs to be done.

7.

CONCLUSION

We have presented a new method of digital photography which uses multiple images to produce a single image that in many cases would need signicant studio-type lighting. Much of the method demonstrated was inspired from the idea of using the test button on a ash as a tangible bit. Rather than simply ring the ash, the test button also actuated the camera. We created two prototypes which embraced this metaphor and gave the prototype both to experienced photographers, as well as a group of children in day care. The experienced photographers who tested the handheld remote ashes appreciated the possibilities oered by the interface. Furthermore, the children who used the interface found it simple and fun to use. Several times, the children produced new forms of visual art simply by playing with the interface. Interestingly, the second prototype developed (the one with no wearable component) was found easiest to use by the children, and was favored by the experienced photographers which used the devices.

[2] B. Ullmer and H. Ishii. Emerging frameworks for tangible user interfaces. IBM Systems Journal, 393(3):915931, 2000. [3] Hiroshi Ishii and the Tangible Media Group. Tangible Bits: Towards Seemless Interface between People, Bits and Atoms. NTT Publishing Co. Ltd., June 2000. ISBN 4-7571-0053-3. [4] Harold E. Edgerton. Electronic ash, strobe. MIT Press, Cambridge, Massachusetts, 1979. [5] ISO Technical Committe 42 (chair: Philip Wychorski). Photography electronic still picture imaging picture transfer protocol (ptp) for digital still photography devices, August 26, 2006. [6] S. Mann. Comparametric equations with practical applications in quantigraphic image processing. IEEE Trans. Image Proc., 9(8):13891406, August 2000. ISSN 1057-7149. [7] C. Manders, C. Aimone, and S. Mann. Camera response recovery from dierent illuminations of identical subject matter. In Proceedings of the IEEE International Conference on Image Processing, pages 29652968, Singapore, Oct. 24-27 2004. [8] F. M. Candocia. Synthesizing a panoramic scene with a common exposure via the simultaneous registration of images. FCRAR, May 23-24 2002. [9] Steve Mann. Intelligent Image Processing. John Wiley and Sons, November 2 2001. ISBN: 0-471-40637-6. [10] Charles Poynton. A technical introduction to digital video. John Wiley & Sons, 1996. [11] F. M. Candocia. A least squares approach for the joint domain and range registration of images. IEEE ICASSP, IV:32373240, May 13-17 2002. [12] S. Mann and R. W. Picard. Virtual bellows: constructing high-quality images from video. In Proceedings of the IEEE rst international conference on image processing, pages 363367, Austin, Texas, Nov. 13-16 1994. [13] C. Manders and S. Mann. Determining camera response functions from comparagrams of images with their raw datale counterparts. In Proceedings of the IEEE rst International Symposium on Intelligent Multimedia, pages 418421, Hong Kong, Oct. 20-22 2004.

8.

REFERENCES

[1] H. Ishii and B. Ullmer. Tangible bits: Towards seamless interfaces between people, bits and atoms. In Proceedings of Conference on Human Factors in Computing Systems (CHI 97), pages 234241. ACM SIGCHI, March 1994.