Aesthetics of

Computation
Visualizing Software

Presented by Ray Garcia

March 7, 2007
University of Arizona
rgz@email.arizona.edu

rgarcia@alterwork.com
Noesis Challenge
 Noesis = Perception + Learning + Reasoning

 Software is not yet a mature engineering

profession and needs to strive to be an
artistic craft that satisfies utility (function) as
well as beauty (form) as co-evolving goals
while enabling anyone to learn the craft.
Software as Art
 Produces Artifacts, sometimes unpredictably
 Malleable Material but limited by the tools
 Variety of interactive and temporal expressions of a concept
 User creates their own meaning
 The process requires
 Understanding, abstraction, insight, and interpretation
 Mastery of a craft
 Representational patterns and forms
 Creativity and inventiveness
 For general public appreciation it lacks
 Inaccessible as a communications medium
 Visibility of computation
 Aesthetic Sense and Critical Review
 Cultural constructs
Visible computation
 “Programs should be written for people to read,
understand, and manipulate, and only incidentally
for computers to execute”
 Visible computation allows for direct observation
and interaction with the code execution supporting:
 Ease of consumption (reading)
 Ease of comprehension (thinking)
 Ease of construction (writing)
 Software should be a communications medium of
design and its intended utility
Computation is invisible
 Programming is the craft of manipulating
computation
 Programming systems enable abstract specification
of the solution in text and diagrams (code & UML)
 The problem, intent, and context is lost in the
specification
 The specification is not the execution
 The diagrams and the text are not inter-changeable
and require annotation
 IDE’s and Modeling tools are not enough
 Execution is what matters most yet it is obscured
Programming Systems
 Programming tools prevent computation from being
visible and accessible by the novice or curious
spectator
 Programming concepts are generally easy to
understand
 Data structures, state manipulation, conditionals, recursion,
algorithms, …..
 Incomplete set of tools and representation for
dealing with computation
 The mysteries of computation should be exposed
Current Tools are Incomplete
 Specification is not computation
 Anatomy is not a not alive
 Blueprints are not a building
 A score is not music
 Code is not computation
 What is visible are the remnants or output of
computation
 UI, reports
 Debuggers, tracing, footprints
 Logs files, profilers
 Static snapshots are not enough
Thinking of Computation
 Human skim/scan faster than reading
 Human process images and sound faster than text
 People think in visual spatial terms for many
technical problems
 The reader needs to reconstruct a mental map that
existed in the creators mind
 Novice programmer construct their own internal
representation of computation through trial and error
 Need to think of computation as a dynamic process
instead of as a static specification language
Unnatural Code
 Computation is a multi-dimensional and non-linear
dynamic structure
 Code may have parallel or multiple execution paths
 Navigating through one dimensional text is like
unraveling a contorted knot
 Code relies on understanding written language
 Even commented code is difficult to understand
without the reader exerting a lot of effort
 Code strays too far from natural language
 Annotated diagrams don’t go far enough
Representations of
Computation
 Why isn’t programming more like interactive
hypertext with diagrams and animation that can be
manipulated directly?
 The syntax of code has no relationship to the
runtime process, it is a constraint of the parser
 Generating names for identifiers is simpler than
creating a picture of the same information
 Diagrams can more easily represent structure and
flow
 The representations should be designed to conform
to the limits of Human perception
Programming as a Craft
 Requires easy tools which makes the craft possible
and accessible to the novice
 The craft needs to conform to the limitations of human
cognition instead of forcing people to think like
computers
 Lack of visible computation negatively impact ease of
use and communication
 Visible computation would facilitate learning
 An aesthetic of computation is needed to advance the
craft of programming
 Craft = Literate programming + Visual Modeling +
Visible Computation + Aesthetics
Literate Programming
 Program expressively to communicate to an
audience instead for a computer to execute
 Tool Support
 Syntax directed IDE
 Text Mapping
 Code Navigation
 Code Hypertext Documentation
 Testing and tracing frameworks
 Code coverage
 Cross hatching with Aspect Oriented programming
 Meta Coding
 Editorial Code Refactoring
 Code generators
 Domain Specific Language development
 Evolutionary genetic programming
Visual Programming
 Visual programming
 Computation
 material data
 process
 Visual Interface
 visual language
 Construction
 navigation
 Aesthetics
 Patterns
 critical theory
Computational Processes
 Object oriented – interactions between objects that
contain state and procedures (smalltalk)
 Imperative – execution of sequential commands with state
 Procedural – combination of imperative and functional can
affect state
 Functional – calculation of mathematical functions
without state (ocaml)
 Constraint based – rules of truth and logic
statements (prolog)
 Multi-paradigm languages (mozart)
 NLP = ontology + behaviors + natural language
Visual Design
 Human Perception needs continuity
 Human mind is not configured to follow discrete
transitions and needs smooth continuous
representations i.e.. animations
 Gradual updates are easier to perceive
 Computation is quantized therefore poses a
design challenge
 Clear spatial relationships

 Minimal context switching

Visual language
 The visual language should be used for
specification, interaction, and execution
 Show cause and effect

 Computation should appear as an animated

interactive film that may be edited
 Challenge to invoke rhythm and drama in the
animation of computation
Visual Modeling (UML)
 Remaining Problems
 UML is limited to a static representation of mixed
perspectives on the specification
 Managing scale and levels of abstraction
 Context and transparency of components
 On the Horizon
 Executable UML
 Pattern Languages
 Model Driven Architecture
 Model Driven Development
Possible Solutions
 Invent a new visual language design and execution
environment
 Extend UML to support code generation, execution
and interactive animation
 Parse and instrument or probe existing code to
generate animation code to view the computation
with annotation links to the source code
 Create a language design environment that
supports human cognition using the full multimedia
capabilities
Complexity, Crowds, Chaos
 Size and complexity
 Software maintenance and evolution
 History through versioning
 Open source failure rate fosters innovation
 Peer review and collaborations
 Chaos due to rate and impact of change
 Software development as a social phenomenon
 Learning theory applied to software development
Scope, Scale, and Perspective
 Amplified perception
 Augmented human & social intelligence
 Audience support for users and creators
 Information hiding and abstraction
 Software comprehensibility through visualization
& sound synthesis of computation execution,
interaction, and navigation
 Algorithm animation
 Software Aesthetics
 Critical Theory for software
Algorithm Animation Examples
 cyclic queue
 train sorter

 dijkstra shortest path

Further study
http://www2.hig.no/~algmet/animate.html
Visual Programming Examples
 Marten [Mac osx]
http://www.andescotia.com

 National Instruments LabView http://www.ni.com/labview/

[device control]
 Pictorial Janus http://users.encs.concordia.ca/~haarslev/pjmovies/

 ToonTalk http://www.toontalk.com [Kids visual

programming]
 PureData PD http://puredata.info/ [Media graphical
programming environment]
 Pictorial Janus Syntax

Port empty Constan Link Channel

List t

Function Constrain Agent Rule

t

Call
Configuratio
n

c b a List ([a,b,c | [])

Pictorial Janus Example
 Append 2 List
 Distributed Queue
Animated UML
 OOP Eggs display – using biomimicry to
visualize software

 Visio Example of animated UML diagram

 Future possibility UML X3D

Closing Remarks
 Enormous opportunity exist to define and
advance the craft
 The current global expansion of the craft is
positive growth in the market
 The software craft won’t progress without
appreciation and participation by the general
public