Journal of Visual Languages and Computing (1991) 2, 217-246

An Integrated Data Flow Visual Language and Software Development Environment

JoHN R. RAsur, e AND CARLA S. WILLmMS

Department of Electrical and Computer Engineering, University of New Mexico, Albuquerque, New Mexico 87131, U.S.A. Received 27 September 1990 and accepted I0 April 1991
The current generation of data flow based visual programming systems is all too often limited in application. It is our contention that data flow visual languages, to be more widely accepted for solving a broad range of problems, need to be more general in their syntax, semantics, translation schemes, computational model, execution methods and scheduling. These capabilities should be accompanied by a development environmentthat facilitates information processing extensions needed by the user to solve a wide range of appllcation-specific problems. This paper addresses these issues by describing and critiquing the Khoros system implemented by the University of New Mexico, Khoros Group. The Khoros infrastructure consists of several layers of intdracting subsystems. A user interface development system (UIDS) combines a high-level user-interface specification with methbds of software development that are embedded in a code generation tool set. The UIDS is used to create, install and maintain the fundamental operators for cantata, the visual programming language component of Khoros.

1. Introduction
VISUALPROGRAMMINGis a subject of current active research aimed at reducing some of the difficulties involved in creating and using computer programs [1, 2]. It represents an attempt to exploit our non-verbal capabilities by combining the disciplines of interactive graphics, computer languages and software engineering. Visual languages have the potential to enhance dramatically the communication between human and computer. However, the current generation of visual programming systems is all too often limited in application. One approach to solving this problem is to provide a generalized visual programming environment which may be customized and retargeted for new application domains. The Khoros system, implemented by the Khoros Group at the University of New Mexico as an ongoing research project, is an attempt to meet this need. The goal of the project is to create a programming environment which can be used by a broad audience for a wide variety of applications. Cantata, the visual language component of Khoros, is a general-purpose, processing domaitl-independent, programming language based on the data flow paradigm. Language features include control constructs for conditionals and iteration and a hierarchy of subprocedures. The Khoros user interface development system provides a comprehensive environment for creating interfaces to user-defined applications which can then be integrated into the existing system to extend the visual language. 1045-926X]91]030217 + 30 $03.00/0 (~) 1991 Academic Press Limited

218

1. R. RASURE & C. S. WILLIAMS

1.1. Related Work

A number of visual programming systems based on data flow have been implemented and described in the literature, including ConMan [3] for interactive graphics, Gabriel [4] for real-tim e signal processing, AVS [5] and apE [6] for scientific visualization and the Visualization Workbench [7, 8] for image processing. These systems are all similar to cantata in that they take a large-grained [9] approach to programming with data flow graphs. However, since none of them include the ability to handle conditionals and iteration they cannot qualify as true programming languages [2]. Specific comparisons will be made throughout the paper between cantata and these, as well as other, systems. But what distinguishes cantata is its integration into the Khoros software development environment. We have seen no other implemented system reportedln the literature that combines all of cantata's language features with an associated environment that is intended to extend the language to new application domains. It is this emphasis on extensibility that is an important feature of the Khoros system. Unlike grammar-based environments for creating new visual programming languages [10, 11], Khoros is intended for use by non-professional programmers who have neither the time, training nor interest in creating a new interface. Knowledge of formal specification of languages or graphics design is not needed, since the programming mechanisms and visual appearance are already built into cantata. This provides a powerful framework within which application ~pecific operators can be added, modified or replaced as needed without changing the essential nature of the interface. A number of software tools have been developed over the past few years to aid in the development of user interfaces. Recent surveys of user interface development systems (UIDS) [12] and tools [13] provide an excellent description of such systems and their strengths and weaknesses. In the terminology of Myers [12], the Khoros UIDS would fit into the category as automatic creation, which 'attempt to create an interface automatically from a specification of the application's semantics, thus overcoming designer resistance to UIDS technology'. 1.2. Khoros System Overview Khoros (chorus)--Any speech, song or other utterance made in concert by many
people.

Khoros is a comprehensive system which relies on existing standards (X Windows and UNIX) ~. The Khoros infrastructure consists of several layers of interacting subsystems. It incorporates .a UIDS which is composed of code generators, interactive development tools and maintenance,software, plus a visual programming language user interface to libraries of processing routines. Figure 1 giveslsome idea of the interrelationship between these elements within the software structure of the Khoros system. In simplest terms, a Khoros application consists of domain-specific processing routines which are accessed by an automatically-generated user interface based on a
*The X Window System is a trademark of MIT. U N I X is a trademark of AT&T Bell Laboratories, Inc.

INTEGRATED DATA FLOW VL

219

UI Specification Editor&Interpreter

User Interface

(UI) Specification ~

fl'rogram \ Specification

Graphical Form ~--(wGU~I CodeGenerator

Command Line Code Generator

GUI

Applicati

*"

sual Language "~---

CANTATA

Processing

Routines

Figure 1. Software structure of the khoros system

User Interface Specification (UIS). The Khoros environment supports the creation of three types of user interfaces: graphical (forms-based), textual (command~line) and the cantata visual language. In Figure 1, shaded rectangles represent specific UIDS tools within the Khoros software structure. Plain rectangles are utility and development libraries. Shaded ovals identify input to the Khoros tools, while plain ovals indicate generated programs. These programs include many of the Khoros tools themselves, in addition to programs created by application developers using the Khoros system. Dotted lines serve to illustrate that applications developed using Khoros may be referenced from within cantata. Because of the breadth of Khoros, a description of those who .work with the system may be helpful to the reader. While these are presented as if they are separate individuals, in reality, many users find themselves working across several levels of the system. End-users of the applications may use any of the three interfaces (shown at the bottom of Figure 1) to program solutions to, information processing problems, although the visual language is most often the preferred choice. Application developers (working f r o m top to bottom in Figure 1) use the various tools provided by the Khoros environment to extend the capabilities of the system, add new routines to be accessed by the visual language or create interactive graphical applications. Originally based on a software environment created for research in image processing [14], Khoros is currently being used as a research and development tool to

220

j . R . RASURE & C. S. WILLIAMS

solve a variety of problems at numerous sites throughout the world, b Applications include: medical imaging, ecosystem analysis, remote sensing, geographic information systems, database management, surface visualization and control systems. To accomplish this, the cantata visual language provides an end-user with access to several information processing libraries including: matrix algebra (via LINPACK and EISPACK), image processing, pattern recognition, signal processing and surface processing. In addition, Khoros includes interactive application programs for displaying/editing one-dimensional, two-dimensional and three-dimensional (surface) data. (At the present time, Khoros does not have a three-dimensional volume rendering or ray tracing library as do the AVS and apE systems.) The following sections are devoted to a discussion of the components of the Khoros system. Section 2 describes the visual programming language, cantata, in detail. While a complete description of the Khoros UIDS is not appropriate within the context of this paper, the specifics of the UIDS tools which provide for extending the information processing capability of cantata are presented in Section 3. 2. T h e C a n t a t a Visual P r o g r a m m i n g E n v i r o n m e n t

Cantata--Setting of a narrative for chorus [Khoros].

The subset of a software development system directly related to the programming process of a single programmer is referred to as a progf'amming environment. Language-centr~:d environments are built around a specific programming methodology and provide a tightly integrated tool set suited to a particular language. The fundamental components which comprise the cantata programming environment are a user interface (Section 2.1), the programming language syntax and semantics (Sections 2.2 and 2.3) and language support mechanisms (Section 2.4). 2.1. User Interface Cantata [15] is a multiparadigm visual language which provides a high-level interface to domain specific operators. The paradigms employed are a graphically-expressed data flow programming model, forms with various buttons and sliders and textual programming. Invoking cantata presents the user with a workspace canvas and an associated top-level menu. Interactions with the user are controlled by a visual editor operating within a hierarchy of workspaces to construct a data flowgraph. Access to the visual language is provided by a forms based interface. In Figure 2 a cantata workspace is shown containing an example program to solve an image processing application.

2. I. 1. Programming Paradigm
The user interface to a visual language is determined in part by the programming paradigm chosen for the language. Languages which are based on a single paradigm offer a cohesive view of the solution to a problem and thereby help to clarify thinking. However, a single paradigm is often too narrowly focused to represent all aspects of a
b Khoros has been made available to the public at no cost via a n o n y m o u s ftp. Email khoros-request at chama.unm.edu for more information.

INTEGRATED DATA FLOW VL

C~,NT~T~ V i s ~ l Fro3ra~ivg E~irorm-~t For th~ K~,,~'~ ~ste~

221

UmMm
Figure 2. Cantata workspace with application flow graph. Use of a count loop to average 5 noisy images

large, complex system. To help solve this problem, multiparadigm languages and environments are being developed which do not restrict the programmer to a single representation technique [16]. By building systems using as many paradigms as needed, each paradigm can address those aspects of the system for which it is best suited. There are many possible ways to create a multiparadigm system. One approach, used by the Visual ToolSet [17], is to build a programming environment consisting of an interacting collection of simple visual languages. The approach taken by cantata is to integrate several representation styles into a single language, with each paradigm used where it can provide the most leverage. Traditional textual programming, graphically expressed data flow and forms are combined to provide different levels of interaction with the user. Graphical techniques are most useful to describe the relationships between objects that are difficult to express in a one-dimensional textual fashion. They can be used to make the invisible, but conceptually important aspects of a prdgram, visible. The principal visual programming paradigm employed by cantata is a data flow graph. Data flow is an inherently visual approach in which a program is described as a directed graph, where each node represents an operator (or function) and each directed arc represents a path over which data tokens flow [18]. Firing of a node is enabled whenever all its input data tokens are available. The result is placed on the node's output arc(s) and flows on to the next node for processing.

222

3. R.

RASURE & C. S. WILLIAMS

One of the lessons learned from early 'small-grain' visual programming environments, such as PICT [19], is that it is not a good idea to try to visualize everything. Some things, like mathematical formulas and numerical algorithms, are handled better with text. It is a waste of time and screen space to visualize these details, except perhaps for teaching at an introductory level. The choice was made in cantata to implement these details in an appropriate textual language. Thus, parameters to operators may be entered in textually as mathematical expressions to be evaluated by an expression parser (described in Section 2.3.2). Large-grain data flow [9], where nodes carry out significant computations compared to their scheduling overhead, is a compromise between data flow and traditional programming. As used in cantata, it also provides a compromise between graphical and textual programming. Thus, operators are written as processing routines in a traditional language, such as C or FORTRAN. An additional advantage is that many algorithms are already available in libraries of routines (like LINPACK, EISPACK, etc.). These modules can be incorporated directly, by using the Khoros UIDS, to become fundamental building blocks of the data flow language. The interrelationship between these operators is the aspect of programming that is graphically visualized by cantata. This relationship is often not one-dimensional and so is not conveniently expressed in a linear, textual fashion. Forms are a natural means of collecting information and are used as the basis for several visual programming languages. In cantata, this third paradigm is used to access language features and to facilitate interaction with the workspace environment (see Section 2.1.3). Forms also provide the interface between the visual data flow graph and the textuai~nodes. In order to select a specific operator and to choose the parameters required by the corresponding routine, cantata presents the user with a subform/pane representing that node. This is described in Section 2.2.1.

2.1.2. Worksapce/Visual Editor

The cantata workspace consists of a large blank canvas upon Which an application program is assembled (the gray area with background grid in Figure 2). The data flow graph can be constructed in any order, mixing execution and construction of the graph. To aid the user creating large program graphs, the entire workspace is actually several times larger than the window visible on the screen and is accessed by scrolling the window. A background grid (which may be selected as visible or not) assists in the alignment of elements on the workspace (grid size is set by the user). Once flow graph nodes are placed in the workspace, they may be moved by dragging. Except for the specification of certain numeric and string parameters for the routines, all interaction with cantata is accomplished through mouse movements and button clicks. This reinforces the notion that the user has complete control in directly manipulating the elements in the workspace to create a problem solution. The user may SAVE an efitire workspace, including the data flow graph of nodes and connections as well as intermediate processing results (if desired), at any point during a programming session with cantata. These can then be restored at a later time for continued use by the programmer.

2.1.3. Language Feature Access

The Khoros Graphical User Interface (GUI) library (briefly described in Section 3) is the basis for communication with the cantata user. The master form is used to initiate

INTEGRATED DATA FLOW VL

223

! :

.~r': |~zz

! :

Figure 3. Cantata masted'form (top and left), subform (center), an(] node or glrph (right)

the user's top-level interaction with the workspace editor (see Figure 3). Groups of operators are listed across the top of the master form while general cantata environment functions are controlled by the'buttons down the left side. A specific operator is selected from a hierarchy of menus. Cantata allows up to three levels of menu access. These levels are: (1) the top-level menu; (2) drop-down menus; and (3) subform menus. The relative size (number of operators) and the natural grouping of operators for an application domain determine the appropriate balance between depth and breadth of the menuing system. For a small application (less than 25 operators), a top-level menu is all that would be needed. For a large application (more than 150 operators), three levels should be used. Cantata (with over 250 operators) currently uses a combination of (2) and (3). Data flow languages such as Gabriel and apE, which use unique icons to represent individual nodes, typically present the user with a palette of icons from which to choose. For application domains which contain a limited number of operators, this is possibly the best approach. General programming mechanismg and execution control are provided by buttons along the left side of the master form. The Wo~kspace subform allows the user to interact with the cantata workspace, configuring attributes of the environment, saving and restoring cantata programs, subprocedures, results, etc. Access to the cantata parser, which is used to define variables and evaluate expressions, is provided by the Variables form (discussed in Section 2.3.2). The ROUTINES button presents the programmer with an alphabetized list of all the operators available from the Khoros libraries of processing routines as an alternate to using the hierarchy of menus. Control nodes, selected from the Control subform, are used to create more

224

J. R. RASURE & C. S. WILLIAMS

sophisticated processing flow graphs, which include looping and branching (as described in Section 2.3.1). The Comment button allows the user to add documentation to flow graph programs. Routines external to the Khoros libraries may be accessed within cantata through a generic glyph provided by the Command button. RUN/STOP,RESET,and CHECKare used to control execution of the program flow graph (see Section 2.4.4). REDRAWrefreshes the drawing of connections for the flow graph.

2.2. Syntax of the Language

Broadly speaking, the syntax of a language is concerned with the way that a program is written, in other words, its outward appearance. The syntactic components of cantata which are used to represent the elements of a data flow graph are subforrn/panes, glyphs and connections. In cantata, the flow graph nodes are represented by glyphs corresponding to processing routines from the Khoros libraries. The connections represent data to be processed by the nodes. To create a cantata application, the user selects the desired processing routines (and control structures, as needed) from the subform menus, places the corresponding glyphs on the workspace, and interconnects these elements to indicate the flow of processing from node to node.

2.2.1. Subforrns/Panes
Flow graph operators are typically accessed by using the subform buttons which are arranged under general categories across the top of the master form. Each subform contains the information for a group of related routines from the Khoros libraries. Figure 3 shows the cantata master form with the pull-down menu for ARITHMETIC displayed. The subform button Binary Arithmetic has been selected and is seen highlighted. Once a subform button has been chosen, the corresponding subform appears in the workspace. The subform which corresponds to the selected Binary Arithmetic is also shown in Figure 3. The left portion of the subform contains the guide pane which includes guide buttons for each of the routines described on that subform, as well as general HELP for using the subform, and QUIT. When a guide button is selected, a pane corresponding to that specific routine is displayed in the right hand portion of the subform. Input/output filenames as well as required/optional parameters may be entered here. Numeric values may be expressed as constants, variables or expressions (non-constant values are evaluated by the cantata expression parser, Section 2.3.2). Detailed Help for that routine, formatted as a UNIX manual page, is also available. The subform in Figure 3 has the Multiplication pane currently displayed. Clicking the mouse on the Glyph button will replace the form with a glyph, a more compact representation of the routifie corresponding to that selected pane. An alternate to the use Of subform/panes would have been to present the information for an operator in a separate workspace (as done by AVS, for example). It is felt by the authors that the subform/pane arrangement provides several advantages. A greater sense of direct manipulation is obtained by having the subform/pane actually replace the glyph within the workspace. Several glyphs may be opened to their corresponding subform/pane representationat the same time. The user may browse through the related routines presented on a single subform, reading

INTEGRATED DATA FLOW VL

r

225

descriptions, etc., to find the one most appropriate for the given task. Selecting an alternate choice requires but a single mouse click instead of returning to the top-level
menu,

222. Glyphs Flow graph nodes in cantata are called glypbs and are represented by a stylized block with a textual label. A glyph corresponds to a specific instance of a processing routine operator selected from a subform/pane. In Figure 3, the vmul glyph is shown which corresponds to the displayed Binary Arithmetic/Multiplication subform/pane. Based on the information provided in the User Interface Specification (UIS) for a selected routine (described in Section 3.2), cantata uses the GUI library to automatically create the corresponding glyph. Pictorial iconic shapes were not chosen to represent cantata operators for several reasons. First, there are the problems associated with sheer numbers. The current Khoros libraries contain over 250 routines which can be accessed as operators within cantata. Designing unique, identifiable bit-maps to represent the different nodes would be a formidable, if not impossible, undertaking. Then, displaying the icons for easy selection within the confines of a workstation screen presents its own problems. Second, the correct choice of an iconic representation is a difficult task, often requiring the skills of a graphics designer. For application domains which contain only a small number of operators (like apE), creating bit-map icons to represent individual nodes would b&feasible and might increase user acceptance, though not necessarily performance. This capability should be added to cantata sometime in the future. By supporting two alternate, easily convertible representations of the primitive building blocks of the language through the GUI library, glyphs and subform/panes, cantata is able to balance the best features of both. Glyphs provide the simplicity of design necessary for the building of complex application flow graphs consisting of many components. At the same time, the panes retain all the detailed information needed to specify completely the interface to the routines. Each glyph contains input a n d / o r output connections, represented by buttons located on the left and right sides of the glyph, respectively (see Figure 3). Input buttons have arrows which point into the glyph and output arrows point out. If the arrow is dark, then its corresponding input/output is required for the routine. If the corresponding arrow is a lighter color, then it is optional. Required inputs must always be connected in the processing flow graph. The three buttons at the top of the glyph (from left to right) are used to destroy the glyph, to return to the subform and to execute the routine represented by the glyph. Control glyphs have an extra RESET button so they can be re-executed using the specified initial conditions. The glyphs are dynamic in their representation to" reflect their current execution status. If the corresponding routine is running, the glyph is' displayed in reverse video .which reverses back when execution terminates. This provides animated feedback as to the execution state of the elements of the program flow graph. 2.2.3. Glyph Connections A cantata application program is constructed by connecting' together glyphs in the form of a data flow graph. Connections represent the flow of data tokens between

226

J. R.

RASURE & C. S. WILLIAMS

operators corresponding to routines from the Khoros libraries. The connections are currently uni-directional, that is, data flows in only one direction The data type of the token is determined by the routines themselves and is therefore application-specific. Glyphs are connected together by using the mouse to click on the output button of one glyph and then on the input button of another glyph (or vice versa). A line is then drawn between the two glyphs and a unique connection name is generated by cantata and automatically stored in the correct input/output fields of the associated subform/panes. These temporary names may be implemented via disk files or the Khoros shared-memory file system. When a glyph is executed, the output data it creates will be stored under this name. Clicking on a line provides the name of the connection and the connected glyphs, as well as the opportunity to delete the connection or to save the data permanently. Intermediate data is typically erased on exit from cantata, but may be saved by the programmer for reuse at a later time. Once a connection has been made between two glyphs, it can be changed by either connecting the glyph input to a different glyph or by deleting it. Fan-out from a node is supported in cantata. Glyphs accept multiple inputs, but not fan-in into a single input arrow.

2.3. L a n g u a g e Semantics The meaning~oscribed to the syntactic components of a language is defined by its semantics. Using the graphical elements just described, which form the basic syntax of the cantata visual language, data flow graphs can be constructed to perform powerful information processing tasks. Source glyphs provide input to the program to be acted on by the processing glyphs. Output glyphs allow the user to interact with the program and view results or save the results in files. In order to provide some method of making run-time, data-dependent decisions, control nodes, including conditional (IF) constructs and iteration (LOOPS), extend the functionality of the underlying data flow methodology. An expression parser is used to evaluate numerical, logical and character expressions. To help manage the problems of visual complexity, a visual hierarchy of workspaces is also provided within cantata.

2.3.1. Glyph Types

Process (Transfer) Glyphs. Process glyphs represent the processing routines in the associated Khoros libraries. Their purpose is to receive as input one (or more) data tokens, perform the indicated operations and produce an output token which is then passed on to the next node in the flow graph for further processing or visual output. The vmul glyph in Figure 3. is an example of a process glyph. Input (Source) Glyphs. Input glyphs are the source nodes which, provide the initial data tokens for the data flow graph. Initial execution of a cantata program identifies the direct descendants of these nodes as having valid data to be processed. The current libraries contain both sample images and signals. Access is also provided for user supplied data files. In addition, routines are available which can create custom multidimensional signals of the required format based on user-specified parameters. The input glyph (see Figure 2) may also be included in a loop to provide a succession of image files to be processed sequentially.

INTEGRATED DATA FLOW VL

227

Output (Terminal) Glyphs. To display the result of processed data, various output glypbs may be connected to a processing flow graph. These include image output in the form of an icon or a full-sized window, and the output of scalar values in the form of an ASCII file. 2-D/3-D plot, surface visualization and animation packages, and an interactive image display/editor may also be accessed from within cantata. Extractor Glyphs. In addition to the standard processing of data tokens to produce another data token, it is often desirable to extract scalar information to be used as either output to the user or internally to provide information upon which data dependent flow control of the processing program may be based. An extractor glypb is used to obtain this information from a data token (connection) and provide the result to the expression parser via UNIX interprocess communication (see Section 2.3.2). IF GIyph. Conditional constructs are used in data flow programs to provide selective routing of data tokens among nodes. The interface to the IF=THEN-ELSEglyph is a subform consistent in usage with other subforms in the system [see Figure 4(a)]. Control is achieved through the use of the expression parser to evaluate, as either 'true' or 'false', a conditional expression that has been entered on the subform/pane. The result of this evaluation determines the flow of available data down the graph. As described above, specialized glyph routines which extract information from a connection provide the necessary input to the expression parser to be used as the basis for data dependent decision-making. In Figure 4(b) an IF-THEN-ELSEglyph is combined with a MERGE glyph (described in the next section) to create the standard double alternative decision structure. A single alternate (IF-THEN) structure may be implemented by just ignoring the 'else' output connection (which is optional). The glyphs with the double border are subprocedure glypbs and represent hierarchical workspaces (described in Section 2.3.3). Here they are used just to indicate the portion of the flow graph corresponding to the alternate paths for 'true' and 'false'. MERGE Glyph. The MERGE glyph was created to join together two mutually exclusive data flow paths in an application program. For example, it can be used to rejoin the 'then' and 'else' paths after an IF-THEN-ELSE as illustrated in Figure 4(b). When the glyph is executed, the input filename which is currently active will be passed to the output filename. Note that it is assumed that only one of the two input connections will be valid at one time. If, by chance, both are valid at once, the input from the top arrow will be passed on. The second input is optional. LOOP Glyphs. Iteration can be accomplished through the use of cyclic data flow graphs. COUNT-LOOP and WHILE-LOOP glyphs are higher-level control nodes which have been designed to combine the two functions of 'selector' and 'distributor' required for constructing conditional loops for traditional data flow graphs [18]. The arrival of input to the LOOP glyph initiates the iteration sequence. Output, produced by the body of the iteration, is cycled back on.~ feedback path until the conditional expression, as specified on the subform, is satisfied. Execution then continues down the exit path from the loop. The RESET button is used to set the glyph so that, when run again, the initial input to the loop will be accepted and the initiation sequence re-executed. By consolidating all the information needed to control a loop into a single pane [see Figure 5(a)], these constructs should simplify the programming of loops while maintaining the flavor of process flow. They enforce the use of standard control structures, long held to be desirable for consistent 'structured' programming.

228

j.R. RASURE & C. S. WILLIAMS

Standard Control Structures

Choose Selection

If-Then-Else Control Structure

J
I

Count-Loop

J
I

ile-Loop

If Conditional Expression Control Path Connections:

[test_value > count

f-Then-E~s~ Herge

Input [data_in~ Then (TRUE) Output Else (FALSE)

Vusr/tmp/if_eIgf~qa07093

HELP

[]

Output ~usb/tmp/it_eICAAa07093

QUIT

.Help

ELSE-PART
Figure 4. (a) Subform for the IF-THEN-ELSE glyph. (b) Conditional construct using the MERGE glyphs

IF-THEN-ELSE and

Figure 5(b) illustrates the use of the WHILE-LOOP glyph (use o{ the COUNT-LOOP is very similar). Initial input (if needed) comes in through the top input arrow. If the loop body is to be executed, this input is then passed out the bottom output. Loop feedback comes in through the bottom input, and the final output (if any) appears at the top. The program flow graph shown in the workspace of Figure 2 illustrates the use of

INTEGRATED DATA FLOW VL

229

StandardControlStructures
Choose

I
I

k~ile-LoopControl Structure While-LoopParameters: Cou~t-Loop I [ ] Initial Expression Itime= 0 hile-Loc, F [ ] UpdateExpression Iti~e : ttme~ delta Test Condition Itime^< stop IF-Then-Else I

Selection

Merge

I
Control PathConnections: [ ] Initial input Ibe9in_data~ Loopinput ~usr/tmp/IF_elCQQa07093 [ ] Exit output ~ Loop output llusr/tmpliF_elBAQa07093^

HELP

I 1
I

Help

/
INITIQLIZE

FINQLIZE

while_loop

LOOP-BO~Y
Figure 5. (a) Subform for the WHILE-LOOPglyph. (b) Loop construct using the WHILE-LOOPglyph

the COUNT-LOOP to average a sequence of five noisy images. The problem is to process five different images of the same picture, which contain zero mean random noise (like the first icon image at the bottom of the figure), and average out the noise. Initial input to the loop is a zero image. Each time through the loop a new noisy image is added to the sum and the result is displayed. The successive results are shown in the icon images at the bottom of the figure. The final result is displayed in the large

230

J . R . RASURE & C. S. WILLIAMS

image to the right. Feedback from the loop comes from the output of the vadd glyph. Since connections are drawn point-to-point, they can be difficult to distinguish. A much needed enhancement to cantata is the ability to route connections around glyphs so that the flow is more visually apparent.

2.3.2. Expression Parser

The cantata expression parser is used to define workspace variables and to evaluate numeric and logical expressions. Expressions may be used in any numeric input parameter field on the subform panes. During execution, the parser is used to evaluate the expression and assign a value to the field based on the current value of defined variables. Conditional expressions, used to determine the behavior of control nodes, are also evaluated by the parser. The cantata programmer may access the parser directly by using the Variables form via the button on the master form. Variables may also be defined on any numeric field on operator panes or by control or extractor glyphs. The scope of a variable is defined by its place of declaration within the hierarchy of workspaces. Normally, the context of a variable is local to the workspace in which it is defined. As an option, a workspace may search its parent(s) for the definition of variables not found locally. (See Section 2.3.3 for a description of workspace hierarchy.) The value of all currently defined variables for a workspace is available by using the LIST button. Figure 2 shows the Variables form (upper left corner of the workspace) being used to define the limit value, " C o u n t - 5 " , for the COUNT-LOOP~.The expression parser was written using LEX and YACC and supports a superset of the C language and U N I X maths library including standard arithmetic and logical operators, constants and mathematical functions. Character variables, which can be used to represent a sequence of connection names, are also evaluated by the parser. The ability to use textual expressions as parameters within cantata provides a signficant increase in functionality without a corresponding increase in visual clutter. Functionality at this level is either not available in other visual systems, like AVS, or is provided only at the graphical level, as in HI-VISUAL [20]. Using textual entry of mathematical expression instead of pure data flow is simpler and less tedious. It also hides low-level details not typically needed at the conceptual level of the node operators. However, expressions can add another level of data dependencies to the data flow graph which is no longer visible. Even worse, in the current implementation of cantata, these variable evaluation dependencies are not recognized by the execution scheduler, so that it is up to the programmer to make sure that the implied interdependencies follow the explicit flow of data. The solution is for cantata to keep track of the variable interrelationships, treating them the same as flow graph data dependencies. The visualization o f these interconnections could then be provided as an option, to be explicitly shown as a superimposed layer in a different color, or to be hidden, as desired.

2.3.3. Visual Hierarchy

Visual programs consisting of single large graphs are cumbersome to deal with. At some point, as the size and complexity of a program is increased, the advantages

INTEGRATED DATA FLOW VL

231

gained by visually representing the program flow are lost in a clutter of detail. While a scrolling workspace may ease the situation, the problem is an essential one limited more by the human field of vision than by size of the workstation screen. This issue is addressed at two levels within cantata. The keys to managing complexity are to: (1) find the right degree of granularity of the nodes; and (2) provide a visual hierarchy corresponding to increasing levels of abstraction. Based on the concept of large-grain data flow, components of a cantata application program are not single statements, but complete algorithms written in a conventional programming language. Standard routines are available from the Khoros libraries, which are designed to be augmented by the application developer using the code generators as described in Section 3. Processing routines can have arbitrary granularity, but typically range from 50-500 lines of code. Similar to the step from assembly to high-level languages, a visual language provides the end user with a higher level of abstraction. Thus, the cantata programmer can experiment with and develop a complex application program, which applies a number of individual routines, without being concerned with the details involved in implementing a specific processing algorithm. A hierarchy of workspaces can be used to manage further the visual complexity. In much the same way that subroutines and procedures are used in conventional languages, visualprocedures add structure to a visual language. A cantata procedure is created by first bringing up a subprocedure workspace which is virtually identical to the main cantata workspace. Desired routines are selected from the subprocedure

OUTPUT

::

~ .........
r ~ :t t::',' 1:;I., I ,

I~

~ .................

[]

[~

# 1: r : ~:::1 ! : ~

,' : ~ I : : : 1

T ~::T':I ~;~

I .~

~ I:,

[1/I

|:~:~: ;~"

l:,:l

I,
Figure 6. Cantata application program with subprocedure workspace. The ~rr_Filter subprocedure parameters are based on extracted image variance

232

J . R . RASURE & C. S. WILLIAMS

master form in the usual manner, and the corresponding glyphs connected to each other and to glyphs in the main workspace. Valid connections are permitted to either simple glyphs or other subprocedures within that subprocedure's parent workspace. Nesting of subprocedures (creating subprocedures inside of subprocedures) is also supported if those connections follow a tree structure. In other words, glyphs within a given workspace may be connected to other glyphs within another workspace only if the workspace is related as parent, child or sibling (another child of the same parent). Users of cantata have written programs containing over 100 nodes, with eleven subprocedures nested three deep. Glyphing the subprocedure workspace replaces it with a single procedure glyph (identified by a double border) with all its connections to the outer world visible, but with its inner workings hidden. Clicking on the top middle button on the procedure glyph brings back the subprocedure workspace of glyphs. Subforms for routines included in the subprocedure can be opened so that parameters may be selected and modified as usual. Figure 6 shows a cantata flow graph containing a user defined subprocedure called MAR._Filter. The subprocedure was created and saved and then used twice (with different parameters) within the same main workspace program. It appears in both branches after the IF-THEN-ELSE glyph. The subprocedure in the 'then' path is glyphed, while the one in the 'else' path is open to show the internal glyphs and connections. This program also makes use of the extractor glyph varviff to supply the variance of the: image to the expression parser. The variance is then used by the IF-THEN-ELSE gly~ph to determine which version of the Mar_Filter should be run. The two paths are then merged back together to display the result.

2.4. Language Support Mechanisms

To support the creation and execution of visual programs, a variety of mechanisms are required. These include compilers and/or interpreters to translate the program into machine executable form. The computational model for a particular language and/or application must be selected and applied. For languages that do not explicitly state the order in which operations should be performed, a scheduler must determine a sequence for executing a program. Different execution modes must be considered and a representation must be chosen for data tokens. And finally, debugging support must be provided to help prevent, discover and correct errors. There are many factors to be considered when choosing the mechanisms to be supported. The following discussion seeks to identify those capabilities that are needed for a general purpose, data flow based visual language to appeal to a broad audience of users. These have been organized into distinct categories, but there remains much overlap and.many interrelated decisions. Often the choices to be made are naturally arranged as two opposing views where important attributes and limitations of both options may be presented and compromise positions offered, as appropriate. 2.4. I. Translation Scheme Before a computer program can be executed, it must be converted into a form suitable for the machine on which it will be run. This may be done in one of two general

INTEGRATED DATA FLOW VL

233

ways: compilation or interpretation. The right choice for a particular implementation depends a great deal on the purpose of the language, whether it is intended for prototyping or production. The use of an interpreter generally allows for greater flexibility, but at the expense of execution speed. To promote a research environment conducive to free experimentation, interactive interpretive program development support is required. It is important to be able to execute a program as it is being assembled to view intermediate results and incrementally add to the developing program (prototyping mode). For this reason, most visual languages based on data flow operate in an interpretive manner. The exception is Gabriel, the purpose of which is to generate executable code to be down-loaded onto special-purpose DSP processors. On the other hand, once a program has been created and is to be used repeatedly without modification (production mode), it is certainly advantageous to have a single compiled executable. Language environments for Pascal and C often provide a interpretive mode for program development and debugging, and a compiled version for rapid execution of stable code. Cantana is an interpretive language. Selected portions of a flow graph may be executed at any point during program construction. The flow graph nodes represent compiled executable routines of large granularity. Because of the relative size of the nodes, the overhead associated with forking off the process is small compared to the total execution time. Creating a compiled version of a cantata program (as planned for a future implementation), would involve quasi-static scheduling of the flow graph (described in Section 2.4.3)] The 'object' produced by the compiler would consist of a sequence of calls to library routines, interspersed with the code necessary to implement any control constructs and data buffering/storage involved. 2.4.2. Computational Model There are two basic approaches to executing a data flow graph: data-driven, where execution order is determined by the availability of data; and demand-driven, where execution order is determined by the requirements for data. Because of its simpler firing rules, the data-driven model is used most often for both graphical and textual data flow based languages. In cantata, normal execution is based on the data-driven model. Scheduling starts with the input source glyphs (see Section 2.3.1); their output provides valid data to the inputs of glyphs to which they are connected in the flow graph. These glyphs, in turn, provide data to their connections. In this way, computations are driven down the flow graph as data becomes available to successive operators. The disadvantage of the data-driven model is that certain portions of the flow graph may be executed unnecessarily if their contribution is not needed to produce the final result. Consequently, demand-driven execution is available as an option within cantata, so that the user my initiate computation by selecting an output glyph. Glyphs that must be run to provide the desired output are then identified and executed. Demand-driven execution starts with the desired output node and traces back through the flow graph to locate the input sources required to produce that result. Consequently, only the essential computations are performed, which may provide a considerable saving in execution.

234

J . R . RASURE & C. S. WILLIAMS

AVS uses a hybrid computational model which is primarily demand-driven in the 'lazy evaluation' style (i.e. execution occurs only when that data is needed). It is data-driven in that changes to module parameters are reported down the flow graph until a terminal node (typically a display window) is reached. At this point the display realizes that it is 'out-of-date' and, in demand fashion, requests recomputation of its data. Similarly, a batch mode output device pulls the data through as it is needed. The effect is a ripple of requests back through the flow graph from these terminal nodes, and then a ripple of computation forward. This method was chosen over traditional data flow due to the demands of real-time responsive visualization.

2.4.3. Scheduler
The scheduler determines when (and where, in a multiprocessor environment) the nodes of a data flow graph are to be executed. Scheduling may be accomplished either statically (at compile time) or dynamically (at run time or interpretation time). This terminology may also be applied to the linking of libraries. A static scheduler takes a complete program and, based on the data dependencies expressed by the interconnection of nodes, determines a sequence of execution of the nodes. Optimal scheduling onto parallel processor architectures can be accomplished, given a priori knowledge of the run time for each of the nodes. This is important for a specia ! purpose system like Gabriel which compiles assembly code for one or more processors. Unfortunately, this significantly limits the generality of algorithms that can be represented. Because static schedules do not vary at run time, conditional branches and data dependent iteration are not allowed. This inflexibility makes static scheduling unsuitable for a highly interactive, general purpose, programming environment. A dynamic scheduler determines the execution order of nodes at run time. Typically, a scheduling list is maintained of all nodes that are ready to fire, based on the availability of data tokens. In cantata, when a flow graph is executed using the RUN button on the master form (described in Section 2.4.4), the scheduler is called to build a list of all glyphs which are immediately qualified to run. These include source glyphs (which have no inputs) and glyphs for which input data is already available (from a previous RUN). The first glyph on the list is then dispatched for execution (unless "parallel execution' is enabled, in which case all are dispatched). When several nodes are ready at once, it is an arbitrary choice as to which will be dispatched next. Cantata places new nodes at the head of the list to ensure a depth first exploration of the graph. Glyphs representing library routines are executed by forking a process. Control glyphs (IF, LOOPS and MERGE) are interpreted internally by cantata. A glyph representing a subprocedu.re workspace is scheduled independently by a recursive call to the scheduler, so that separate scheduling lists are maintained for each workspace. Once a glyph has been executed, its outputs then validate'the inputs of the next glyph in the flow graph. Glyphs with all inputs satisfied are added to the list. This process continues with the selection of another glyph until all glyphs have been executed and normal termination occurs. Because glyphs are scheduled dynamically, flow graph connections, as well as routine parameters, may be modified during execution. If an error is detected when a routine is run, execution is terminated for

INTEGRATED DATA FLOW VL

235

that portion of the flow graph headed by the corresponding glyph. Unaffected paths continue to run. To avoid synchronization problems, cantata forces the complete execution of a loop before the next instance can be started. This is of special benefit when there is a cycle with a spur branching out as in the loop in Figure 2. It is more efficient to execute the spur before looping back, to avoid having to buffer a backlog of unprocessed executions at the fork. It also insures the integrity of the loop control variable in the case where its value is dependent on data produced in the spur.

2.4.4. Execution Modes

To provide programming support for a broad range of application activities, a variety of execution modes should be included. These fall into two general categories: execution control for an entire flow graph and execution type of an individual node. The fundamental actions needed to control the execution of a program flow graph include Run, Stop, Continue and Rerun, which are provided in cantata by a single button. Initially labeled RUN, when selected the label changes to STOP (displayed in reverse video). Thus, when clicked on again (as STOP) execution of the flow graph terminates, killing any processes currently running. During execution, sufficient information is retained to allow the user to halt execution, modify parameters or connections, and resume execution at the last state not invalidated by the user modifications (see related Tdiscussion in Section 2.4.5). This includes correcting for errors detected during exec~ution, so that the entire flow graph does not have to be rerun. The next time the RUN button is selected it will Continue execution of the flow graph. To Rerun the entire program, the global RESETbutton is first used to force the re-execution of all glyphs, whether they have been modified or not. To just reinitialize loops, the 'reset' button on the individual loop control glyphs may be used. Single-step execution of a program is an important tool in program development and debugging. Since all executable glyphs in cantata have their own 'run' button, it is easy to single step through a flow graph. In fact some users prefer this mode, especially during program development. It is extremely valuable to be able to execute incrementally, view results and add to the flow graph as a problem solution is prototyped. Parallel execution mode is controlled by a toggle on the Wor~l~ce/Environment pane. If parallel paths are detected in the flow graph during scheduling of glyph routines for execution using the RUN action button, they are then dispatched in parallel by the cantata scheduler. The set of all glyphs with fully qualified inputs (and no data dependencies) is executable in parallel on multiprocessor architectures. A level of responsiveness may be established for execution to allow automatic triggering of re-execution of the flow graph: (1) whenever new input data is available; or (2) in response to a modification of the flow.,graph (typically by adjusting a node parameter). Responsive mode is turned off during initial construction of a program, then could be turned on for final testing and use. This feature is controlled by another toggle on the Workspace/Environment pane. A hybrid computational model may provide more efficient execution (see Section 2.4.2). Several data flow based visual languages, for example HI-VISUAL, execute a node immediately upon being connected to input data. This would not be appropriate for a

236

i . R . RASURE & C. S. WILLIAMS

large-grained data flow language where nodes may be expensive to recompute. AVS also supports this execution mode, but allows user control to disable the feature. Within the context of executing an entire flow graph, the behavior of each individual node is governed by its own execution type. For a simple flow graph in which each node is executed at most once, this will have little meaning. But for flow graphs containing loops, or during the responsive re-execution of flow graphs, it is an important consideration. Three primary execution types have been identified for cantata glyphs based on what happens when the node is re-executed: Single Run, Rerun and Always Run. The choice of execution mode is typically a function of user interest. For example, a display node can be run in any of the three modes. Single Run mode terminates the current display and replaces it with a new one; appropriate if only interested in a final result. Rerun mode initiates a new display, leaving the current display intact; useful when comparing results. Always Run mode updates the output within the current display; this provides an animation of the changes between successive results. Input source nodes could be designated in a similar fashion. Single Run would always input/generate the same data. Rerun would input/generate a succession of data tokens (for example, random noise). Always Run would continuously input/generate a stream of data values.

2.4.5. Data Token Representation

Data token transport metllods within UNIX include: files, pipes, shared memory, sockets, etc. Choice of a model for representing the transport of data tokens is affected by several interacting requirements, the primary trade-offs being execution speed, data size and recoverability. In the traditional data flow execution model, when a node fires, it 'consumes' its input data, so that the token is removed from the input arc. There are at least two reasons why this practice may not be strictly adhered to: fan-out and recoverability. In general, to support fan-out from an operator (connecting an output port to multiple input ports), a data token must be copied to provide individual tokens for input to each receiving node. This may involve excessive copying of data which can be avoided by refereffce counting (allowing tokens to be saved until all affected nodes are finished). The issue of recoverability is not so easily addressed. During program construction and debugging, it is very convenient to be able to execute incrementally and add to or modify a program flow graph, recomputing only those portions affected by the change. This is especially important if the flow graph contains operators that take a long time to compute. To solve these problems, cantata takes the most conservative approach. By saving all intermediate results of the most recent execution in files or shared memory, unnecessary recomputing or copying'is avoided. The execution state of the flow graph is thereby maintained and can be recovered. When the user desires to SAVE the current cantata session, the execution state and intermediate results can be stored along with the User Interface Specification (UIS) and the workspace. Upon starting another session and loading the context of the stored one, the user may re-enter the scheduler at precisely the point where execution was terminated in the previous session. A compromise to avoid the problems of excessive memory consumption could have

INTEGRATED DATA FLOW VL

237

simple operators always recompute their results. More complex operators, for which it is expensive to compute but cheap to save, could choose to cache their results in case they might be reused next time. This has not been implemented in cantata. 2.4.6.

DebuggingSupport

Debugging support within a programming environment takes the form of both error prevention and error detection/correction. This support can be provided "at different levels throughout the programming process. The following discussion will be broken into three phases: program construction; pre-execution; and run time checking. Prevention of basic syntactic errors during program construction can be enforced by the visual editor. Since programming consists primarily of connecting modules together, illegal connections can be disallowed. For example, it is impossible to connect two input arrows together in cantata. Also, since fan-in of inputs is not allowed, attempting to make a second connection to an input arrow replaces the first connection. With flow graphs involving more than one type of data token, type compatibility of connections should be checked and flagged with an error message. AVS color codes different type I / O ports and data connections. Polymorphic operators, as used by cantata, accept all legal data types and/or aut9matically convert to correct data formats which simplifies ease of use, as well as prevents the most common errors. ' Parameterized operators must be checked as they are configured. The forms based interface provided by the GUI library simplifies selections of many parameters to that of mutually exclusive choices, toggles and sliders. Numeric constants are collected and automatically checked against legal bounds when a subform is closed. Use of variables and expressions adds extra complexity to the detection of errors since parameter values cannot be checked until run time, although expression are checked for syntactical correctness. Checking a completed flow graph for certain types of errors can be done prior to execution. Cantata provides a CHECK button on the master form which indicates glyphs that have required inputs that have not been connected. 3. T h e U s e r Interface D e v e l o p m e n t S y s t e m The goal of the Khoros User Interface Development System (UIDS) is to provide application developers with the tools needed to create user interfaces to application programs. The role of cantata within the Khoros system is to provide an end-user with a graphical programming environment from which these application programs may be accessed as operators for a data flow visual language. The previous section described the user interface, language specifications and support mechanisms of cantata. The purpose of this section is. to describe how the components of the Khoros software structure outlined in the introduction (see Figure 1) are used to build and extend cantata.

3.1. The Command-line and Graphical User Interface Modes

Khoros applications may be created with two types of user interfaces. The Khoros Graphical User Interface (GUI) library presents the user with a graphical, form/menu-based, mouse-oriented interface and runs on workstations under the X

238

j . R . RASURE &: C. S. WILLIAMS

Windows network transparent window system'.)~A textual, command-line, keyboardoriented, user interface provides quick and straightforward access to programs for expert users who are accustomed to the UNIX operating system and the C shell, and have no need for the mediation of a graphical interface. Both user interfaces are specified using the same User Interface Specification (UIS) file (Section 3.2) and the code to implement them can be automatically generated by the Khoros code generators (Section 3.3). For example, the Khoros libraries of processing routines are made up of programs which have a command-line user interface generated from a UIS file plus a Program Specification (Section 3.4). The GUI library can also be used to interpret a UIS file to create a graphical user interface at run time. Cantata makes use of this interpretive mode. Once a command-line user interface has been generated and used to create a functional processing routine, cantata can interpret the same UIS file to present a graphical user interface in the form of a pane or a glyph. Cantata executes the routine by gathering the parameters (input/output files, slider values, toggles, etc.) from the pane to create a proper command-line string and then forks this as a process.
3.2. User Interface Specification

As seen in Figure 1, the User Interface Specification (UIS) is central to the software structure of the Khoros system. Information concerning thephysical appearance and functionality of an application interface is contained here. The specification is an ASCII file co/nposed of a sequence of lines following a strict syntactical structure, each of which describes a separate part of the user interface. Designed as a high level description, the specification is used to generate directly the code for both the graphical and the command-line user interfaces. In addition, it represents a formal specification of the dialog between the end-user and the application, independent of the user interface mode. The 'look and feel' of the user interface is a function of the Athena widgets and is independent of the UIS. Creation of a UIS file is discussed in Section 3.2.2. The Khoros UIDS was specifically designed to aid in developing the Khoros system and its applications. One of the design goals of the UIDS was to develop tools which are easy to use by application developers. This required a sacrifice in generality that resulted in a user interface specification with a fixed user dialog with limited methods for input and output. Consequently, it is easy to use and powerful when working within the Khoros environment (with its underlying assumptions and design decisions), but the UIDS is limited in its general applicability. Figure 7 presents an excerpt from the UIS file for cantata which illustrates the type of information contained. Each line corresponds to an option or button (or grouping of these) on the forms-based portion of the graphical user interface as displayed in Figure 3. The parameters in each line control the data type, position, size, default values, variable names, command-line switches and action controls for that item. These are described in more detail in the following section.

3.Z1. UIS Components

The forms-based graphical user interface consists, at the highest level, of a master form (corresponding to the -F line in the UIS, see Figure 7). The master form may

- F 4.2 1 0 170 7 + 10 + 20 + 25 + 1 'CANTATA Visual Programming Environment for the KHOROS System' cantata - S 1 0 - d 1 0 10 x 1-5 + 1 + 7 %Xforkpspace' - d 1 0 10 x 1.5 + 1 + 8-5 %'ariables' - n 1 0 10 1.5 + 1 + 10 'ROUTINES"creates glyphs from routine names' accelerate -d 1 0 10 'Control' - d 1 0 1 0 x l . 5 + 1 +13.5 'Comment' -d 1 0 10x1.5+1+15 'Command' -n 1 0 10xl.5+1+17 'RUN"runs the pipeline in order' auto_run -n 1 0 10xl.5+1+17 'STOP"runs the pipeline in order' stop_auto_run - n 1 0 10 x 1 + 1 + 18.5 'RESET''reinitialize pipeline to reexecute' reset - n 1 0 10 x 1 + 1 + 19.5 'CHECK"check I/O connections on glyphs' check - n 1 0 10 x 1 + 1 + 20.5 'REDRAW''refresh the lines connecting glyphs' redraw -H 1 10 x 1-5 + 1 + 22 'HELP''guide help' KHOROS_HOME/doc/cantata/master_help/ help - Q 1 0 1 0 x 1 " 5 + 1 + 2 3 - 5 'QUIT'
~ 1 7 6 . . ~

-D -d -d -d -d -E
~ , , .

1 0 16x2+31+4 1 0 16 1 0 16x1+31+4 1 0 16x1+31+4 1 0 16

+ 0 + 0 'ARITHMETIC'arith_menu 'Unary Arithmetic' 'Binary Arithmetic' 'Logical Operations' 'Matrix Algebra'

-E
, , ,

-M 1 0 100 10 + 20 -F 23 + 1 'Binary Arithmetic Operations' binary - G 1 20 x 38 + 1 + 2 + 2 + 0 'Choose Selection' 'Addition' -g 1 1 18xi+I+1 -g 1 0 18x1+1+2 'Subtraction' -g 1 0 18:<1+1+3 'Multiplication' -g 1 0 18xi+1+4 'DMsion' -g 1 0 18xi+I+5 'Absolute Difference' -g I 0 18xi+I+6 'Blend Images' -H I 18x2+I+11 'HELP' 'guide help' KHOROS_HOME/doc/cantata/ subforms/arith_binary/help -Q 1 0 18x2+1+13 'QUIT' -E
~ 1 7 6

-P 1 0 80x38+22+2 +8+0 -I 1 O0 1 0 1 50x1+1+2 -I 1 O0 1 0 1 50x1+1+3 -0 1 O0 1 0 1 50x1+1+5 -I1 01 001 50x1+1+7

'Multiply Two Images Pixel by Pixel' vmul " I n p u t Image #1' 'first input image' il " I n p u t Image #2' 'second input image' i2 " O u t p u t Image' 'resulting output image' o ' 'Masking Image' 'multiply operation masking image' m -1 1 0 1 0030:<1+1+90+0 1 'Select Scalar or Complex Arithmetic' 'Scalar' 'Complex' 'Complex[T] Scalar[F]' j -R 1 0 1 13x2+11 'Excute' "excute operation' vmul -H 1 13 x 2 - F 3 8 + 11 'Help' 'man page for vmul' KHOROS_HOME/docfmanpages/ vrnul.1 -E +0+0 0+0 0+0 +0+0
~ 1 7 6

-E
~ 1 7 6 1 7 6

-E

Figure 7. Excerpt from cantata User Interface Specification

240

J. R. RASURE & C. S. WILLIAMS

contain (within -S to -E grouping) master action buttons (-n lines) as well as subforrn buttons (-d lines). Master action buttons are used to indicate that some specific action should be performed immediately. Control is passed to the application program to take the appropriate action. For example, the RUN master action button is used to execute the program flowgraph of glyphs contained within the current cantata workspace. Subform buttons are each labeled with the broadest category of operation that is dealt with in the application program. These may be grouped on pull-down menus (-D to -E grouping). See the ARITHMETICgrouping in the UIS of Figure 7. When the user clicks the mouse on the subform button labeled with the desired category, a subform will appear (-M to -E grouping). The Binary Arithmetic subform grouping in Figure 7 corresponds to the GUI subform displayed in the cantata workspace of Figure 3. Each subform contains a guide pane (-G to -E grouping), the long panel on the left of the subform, with buttons labeled with each of the subjects that fall in its category. These guide buttons (-g lines) are titled with subjects that will each require their own pane (-P to -E grouping) in the user interface. The user clicks the mouse on the desired guide button, which brings up the corresponding pane in the right half of the subform. The cantata UIS includes individual pane specifications describing each of the routines to be accessed from the Khoros library of processing algorithms. The Multiply Two Images pane grouping i n Figure 7, is accessed by the Multiplication guide button, as shown on the subform in Figure 3. The name of the actual routine to be executed, vi~ul, is indicated on the run button (-R line). The individual pane is where most input/output of the parameter information takes place. Text selections are provided for entering string (-s), integer (-i), float (-f) and logical (-I) values, as well as input (-I) and output (-O) connections. For entering filenames as connections, a file browser is provided for selecting an existing file from memory by clicking on the 'Filename' prompt. In cantata, except for original input data, filenames are typically not typed in, but are generated by cantata when a connection is made between glyphs. Additional features of the graphical user interface include feedback messages which are provided in pop-up form for errors (including automatic error checking of input data against predefined limits) and general user information. Local and global help is available from help buttons (-H lines) at each level. User customization of colors and fonts is supported at the X Windows level via the .Xdefaults file and the X Resource Manager. Modifications to the UIS are incorporated at start up of execution since the file is interpreted rather than compiled. Also built into the GUI is a journal playback mechanism which records (in a journal file) all user interactions with an application for playback at a later time. This is useful for testing, demonstrations, training novice users, and (in the future) animated help. In the context of cantata, this means that the new user can be provided with examples of how others have used the visual language to solve sample problems. 3.2.2. UIS Editor and Interpreter The first task for the application designer who wishes to develop a Khoros user interface is to create the User Interface Specification. A UIS may be created in one of two ways, or (as often happens) a combination of both. The Khoros graphical UIS

INTEGRATED DATA FLOW VL

241

Editor can be used by an application developer to select interactively appropriate options describing the desired user interface, which are automatically translated into the UIS file format and then interpreted by the GUI and displayed on the screen. The specification editor is not as general as other user interface editors/builders since it only supports user interface items that are part of the Khoros UIS. But it is simpler to use and learn since it does not require any knowledge of X Windows programming. Alternately, a standard text editor can be used to create the UIS directly. In this case a prototype of the graphical interface can be interpreted and then displayed by the UIS Previewer in an active but non-functional state and modified until its appearance is as desired. Syntax checking is automatically done on each entry for the new form and erroneous or illegal specifications are rejected. 3.3. Code Generators The code generators for Khoros use the approach advocated by the Programmer's Apprentice [21]; they do not attempt to provide a completely automatic system. Instead, they act as assistants to the programmer, by providing aid in the tasks that prove to be invariant across applications. This includes generation of modular code to extract user input from the user interface, and creation of the C structures needed to mediate between the application program and either type of user interface plus the code to properly initialize these structures. Also generated are the generic application drivers that may be modified as desired by the application developer, as well as consistently formatted manual pages and usage statements. A popular approach to software engineering states that one should completely define all requirements, parameters and I / O mechanisms before beginning the implementation of a software project. However, this assumption is not realistic: 'The reality is that requirements change continually, often in response to organizational change. These changes are more likely to emanate from experience in the use of the system than from an abstract specification in the early design of the system' [22]. The Khoros code generators are designed with this reality in mind. Initially, the programs can be compiled and executed with only the code as created by the generators for a n e w application program. Both generators will create the drivers, interpretive structures, initialization code and code to provide the links between the application program and the user interface. Thus, application developers may generate all the user interface code that is needed by their program without having to know anything about the mechanics of getting input from the command line, or how to program using X Windows or the X Toolkit. Productivity can be significantly boosted, since the application designer is freed from the details of implementing the user interface and can concentrate on developing the functionality of the application program. Just as important, the code generators were developed specifically to minimize the amount of work necessary to do maintenance on the application programs. Because the user interface code is separated from the actual functionality of the application program, and because the code generators were implemented to function both as design tools and as maintenance tools, once the application developer has begun to add to or modify the existing UIS, either of the code generators may be re-run to update the existing structures and code. Therefore, the code generation programs are

242

j.R. RASURE & C. S. WILLIAMS

-AUTHORS Scott Wilson -AUTHORS_END -SHORT_PROG_DESCRIPTION Multiply two images, pixel by pixel. -SHORT_PROG_DESCRIPTION_END -MANI_LONG_DESCRIPTION vmul computes the product of two input images pixel by pixel. The optional input mask is used to gate the operation. A non-zero mask pixel enables the output pixel to contain the sum. A zero mask pixel transfers value of corresponding pixel of first input image to the output pixel. -MANI_LONG_DESCRIPTION_END -MANI_EXAMPLES vmul - i l ball~xv - i 2 feather.xv - o outPut.xv -j 0 Multiplies images "balb~v" and "feather,xv" with the resulting image written to "outimage=xv" The " - j 0" says NOT to use complex arithmetic. -MANI_EXAMPLES_END -MANi_RESTRICTIONS vmul will not operate on BIT data types. -MAN 1_RESTRICTIONS_END -INCLUDE_MACROS #define CHECKINPUT(program, imgl, img2) #define MATCHINPUT(program, imgl, img2) (INCLUDE_MACROS_END -MAIN_VARIABLE_LIST struct xvimage *imgl, *img2, *img3, *readimageO; -MAIN_VARIABLE_LIST_END -MAIN_BEFORE_LIB_CALL /* Verify input images */ if (check_argsO) exit (1); imgl = readimage (vmul - > il_file); if (imgl = = NULL) exit (1); img2 = readimage (vmul - > i2_file); if (img2 = = NULL) exit (1); if (vmul - > re_flag) img3 = readimage (vmul - > m_file); if (vmul - > re_flag && (img3 = = NULL)) exit (1); CHECKINPUT(program, imgl, img2); if (vmul - > re_flag) MATCHINPUT(program, imgl, img3); -MAIN_BEFORE_LIB_CALL_END -MAIN_LIBRARY_CALL if (!lvmul (imgl, img2, img3, vmul - > m_flag, vmul - >j_logic))
Figure 8. Excerpt from Program Specification for vmul

INTEGRATED DATA FLOW VL

243

{ )
(void) fpfintf(stderr, "lvmul failed"); exit (1);

-MAIN_UBRARY_CALL_END -MAIN_AFTER._LIB_CALL (void) writeimage (vmul - > o_file, imgl); -MAIN_AFTER_LIB_CALL_END -SHORT_UB_DESCRIPTION MulUply two images, pixel by pixel. -SHORT.UB_DESCRIPTION_END -MAN3_LONG_DESCRIPTION vmul computes the product of two input images
. o .

-MAN3_LONG_DESCRIPTION_END -MAN3_RESTRICTIONS lvmul will not operate on VFF_TYP_BITdata types. -MAN3_RESTRICTIONS_END -UBRARY_INPUT imgl first xvimage structure img2 second xvimage structure img3 gating mask image mflg flag set (1) if gating image mask to be used jflg scalar (0) or vector (1) complex arithmetic Note: all inputs must be the same data type and size. -HBRARY_INPUT_END -UBRARY_OUTPUT imgl holds the result of the multiply operation. This is done to save space, but care must be taken not to overwrite important data. Output data type is the same as input data type. Return Value: 1 on success, 0 on failure. -LIBRARY_OUTPUT_END -UBRARY_DEF int lvmul (imgl, img2, img3, mflg, jflg) slruct xvimage *imgl, *img2, *img3; int mflg, jflg; -UBRARY_DEF_END -LIBI~d~Y_CODE

. . .

switch (imgl - > data_storage_type)

case VFF_TYP_BIT: ~dn~(stderr, "cannot multiply BIT images"); return (0); break;
Figure

8---(contd.)

244 case WF_WP_I_BYTE:

. , . . o .

J.

R. RASURE & C. S. WILLIAMS

} }
return (1); -LIBRARY_CODE_END
Figure

8--(contd.)

not only an important aid in the initial development of an application program; they are invaluable tools for the ongoing maintenance of such programs. 3.4. Program Specification In addition to the UIS file, the Command-line Code Generator may also take as input a Program Specification to generate a complete program or library routine. In this file, the application developer provides additional information that cannot be included in the UIS, such as program header documentation, syntax for library calls, and specific code segments that are to appear in the application program. Figure 8 contains an excerpt from the Program Specification for vmul. This was used, along with the UIS for the Multiplication pane (see Figure 7) to generate the complete code for the library routine. When maintenance is required, it is only necessary to update the Program Specification file or the UIS file, or both, as needed. Re-running the code generator updates the C code that controls the command-line user interface. The automatic generation is bi-directional; the application programmer may use the Program Specification to update the C code and manual pages, or may take the opposite approach, using the C code and" manual pages with a companion program to automatically update the Program Specification file. In this way, the system is made flexible enough to accommodate the needs of a wide variety of approaches on the part of application programmers, many of whom may work on a particular application program in the course of its history The use of the Program Specification promotes standardized styles and methods of programming, and encourages consistent implementation of applications across the system. 4. C o n c l u s i o n a n d C o m m e n t s This paper addresses many of the issues related to designing a data flow visual programming system that can be used to solve a broad range of problems. Our approach is to create a general-purpose, processing domain-independent, languagecentered visual programming environment with a complete set of language support mechanisms. And then to provide the application developer with a user interface development system to enable application-specific extensions. As research proceeds in visual programming languages, code generation and user interface development systems, there needs to be an integration of these efforts, so that users of a visual programming environment can modify and extend it to better

INTEGRATEDDATAFLOWVL

245

suit their own requirements. It should not be necessary to modify a large C program to do this--a high level language or specification should be provided. These concepts, along with the hope of wide collaboration, have provided the motivation behind the development of Khoros. Other researchers who are also pursuing this goal can view Khoros as one example of how the many paradigms involved in the system can be integrated. Acknowledgments The authors would like to acknowledge the contributions of the following individuals to the Khoros project. The cantata visual language and the GUI library were implemented primarily by Mark Young and Danielle Argiro. The code generators were implemented by Danielle Argiro. The tools were written by Charlie Gage and Tom Sauer. The routines in the image processing library were written primarily by Scott Wilson, Charlie Gage and Tom Sauer. The signal processing library was written by Ramiro Jordan, Jeremy Worley and Donna Koechner.
References

1. S.-K. Chang (1987) Visual languages: a tutorial and survey. IEEE Software 4, 29-39. 2. B. A. Myers (1990) Taxonomies of visual programming and program visualization.Journal of Visual Languages and Computing 1, 97-123. 3. P. E. Haeberli (1988) ConMan: a visual programming language for interactive graphics. Computer Graphics 22, 103-111. 4. E. A. Lee, W. H. Ho, E. E. Goei, J. C. Bier & S. Bhattacharyya (1989) Gabriel: a design environment for DSP. IEEE Transactions on ASSP 37, 1751-1762. 5. C. Upson, T. Faulhaber, Jr., D. Kumlns, D. Laidlaw, D. Schlegel, J. Vroom, R. Gurwitz & A. Van Dam (1989) The application visualization system: a computational environment for scientific visualization. IEEE Computer Graphics & Applications 9(4), 30-42. 6. D . S. Dyer (1990) A dataflow toolkit for visualization. IEEE Computer Graphics & Applications 10 (7), 60-69. 7. C. G. Masi (1989) Imaging with icons. Test and Measurement World December, 85. 8. A. Wilson (1989) A picture's worth a thousand lines of code. Electronic System Design July, 57-60. 9. R. G. Babb (1984) Parallel processing with large-grain data flow techniques. IEEE Computer 17(7), 55-61. 10. S.-K. Chang, M. J. Tucber, B. Yu & J. S. Yu (1989) A visual language compiler. IEEE Transactions on Software Engineering 15, 506-525. 11. E. J. Golin & S. P. Reiss (1990) The specification of visual language syntax. Journal of Visual Languages and Computing 1, 141-157; 12. B. A. Myers (1989) User-interface tools: introduction and survey. IEEE Software 6, 15-23. ' 13. E. Lee (1990) User-interface development tools. IEEE Software 7, 31-36. 14. J. Rasure & S. Hallett (1988) XVision: a comprehensive software system for image processing research, education and "applications.,In: ACM SIGGRAPH User-Intelface Software Symposium. Alberta, Canada, October. 15. C. S. Williams & J. R. Rasure (1990) A visual language for image processing. In: 1990 IEEE Workshop on Visual Languages. Skokie, Illinois, 4-6 October, pp. 86-91. 16. B. Hailpern (1986) Multiparadigm languages and environments. IEEE Software 3, 6-9. 17. J. A. Borges & R. E. Johnson (1990) Muhiparadigm visual programming language. In: 1990 9IEEE Workshop on Visual Languages. Skokie, Illinois, 4-6 October, pp. 233-240. 18. A. L. Davis & R. M. Keller (1982) Data flow program graphs. IEEE Computer 15(2), 26-41.

246

I.

R. RASLIRE & C. S. WILLIAMS

19. E. P. Glinert & S. L. Tanimoto (1984) PICT: an interactive, graphical programming environment. IEEE Computer 17(11), 7-25. 20. M. Hirakawa, S. Iwata, I. Yoshimoto, M. Tanaka & T. Ichikawa (1987) HI-VISUAL iconic programming. In: 1987 Workshop on Visual Languages. Linkoping, Sweden, August, pp. 305-314. 21. C. Rich & R. C. Waters (1988) Research overview--the programmer's apprentice. IEEE Computer 5(11), 11-24. 22. N. F. Schneidewind (1987) The state of software maintenance. IEEE Transactions on Software Engineering 13, 303-309.