You are on page 1of 29

Final Report on the GIS Demonstration Project

National Center for Atmospheric Research Terri L. Betancourt Olga Wilhelmi

1 October 2003

Executive Summary
Between June 2002 and September 2003 a research project was conducted at the National Center for Atmospheric Research (NCAR) to explore interoperability technologies currently available in the realm of Geographic Information Systems (GIS) and to advance the scientific application of these technologies. In particular, the project was designed as a demonstration of technology options from the OpenGIS standards community as well as from ESRI, the leading commercial vendor of GIS software in the United States. This report summarizes the findings of that research project, subsequently referred to as the Demonstration Project. The atmospheric context of the GIS Demonstration Project was tied to the International H2O Project (IHOP). Perhaps the largest atmospheric field experiment ever conducted, IHOP took place over the US Southern Great Plains during May and June of 2002. The major goal of IHOP was improved characterization of the 4D distribution of water vapor and its application to improve the understanding and prediction of convection. The extensive array of heterogeneous data collected during IHOP provided an exceptional opportunity to address the question, How can GIS and related interoperability technologies address the need to integrate a wide range of existing atmospheric data into a common geospatial framework? To address this question the Demonstration Project identified a two-track implementation path. The first track focused on developing OpenGIS client/server capabilities for integrating IHOP data in its native form into existing NCAR software. The second track focused on implementing a commercial-based solution, converting IHOP data into a form suitable for ESRIs GIS environment. Although the project design called for and focused on two parallel technology tracks, the most promising outlook for long-term GIS interoperability in the atmospheric domain appears to be a hybrid architecture incorporating cross-linkages of OpenGIS and ESRI Web services technology. The driving needs for spatial data interoperability differs quite significantly between the atmospheric science community and the general GIS user community (and between specialized subsets of users within these communities). To some extent the technologies for effectively addressing the multiplicity of user community needs varies accordingly. This report attempts to identify significant user segments, associated interoperability needs, and relevant technologies. It is envisioned that the user-centric perspective presented here serve as a launching point for defining an implementation plan for future projects wishing to incorporate elements of GIS interoperability.

Scope and Intended Audience


This report is not intended to cover the comprehensive set of GIS interoperability technologies, but rather to reveal technical issues uncovered in the implementation of a limited subset during the Demonstration Project. The report identifies specific OpenGIS and ESRI software components for implementing interoperable solutions over the Internet. It describes some of the technical hurdles and limitations of these components in the context of the atmospheric domain. As such, this document can be used as a

1 October 2003

Page 2

preliminary guide for those interested in implementing software solutions for atmospheric GIS interoperability. Readers are assumed to have a general understanding of atmospheric data, geographic information systems, the role of OpenGIS standards, ESRI products, and the Internet protocols related to these topics. For information on these and other related topics not covered in detail by this report, the reader is referred to: http://www.atd.ucar.edu/dir_off/projects/2002/IHOP.html for information on the IHOP experiment and field data http://www.opengis.org for information on OpenGIS standards http://www.esri.com for information on GIS and ESRI products http://www.w3.org for information on Internet protocols

A Survey of Perspectives
To understand the rationale behind the two-track approach of the GIS Demonstration Project, it is useful to distinguish between geographic information systems and other information systems employed in the atmospheric sciences, which we will refer to in this report as Atmospheric Information Systems (AIS). Both GIS and AIS fit the classic definition of a geographic information system in that they both are used for managing, analyzing, and visualizing geo-located data. However, for the purpose of the work described in this report, differences in the software design strategies behind GIS and AIS become significant. GIS, as defined in the context of this report, represents a class of information system that has general purpose in its design. In other words, GIS software is not developed for any particular user domain. Instead, GIS tools are designed for a broad suite of industry and research applications ranging from urban utilities management to demographic studies of migration patterns. These tools tend to be highly developed in their cartographic capabilities and are becoming increasingly reliant upon relational database management systems (RDBMS) as their data storage mechanism. As a result of the broad community of users and data developers, GIS systems can be described as geospatial data fusion environments. In contrast to GIS, AIS represent a more specialized class of information system developed particularly for the atmospheric sciences. AIS systems are designed for efficiency in managing and analyzing weather and climate data, which tend to reside in file-based storage and in data formats that are not well known to the broader GIS community. The software design strategies behind AIS often favor adept representation and manipulation of time-varying, volumetric data over the integration of a wide variety of datasets.

1 October 2003

Page 3

Given that both GIS and AIS designs have evolved appropriately for the audiences that they serve, it is unlikely and undesirable for one to replace the other. It is, however, desirable for each of these systems to have access to the components and capabilities of the other. For example, rather than re-implementing a comprehensive, robust map projection library, it would be more desirable for AIS to have direct access to a coordinate reference system component of GIS, where map projections are typically a strong suit. Such an approach to modularity and division of responsibilities is the ultimate goal of interoperability. From the wide array of interoperability needs between atmospheric and geographic information systems, we can derive a matrix of goals from both GIS and AIS perspectives. Decomposing software information systems into their two primary components a data component and a process component allows us to identify four interoperability goals grouped by user community (Tables 1 and 2). Table 1: Interoperability from the AIS perspective Target User Data Component Process Component Atmospheric Scientist Atmospheric Scientist Interoperability Goal Access to a wider range of data Access to geospatial processing capabilities

Table 2: Interoperability from the GIS perspective Target User Data Component Process Component Broad GIS community, impacts researchers Atmospheric Scientist Interoperability Goal Access to atmospheric data Access to atmospheric models

Table 1 represents the community of users who conduct research and analysis inside specialized atmospheric software systems such as CCSM, WRF, and ESMF. A challenge for these systems is to implement access to a wider range of data, for example, land cover classifications for establishing boundary conditions. Table 2 represents the community of GIS users who rely upon the data fusion capabilities of general-purpose GIS to conduct their work and who face the challenge of incorporating atmospheric data into the GIS environment.

1 October 2003

Page 4

AIS and GIS users are confronted by limitations in both data interoperability and process interoperability. Nevertheless, to focus the activities of the GIS Demonstration Project, priority was placed first and foremost on data interoperability and its associated goals: AIS Goal: Access to a wider range of data in an AIS environment GIS Goal: Access to atmospheric data in a GIS environment

Addressing issues of data interoperability in both AIS and GIS environments requires navigating a complex landscape of tools and their associated data formats. The following two tables show a small sampling of tools and formats in atmospheric software (Table 3) and ESRI software (Table 4). Table 3: Sample tools and data formats from the AIS perspective Tools WRF CCM NCL TITAN Data Formats NetCDF History files NetCDF, GRIB, HDF-EOS, HDF, CCM history files MDV, SPDB

Table 4: Sample tools and data formats from the ESRI GIS perspective Tools ArcGIS ArcIMS ArcSDE Data Formats Shapefiles, geodatabase, ArcInfo coverage, ESRI Grid Shapefiles, Images (JPG, PNG), SDE layers Relational database tables

Note the lack of overlap in the data formats identified in Tables 3 and 4. The goal of the GIS Demonstration Project was to explore technologies for bridging the gap between these non-overlapping sets.

Methodology
The design of the GIS Demonstration Project involved a two-track implementation approach to data interoperability (Figure 1). Track 1 focused on the use of OpenGIS Consortium (OGC) protocols and services, while Track 2 focused on ESRI GIS technology.

1 October 2003

Page 5

Figure 1. The two-track approach of the GIS Demonstration Project. The first step in the Demonstration Project was an examination of field data collected during the IHOP field experiment. A subset of IHOP field data was identified for inclusion in the Demonstration Project based on the following criteria: Availability of associated geo-referenced meta-data Level of dataset demand among IHOP researchers Applicability to scientific collaborators involved in this GIS demonstration project (e.g., land surface atmosphere interactions) Ease of mapping to supported spatial data types, e.g. points, lines, grids. Based on these selection criteria, the Demonstration Project dataset included upper air observations (radiosondes and dropsondes), surface observations from ISFF stations, aircraft observations from the Wyoming King Air, and radar reflectivity mosaics. Following IHOP data selection, OpenGIS-based activities in Track 1 involved implementation of client/server middleware for encoding and decoding these data. The ESRI-based Track 2 approach involved conversion of the IHOP data into formats compatible with the ESRI software system. Implementation details of these two tracks are discussed in subsequent sections of this report. The two tracks of the Demonstration Project correspond directly to the two user communities represented in Tables 1 and 2. Track 1 aims to meet the AIS interoperability goal identified in Table 1 (access to a wider range of data in an AIS), while Track 2 aims to meet the GIS interoperability goal identified in Table 2 (access to atmospheric data in a GIS).

Focus on Data Interoperability


Focusing on the data component of interoperability has two noteworthy advantages: 1) it allows us to address needs of both atmospheric and non-atmospheric audiences and 2) it 1 October 2003 Page 6

allows us to conduct a demonstration based largely upon software that is developed and available as OpenGIS and ESRI components. To identify specific interoperability components or tools appropriate for addressing our goals, it is useful to categorize each of our two goals by geospatial task and communication environment. The geospatial task indicates whether the data are required only for visual display or for subsequent analysis. For either of these tasks (visualization or analysis), data may flow between localized desktop applications or may rely on a distributed communication environment. This categorization results in four possible usecases. The software tools available for each of these four use-cases vary, once again, between AIS and GIS environments (Tables 5 and 6). Table 5: Data Interoperability from the AIS Perspective Communication Environment Local Desktop Distributed Geospatial Task Display Only Subsequent Analysis N/A Web Mapping Service

Simple Features for Web Feature Service OLE/COM, CORBA, SQL

Table 6: Data Interoperability from the GIS Perspective Communication Environment Local Desktop Distributed Geospatial Task Display Only Subsequent Analysis N/A ArcObjects (COM) ArcIMS Image Service ArcIMS Feature Service, ArcSDE

Tables 5 and 6 can serve as a simple reference guide for selecting the appropriate form of software tools according to the geospatial and communication function to be served. Because the technologies developed for a distributed environment are often extensions of those developed for the local desktop environment, project efforts described in this report focused on exploring and demonstrating distributed interoperability capabilities: Web Mapping Service (WMS), Web Feature Service (WFS), ArcIMS (Image and Feature Service), and ArcSDE (Spatial Data Engine). In addition to leveraging off the powerful technologies being developed for Internet communications, focusing the Demonstration Project efforts on the distributed communication environment promotes a flexible client/server approach to implementation. The client/server strategy is particularly well suited for atmospheric

1 October 2003

Page 7

data where large volume repositories limit the practicality of copying datasets to a local desktop environment.

Track 1: OpenGIS AIS Implementation


The primary motivation for extending atmospheric information systems to include OpenGIS conformance for Web Map Service (WMS) and Web Feature Service (WFS) is to provide the basis for a standard encoding for atmospheric data so that software systems that are likewise compliant can interoperate. For developers of AIS, an important aspect of the standards defined by the OpenGIS Consortium (OGC) is the philosophy that data storage formats are outside the realm of the OGC and should be defined by the users of those data. This philosophy allows for the development of OpenGIS compliant AIS that manage existing data repositories in native atmospheric data formats such as NetCDF, GRIB, METAR, etc. Instead of establishing data format standards, the OGC defines standards for request protocols and data encodings for communication between software systems. A significant advantage of this approach is that existing repositories of atmospheric data need not be reformatted, i.e., converted to yet another persistent file format. Relying on data format conversion as a means to data exchange results in duplication of datasets, which in the atmospheric domain can lead to significant disk storage requirements. Rather than data reformatting, the primary task in developing an OpenGIS-based solution for Track 1 is client/server software development. WMS Concepts The OGC WMS specification is intended for graphical display of data, not data analysis applications. WMS data encodings are simply images, such as JPEG, GeoTIFF, and PNG, and do not encode actual data values. In the context of atmospheric information systems, WMS servers are suitable for generating base maps over which atmospheric data can be displayed. As of this writing, the current release of the WMS specification is version 1.1.1 and can be found at: http://www.opengis.org/techno/specs/01-068r3.pdf. The specification requires that all WMS servers are capable of responding to two HTTP requests: GetCapabilities and GetMap. The response to a GetCapabilities request must be an XML document that contains metadata about the WMS services provided. The capabilities document describes metadata such as the spatial extent, map layers, e.g., political boundaries and rivers, image formats, and coordinate systems supported by the WMS server. In addition to the request for capabilities, a WMS server must respond to a GetMap request by delivering an image that corresponds to the HTTP-CGI request. Although many image formats are permissible (JPEG, TIFF, GeoTIFF, PPM, WBMP, GIF) only GeoTIFF support is required by the specification. In communicating with a WMS server, a client must be capable of generating HTTP-CGI requests and rendering at least one of the image formats supported by the target WMS server. An example of a generic WMS client capable of rendering maps from multiple,

1 October 2003

Page 8

distributed WMS servers is NASAs digital earth viewer that can be seen at: http://viewer2.digitalearth.gov. WFS Concepts Unlike the image-based encodings of the WMS, the OGC WFS specifies a request protocol that allows for the transmission of spatial data along with their attribute values. Like WMS, a WFS server must respond to a GetCapabilities request. Unlike the image encoding of a WMS GetMap request, the data returned from a WFS GetFeature request is an encoding of data attributes and their values. Access to WFS services is suitable for atmospheric display applications that provide specialized client-side graphical rendering capabilities, such as Unidatas IDV application. The current release of the WFS specification is version 1.0 and can be found at: http://www.opengis.org/techno/specs/02-058.rtf. As a minimal requirement, WFS servers must support the Geography Markup Language (GML) as an output encoding type, in response to a GetFeature request. Although WFS servers and clients are free to support other data encodings, relying upon specialized encodings defeats, to a large extent, the broader interoperability goal. For example, a valid response to a WFS GetFeature request could include data in a GRIB encoding, providing the client application understood how to interpret GRIB data; however, the interoperability of such a system would be no more advanced than is currently the case. A more effective approach to interoperability involves the design of a data model (discussed below in Track 2 implementation). Various data formats (GRIB, NetCDF, etc.) are then translated into a GML encoding of the domain data model, which is understood by both the client and server. Server Implementation Rather than developing a new WMS and WFS for the Demonstration Project, we chose to install existing open source implementations from http://www.deegree.sourceforge.net (WMS) and http://www.geoserver.sourceforge.net (WFS). These particular implementations were selected because of their open source availability as well as their official status as OGC reference implementations. Both the Deegree WMS and GeoServer WFS are designed for extensibility in that new data connectors can be added for server access to various data storage types. The storage types currently supported by the Degree WMS are: shapefiles, Oracle Spatial RDBMS, and ArcSDE. Supported GeoServer storage types include: shapefiles, GML, PostGIS RDBMS, MySQL RDBMS, and MIF/MID files from MapInfo. Since the IHOP data are not supported by any of the existing Deegree WMS or GeoServer WFS data connectors, new connectors are required for the Demonstration Project. A NetCDF data connector that will operate for both the WMS and WFS is currently under development. Completion of this connector is dependent upon the next scheduled release of NetCDF version 4, which is under development in Unidata, and is

1 October 2003

Page 9

proposed to extend the NcML capabilities to include a GML generator (http://www.unidata.ucar.edu/packages/netcdf/ncml). A schematic of the target architecture for Track 1 is shown in Figure 2, including the NetCDF data connector currently under development. It is envisioned that the NetCDF connector will provide an OpenGIS framework for serving not only IHOP data from native NetCDF format, but also CCSM data from the upcoming IPCC model runs.

Internet WMS
Oracle
Connector

Oracle RDBMS

GetMap

Image

NetCDF Connector

NetCDF files

WFS OpenGIS Client GetFeature


NetCDF
Connector

shapefile
Connector

shape files

GML

PostGIS
Connector

PostGIS RDBMS

Figure 2. System architecture for Track 1. Adopting the architectural framework of the Deegree WMS and GeoServer WFS was an appropriate implementation choice for a new project without an existing code base. However, for existing AIS that already have developed their own client/server software framework, a different approach to implementing OpenGIS capabilities may be necessary. For example, the DODS/OpenDAP architecture, like the OGC service architecture, is designed for distributed access to data independent of data storage formats (http://www.unidata.ucar.edu/packages/dods). Because of the large legacy of software built around OpenDAP, a migration to OpenGIS standards within OpenDAP would more likely involve an extension to the existing middleware code rather than adoption of OGC reference servers such as was done in the GIS Demonstration Project. Undoubtedly, there are, within the atmospheric community, a number of existing software systems like OpenDAP where the details of enabling OpenGIS services capability depend completely upon the individual AIS architecture.

1 October 2003

Page 10

Client Implementation As is the case with existing AIS server-side implementations, the requirements for extending existing AIS clients to enable OpenGIS compliance are entirely dependent upon the details of the client architecture. For purposes of the GIS Demonstration Project, two atmospheric display applications were extended to support WMS image rendering capability. In each case, support was added to the client application for generating a WMS GetMap request along with client-side image rendering capabilities. As a result, these two weather display applications are now capable of requesting, for example, a shaded relief terrain map from a WMS server and displaying the resulting image as backdrop for weather data. These two weather displays include TITAN/CIDD, a C/C++ application used for realtime weather operations (Figure 3), and a JAVA-based prototype display application (Figure 4), both developed in NCARs Research Applications Program (RAP).

Figure 3. MM5 model data displayed in CIDD, C++ AIS, over topographic data from a WMS server.

1 October 2003

Page 11

Figure 4. MM5 model data displayed in JADE, Java AIS, over topographic data from a WMS server. Observations on OpenGIS Implementation Much remains to be done in the Demonstration Project to fully implement an OpenGIS solution for AIS. WMS image rendering in an AIS client represents only a small fraction of what is possible. OpenGIS standards present a flexible and extensible approach to interoperability for atmospheric software systems; however, significant software development resources are required. In particular, software engineering skills in Web and XML Schema technology are a prerequisite. Because atmospheric systems are often developed on UNIX platforms and Web and XML Schema tools under UNIX are frequently built on JAVA technology, software engineering skills in JAVA become, in practice, a prerequisite for OpenGIS implementation. On top of this knowledge base, an understanding of the OGC specifications is fundamental. Without membership in the OGC and active participation in the OGC standards development process, in-depth insight into the intended use of the OpenGIS specifications can be difficult. As with any technology standard, formal specification documents are essential, but do not function well as an educational tool for software developers. Lamentably, developers looking to implement OpenGIS standards suffer from a lack of comprehensive implementation guidelines. Thus, much of the useful insight into OpenGIS standards remains within the standards organization itself.

1 October 2003

Page 12

Some useful OpenGIS guideline documentation does exist. For example, the WMS Cookbook (http://www.ogcnetwork/docs/03-050r1.pdf) provides an introductory primer, and a call for contributions to a WFS Cookbook has recently been announced by the OGC. For its part, UCAR is continuing to expand its OpenGIS knowledge base, through the work being conducted in Unidata and RAP, while NCAR is in negotiation with the OGC for membership status.

Track 2: ESRI GIS Implementation


In contrast to the OpenGIS AIS implementation in Track 1, the GIS implementation of Track 2 benefited from an extensive ESRI knowledge base. In addition to the readily available documentation on ESRI technology, telephone access to technical support along with the broad base of users and professionals in the ESRI community provided notable assets for Track 2 implementation. These assets helped significantly in identifying system requirements and designing system architecture. Figure 5 illustrates the highlevel system architecture used in Track 2.

Figure 5. High-level system architecture for Track 2. The two major components of Track 2 are ArcSDE (Spatial Data Engine) and ArcIMS (Internet Mapping Server). ArcSDE facilitates management of spatial data in a RDBMS and provides enterprise-level data access to ESRIs primary desktop GIS client, ArcGIS. ArcIMS is a data distribution mechanism that provides Web client access to data stored in ArcSDE and to other non-relational ESRI data such as images, grids, and shapefiles. In contrast to Track 1 where the architecture required a focus on software development, the Track 2 architecture necessitated a focus on data reformatting and data modeling. The following sections will explore some of the data integration details of the GIS 1 October 2003 Page 13

Demonstration Project and will discuss the roles of ArcSDE and ArcIMS in the context of these activities. Data Reformatting One feature of GIS as a data fusion system is support for access to or conversion of geographic data sets. Currently ESRI software supports over forty data formats via data converters or direct access from ArcGIS (Appendix A). The majority of ESRI supported data formats and data conversion tools evolved from the requirements of a GIS user-base that traditionally did not include atmospheric scientists and atmospheric data. As a result the 3- and 4-dimensional nature of atmospheric data is not well supported in ESRIs current suite of supported data formats. In Track 2 converting and importing the selected IHOP datasets into the GIS environment was one of the major tasks. This data-centric focus is in keeping with typical GIS efforts, where data preprocessing and migration are usually the most time consuming part of a project. Several data software applications were developed to convert data from their native IHOP form to file formats suitable for import into the ESRI environment. Reformatting steps for the Demonstration Project data are shown in Table 7. Table 7. Reformatting of the Selected Datasets Data Type IHOP format Converter GIS ingest format GIS internal format SDE
Radar Mosiac Upper Air (radiosondes and dropsondes) Aircraft (UW King Air) ISFF Stations ATD sweepfiles JOSS glass NetCDF (NCARRAF/nimbus convention) ASCII columnar SweepToAscii ClassToShapefile ESRI ASCII grid Shapefiles ESRI GRID, BSQ Geodatabase feature class Geodatabase feature class Geodatabase feature class 8-bit PNG Image

IMS

NetCDF2Shapefile (N/A)

Shapefiles ASCII columnar

The shapefile spatial data format is an open format developed and published by ESRI. This format is suitable for data representation and exchange of simple spatial feature (points, lines, and polygons). For spatial data exchange the shapefile format is becoming more widely used and is capable of accommodating more complex spatial data. Open source APIs (Application Programming Interface) for reading and writing shapefile data are readily available. As a result, many GIS and some AIS (e.g., Unidatas IDV) applications support shapefiles as a datastore.

1 October 2003

Page 14

Other file formats are required for representing and importing 2-dimensional gridded data. ESRIs internal binary grid representation is not a published format, and many of the data format converters supported by ESRI are used to convert gridded data into the ESRI GRID format. In the Demonstration Project the data conversion steps for gridded radar data were somewhat simplified in Table 7 and are described more completely in Figure 5. Radar Mosaic Data Conversion Processes SweepToMDV converter MdvToAscii converter Data File Formats RAP MDV GRID ESRI ASCII GRID for ArcGIS

ATD sweepfiles

ASCII GRID import

ESRI GRID format

GRIDIMAGE converter Figure 5. Conversion process for gridded radar data

BSQ Image format

for ArcIMS

Unlike the multi-step process of gridded radar data, feature data underwent a two-step process: 1) converting from native to shapefile format, then 2) loading ArcSDE from shapefiles. An alternative to the two-step conversion process involves data converters written with the ArcObjects API. The single-step ArcObjects approach directly populates the geodatabase and circumvents the data redundancy of generating intermediate shapefiles, which can become a problem with very large datasets where duplicate data storage is not practical. On the other hand, the intermediate step of converting data into shapefiles serves three useful purposes. First, shapefiles provide an easy mechanism for distributing data to external users. Second, the two-step conversion/loading approach may be more appropriate for the use of scientific data by non-scientific GIS users. In this scenario, data conversion is performed by scientific staff more familiar with the data content, thereby promoting more appropriate use of the data. Finally, separating the data conversion and geodatabase loading processes allows for independent and parallel development of the database schema. In other words, the relational data model does not have to be complete before data preprocessing and conversion can proceed.

1 October 2003

Page 15

Data Modeling and ArcSDE In parallel with the data conversions described above, a major step in Track 2 was designing a data model (here referred to as the IHOP Data Model) for the ESRI geodatabase. In addition to specifying the relational schema, i.e. table definitions, required for RDBMS implementation, data models facilitate communication, understanding, and data interoperability in non-relational environments as well. Community-based or domain-specific data models are a critical component of peer-topeer interoperability, for without a common understanding of the underlying schema, software applications are unable to exchange data directly. Data models describe objects (or classes), their attributes, and the relationships between these. Like the IHOP Data Model described in Appendix B, data models are commonly communicated using UML (The Unified Modeling Language). The UML diagrams in Appendix B present an observational data model that now can serve as a prototype to future atmospheric field projects. The development and community adoption of such data models provides the basis for software application interoperability. Data modeling tends to be a highly iterative design process. The use of design tools, such as Rational Rose and Microsoft Visio allows for convenient coupling of the UML design language to the underlying data representation. Thus, changes to the UML data model can be easily propagated to the affected data structures, e.g., RDBMS tables or Java classes. Microsofts Visio (Enterprise Edition) was the first component of the CASE (Computer Aided Software Engineering) tool used for developing the IHOP Data Model. The second component was the Geodatabase Schema Wizard tool found in ArcCatalog. Visio exports the data model as an XML file that the Geodatabase Schema Wizard then reads to generate the required relational tables for ArcSDE. In contrast to Track 1 where the data model representation is format-independent, the use of ArcSDE in Track 2 operates specifically on a RDBMS datastore. ArcSDE supports connections to four relational databases Microsoft SQL Server, Oracle, IBM Informix, and IBM DB2. All four of these databases provide sufficient scalability, security and data control mechanisms required within the scope of the Demonstration Project. Microsoft SQL Server was selected as the Demonstration Project relational database for three reasons: (1) its low cost relative to the other three RDBMS options, (2) available skill in SQL Server administration among the Demonstration Project staff, and (3) availability of operating system platform (Microsoft Windows). Although SQL Server was identified as an appropriate choice for the scope of the GIS Demonstration Project, an enterprise-wide solution may call for alternative RDBMS choices. Factors affecting this decision include support for multiple operating systems, scalability for very large databases, more extensive configuration and administration tools, and native spatial data extensions, e.g., Oracle Spatial or Informix Spatial Data Blade.

1 October 2003

Page 16

Web Distribution and ArcIMS Following data conversion and SDE data loading, configuring and customizing ArcIMS provided a mechanism for distributing data across the Web in a GIS format. Like the OGC specifications, ArcIMS distinguishes between image and feature services. ArcIMS image servers return maps to the client in image formats (JPEG or PNG). This approach allows for the use of thinner clients and does not require clients to have special software plug-ins in order to view published map images. Use of ArcIMS image servers allows for simple, out-of-the-box functionality, results in limited capacity for customized clientside applications, and puts the majority of the weight of data querying and rendering on the server. ArcIMS Feature Servers, on the other hand, return data and attribute values in ArcXML encodings to thick clients. This approach shifts the weight of data filtering and rendering onto the client and results in the typical tradeoffs between thick and thin clients, e.g. client flexibility vs. lightweight simplicity, server scalability vs. data transmission capacity, etc. ArcIMS supports two types of client graphical user interfaces: 1) a light-weight Web browser, HTML client and 2) a heavy-weight Java application. Accessing an ArcIMS feature server from the Web browser application requires downloading and installing two Java components: Java Browser Plug-in and Java Runtime Engine (JRE). For the Demonstration Project an ArcIMS image service was implemented in conjunction with a customized Web browser client interface. The customized ArcIMS interface supports viewing multiple reference and data layers, direct access to feature information, and provides data querying capabilities by user-defined areas of interest, data type, altitude or time of interest. A view of the ArcIMS browser interface for the Demonstration Project is shown in Figure 6.

Figure 6. ArcIMS client interface to IHOP Demonstration Project data 1 October 2003 Page 17

This interface allows for Web access to the IHOP Demonstration Project database (SDE geodatabase and radar image files). However, due to firewall security issues, data access is currently available only to internal UCAR users via intranet at: http://topo.rap.ucar.edu/website/IHOP/viewer.htm. Data Analysis and ArcGIS While ArcIMS provides Web access to GIS project data, it is limited in its usefulness for complex data analysis and sophisticated cartographic symbology. ArcMap is ESRIs flagship graphical user interface for integrated data display and analysis. ArcMap, along with ArcCatalog and ArcToolbox, comprise the interactive desktop environment, ArcGIS. Through local area network connections to the ArcSDE Demonstration Project database, ArcGIS provides a desktop GIS environment for integration and further analysis of IHOP data. In an effort to demonstrate an end-to-end GIS solution, ArcGIS will be used to assist in the spatial analysis of select IHOP investigations. Figure 7 depicts an example of the data integration capabilities in ArcMap.

Figure 7. Radar mosaic and NSSL Mobil Class temperature data in ArcMap over IHOP field site Observations on ESRI Implementation Implementation of ESRI GIS technology in the context of atmospheric science presented a challenging task exacerbated in part by constraints in operating system support. Two decision points in which operating system platform played a significant part of Track 2 architecture were: (1) installation of ArcIMS and (2) selection of RDBMS. Linux support for ArcIMS is currently available only under RedHat 7.1 while the current distribution release is at RedHat 9. It is expected that porting to subsequent OS releases 1 October 2003 Page 18

always takes some time and that commercial software products will always lag behind the current OS release. However, the significant security upgrades that have taken place between RedHat 7.1 and version 9 pose a problem for exposing an ArcIMS server outside the NCAR firewall. Selection of a RDBMS for ArcSDE also involved tradeoffs related to operating system support. Although ArcSDE can connect to four different RDBMS vendor products, the combination of platform support is rather restricted. For example, IBM releases Informix under Linux, but the ArcSDE connection to Informix is supported only on an IBM/AIX platform. One challenge that was encountered during the data conversion task was processing 4dimensional, multi-attribute data for conversion into a relational database. In particular, some aircraft data were collected at a sub-second time scale. Bringing these temporal clusters of observational datasets into a relational database presents a data modeling challenge. One solution that is being investigated is creating a separate dataset for each temporal subset. Additional challenges were presented with batch import/conversion processes and application of color scales to radar data. This was a multi-step process. The radar data was converted to an ASCII file which was then imported into an ESRI Grid through the conversion tools of ESRIs ArcGIS. The Grid was converted into a BSQ file format and finally imported into ArcSDE as an SDE raster catalog. A .bsq file is a Band Sequential raster file format. The .bsq stores information for images one band at a time. The reason for converting the ESRI Grid to the .bsq file was to apply the colormap. ESRI Grid colormaps are lookup tables which define the colors to be used for displaying cell values. The colormap files, however, can only contain numbers greater than zero. This lead to some issues with the radar data which has many negative values defined. We, therefore, used a RECLASS command in Workstation ArcInfo to reclass the negative values to positive numbers in order to apply our colormap file to the ESRI Grid in order to create the .bsq file. The .bsq file was imported into ArcSDE at an embedded image catalog. The embedded image catalog allows for an efficient means of storing raster data in a relational database. Each image is stored in one row of the raster table, opposed to storing each image as a separate table in the database. All of these processes were accomplished through the use of AML (Arc Macro Language) and SDE command line. AML is the scripting language for Workstation ArcInfo. It is used often for automating frequent or repetitive tasks. Technical challenges notwithstanding, the availability of commercial-grade products complete with technical support and documentation resulted in our ability to demonstrate a fairly complete picture of the interoperability solutions available in Track 2. The Track 2 implementation successfully demonstrates a full range of data management activities from atmospheric data conversion, GIS-formatted data ingest, relational database modeling, local area network access to the GIS database model, and HTTP distribution of the data.

1 October 2003

Page 19

Resource Requirements for Tracks 1 and 2


As with any information system implementation, the resources required are largely dependent upon the system architecture. Nevertheless, identification of general resource requirements for Tracks 1 and 2 may be useful for planning future GIS-based projects. Because the target platform for ESRI client software is Microsoft Windows, most of the Track 2 implementation was targeted for this platform while Track 1 was implemented under Linux. Although some ESRI server-side products (ArcIMS and ArcSDE) are supported under various flavors of UNIX, the primary ArcGIS clients are strictly Microsoft Windows based. For purposes of project planning, this distinction has implications in determining the type of computer administration support required. For Track 1 the Demonstration Project relied heavily on Windows system administration, while Track 2 required support from UNIX systems administrators. This point is particularly relevant to NCAR scientific divisions where UNIX operating system platforms and administrative support are readily available. As previously mentioned, Track 1 activities focus on client/server implementation while Track 2 focuses on data reformatting. Both of these activities require software development. Thus, in the atmospheric domain where data are not readily available in GIS-friendly formats, it is important to recognize that either approach to GIS integration will require software engineering project resources. Software engineering skills useful for this Demonstration Project included knowledge of Web server architecture components (such as Apache, Tomcat, Java servelets), Web development languages (such as HTML, XML, HTTP-CGI, JavaScript, Java), and application or scripting languages (such as C++ or perl) for developing data converters. Beyond computer system support and software engineering, additional staff resources should be mentioned. Both OpenGIS and ESRI have steep learning curves fraught with up-front implementation options that have long-term implications. Thus, projects that plan to implement these technologies would benefit greatly from experienced staffing well versed in the OGC specifications (for Track 1), and ESRI products and relational database systems (for Track 2).

Cross Linkages Between Tracks 1 and 2


Although not addressed within the scope of the GIS Demonstration Project, some promising interoperability options exist in the cross linkages between Tracks 1 and 2. Because ESRI is developing OpenGIS compliance into some of their products, it is possible to implement a hybrid system architecture of OpenGIS and ESRI clients and servers. Currently, a hybrid configuration is possible using ArcIMS as the data service layer with OpenGIS clients requesting OGC-compliant capabilities, maps, and data features from the ArcIMS server. For the atmospheric community however, a much more powerful configuration, which is not currently supported, would allow for ESRI clients (such as ArcMap) to request OGCcompliant capabilities, maps, and data from OpenGIS servers that have been implemented to handle the data management needs particular to atmospheric data stores. 1 October 2003 Page 20

The real strength of this configuration is that it provides GIS users with powerful cartographic, visualization, and spatial analysis tools of ArcMap while allowing the atmospheric community to focus their software implementation efforts on the data distribution goals of providing easy access to climate and weather data for the GIS user. The current release of the ArcMap client does not support connections to OpenGIS servers, only to ArcIMS servers. However, an ESRI mapping client that does seamlessly support both ArcIMS and OpenGIS servers has been developed at ESRI as part of their involvement in the Federal Geospatial One-Stop project. This new client, demonstrated at the July 2003 ESRI User Conference, is a Web browser client and foreshadows a promising architecture for AIS and GIS interoperability.

Internet

ESRI Services ArcIMS ArcSDE

IPCC RDBMS

DayMet RDBMS

OGC Services ESRI OpenGIS Mapping Client


NetCDF
Connector

NetCDF files

METAR
Connector

Surface obs

Radar
Connector

Radar data

Figure 8. A future architecture for OpenGIS AIS and ESRI GIS interoperability.

Ongoing Efforts and Recommendations for Future Work


The GIS Demonstration Project has served a pivotal role at NCAR in several ways: Expanding the vision of the NCAR GIS Initiative; Providing research-focused (rather than production-focused) funding for exploring multiple technologies in GIS interoperability; Seeding related activities that will become the basis for longer-term solutions to GIS interoperability in atmospheric science. Presented in the following sections are some of the ongoing activities seeded by the GIS Demonstration Project along with recommendations for further activities that were

1 October 2003

Page 21

identified during the Demonstration Project as having important bearing on the future of GIS interoperability in the atmospheric domain. GIS User Needs for Atmospheric Data A common concern raised in discussions of GIS in atmospheric science is that of data complexity. Atmospheric data complexities include 3-dimensional volumes, temporal continuity, irregular grids, and non-planar 2-dimensional surfaces. For the atmospheric scientist these are important data constructs that must be faithfully represented in atmospheric information systems. However, for GIS users, simple representations of atmospheric data subsets are often quite sufficient. Thus, it is useful to return to Table 2, where we identified the primary goal of Track 2 as Access to atmospheric data in GIS, and to identify the subset of atmospheric data most useful to the GIS community at large. Having a clear understanding of the atmospheric data needs for the GIS community would also serve to establish requirements for data download services. For example, while the needs of an emergency planner may be met with ArcIMS access to a flood risk map, an impacts researcher may, on the other hand, need full download capabilities to the same dataset for further analysis. Efforts to quantify the need for weather and climate data and information within a GIS community are currently underway. A survey under development by the GIS Initiative aims to: 1) assess current and potential uses of weather and climate information by GIS users, 2) gather information on access of atmospheric data, preferred data formats, dimensionality, and representation, and 3) evaluate how the non-atmospheric science community uses weather and climate information in GIS applications. Data Modeling Also underway as a result of the GIS Demonstration Project is a community-based atmospheric data modeling effort. This work is being facilitated by the authors and coordinated through the atmospheric Special Interest Group (SIG) established at the July ESRI International User Conference. Atmospheric community members of the newly formed SIG represent government, academic, commercial, and research sectors. With the common interest of integrating atmospheric data into the GIS environment, SIG members have plans to conduct a two-day data modeling workshop January 16-17, 2004 in conjunction with the annual meeting of the AMS in Seattle. Questions to be addressed by the community data modeling effort include: How well does the IHOP data model serve as a prototype for a more comprehensive atmospheric data model? How would an atmospheric data model relate to the established marine and hydrologic data models (http://www.esri.com/software/arcgisdatamodels)? Given the time series representations defined by the Marine and Hydrologic data models, how might time-varying atmospheric gridded surfaces best be modeled? Data modeling efforts may provide a crucial cross-link between Track 1 and Track 2 approaches by providing atmospheric data representations that can be implemented in an

1 October 2003

Page 22

ESRI geodatabase as well as an OpenGIS GML. Development of GML encodings for atmospheric data is the next step required in a full implementation of Track 1. Thus, it is expected that the data modeling efforts of the atmospheric SIG will serve both AIS and GIS communities. Gridded Data Gridded surfaces of atmospheric model and observational data present a challenge even in 2-dimensional space. Although support for gridded data exists in both ESRI and OpenGIS realms, the support is preliminary (as in the case of OGC Web Coverage Service) or highly constrained (as in the specification of color scales in ArcGIS). Given the state of technology for managing gridded data, an area of further investigation is warranted to identify best practices for GIS integration of gridded atmospheric data. Atmospheric Modeling in a GIS Environment Once again revisiting Table 2, we see that Track 2 of the Demonstration Project focused on the data component within a GIS environment in an attempt to provide atmospheric data to a broad GIS user community. There is, however, a class of atmospheric scientists who see benefit in conducting analytical research within a GIS environment, i.e., the process component in Table 2. For the benefit of the scientific GIS users, one area for future work involves identifying and perhaps developing GIS tools for atmospheric research.

Conclusions
Under the efforts of this Demonstration Project, a more complete solution was achieved in Track 2. Some of this progress is accounted for by increased staff resources and availability of reference materials and software support. It is also recognized that the implementation challenges of Track 1 increased with the complexities inherited when connecting several independent open source projects into a coherent software system. Any comparison of the success or appropriateness of either Track 1 or Track 2 begs the question: What are the metrics for evaluating the usefulness of GIS interoperability in atmospheric science? Recent NCAR discussions have proposed software system metrics such as: number of software package downloads, number of data access requests, number of users supported, petabytes of data managed, etc. It remains a challenge to relate cause and effects between these metrics and the effectiveness of GIS technology in atmospheric research. Ultimately, the success of any interoperable information system is the extent to which it serves user needs. As we have seen, these needs vary significantly depending on the user perspective under consideration (AIS users vs. GIS users). Similarly, the available technologies for interoperability vary between these two environments. Thus, it becomes imperative to identify clearly the audience to be served before designing an organizational or project level approach to GIS integration. What we find is that both Track 1 and Track 2 provide solutions to interoperability each with benefits and limitations. Furthermore, we posit that for atmospheric data the most

1 October 2003

Page 23

productive area of focus in interoperability is in the cross linkages as depicted in Figure 8 above. It is critical to recognize that the user community served by this approach does not include, in the short run, the sizable body of atmospheric scientists that conduct research outside of the GIS environment. Moving towards a hybrid architecture does, however, put into place an essential server-side component required for future interoperability in AIS. This architecture more immediately highlights NCAR as a source of valuable data resources and serves the needs of the GIS user community interested in atmospheric data, including the important role of impacts assessment researchers.

1 October 2003

Page 24

APPENDIX A: ESRI Supported Data Formats Direct Read


Vector data: Shapefiles, Coverages, Geodatabase ArcIMS Map Services, ArcIMS Feature Services, Geography Network Feature Service, Smart Data Compression (SDC), PC ARC/INFO coverages, ArcSDE 3.x, VPF CAD data: DXF, DWG, DGN Raster data: Geography Network Map Service, ESRI grids, ArcSDE rasters, ESRI image catalogs, ERDAS IMAGINE (IMG), ERDAS 7.5 LAN (LAN), ERDAS 7.5 GIS (GIS), ERDAS Raw (RAW), ESRI raster catalogs, ESRI band interleaved by line (BIL), ESRI band interleaved by pixel (BIP), ESRI band sequential (BSQ), ESRI grid stack, ESRI grid stack file (STK), Windows bit map (BMP), Controlled Image Base (CIB), Compressed ARC Digitized Raster, Graphics (ADRG), ADRG image (IMG), ADRG overview (OVR), ADRG legend (LGG), DTED (levels 1 and 2), ER Mapper (ERS), Graphics interchange format (GIF), JPEG file interchange format (JFIF), National Image Transfer Format v2.0 and 2.1 (NITF, NTF), Portable Network Graphics (PNG), LizardTech MrSID (SID), Tagged image file format (TIFF) Other data: Geostatistical layers, TIN, DBF, TXT, INFO, ODBC, Microsoft Access

Data Conversion Tools


Import: CAD formats (DXF, DGN, DWG) to geodatabase Coverage to personal geodatabase Route event table to feature class (wizard) Shapefile to geodatabase Table to geodatabase VPF to geodatabase ASCII to grid DEM to grid DTED to grid Floating-point data to grid SDTS raster to grid AGF to shapefile Geodatabase to shapefile MIF to shapefile Geodatabase to table OLE DB to table Table to point features ArcInfo Export (E00) to coverage SDTS point file to coverage Export: Geodatabase to shapefile Geodatabase to table Raster to MrSID Raster to grid Raster to ERDAS IMAGINE Raster to TIFF Shapefile to AGF Shapefile to DXF Shapefile to geodatabase INFO to dBASE Table to geodatabase DXF, DGN, DWG to geodatabase Coverage to geodatabase

(source: ArcGIS Desktop Products Data Sheet. May 2003. http://www.esri.com/library/brochures/pdfs/arcgisdesktopsheet.pdf)

1 October 2003

Page 25

APPENDIX B: IHOP Data Model

This Visio Diagram contains the feature classes of the Demonstration Project IHOP Data Model. Feature classes are entities that have a geographic location. These classes are logical representations of the data and have a one-to-one mapping to relational tables in the physical representation of a RDBMS datastore.

1 October 2003

Page 26

This Visio Diagram contains the relationship classes of the Demonstration Project IHOP Data Model. Relationship classes are logical representations of relationships between feature and object classes. Each logical relationship class maps to a relational table in the physical representation of a RDBMS.

1 October 2003

Page 27

Acronyms used throughout the text AIS Atmospheric Information Systems (here used to describe NCAR-built software packages) AML Arc Macro Language ArcIMS ESRI Internet Mapping Service ArcSDE ESRI Spatial Data Engine BSQ Band Sequential CASE Computer Aided Software Engineering CCM Community Climate Model History Tape Format CCSM Community Climate Systems Model CIDD Cartesian Integrated Data Display COM Component Object Model DODS/OpenDAP Distributed Oceanographic Data System EOS Earth Observing System ESMF Earth System Modeling Framework ESRI Environmental Systems Research Institute GIS Geographic Information Systems GML Geography Markup Language GRIB Grids In Binary HDF Hierarchical Data Format IHOP International H2O Project ISFF Integrated Surface Flux Facility JRE Java Runtime Environment MDV Meteorological Data Volume NcML NetCDF Markup Language NetCDF Network Common Data Format OGC Open GIS Consortium OS operating system RDBMS Relational Database Management System SPDB Symbolic Product Data Base TITAN Thunderstorm Identification Tracking and Nowcasting

1 October 2003

Page 28

UML Unified Modeling Language UWKA University of Wyoming King Air WFS OGC Web Feature Service WMS OGC Web Mapping Service WRF Weather Research and Forecasting Model XML Extensible Markup Language

1 October 2003

Page 29

You might also like