Performance Impact Analysis with KPP using Application Response

Measurement in E-Government Systems

Namho Yoo, Ph.D.
DoD/HA
Falls Church, Virginia, USA
Namho.Yoo@us.army.mil

Abstract
In this paper, the performance impact analysis of egovernment systems with key performance parameters
is being considered. Meaningful impact analysis in
sustained government systems is required for
considering
non-functional
requirements
and
functional requirements. Performance requirements
are a critical component of non-functional areas. For
example, if a new system change is set to the system,
the impact in terms of the response time must be
implemented in each sub-system. In this paper, an
XML-based framework can be used to analyze
performance impacts on sub-systems and can provide
a scheme to enhance impact analysis by performance
monitoring using application response measurement.
Through a health system example as a case study, a
performance requirement model to describe extended
trees and adapting analysis result of performance
monitoring using application response measurement
and XML tree representation are addressed. This
paper also proposes a scheme for prioritized
processing and an algorithm for effectively enhancing
impact analysis in a timely fashion.

1. Introduction
In a sustained system, impact analysis is required to
make decisions prior to making system changes
associated with new requirements [1]. System
engineers are faced with the problem of analyzing nonfunctional impacts as well as functional impacts. There
is a growing recognition of the role of the system
engineer in determining the quality of a system. This
analysis is driven by non-functional requirements such
as performance, security, reliability, modifiability,
reusability, and economic constraints. Among these
requirements, the performance requirement is a critical
component in the non-functional area. Most

performance failures occur due to a lack of

consideration of performance issues in the early stage
of analysis. As impact analysis is covered by several
systems connected via interface in a sustained system,
several supporting engineers would be involved in the
engineering process suggested by the technical
committee to meet the users requirement [2].
One of the most typical requirements from the user
with respect to performance is application response
time. Prior to deployment, performance testing to
measure the response time must be conducted for a
thorough engineering impact analysis.
As system environments change rapidly and
engineering efforts for impact analysis increase
significantly, performance specifications require
careful examination. One of the key documents for
such performance engineering efforts is the
Operational Requirement Document (ORD), including
the Key Performance Parameter (KPP) that contains a
description about performance requests/requirements
and how they should be implemented and evaluated.
In this paper, the experience gained during
engineering projects for impact analysis is discussed.
Performance requirement are commonly applicable to
all sub-systems. In this paper, a typical government
health system as a case study. Thus, this paper suggests
a customized process to monitor performance
requirement during the impact analysis. As shown in
Figure 1, the performance requirement is described as
one of the non-functional requirements. To focus on
the response time, application response measurement
(ARM) for performance monitoring is discussed as a
lightweight tool [3]. The proposed scheme and
implementation are discussed in order to connect the
users requirement described in the document and
system environment for performance monitoring. In
this paper, Extensible Markup Language (XML)
technology is proposed for supporting performance
engineering efforts for impact analysis in sustained
systems as a common vehicle for information
exchange.

503
978-1-4244-4828-9/09/$25.00 2009 IEEE

Proc. ICSM 2009, Edmonton, Canada

Change
Case
1
2
3
4
5
6

NFR Types

Performance

Time

Space

Security

Availability

System
A
Insert
Delete
Update
Volume
Environment
Combined

System
B
Yes/No
Yes/No
Yes/No
N/A
N/A
N/A

Confidentiality

Integrity
Response Through Main
Secondary
Time
-put
Memory Storage

Figure 1- Non-Functional Requirement (NFR)

Types Tree

XML is widely used as a standard for efficient

information exchange on the World Wide Web. An
efficient data management scheme associated with
performance requirement is discussed and algorithms
are proposed for decision support [4].

2. Background and Problem Statements

System impact analysis is essential for good
decision support for given change requirements. If
change requirements are given as external inputs which
should be applied to the entire system, it is obvious
that change impacts between sub-systems must be
carefully examined. Moreover, in case a given
requirement has performance parameters for
application, it should be efficiently examined with the
overall view as well as the technical view.
In table I, it takes some time to find the result of
whether or not to analyze the impact in change cases
(1-3). The analysis result is dependent on the
knowledge and experience of the System Engineer and
there are many cases in which the result is uncertain. If
change cases (4-6) are considered, the situation for
system impact analysis is harder than the previous
case. System Engineers need to conduct system impact
analysis and evaluate testing results against the ORD
with KPP. In sustained systems, performing testing and
evaluating the results take up a lot of time and the
project schedule is delayed accordingly. In the current
situation, less consideration is given for reducing the
time for testing and mitigating the inaccuracy of
performance testing result. The customized process to
support the System Engineers efforts for performance
impact analysis is required.
Table I- Change Case Table for The System

504

System performance requirements are categorized

as threshold or objective, and some threshold
requirements are further defined as key performance
parameters. The threshold value is the minimum
acceptable value that, in the users judgment, is
necessary to satisfy the need. This paper suggests the
usage of a performance monitoring tool to improve the
process for performance testing.
ARM is defined as Application Response
Measurement. It comprises of an open standard
application program interface (API) for performance
monitoring in applications. The ARM standard is
administrated and overseen by the Computer
Measurement Group (CMG) and is an open standard,
derived with input and agreement, from multiple third
party vendors[2]. Consequently, the ARM API is
supported by a number of third party commercial offthe-shelf (COTS) products targeting application
performance monitoring, including Tivoli, Candle, and
others.

arm_init : initializes ARM environment for

the application.

arm_getid : Names each transaction that will

be monitored.

arm_start : Signifies the start of a unique

transaction instance.

arm_update : Updates statistics for a long

running transaction (optional)

arm_stop : Signifies the end of a unique

transaction instance.

arm_end : Cleans up the ARM environment

prior to application shutdown.
Although the ARM environment is recommended
for performance impact analysis, implementation
difficulties still exist for evaluation against KPP, based
on the operational requirement document. This brings
up several questions:
1) How does one map the correlation between the
ARM result and threshold?;
2) How does one communicate each of the other
system for efficient performance impact analysis?;
3) How does one track the measured result and
update the specification on performance requirements
in accordance with the monitored result?; and
4) Are there any simple and powerful ways to
execute performance impact analysis?

3. XML-based performance monitoring

with ARM
To monitor performance with ARM, XML is a good
vehicle for supporting engineering environments and
enhancing the process of performance impact analysis.
To achieve this goal, in this paper, an XML-based
representation
of
gathered
specification
for
performance impact analysis is discussed. By
customizing and implementation scripted by Perl
programming, as well as applying ARM
instrumentation, the metrics are produced in
accordance with the users requirement for
performance evaluation.
Active
Impact
Analysis
Report
Database

NonFunctional
Analysis

ARM
XML

Application

Document XML

Documents

Parsed
Data

Profile
XML

Security Input

Functional
Analysis

Profile XML is related to the non-functional

requirement but the security issue is taken care of the
full version of the paper. Log XML and Message XML
are also omitted intentionally.

4. Enhancing performance impact analysis

A manually performed impact analysis is time
consuming and prone to defect. The System Engineer
needs information about all related documentation
entities of a selected requirement. Using a performance
monitoring tool, the time and efforts for analysis, as
well as potential error, is mitigated. But in order to
increase practical usability for performance impact
analysis, the end users performance requirement based
on engineering document must be considered with the
tool. Based on an example in Table II, the data for ten
transactions are demonstrated in terms of response
time. The data on threshold is defined in accordance
with KPP in the ORD. The items are identified as use
cases to be measured and are comprised of the
relevant performance requirement. Specifically, they
are like follows:
KPP 1-1(response time): User Actions within a
screen(1 sec)
KPP 1-2(response time): Person Data Retrieval
Multiple Results/Multiple Tests(25 sec)
KPP 1-3(response time): User actions to change
screens(6 sec)
KPP 1-4(response time): Person Data Retrieval
Single Result/Single Test(6 sec)

Mapped Data

Queue

Log XML

For the purpose of initial verification, the stopwatch testing at engineering lab can be used also as
seen duration column of the table. Table II also
includes Run #, which means the number of runs tested
with stop-watch. Based on the experience, stop-watch
testing increases the reliability of ARMed result and is
useful to mitigate the users concern. This information
that has metric can be maintained using XML as well.

Decoded Data
HL7 Msg

Message
XML

Log File

HL7
Engine

HL7 Msg

Comm.
Ack

Ack

Figure 2- Proposed XML-based PIA

Architecture Design

Each row contains user name, transaction name,

workstation IP address, transaction start time,
transaction end time, response time, and ARM status.
To propose PIA architecture, Figure 2 (left hand side)
shows the XML-based information architecture
mapping to each offline or online components. Among
system component, the relevant entity associated with
performance impact analysis is considered. The focus
is on the application and ensuring that the ORD has
performance information. Using related artifacts, the
Document XML based on key performance parameter
metrics of the ORD, is generated. By embedding ARM
API functions to relevant component of application,
the measured duration of response time is obtained.

505

Table II- Sample Response Time (Manual and ARMed)

Number
1
2
3
4
5
6
7
8
9
10

Transaction
Security-User logon
Change User
Open-Review
Open-Appointment
Cancel-Search Dlg
Run-Search
Select Person
Close-Module
Open-Item
Run-Calculate Item

Run # Duration* ARMed Threshold Pass/Fail

3
1
3
2
2
3
2
1
3
1

7.53
17.09
1.44
1.72
0.47
2.28
1.56
1.07
11.62
0.72

6.45
14.99
1.81
1.32
0.38
2.25
1.73
1.36
10.95
0.91

25
25
6
6
1
6
6
6
6
1

Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Fail
Pass

* Using Stop-Watch

For reporting purposes on the implementation, the

response time falling within a certain range near each

threshold meet the threshold. More significantly, in

order to answer about the relationship between the
systems, it is sufficient to consider the path
information from system to system.

prioritizing each component in the system and

predicting the impact for better decision support based
on ARMed data gathered.
Table III- Performance Impact Analysis Support Levels

A l g o rith m P I A
In p u t : A R M e d m e tric s In fo r m a tio n , P e rfo r m a n c e -b a s e d
d o c u m e n t & H L 7 L o g F ile
O u tp u t: U p d a tin g P e rfo r m a n c e I m p a c t A n a ly s is D o c
1 ) A c c e s s E n g in e e rin g la b S e rv e r fo r te s tin g
2 ) M o v e to re le v a n t d ire c to ry w ith r e s p e c t to th e s y s te m
c h a n g e in fo rm a tio n
3 ) C h e c k th e p o s s ib ility o f H L 7 m e s s a g e a ffe c te d b y
change.
4 ) I f y e s , th e n d o fo llo w in g b e lo w p ro c e d u re . O th e r w is e
e x it.
5 ) E x tra c t th e A R M e d d a ta o f re le v a n t s ys te m .
6 ) P a rs e th e H L 7 lo g a c c o rd in g to th e r u le b a s e d o n
s y s te m c h a n g e d
7 ) P a rs e th e re le v a n t K P P m e tric s in fo r m a tio n
8 ) C o n v e rt A R M m e tr ic , K P P m e tric s , a n d p a rs e d H L 7
lo g to X M L
9 ) C h a n g e X M L fo rm a t to X M L D O M tr e e re p re s e n ta tio n
1 0 ) S e a rc h a n d C o m p a re th e re le v a n t in fo rm a tio n .
1 1 ) C a lc u la te th e p a th in fo r m a tio n u s in g X P a th w ith
c o n s id e rin g p rio r ity
1 2 ) C o m p a r e it w ith th e p r e v io u s in fo rm a tio n .
1 3 ) W ith re g u la r b a s is , u p d a te th e h is to ry d a ta .
1 4 ) S y s te m e n g in e e r p ro d u c e th e m e tr ic in fo r m a tio n
b a s e d o n X M L D O M d a ta .
1 5 ) A p p ly th e re s u lt fo r th e P e rfo rm a n c e I m p a c t A n a l ys is .
1 6 ) U p d a te P e rfo rm a n c e I m p a c t A n a ly s is D o c u m e n ta tio n .

*
Understandability
Reliability
Productivity
Maintainability
Customizability
Interoperability

L0
C
V

L1
T
V
V

L2
M
V
V
V

L3
S
V
V
V
V
V

L4
A
V
V
V
V
V
V

*C: Current, T: ARM Applied, M: Model Applied,

S: Scheme Applied, A: Algorithm Applied

Using XML based information, supporting engineers

save time and are able to support the customer with
more value added service.

7. Conclusions and future work

Figure 3- PIA XML Algorithm

For integration purpose with relevant artifacts using
proposed XML-based approach, the requirement is to
define the correlation and the procedure. Thus, this
paper proposes a PIA algorithm based on XML shown
in Figure 3. As a result, the maintainability and
customizability are increased and the error rate is
decreased. In addition, the time-consuming effort for
investigating and integrating information for
performance impact analysis are decreased, so the
users requirements with time constraint are met for
performance evaluation. To show the quality of system
applied in this approach, check the results based on
empirical analysis intuitively describe in Table 3. It is
also meaningful case for implementing the Egovernment [5]. As shown at the Table 3, the
expecting result for engineering work of impact
analysis by performing impact analysis on performance
can be gained. The potential risk factors are reduced by

506

In this paper, some issues raised in maintaining

sustained systems are examined. In particular,
performance impact analysis in terms of response time
associated with key performance parameter in large
scaled sustained system is focused on. Through a
health system example, performance requirement
model is proposed for prioritized process and an
algorithm for effectively enhancing impact analysis in
timely fashion.

8. References
[1] R. Arnold, S Bohner, Impact Analysis Toward A
Framework for Comparison, Proc. of Conf. Software
Maintenance, September 1993 pp 27-30.
[2] N. Yoo, An SOA-Based System Interface Impact
Testing Improvement for E-Government, CSREA EEE
2007 pp 60-66.
[3] The Open Group, Application Response Measurement,
Open Group Technical Specification.
[4] N. Yoo, A SOA-Based Layered Impact Analysis
Algorithm in E-Government, IKE 2007 pp 205-214
[5] N. Yoo, A SOA-Based Vulnerability System
Engineering for E-Government Solution, ICSENG, 2008