A Performance Evaluation Model for

E-government Project
Yang Yang1

Kejin Hu2

1. Electronics and Information Engineering College

Tongji University
Shanghai, China

2. School of Economic and Management

Tongji University
Shanghai, China

yycisa@263.net
AbstractProposed a performance evaluation model for e-

government project management based on the

investigation on 15 various projects in Shanghai
government from 2005 to 2006. The model includes: (A) a
performance indicator system consists of 5 domains, (B) a
nonlinear evaluation algorithm based on project risk
theory. The model has been verified and applied in related
departments in Shanghai government since 2007. The
paper discussed the indicators design and the evaluation
algorithm, and compared the evaluation results between
the model and professional groups.
Keywords- E-government, Performance Evaluation, Project
Management

I.

RESEARCH BACKGROUND

E-government is one of the most important tasks to many

developing countries today. Unfortunately, most such projects
did not return benefits as has expected, which has delayed the
whole process of e-government constructing. To resolve the
problem and improve the effect of e-government project
management, the informational administration departments of
Shanghai government invited us to make a long-term
investigation on major e-government projects and propose a
solution framework.
During the investigation from 2005 to 2006, we have
examined 15 various e-government projects in 5 sectors in
Shanghai, and table 1 shows the distribution of the sample. The
investigating methods include field-inquiry and questionnaire,
snapshot review in production system, data integration test on
production system, walk-through test in function line, and
Delphi review method.
TABLE I.
Application
Type
Budget
Scale
(RMB
1,000)
Main
Sponsor

THE INVESTIGATION SAMPLE DISTRIBUTION

G2C
3
Less than
500

G2B
5

Internal OA
6

500-1,000

1,000-2,000

Other
1
Greater
than 2,000

Local
government
10

Central
government
3

Public
organization
1

Private
organization
1

Project
Type

New
Application

Upgraded
Application

New IT
Framework

Upgraded
IT
Framework
1

Through the investigation, we concluded that the lack of a

effective performance evaluation model has obviously affected
the effect of more than 50% e-government projects in three
ways: (1) Redundancy construction, includes systemredundancy among departments or regions, functionredundancy among different systems in a same department or
region, and data-redundancy among different departments. (2)
Low customer satisfaction, includes complaints from citizens
and enterprises for inconvenient program and useless
information, and the complaints from operators or system
managers for the complex workflows which reduce work
efficiency. (3) Low sustainability, includes low capacity to fit
the change of environment and to be the base platform for
future extending.
We have proved and analyzed these issues in ref. [1] (2006),
and proposed a performance budgeting framework as a solution.
In ref. [2] (2007), we expounded the basic architecture and
principles of the framework. While this paper described our indepth study on the performance evaluation model of the
framework. The model proposed has been applied in the egovernment management practice in Pudong sector and Putuo
sector in Shanghai.
II.

BRIEF INTRODUCTION OF E XISTING RESEARCHES

Performance evaluation and related budgeting method has

been applied into public administration for more than 50 years
in developed countries such as USA and UK. From 1990s, egovernment projects have been a huge investment for these
governments, and required an advanced special performance
based budgeting framework to meet the complexity and
specialization of information systems. In 2002, USA
government proposed the Performance Reference Model
(PRM) as a part of the Federal Enterprise Architecture
(FEA), which can be used in e-government project
performance evaluation in collaboration with the form A300
and the other models in FEA (ref [4]). This model provides a
series of general indicators, which should be adapted to
individual forms by each evaluated department. This model has
been applied in federal budgeting decision since BY 2004, and
obviously improved the manage efficiency. In recent years,
many developing countries such as India also noticed this

978-1-4244-2108-4/08/$25.00 2008 IEEE

requirement, and proposed their own frameworks. In 2003,

India government designed the E-government Assessment
Framework (EAF), which is more detailed than USAs PRM.
The framework consists of a series of detail indicators, an
evaluation algorithm based on weighted mean method, and a
yardstick for future budget decision based on evaluation
score (ref. [5]). In China, such researches are few. Peng
Xizheng discussed the basic concept of e-government
performance evaluation in ref [6]. Mi Hong proposed a preassessment index system in ref [7]. Some businesses such as
sina.com also provided their evaluation methods for
government portal sites.

Reduction of
Transparency.

In the investigation in Shanghai, we analyzed all these

frameworks or models, applied them in those projects, and
discussed their results with the related administration
departments. As the conclusion, although these researches can
provide many useful experiences, none of them can be applied
perfectly in developing countries such as China. The PRM
model provides only indicators but evaluate method, and the
indicators are too abstractive for un-experienced administrators
in developing countries to individualize and use. The
evaluation method in EAF of India can not reflect the nonlinear diversity between performance levels because of the
inherent shortcoming of weighted mean method. And the
yardstick of EAF is too fixed to apply in different decisions.
The other evaluation frameworks all focused one special type
of e-government project and can not be used in other types. So
we designed the new framework based on the investigation
results and the related existing researches.

There are many ways to calculate the scores of basic

indicators and domains based on detail indicators. Based on the
results of investigation, we found the weighted mean method is
effective and easy to apply. Here we select the well-known
AHP method and Delphi method to decide weights. Table 2
shows the indicator method and the weights for detail
indicators.

III.

THE INDICATOR SYSTEM

To reflect the whole performance of an e-government

system, the performance evaluation model contains 15 groups
of indicators (66 detailed indicators totally) of 5 domains:
Service domain. Service is a direct performance factor as
the final output of e-government system. To contain all key
aspects on service chain in various systems, this domain
includes Service Effect and Efficiency, User Convenient,
and Service Focus.
Technology domain. Technology is the inner driven power
of e-government system. The characters of technology can
disclosure many potential performance trends which can not be
shown by service domain alone. Here we divided the
technology indicators into Architecture, Standard
Compliance, Security, Reliability, Extendibility.
Sustainability domain. Sustainability focused on the longterm strategic performance. This domain consists of three
environment factors that will influence the projects:
Organizational Environment, Business Environment (this
indicator is valid only for the private-participate-pattern mode),
Law Environment.
Cost-benefit domain. Compare between the cost and the
benefit can directly reflect the efficiency of investment. Since
the main purpose of most e-government systems is to gain
social benefit but economic benefit, we transform most cost
and benefit indicator into non-monetary form, such as

Process

Time,

Improvement

of

Reuse-ability domain. The ability to reuse a system in

different environments is very important to a large-scale
government. Since most regular transactions in different
departments are similar, developed a powerful system and
reused it into all needed departments will obviously decrease
the redundancy constructing and save the total budget. So we
designed three important indicators to examine the reuse-ability
performance: Technology Reuse, Function Reuse and
Business Mode Reuse.

TABLE II.

Domain

Basic
Indicators

THE INDICATOR SYSTEM WITH WEIGHTS

Weights
Detail Indicators
G2C G2B G2G Special

'HOLYHU\VSHHG

 

6HUYLFHLQWLPH

 

6HUYLFHTXDOLW\ 
(IILFLHQF\
(DV\WRXVH

6HUYLFHVFRSH

6HUYLFHIRUEDFNZDUG

XVHUV
/RFDWLRQOLPLW 
7LPHOLPLW

6HUYLFH
7UDQVDFWLRQV

8VHU
6HUYLFH,QWHJUDWHG 
FRQYHQLHQFH
([FHSWLRQ+DQGOH 
(DVHIRUEDFNZDUG

XVHUV
)ULHQGO\LQWHUIDFH 
&UHDWLYH

6HUYLFH)RFXV
8VHUFRPSODLQW 
6HUYLFHVNLOO

'HPDQGDQDO\VHV 
5HVRXUFHV

$UFKLWHFWXUH ,QWHJUDWHGGHVLJQ 
3URMHFWFRQWURO 
,QWHURSHUDWLRQDO 
7HFKQRORJ\VWDQGDUG 
6WDQGDUG
'DWDVWDQGDUG

Technolog
&RPSOLDQFH
y
([FKDQJHLQWHUIDFH 
6HFXULW\IUDPHZRUN 
6HFXULW\LQYHVWPHQW 
6HFXULW\
6HFXULW\FRQWURO 
6HFXULW\SUDFWLFH 
%&3FDSDELOLW\ 
([WHQGLELOLW\
6FDODEOH


978-1-4244-2108-4/08/$25.00 2008 IEEE






























+DUGZDUH
 
FRPSDWLELOLW\
5XQQLQJHQYLURQPHQW  
077)
 
DFFXUDF\
 
5HOLDELOLW\
5HVSRQG7LPH
 
5HFRYHU\7LPH
 
2UJDQL]DWLRQPRGHO  
3RVWGHILQLWLRQ  
2UJDQL]DWLRQ VWDIIHGXFDWLRQ  
(QYLURQPHQW 6WDIILQYROYHPHQW  
/HDGHUV6XSSRUW  
)HHGEDFNSURFHVV  
Sustainabil
2XWVRXUFHFRQWUDFW  
ity
2XWVRXUFHFRQWURO   
%XVLQHVV
(QYLURQPHQW 3DUWQHUFDSDELOLW\ 

Costbenefit

Reuseability

([SLUHGWLPH
  
&RPSOLDQFHZLWKODZV 
/DZ
(QYLURQPHQW 5HJXODWLRQLPSURYLQJ 
8VHUFRVW
 
*RYHUQPHQWFRVW  
8SGDWHFRVW
 
3URFHVV2SWLPL]HG  

5HGXQGDQF\
 
,QYHVWPHQW
 
5HYHQXHDGGHG
 
$QWLFRUUXSWLRQ   
)XQFWLRQ 8QLYHUVDOIXQFWLRQ  
UHXVH
FRPPRQSURGXFWLRQ  
&URVVSODWIRUP  
(DV\GHSOR\PHQW  
&XVWRPL]DWLRQ
 
7HFKQRORJ\
&RPSRQHQWGHVLJQ  
UHXVH
4XDOLW\RI'RFXPHQWV 
&DSDELOLW\OLPLW  
'DWD5HXVH
 
  
%XVLQHVV0RGH %XVLQHVVPRGH
5HXVH
0DUNHWSURVSHFW   

To get accurate and quantitative scores for each indicator,

we provide a detailed guideline which lists concrete score
standards for all detail indicators considering most possible
cases. Farther more, by using evidential-reasoning algorithm,
the operator can get an accurate score even when he (her) can
not assure the reference case in guideline exactly (ref. [8]).
IV.

THE EVALUATION ALGORITHM

A. Nondimensionalization of Indicators.
By analyzing the investigation data, we found many of the
indicators levels belong to non-linear distributing. Table 2
shows an example. In this sample, we can hardly reflect the
difference
among
each
project
with
a
linear
nondimensionalization method such as weighted mean method
adopted in EAF. If we set 41% as the lower limit, the better
systems (11 in sample) whose compliance rate greater than
91% will have the same score. Otherwise if we set 91% as the
lower limit, then we are not able to compare the systems whose

rate less than 91%. So in both cases, the evaluation score will
be useless.
TABLE III.
Compliance
rate
Sample count

THE SAMPLE DATA OF INDICATOR COMPLIANCE WITH

PROMISED RESPONSE TIME
0%40%
0

41%50%
1

51%60%
1

61%70%
0

71%80%
1

81%90%
1

91%100%
11

For the reason, we use a fuzzy nondimensionalization

algorithm (formula 1) which can clearly reflect the non-linear
distribution of indicators. Figure 1 explains that.

xj max + xj min

(xj
)
, xj min < xj < xjmzx (1)
0.5 + 0.5sin
x
x
2

Rj (x) =

j max j min

, xj xj maxxj xj min
0

x j is the original value of the indicator x , R j (x) is the

dimensionless value x j max is the reasonable upper limit
x j min is the reasonable lower limit. This formula has been used
in the performance evaluation of supply-chain system
successfully (ref. [9]), and so did in our investigation.

Figure 1. Fuzzy nondimensionalization algorithm

B. Evaluation Algorithm.
The evaluation algorithm will integrate all the
dimensionless indicator values into one integrated score for
understanding the whole performance of one project or
comparing the performances among different projects.
The first step is to integrate indicator values in each domain
by weighted mean method. Since the non-linear problem has
been solved by formula (1), the weighted mean method can
work well in this step. After this step, we can get a vector of
scores of five domains. Because of the obvious orthogonality
of pn , we need an algorithm which can integrate the five
scores reasonably.
Suppose P as the integrated score, p n (n = 1,...,5) as the
vector of scores of domains, R as the failure risk of whole
system, rn (n = 1,2,3,4,5) as the failure risk of each domain, we
can get hypothesis below:

978-1-4244-2108-4/08/$25.00 2008 IEEE

Hypothesis 1: P positively correlate with pn , and

negatively correlate with R .
Hypothesis 2: pn negatively correlate with rn .
Hypothesis 3:
The general risk control formula
is
also
suitable for R and rn .
R = 1 (1 rn )
Hypothesis1 and 2 can be accepted because of the general
definition of performance and risk. Hypothesis3 has been
accepted in IT manage, control and audit for many years,
which make it acceptable here too. Then we get formula (2).

R = 1 (1 rn ), R = f (P)
, rn = g( pn ), n = (1...5)
set

According to the normal risk control model r = 1 c ,

,
,
then
f ( P)= 1 P +
g ( pn )= 1 pn +

P = ( p n ) + . Formulate P into [0,1], and we can get

the final evaluation formula (3):
P = [( pn ) + 5 ]
[(1 )5 + 5 ] n = (1,...,5)

(3)

C. Verification of the Evaluation Model

We selected 35 various e-government projects as the
sample to verify the evaluation model. Table 3 shows the basic
attributes of the verify sample.
TABLE IV.
Application
Type
Budget
(RMB 1,000)

Figure 2. Compare of two evaluation results

(2)

THE VERIFY SAMPLE

G2C

G2B

Internal OA

11

18

Other

Less than
500

5001,000

1,000-2,000

Greater
than 2,000

12

We first collected the original scores of detailed indicators

by field examination and questionnaires, and then calculated
the integrated scores using the model. Considering the lack of
experience in first time use, we set the risk adjust factor
equals 0.25, and set the weights using AHP method. We also
asked a group of professionals and experienced administrators
in government to evaluate these projects independently. We
compared the professional-groups evaluation results with that
of our model. Figure 1 shows the result. The Pearsons
correlation coefficient is 0.835, and the significance level is
less than 0.01.

V. CONCLUSION
Based on the investigation data and the compare research
on different models, we proposed a performance based
budgeting framework suitable for developing countries like
China. The advantage of the performance evaluation model is
its practical and objective, which has been proven by trial in
Shanghai government. And the budget decision model has to
be improved and verified in future because of the lack of
historic data now. We will keep the research go on.
[1]

Yang Yang, Hu Kejin. The Analyze and Solution to The Redundancy Egovernment Development in China. Computer Engineering and
Applications, 2006 2(4):13-15(in Chinese)
[2] Yang Yang, Hu Kejin, Shi Da, A Research on Performance Based
Budgeing for E-Government Project. International Conference on Public
Administration, 2007
[3] Andres. Authority, Acceptance, Ability and Performance Based
Budgeting Reforms. The International Journal of Public Sector
Management, 2004(4/5): 332-344..
[4] Willoughby.K, Melkers.J. Implementing PBB: Conflicting Views of
Success. Public Budgeting and Finance, 2000(1):105-120.
[5] OMB USA. Federal Enterprise Architecture [S]. 2005:1-17
[6] India Government. E-governance Assessment Frameworks CEG &
NISG [S]. 2003:1-50
[7] Peng Xizheng. A Discussion on E-government Performance Evaluation
Framework. Chinese Informatization, 2005 6.27. (in Chinese)
[8] Mi Hong. Studies on Index System and Comprehensive Evaluation of
Pre-assessment in E-government Item. Journal of Xiamen University,
2004(8):279-283 (in Chinese)
[9] Yang, J. B, Xu, D. L. Evidential Reasoning Algorithm for Multiple
Attribute Decision Analysis under Uncertainty. Systems, Man and
Cybernetics, Part A, IEEE Transactions on, 2002 (32/3):289-304.
[10] Ho Jiazhen, A research on Performance Evaluation of Integrated Supply
Chain, Beijing Tsing Hua University Press2005:75-76 (in Chinese)

978-1-4244-2108-4/08/$25.00 2008 IEEE