Acquisition of SW and HW (software and hardware)

Acquisition process improvement has been defined as the analysis of current software
acquisition processes for deficiencies and implementing new/modified processes to
correct those deficiencies. The software acquisition process improvement, defining it as:

• A documented process for software acquisition planning, requirements

development/management, project management/oversight and risk management
• Efforts to develop appropriate metrics for performance measurement and
continual process improvement
• A process to ensure that key program personnel have an appropriate level of
experience/training in software acquisition
• A process to ensure that each military department and defense agency select,
implement and adhere to established processes and requirements relating to software
acquisition

One of the better descriptions of the characteristics of a software acquisition process that
is successfully being used by leading software acquirers/developers is contained in the
“Stronger Management Practices. Three fundamental management strategies are
:
· Evolutionary Environment

· Disciplined Development Processes

· Collecting/Analyzing Meaningful Metrics

These strategies limit software development efforts to what can be managed and result in
decisions based on sufficient systems engineering knowledge to adequately assess risks.
In addition, business decisions are consciously made to invest time/resources in achieving
high software process maturity levels. Strong upper-level management support is a
strong enabler for ensuring that these combined strategies are successful.

Essential conditions in acquisition of SW AND HW

“Organizations that have a strong, consistent evolutionary environment and practices for
setting product requirements, maintaining a disciplined development process and using
metrics to oversee development progress achieve favorable cost, schedule and quality
outcomes.”

1
Acquisition

The conceptualization, initiation, design, development, test, contracting, production,

deployment, Logistics Support (LS), modification, and disposal of software systems,

Acquisition environment

Internal and external factors that impact on, and help shape, every software acquisition
program. These factors include political forces, policies, regulations, reactions to
unanticipated requirements and emergencies.

Acquisition life-cycle

The life of a software acquisition program consists of phases, each preceded by a

milestone or other decision point, during which a system goes through Research,
Development, Test and Evaluation (RDT&E) and production.

Acquisition program

A directed, funded effort over the software acquisition life-cycle that provides a new,
improved, or continuing materiel, weapon or information system or service capability in
response to an approved need.

Acquisition process

The steps involved, and practices employed, in the planning, contracting,

implementation, acceptance and follow-on activities of a software acquisition. The
acquirer will bear most of the responsibility for the results from the planning, contracting,
and follow-on phases of the acquisition, while the developer is most accountable for the
results coming out of the implementation and acceptance phases of the acquisition.

Development process

The process of working out and extending the theoretical, practical, and useful
applications of a basic design, idea, or scientific discovery, characterized by the design,
coding, modification or improvement of software.

2
The SW&HW Acquisition Process

It is necessary to first understand what a software acquisition process is before steps can
be taken to improve In following figure a generic nine-step process for software
acquisition, summarized in Figure 1.

3
4
Figure 1: IEEE Std 1062, 1998 Edition S\W AND HW Acquisition Process Steps

There are three steps that comprise the planning phase of a software acquisition. During
this phase, the acquisition strategy should be planned based on a review of the acquirer’s
objectives. A formal Acquisition Strategy document is developed to guide the
acquisition.

Implementing the acquisition strategy into the organization’s process is the next step.
Appropriate contracting practices need to be included in the process. The final step of the
planning process encompasses the determination of software requirements. In addition to
defining the software being acquired, this step includes the preparation of the quality and
maintenance plans for accepting the supplier’s/developer’s software. The release of a
Request for Proposal (RFP) is the completion milestone for the planning phase of the
software acquisition life-cycle.

The contracting phase of the software acquisition is also comprised of three steps. The
first step is the identification of potential suppliers. Suppliers should be selected who
will provide documentation for their software, demonstrate its performance, and submit
formal proposals. Failure to perform any of these actions may serve as a basis for
rejecting a potential supplier.

Past performance data from previous contracts should be reviewed for all potential
suppliers. The next step is to actually prepare the contract requirements. The quality of
work (acceptable performance and acceptance criteria) should be described, and contract
provisions that tie payments to deliverables should be prepared. Legal counsel should
also be sought to review and approve contractual language/requirements. Finally,
proposals will be evaluated and a qualified supplier(s) will be selected. If appropriate, an
alternate supplier should be negotiated with, primarily as a risk reduction measure.

Supplier selection completes the contracting phase of the software acquisition life-cycle,
which started with the release of the RFP and ends with the contract signing.
Once the contract is signed, software development can begin and the product
implementation phase of the software acquisition life-cycle is initiated. During this
phase, the acquirer must manage supplier performance. The supplier’s progress is
monitored to ensure all milestones are met. The acquirer also must approve all
appropriate work products. Note that, during this phase, the acquirer has the
responsibility to provide all acquirer deliverables to the supplier when required. The
product implementation phase of the software acquisition is completed when the software
product is delivered to the acquirer (pre-approval).

The next phase of the software acquisition life-cycle is product acceptance. Adequate
testing needs to be performed, and a process for certifying that all discrepancies have
been corrected and all acceptance criteria have been satisfied must be established.
Acceptance of the software product by the acquirer based on mutually agreed-to pre-
defined criteria signals the end of the product acceptance phase.

5
The follow-on phase of software acquisition covers the period between software
acceptance and software retirement, when it is no longer in use. A follow-up of the
software acquisition contract should typically be performed to evaluate contracting
practices, record lessons learned, and evaluate user satisfaction with the software. During
this final phase of the software acquisition life-cycle, the acquirer should record and
retain supplier performance data to use for the acquisition of future software products.
More recently, the Software Engineering Institute (SEI) has released the Software
Acquisition Capability Maturity Model to represent the important phases and elements of
the software acquisition processes.

Key areas necessary in SW &HW acquisition

The areas necessary to be considered while planning for acquisition of SW &HW or

hardware includes many sub-components as mentioned in table -

Table 1: Key Process Areas

LEVEL FOCUS KEY PROCESS AREA

5 Continuous Process · Acquisition Innovation Management
Optimizing Improvement · Continuous Process Improvement
4 Quantitative Management · Quantitative Acquisition Management
Quantitativ · Quantitative Process Management
e
3 Process Standardization · Training Program
Defined · Acquisition Risk Management
· Contract Performance Management
· Project Performance Management
· User Requirements
· Process Definition and Maintenance
2 Basic Project · Transition to Support (Acceptance/Follow-On)
Repeatable Management · Evaluation (Implementation/Acceptance)
· Contract Tracking and Oversight
(Implementation)
· Project Management (Implementation)
· Requirements Development and Management
(Planning/Contracting)
· Solicitation (Planning/Contracting)
· Software Acquisition Planning (Planning)
1 Competent People and Heroics
Initial

6
What is Software Acquisition Process Improvement?

Acquisition process improvement has been defined as the analysis of current software
acquisition processes for deficiencies and implementing new/modified processes to
correct those deficiencies. The most comprehensive definition of software acquisition
process improvement, defining it as:

• A documented process for software acquisition planning, requirements

development/management, project management/oversight and risk management

• Efforts to develop appropriate metrics for performance measurement and

continual process improvement

• A process to ensure that key program personnel have an appropriate level of

experience/training in software acquisition

• A process to ensure that each military department and defense agency select,
implement and adhere to established processes and requirements relating to software
acquisition

SW AND HW acquisition/development life-cycle:

· Acquisition planning
· Requirements development/management
· Configuration management
· Risk management
· Project management/oversight
· Test & evaluation
· Integrated team management
· Solicitation/source selection
· Performance measurement

Factors supporting software acquisition process improvement

· System/software objectives and constraints are adequately defined /validated

· System/software acquisition strategies are appropriate /compatible

· Success-critical stakeholders have committed adequate software

Capability / resources to perform their software-related tasks

· Software product/process plans are feasible/ compatible

7
· Feasibility Rationale provides convincing evidence that software progress is
satisfactory

• Specified improvements include –

(1) Documenting agreements between the software developer and acquirer that contain
baseline requirements based on systems-engineering knowledge,

(2) Gated reviews during the software development process

(3) obtaining meaningful metrics from the developer for managing the program,

(4) attaining knowledge about the technology that is planned early in the development
process,

(5) ensuring an appropriate balance between requirements and available resources,

(6) ensuring that the software design is mature before it is released, and

(7) having all processes under control prior to release. In general, the more that
acquisition managers know about software development processes and metrics, the better
equipped they are to improve the processes for software acquisition.
(8) provide applicable improvement program administration and compliance guidance
and ensure that secretaries of the departments and selected agencies comply with that
guidance and

(9) assist departments/agencies with their respective improvement programs by ensuring

the use of applicable source-selection criteria and access to information regarding
software acquisition/development best practices in both the public and private sectors.

Three fundamental management strategies for SW AND HW acquisition are

· Evolutionary Environment

· Disciplined Development Processes

· Collecting/Analyzing Meaningful Metrics]

These strategies limit software development efforts to what can be managed, and they
result in informed decisions based on sufficient systems engineering knowledge such that
risk can be adequately assessed. In addition, business decisions are consciously made to
invest sufficient time and resources in order to achieve high software process maturity
levels (high quality, on-time, within budget). Committed upper-level management
support is an effective enabler for ensuring that these combined strategies are successful.

8
Suggestions for strengthening acquisition plans included specific
criteria are –

· Requirements baselines based on systems engineering are documented/agreed to

by both the acquirer/developer before a program is initiated

· Cost/benefits analyses are required when new requirements are proposed

· Software acquirers/developers make efforts to continually improve practices over

time
· Gated reviews and deliverables are integrated into development processes

· Requiring software developers to collect/analyze metrics (including earned value)

to obtain knowledge of progress and to manage risk

Evolutionary Environment:

An evolutionary software acquisition environment is characterized by incremental

improvements to software performance, as opposed to succumbing to programmatic
pressures to set unrealistic expectations, in order to improve software processes on a
continual basis. An evolutionary environment limits software development to what is
possible to manage by adhering to well-understood, well-defined, manageable
requirements while simultaneously encouraging continuous process improvement. An
evolutionary product development is one of the fundamental elements of a manageable
acquisition environment. The concept of continuous improvement is a critical part of the
evolutionary environment, and must be supported by both the environment and the
organizational culture (including senior management) in order to be successful. Ad hoc
processes make it impossible to understand exactly when and how defects in the process
occur.

Disciplined Development Processes:

Developers must follow disciplined, structured management and development processes
in order for the software acquisition process to improve. Each phase of the software
acquisition should end in a management review/gate to ensure that the project is on
track. To pass these gated reviews, developers must demonstrate that acquirer’s
expectations and quality standards are met before proceeding to the next phase. Peer
reviews to check each other’s work and remove defects, beginning at the earliest stages
of software development, are necessary to prevent costly rework and schedule delays
resulting from downstream defect detection. Disciplined software development processes
that are characterized by their reliance on configuration management, peer reviews and
quality assurance significantly increase the probability of a successful acquisition, and
have been proven to assist leading organizations in identifying opportunities for software
process improvement. As a result, areas of risk can be identified early and actions can be
taken to control them.

9
Figure 4 highlights a disciplined knowledge-based software development process that can
serve as a basis for software acquisition process improvement, describing the information
needed, the relevant activities and the deliverables for each phase of the process.

Figure 4: Highlights of the Knowledge-Based Software Development Process [GAO,

2004]

Requirements need to be managed and controlled before design work begins and all
lower-level design elements should be adequately defined before the start of coding. It is
critical that the software acquirer and developer work closely together and have open and
honest discussions. Requirements management should be viewed as one of the most
critical development tasks for ensuring a successful acquisition.

Requirements should be sufficiently documented and validated prior to preliminary

design. In addition, acquirer/developer stakeholders must use sound systems engineering
techniques to establish the software requirements, and aggressively manage and control
requirements changes.

To enhance software coding and testing quality, requirements should be well-written and
achievable. By this phase of the acquisition, designs should be sufficiently detailed.
Other processes that are critical to achieving high quality software include peer reviews,

10
documented coding standards, frequent unit testing, access to a reuse library, and the use
of program languages that enable code documentation to facilitate future understanding.
Test plans should not be developed until after requirements are considered stable, and
there should be a minimum of one test to verify/validate each requirement.

One of the most widely-known and best-structured software acquisition and development
processes is the SEI’s CMMI® series of processes. A very recent addition to this suite of
tools is the CMMI Acquisition Module (CMMI-AM.

The focus of the CMMI is on “what” should be done, not “how” it should be done, so it
allows a great deal of latitude in terms of implementation of the CMMI Key Process
Areas. As with the other CMMI documents, a primary emphasis of the Acquisition
Module is on Integrated Product and Process Development (IPPD) concepts. These
include the effective use of cross-functional or multidisciplinary teams; leadership
commitment (particularly of senior management); appropriate allocation/delegation of
decision-making authority; definition of organizational structures that reward team
performance; the design of downstream processes (transition to Operations and Support –
O&S) during the acquisition; a strong focus on customer needs throughout the software
life-cycle; timely and effective collaboration between all relevant stakeholders; the
continuous and proactive identification and management of risk; and a focus on the
measurement and improvement of processes to develop/deliver a software product.

A graphical representation of the CMMI-AM® process is shown in Figure 5. Following

that figure is Table 2, which includes checklist-type questions that can be used to self-
assess coverage of each defined CMMI-AM® process area. The reader is referred to the
CMMI-AM® document to get a more in-depth discussion of each of the process areas
defined in Figure 5 and Table 2.

11
Figure 5: Graphical Representation of the CMMI-AM® Process

Table 2: CMMI®-AM Acquisition Process Areas [CMMI-AM, 2004]

Process Area Questions
Organizationa § Is infrastructure provided to establish fully functional teams from
l Environment among all stakeholders?
for Integration § Does the acquisition project have two focuses on integration – technical
and organizational?
§ Are technical capabilities/requirements examined/understood as they
relate to overall project goals/objectives?
§ Do organizational project entities operate with a single vision and
cooperative purpose?
§ Do project members understand the interrelationships of all project
aspects and how they relate to specific goals/objectives?

12
Integrated § Are project management processes consistent with/tailored from the
Acquisition organization’s standard processes?
Project § If a standard set of processes does not exist, has the project defined its
Management own processes to meet project needs?
§ Is there an integrated management process that incorporates/involves all
stakeholders?
§ Is the project management process defined in an overall Project
Management Plan, or equivalent document?
§ Are formal interfaces used among stakeholders, in the form of
Memorandums of Understanding (MOUs), Memorandums of Agreement
(MOAs), contractual commitments, associate contractor agreements, and
similar documents, depending on the nature of the interfaces and involved
stakeholders?
§ Does the project’s defined process include all life-cycle processes
(including IPPD) that are applied by the project?
§ Do the project’s defined processes include (1) selecting the team
structure, (2) allocating personnel resources, (3) implementing cross-
integrated team communication, and (4) conducting issue-resolution
processes?
§ Does development of a comprehensive project plan or the nature of the
project’s defined process require an iterative effort due to a complex,
multilayered, integrated team structure?
Acquisition § Does planning start by setting the acquisition project strategy, then
Project planning the acquisition process in increasing levels of detail?
Planning § Does the acquisition project review potential suppliers’ planning
processes for adequacy and compliance?
§ Are potential suppliers’ plans reviewed for consistency with system
acquisition plans?
§ If the acquisition project involves replacement of an existing system,
has operation/disposal of the existing system been considered as part of the
acquisition planning for the new system?
§ Are transition activities included in the acquisition project plan?
§ For integrated teams:
§ Does acquisition project data include that developed/used solely by a
particular team, as well as data applicable across integrated team
boundaries?
§ Does acquisition project resource planning consider staffing of the
integrated teams?
§ Is stakeholder involvement planned down to the integrated team level?
§ Is special attention paid to resource commitments?
§ Do integrated team plans get buy-in from team members, interfacing
teams, the acquisition project, and the acquisition process owners whose
standard processes are selected for tailoring?
Solicitation & § Does the project comply with applicable federal, departmental and
Contract service acquisition regulations and policies?
Monitoring § Does the solicitation address issues appropriate to the system domain or

13
 acquisition environment (e.g., supplier process evaluations, operational
safety suitability/effectiveness, safety, certifications, architectural
evaluations and interoperability)?
§ Are representatives responsible for functional disciplines within the
acquisition project or stakeholder organizations consulted for appropriate
inclusion of those disciplines into the solicitation and contract monitoring
process?
§ When integrated teams are formed, is team membership negotiated with
suppliers and incorporated into an agreement?
§ Do teaming agreements identify integrated decision-making, reporting
requirements (business and technical) and trade studies requiring supplier
involvement?
§ Are supplier efforts orchestrated to support the acquisition project’s
IPPD efforts?
Requirements § Does requirements management include the direct management of
Management acquirer-controlled requirements and oversight of supplier requirements
management?
§ Are requirements managed/maintained such that changes are not
implemented without assessing the impact on the project?
§ Does the requirements management process define “approved”
requirements providers and an approved path by which requirements are
provided to the supplier?
§ Are suppliers prevented from receiving requirements changes from
unauthorized sources that are outside the flow of the acquirer’s established
requirements management process?
§ Are commitments to requirements by acquisition project participants
demonstrated by having coordinated/approved documents that define
requirements?
§ Is each change to a controlled requirement assessed for impact on
acquisition project performance, cost and schedule baselines, and project
risk?
§ Are existing cost, schedule and performance baselines changed, as
required, to accommodate a requirements change?
Requirements § Are customer requirements grouped/coordinated into a set of
Development requirements that will define the scope/direction of the acquisition?
§ Do customer requirements and additional acquirer requirements become
the basis for the processes used by the supplier’s organization?
§ Do requirements flow from the stakeholders/customer to the system
level, to multiple subsystem levels and, eventually, to software component
levels?
§ Is the responsibility for requirements development/flowdown split
between the acquirer and developer, and who is responsible for which
levels?
§ Is there an iterative process of requirements allocation, high-level
design and requirements definition for the next lower level?
§ Does the acquirer exercise responsibility for defining/baselining the

14
 requirements levels under their control?
§ Does the acquirer exercise responsibility for monitoring the supplier’s
definition of lower-level requirements, and reviewing/ensuring that
required work products are being developed.?
§ Does the acquirer exercise responsibility for monitoring the lower-level
requirements development process to ensure that requirements produced at
each level result in a system that satisfies customer/operational
requirements?
§ In addition to functional/performance requirements, are interface
requirements also covered?
§ Do requirements also include non-functional requirements such as data
rights, delivery dates, milestone exit criteria, and attributes such as
evolvability, maintainability and reusability?
§ Is the same requirements process used for acquiring services instead of
products?
Integrated § Does integrated teaming consider the overall scope of/requirement for
Teaming participation of stakeholders from users; acquisition executives; acquisition
organizations; developers (primes, subcontractors, suppliers, vendors); test
organizations; and other support organizations?
§ Should the team adopt a common process for team operation or rely on
each team member to use his/her own organization’s processes?
§ Are various team member processes compatible at the interface points
of the processes?
§ Do life-cycle support considerations drive the selection of processes
across the team members (tools, support data commonality/compatibility)?
§ Are integrated information infrastructures needed to
facilitate/coordinate within/external to the team, especially for
geographically dispersed members/stakeholders?
Validation § Is validation performed early and continuously throughout the
acquisition life-cycle?
§ Are validation processes used to demonstrate that acquisition process
work products fulfill the acquisition strategy, and that the processes will
successfully acquire products/services?
§ Are validation processes used to ensure that received products/services
fulfill their intended use?
§ Is the test community treated as a major stakeholder in the validation
process, beginning with up-front planning through final system validation?
§ Has the acquisition project defined the degree to which validation is
required, both early in the acquisition project definition and when
products/systems are received?
§ Do acquisition project plans identify adequate resources to execute
validation activities?
Verification § Is verification performed early and continuously throughout the
acquisition life-cycle?
§ Does the acquisition project ensure that selected work products meet
project requirements?

15
 § Does the acquisition project ensure that the evolving acquired products
satisfy contractual requirements?
Transition to § Does acquisition planning for transition include establishing support
Operations & strategies through organic support infrastructures, contractor logistics
Support support, or other sources?
§ Are the roles/responsibilities of the acquirer, supporter and user defined
in the life-cycle support of the system?
§ Has the acquirer determined whether it will execute the transition
function directly, or as a result of the acquisition itself?
§ Are responsibilities for capability enhancements during the support
phase defined, taking into account the magnitude/complexity of the
envisioned change?
§ Is the organization responsible for support (i.e.,” level 1 maintenance”)
and enhancements (i.e., “level 2 maintenance”) explicitly identified?
§ Has the acquisition project assigned responsibility for implementation
of process improvement practices?
§ Is the acquisition project working with operational units to plan for
product transition into operational use and eventual disposal of the product
(or technical/functional obsolescence of the software)?
Project § Are monitoring/control functions implemented in the acquisition project
Monitoring & as acquisition planning is performed and the acquisition strategy is
Control defined?
§ Does the acquisition project have internal processes, plans and work
products that should be monitored for progress/satisfactory completion?
§ Do the monitored work products include specifications, plans, Request
for Proposal components, etc.?
§ Do monitored items include staffing levels/qualifications, system
performance objectives/thresholds, infrastructure readiness (tools,
networks, etc.) and other project planning activities/products?
§ Is acquisition project risk actively identified and mitigated?
§ Is corrective action applied when execution does not match acquisition
project planning (e.g., internal staffing, project plan completion dates,
draft/final solicitation & contract award milestone slips)?
§ Are corrective actions required to resolve variances from acquisition
project plans tracked to closure?
§ After supplier selection/contract award, is monitoring and control still
applied to the internal acquisition project
§ After supplier selection/contract award, does monitoring and control
also cover supplier performance against the supplier’s project plan?
Acquisition § Have the acquisition project products and processes (e.g., solicitation
Process & packages, etc.) been evaluated?
Product § Has the solicitation package been developed per the standard/format
Quality agreed to by the project team?
Assurance § Does the solicitation package conform to all applicable policies and
laws?
§ Does the acquisition project’s risk management process conform to that

16
 called out for in the risk management plan?
Risk § Is the acquisition strategy influenced by risk identification and
Management estimation of risk probability/impact?
§ Does the acquirer assess/manage overall acquisition project risks over
the duration of the project?
§ Does the acquirer assess/manage risks associated with supplier
performance?
§ For acquisitions using integrated teams, are risks associated with loss of
inter- or intra-team coordination considered?
Configuration § Are acquisition work products (e.g., solicitation packages) placed under
Management internal CM control?
§ Are interim/final primary/subordinate supplier work products monitored
to ensure that project goals are met?
§ Are provisions made for conducting CM of supplier
products/documentation?
§ Are methods established/maintained to ensure that acquisition project
data is complete/consistent?
§ Has the acquisition project determined which work products should be
CM-controlled?
Decision § Is there a repeatable, criteria-based decision-making process used for
Analysis & making critical decisions that define/guide the acquisition process itself?
Resolution § Is there a repeatable, criteria-based decision-making process used for
making critical decisions with the selected supplier?
§ Does the process for decision making document the acquisition project
decision rationale?
§ Can criteria for critical decisions be revisited when changes/technology
insertion decisions impacting essential requirements are considered?
Measurement § Does the acquisition project have the information it needs for
& Analysis determining status of its activities throughout the acquisition life-cycle, the
supplier’s activities per contractual requirements and the status of the
evolving products acquired?
§ In acquisition projects requiring multiple products or teaming
relationships, is additional information needed/available to ensure
programmatic, technical and operational interoperability objectives are
identified/measured/achieved?

The use of checklists such as the one above helps maintain discipline and structure in the
software acquisition process, and can cover any phase of a software acquisition life-
cycle-
Table 3: IEEE Std 1062, 1998 Edition Software Acquisition Checklists [IEEE, 1998]

17
No Category Subcategories
.
1 Organizational Strategy
2 Define the Software
3 Supplier Evaluation · Financial soundness · Maintenance service
· Experience/capability · Product usage
· Development/control · Product warranty
processes · Costs
· Technical assistance · Contracts
· Quality practices
4 Supplier/Acquirer
Obligations
5 Quality/Maintenance Plans · Contents of a quality · Contents of a
plan maintenance plan
6 User Survey · Operation · Installation
· Reliability · Costs
· Maintenance service · Security
· Performance · Documentation
· Flexibility · Miscellaneous
7 Supplier Performance · Performance criteria · Correction of
Standards · Evaluation/test discrepancies
· Acceptance criteria
8 Contract Payments
9 Monitor Supplier Progress
10 Software Evaluation · Functionality · Serviceability
· Performance · Ease of installation
· Reliability · Ease of use
· Availability · Adequacy of
· Ease of modification documentation
· Cost to acquire/use
11 Software Test
12 Software Acceptance · Rate actions for · Rate remedies if
certification supplier fails

Finally, the use of checklists can benefit specific monitoring and oversight activities of
software acquisition such as risk management. Gallagher [Gallagher, 1997] defines a
software acquisition risk management Key Process Area (KPA) that can be put into a
high-level checklist form (see Table 5) in order to assess whether Goals, Activities,
Commitments, etc., associated with risk management have been covered in the software
acquisition process.
Table 5. Software Acquisition Risk Management KPA [Gallagher, 1997]

18
Category Goal Description Check
Goals 1 SA risk management is an integral part of the project’s
defined SA process ______
2 The project identifies/deals with risk in a positive manner,
such that identification is recognized/rewarded, and results
in effective risk handling ______
Activities 1 SA risk management activities are integrated into SA
planning ______
2 The Software Acquisition Risk Management Plan is
developed in accordance with the project’s defined SA
process ______
3 The project team performs its SA risk management
activities in accordance with documented plans ______
4 Risk management is conducted as an integral part of the
solicitation, project performance management, and
contract performance management processes ______
5 SA risk handling actions are tracked and controlled until
the risks are mitigated ______
Commitments 1 The acquisition organization has a written policy for the
management of SA risk ______
2 Responsibility for SA risk management activities is
designated ______
Abilities 1 A group that is responsible for coordinating SA risk
management activities exists ______
2 Adequate resources are provided for SA risk management
activities ______
3 Individuals performing SA risk management activities
have experience or receive required training ______
Measurement 1 Measurements are made/used to determine status of the
acquisition risk management activities and resultant
products ______
Verification 1 Acquisition risk management activities are reviewed by
acquisition organization management on a periodic basis ______
2 Acquisition risk management activities are reviewed by
the project manager on both a periodic and event-driven
basis ______

Collecting/Analyzing Meaningful Metrics:

Meaningful metrics are collected and analyzed by both software acquirers and developers
in order to track progress, confirm knowledge, manage risk, improve processes and
ensure that all stakeholders are well-informed. Meaningful and suitably tailored metrics
should cover cost, schedule, software size, requirements, tests performed/completed,
number of defects detected/corrected, and quality attributes. Acquirers can apply some of
these same metrics to assess whether the developer will be able to deliver software within
cost, schedule, performance and quality constraints. The use of earned value analysis

19
tools to track cost/schedule and help mitigate risk is an important driver in determining
meaningful metrics, supporting the ability of a program to maintain consistent,
predictable practices and acceptable process outcomes. Sharing of metrics between
developers and acquirers is an important aspect of a successful software acquisition
process.
Leading software developers have been described as relentless in their efforts to collect
metrics to improve project processes and results [GAO, 2004]. They typically also
establish goals and track metrics for company-wide initiatives, such as cost reduction
efforts and customer satisfaction. These leaders have continually emphasized the critical
nature of measuring processes, collecting metrics and using them to improve performance
in their workforce through training. In addition, a good part of their success can be
attributed to their use of an Earned Value Management System (EVMS) and a project
time-tracking system (to record time spent on “cost of quality” and “cost of poor
quality”). The cost of poor quality is a direct reflection of the relative level of
effectiveness of an organization’s software acquisition and/or development processes.
Typical metrics used by leading software developers are indicated in Table 6.

Software and hardware evaluation criteria

1)

Table 6: Metrics Used by Leading Software Developers [GAO, 2004]

Major Examples Usefulness
Metric
Cost · Cost per phase · Products of an earned value
· Effort per phase management system, indicating
· Planned vs. actual cost progress towards completing software
· Cost performance index development against the plan
· Poor cost performance index
indicates problems meeting cost goals
· May need to consider project scope
reduction to meet release dates, or
canceling the program
Schedule · Planned vs. actual delivery dates · Products of an earned value
· Schedule estimation accuracy management system, indicating
· Percentage of project on time achieved schedule progress against
· Schedule performance index the plan
· Used over the development life-
cycle to gauge progress toward
developing key products or meeting
critical milestones
· Close attention to schedule
deviations identifies the ability to
meet project goals, and if/when
additional resources are needed
Size · Amount of new, modified and · Used by management to compare

20
 reused code
· Size estimation accuracy
Requirements · Total requirements/features
committed for delivery
· Percentage of requirements
completed
· Number of requirements changes by changes, can impact cost/schedule
phase
Tests · Number if tests planned, completed, · Determine extent to which planned
passed
· Percent of plan tests completed
· Percent of planned tests passed
Defects · Number of defects per phase
· Phase defect originated vs. phase
found
· Cost to fix defect
· Severity of defect
· Total unresolved defects
Quality · Cost of quality efforts
· Cost of poor quality (rework)
· Number of quality goals
missed/achieved
· Customer satisfaction survey results to assure a given quality level
21
amount of code produced vs.
estimated
· Changes to size estimates indicate
potential cost/schedule problems
· Used to assess progress towards
meeting acquirer’s performance
demands
· Large number of changes, or late
commitments
· Large number of changes, or late
changes, can result in software with
higher defect levels
·
software tests have been successfully
accomplished
· Deviation from planned number of
tests may indicate inadequate testing
and subsequent quality problems,
resulting in costly rework in later
project phases
· Track software problems to (1) the
phase where they were found, (2) the
phase where they should have been
found and (3) determine the cost to
fix
· Defects found after the phase in
which they were created indicate
performance problems that may
increase cost/schedule due to rework
of software and correction of
development processes
Too few defects found may indicate
inadequate test coverage during the
test phase, or insufficient formal
technical review of documents in the
design phase
· Provides information on potential
reliability of delivered software
· Indicates the amount of
money/time invested in development
· Defects found/fixed in the phase
that they occur indicates good process
 quality
· Defects found/fixed in downstream
phases are costly/time-consuming,
and indicate weaknesses in the
development process that will require
corrective action

2)
Table 8: Recommended COTS-Based Software System Acquisition
Best Practices
Practice Comments
Best Practices for Establishing a Program Baseline
Perform Software Architecture Trade · Perform with system architecture trades
Studies · Include major COTS and legacy components
· Supports Government software architecture
baseline selection
· Include user in all trades
Determine Realistic, Independent · Size, effort, cost and schedule
Baseline Software Estimates · COTS, reuse and newly-developed
· Include tasks not reflected in cost models
· Include COTS refresh through both development
and sustainment
Include Software in Systems · Prioritized requirements
Performance Requirements · COTS software support requirements
· Specialty engineering (reliability, maintainability
& availability (RMA), security, safety)
· Key Performance Parameters (KPP)
· Open system architecture
Best Practices for Obtaining Contractual Insight
Require Key Software Technical and · Highest risk reduction potential is in plans;
Management Deliverables requirements and architecture; test plans,
procedures and reports; metrics reports; delivery,
installation and O&M documentation
· Use electronic delivery
Require Timely Electronic Access to · COTS evaluation trade studies
All Software Products · Intermediate and final products (requirements,
architecture and design; implementation (including
code); integration and verification testing)
Require Software-Level Technical · In addition to system reviews
and Management Reviews · Include COTS software experts in reviews
Best Practices for Obtaining Contractual Commitment
Mandate Compliance with Robust · E.g., EIA/IEEE J-STD-016 (commercialized
Commercial Standards version of MIL-STD-498)
· Tailor standard for COTS-based software system

22
 development
Require Contractor Commitment to · Require SDP to include processes for COTS
Software Development Plan (SDP) software
· Require Integrated Management Plan (IMP) to
have adequate system engineering and sustainment
for COTS
· Include commitment to SDP in IMP
Best Practices for Providing Tools for Contract Management
Incentivize Software Quality (Not · Use award and incentive fee plans
Just Cost/Schedule) · Reward adherence to defined software processes
and software process improvement
· Reward timely/acceptable response to
Government comments
· Reward low rework rates
· Reward meeting performance requirements (e.g.,
RMA) post delivery/launch
· Reward architecture development that supports
COTS-based software system evolution
Mandate Periodic Team Software · Relate results and improvement actions directly
Capability Appraisals to award fees
· Explicitly include COTS processes in appraisals
Best Practices for Selecting a Capable Software Contractor Team
Evaluate Software · Individual team member evaluation is
Capability/Processes of Offeror insufficient
Teams · Evaluate software capability as a separate
subfactor under the Mission Capability factor
· Weight according to software risk
Evaluate Teams’ Proposed Software · Corporate and past project process evaluation is
and Related Processes insufficient
· Include COTS software, systems engineering and
logistics processes
Evaluate Realism of Cost and · Be suspicious of productivity extremes, COTS,
Schedule Bids reuse, low lines of code, and short integration
times
· Ensure that all COTS software tasks are included
in the cost and schedule bid
· Ensure bid contains sufficient cost and schedule
margin
Evaluate Software and Hardware
Architecture with System Design
Best Practices for Performing Technical Product Review
Focus Technical Review Resources · IPTs, Technical Interchange Meetings (TIMs),
on Areas of Highest Risk working groups, per reviews, etc.
· Software-level technical reviews
· High risk/critical software products (including
COTS software)
23

Include Users/Operators in All
Technical Review Activities
Monitor Software Integration and
Verification Adequacy
Best Practices for Performing Software Process Review
Review Effectiveness of Team’s
Defined Software and Related
Processes
Perform Periodic Team Software
Capability Appraisals
Review Team’s Adherence to
Defined Software and Related
Processes
Best Practices for Managing the Contract
Use Incentive/Award Fees
Aggressively
Apply Proactive Quantitative
Management
Ensure Adherence to Software-
Inclusive Requirements
Perform Periodic Independent
Assessments
Best Practices That Span the Life-cycle
Software Acquisition Risk
Management
· Key software technical deliverables
· Ensure users/operators understand the evolving
design, including COTS software capabilities and
impacts on O&M
· Begin at the build level
· Focus on areas of highest risk
· Focus on early performance analysis results and
meeting KPPs
· Ensure that COTS software performance is
measured
· Ensure requirements allocated to COTS software
are verified
· Identify process deficiencies (especially across
team boundaries and with COTS products)
· Assist with process improvement
· Individual Level 2 and 3 CMMI/CMM
compliance may not be sufficient
· During contract performance
· Support for significant program or award fee
milestones
· Explicitly include COTS processes
· Identify adherence deficiencies
· Assist in deficiency correction
· Motivate good software and related practices
· Focus on quality an d architecture
· Ensure a comprehensive software/systems
metrics program balanced across information
categories (include leading quality indicators, e.g.,
rework)
· Perform cross-metric analysis
· Earned value alone is insufficient
· Especially RMA, safety and security
· Especially COTS software supportability
· Support for significant program or award fee
milestones
· Act aggressively on findings
· Continuous software acquisition risk
management across all acquisition organization
levels
· Program-level risk management and contractor
development risk management are necessary, but
24

Software Systems Acquisition
not sufficient
· Establish management reserves consistent with
software risks (including/especially COTS
software risks)
· Integrate software acquisition with the system
acquisition process (from mission needs
identification through system retirement, especially
during pre-contract activities)

Figure 7: Use Quality Function Deployment to Translate Successful Acquisition

3) Phase Containment Matrix:

A Phase Containment Matrix is a well-known tool for identifying and reducing software
defects across the various phases of a software life-cycle. It is constructed by identifying
the phases during which a defect is inserted as the column headings, and the phases
where the defect was detected as the row headings (see Figure 8).

Figure 8: Phase Containment Matrix for Tracking Software Development Quality

Within each cell, the number of defects that are detected at that phase in the software life-
cycle are entered. There is a basic, yet critical, assumption that root-cause defect analysis
has been performed in order to accurately determine in which phase of the life-cycle the
defect was actually inserted.
In-phase defects are often not counted as rework, as these defects are assumed to be a
natural part of the process. For example, the rework required to correct the 25 defects

25
that were introduced during the requirements phase, but were also detected and
eliminated in the requirements phase, would not be counted, even though there is still
potential cost and schedule impact associated with the rework. Ignoring the cost of in-
phase rework deludes both the customer and the organization, and deprives the
organization of an opportunity to improve its in-phase processes. Analyzing the cost of
performing rework on 25 in-phase requirements defects might provide significant
justification for improving an acquisition or development process to reduce the number
of introduced defects.

Out-of-phase defects are generally classified as rework and, just as generally, are going to
cost more to fix than in-phase defects. The larger the span (in project phases) between
the insertion point and the detection point, the more expensive it is going to be, on a per
defect basis, to rework and fix it. This “more expensive” factor can be an order of
magnitude of cost between each phase.
The concept of a Phase Containment Matrix could, with some adaptation, also be applied
to improve software acquisition processes by defining an appropriate set of processes
within which software acquisition defects can be detected. Figure 9 provides a simple
example of how this might be accomplished.

26
ANTICIPATED BENEFITS OF IMPLEMENTATION: of SW and HW

Successful implementation of software acquisition process improvement can result in:

§ Realistic, incremental and continuous improvements to software acquisitions – A
process of evolutionary software acquisition avoids the temptation to succumb to
unrealistic expectations in an attempt to force revolutionary changes in software
development.
§ Adherence to well-understood, well-defined, manageable requirements – An
evolutionary software acquisition process ensures that software development is limited to
things that are possible to manage. Only requirements that are well-defined and well-
understood can be adequately managed.
§ Following of disciplined, structured software management and development
processes that provide predictable results – Disciplined, structured software management
and development process phases that end in a management review/gate ensure that the
acquisition will remain on track, and that project course corrections can be identified and
implemented effectively.
§ Well-informed, well-disciplined and knowledgeable software acquisition teams –
The suitably tailored collection and use of process metrics ensures that the software
acquisition is adequately tracked; that project knowledge is confirmed and shared among
stakeholders; that acquisition risk is effectively managed; and that software acquirers are
firmly integrated into the continuous improvement process.

27
§ Increased probability of software acquisition successes and “best value” awards –
Heavy reliance on configuration management, risk identification/management, peer
reviews and quality assurance, coupled with earned value management techniques, helps
ensure a successful and improving acquisition process.

DETAILED CHARACTERISTICS

Key Characteristics of the Acquisition Process Improvement Gold Practice

Characteristic Comments
Manageable · Business decisions must be made to invest suitable
Development time/resources in achieving high levels of software process maturity
Environment is · Honest relationships between the acquirer and developer must be
Defined and established
Implemented · There must be a mutual understanding of software requirements
between the acquirer and developer in order to optimize
setting/managing requirements
· There must be a match between available resources and
requirements, supported by systems engineering analysis and trade-
off analyses that consider the performance, cost and schedule
impacts of major changes to software requirements
· Technical and administrative “knowledge” must be obtained
early in the software development process
· Sufficient requirements knowledge must be demanded by PMs
and other stakeholders at key process points
· In order to ensure that software requirements are appropriately
set/managed, it is necessary to document that software requirements
are achievable based on systems engineering knowledge
Disciplined · Processes that meet program needs must be established and
Development adhered to
Processes, · Acquirers should develop a list of tailored systems engineering
Supported by Gated deliverables that include software based on the results of required
Reviews engineering activities at the appropriate stages/phases of system
development
· A sufficient amount of software development time should be
devoted to requirements-setting activities
· Well-written and achievable requirements must be developed
and managed/controlled before actual software design begins
· Lower-level software design elements should be defined before
any coding begins
· The number and timing of requirements changes should be
aggressively managed
· Test plans that are developed should be based on a stable set of
requirements
· Ensure that the software design is mature before approving
release/production

28
 · All production processes must be under control before
production begins
· Provide for gated reviews of systems engineering baselines on
an event-driven basis
Established and · The collection and reporting of metrics related to software
Leveraged Metrics acquisition and development should be required internally, and from
are Defined and contractors, in order to ensure adequate oversight knowledge of
Utilized software-intensive acquisitions
· There should be relentless pursuit of tailored metrics that are
derived from effective processes
· Require software contractors to collect/report cost, schedule,
size, requirements, test, defect and quality metrics on a per-month
basis, if appropriate, and before program milestones
· Ensure that contractors are using an earned value system that
reports work at a suitably detailed level
· Software acquisition should be enforced with suitable controls
and performance incentives in DoD acquisition policy, software
acquisition improvement plans and software development contracts
Use of Appropriate · Formal processes such as SA-CMM® and CMMI-AM® provide a
Tools and framework for implementing a structured software acquisition
Techniques to improvement process
Support Acquisition · Tools and techniques to be considered include document
Activities templates, documentation standards, checklists, QFD, AHP, Phase
Containment Matrices, earned value tracking, knowledge-sharing
systems, lessons learned repositories, past performance databases,
centralized acquisition resources, software tools that support risk
management, configuration management, source selection, decision-
making, etc.
Use of Integrated · Integrated Process Teams conduct process analyses and identify
Process Teams bottlenecks and non-value-added process structure and flow, with a
(IPTs) That Involve constant focus on measurement/improvement of software acquisition
All Stakeholders processes.
(including End · IPTs keep the focus on the customer’s needs and requirements
Users) · Effective IPTs contribute to collaborative requirements
development, an important attribute of the Air Force “Agile
Acquisition” initiative
· Teams should be cross-functional and multi-disciplinary, and
there must be leadership/senior management commitment to support
the team (with appropriate allocation/delegation of decision-making)
· Organizational structures should be defined such that team
performance is suitably rewarded

RELATIONSHIPS TO OTHER PRACTICES:

29
The Figure below represents a high-level process architecture for the subject practice,
depicting relationships among this practice and the nature of the influences on the
practice (describing how other practices might relate to this practice). These relationship
statements are based on definitions of specific “best practices” found in the literature and
the notion that the successful implementation of practices may “influence” (or be
influenced by) the ability to successfully implement other practices. A brief description
of these influences is included in the table below.

Process Architecture for the "Acquisition Process Improvement" Gold Practice™

30
Other general criteria for evaluation of SW and HWLen Bass
Feb 2, 2010
© 2010 Carnegie Mellon University
• Externally visible properties
• Relationships among elements
• Multiple different structures

Uses
• Guide to construction
• Artifact for analysis
Quality attributes are properties of a system such as
• Functionality
• Reliability
• Usability
• Efficient 2010 Carnegie Mellon University
• Maintainability
• Portability

Acquisition perspective

Given the importance of software architecture to software development,

there are three portions of the life cycle where architectural knowledge
is important
• Requirements
• Proposal evaluation
• Architecture evaluation of system during development
Requirements information must include quality attribute requirements
• Business/mission goals must be available to proposers/designers

During Development

Acquirers should evaluate architecture during development

• To ensure designed system satisfies mission goals
• To ensure conformance
• To enable adaptation to changes in mission

Evaluating integrity and quality f SW and HW

Requirements analysis

� R isk assessment

�Development analysis

�Code review

31
�Independent testing

�Contract verification

�Algorithm analysis

�Software metrics

32
Essentials In system Implementation

�Adaptive
 – tailoring to the scope of the target program

�Integrated
 – working closely with development teams

�Self-improving


Essentials in System Implementation

Depending on the unique circumstances of the implementation process, the status of data
compilation, and the organizational climate, increased productivity is normally reached
between the second and fifth year of implementation. This is identified by the threshold
point. Again, this is dependent on a variety of factors including:

the skills and experience of the staff involved;

the priority and commitment by the organization;

33
 the implementation strategy; and
the status of data compilation.

The Implementation Plan

Implementation can be seen as a six phase process. They are:

Creating an awareness GIS needs to be sold within an organization. The

education of staff is very important. Depending on the way in which GIS
technology is being introduced to the organization the process for creating an
awareness may differ. Technical workshops are often appropriate when a top-
down approach exists, while management workshops are often more relevant
when a bottoms-up approach exists. Education of the new technology should
focus on identifying existing problems within an organization. These often help
justify a GIS acquisition. They include :

spatial information is poorly maintained or out of

date;
spatial data is not recorded or stored in a standard
way;
spatial data may not be defined in a consistent
manner, e.g. different classifications for timber
information;
data is not shared between departments within an
organization;
data retrieval and manipulation capabilities are
inadequate to meet existing needs; and
new demands are made on the organization that
cannot be met with existing information systems.
Identifying System Requirements

The definition of system requirements is usually done in a user needs analysis. A user
needs analysis identifies users of a system and all information products required by those
users. Often a prioritization of the information products and the data requirements of
those products is also undertaken. A proper user needs analysis is crucial to the
successful evaluation of GIS software alternatives.

34
After user needs have been identified and prioritized they must be translated into
functional requirements. Ideally, the functional requirements definition will result in a set
of processing functions, system capabilities, and hardware requirements, e.g. data
storage, performance. Experienced GIS consultants often play a major role in this phase.

System Evaluations

Evaluating alternative hardware and software solutions is normally conducted in several

stages. Initially a number of candidate systems are identified. Information to support this
process is acquired through demonstrations, vendor literature, etc. A short listing of
candidates normally occurs based on a low level assessment. This followed by a high
level assessment based on the functional requirements identified in the previous phase.
This often results in a rating matrix or template. The assessment should take into account
production priorities and their appropriate functional translation. After systems have been
evaluated based on functional requirements a short list is prepared for those vendors
deemed suitable. A standard benchmark, as discussed earlier, is then used to determine
the system of choice.

Justifying the System Acquisition

The proper justification of the chosen system requires consideration of several factors.
Typically a cost-benefit analysis is undertaken to analyze the expected costs and benefits
of acquiring a system. To proceed further with acquisition the GIS should provide
considerable benefits over expected costs. It is important that the identification of
intangible benefits also be considered.

The justification process should also include an evaluation of other requirements. These
include data base development requirements, e.g. existing data versus new data needs and
associated costs; technological needs, e.g. maintenance, training, and organizational
requirements, e.g. new staff, reclassification of existing job descriptions for those staff
who will use the GIS.

System Acquisition and Start Up

After the system, e.g. hardware, software, and data, is acquired the start up phase begins.
This phase should include pilot projects. Pilot projects are a valuable means of assessing
progress and identifying problems early, before significant resources have been wasted.
35
Also, because of the costs associated with implementing a GIS it is often appropriate to
generate some results quickly to appease management. First impressions are often long
remembered.

Operational Phase

The operational phase of a GIS implementation involves the on-going maintenance,

application, and development of the GIS. The issue of responsibility for the system and
liability is critical. It is important that appropriate security and transaction control
mechanisms be in place to support the system. A systematic approach to system
management, e.g. hardware, software, and data, is essential.

2. Strategies concerning how to evaluate

We distinguish between three types of strategies:

• Goal-based evaluation

• Goal-free evaluation

• Criteria-based evaluation

The differentiation is made in relation to what drives the evaluation. Goal-based

evaluation means that explicit goals from the organisational context drive the evaluation.
These goals are used to measure the IT-system. The goal-free evaluation means that no
such explicit goals are used. Goal-free evaluation is an inductive and situationally driven
strategy. Criteria-based evaluation means that some explicit general criteria are used as
an evaluation yardstick. The difference to goal-based evaluation is that the criteria are
general and not restricted to a spe-cific organisational context.

2.1 Goal-based evaluation

Goal-based evaluation can be seen to be formal-rational to its character (e.g. Walsham,

1993). Walsham means that a formal-rational view sees evaluation mainly as quantitative
process of calculating the likely costs and benefits. According to Patton (1990) goal-
based evaluation is defined as measuring the extent to which a program or intervention
has attained clear and specific objectives. The focus is on intended services and outcomes
of a program – the goals. Good et al (1986) claim that evaluations should be measurable
and that the evalua-tion should meet the requirements specification.
One common criticism of the formal-rational view is that such evaluation concentrates on
technical and economical aspects rather than human and social aspects (Hirschheim &

36
Smith-son, 1988). Further Hirschheim & Smithson means that this can have major
negative conse-quences in terms of decreased user satisfaction but also broader
organizational consequences in terms of system value.

We agree with the criticism of Hirschheim & Smithson, but when analysing goal-based
evaluation in an ideal typical way there is no imperative relation be-tween a focus on
technical and economical aspects and goal-based evaluation. Of course, the stated goals
can be of a human or organisational character. However, the traditional way of
understanding goal-based evaluation is often related to harder measurable goals.

Further, there is no imperative relation between a goal-based approach, and a quantitative

process. A judgement of, if the goals have been fulfilled can be evaluated with a
qualitative process. As we see it, the differences between a quantitative and qualitative
strategy is that the quantitative strategy aims to decide if the goals are fulfilled and which
goals that are ful-filled. The fulfilment of the goals will be expressed in quantitative
numbers.

There are also goals of more social or human character. The fulfilment of this types of
goals is preferably expressed in qualitative terms. The qualitative process has also,
besides the if- and which questions, a better possibility to describe how the goals are
fulfilled. This means that the qualitative approach aims at achieving richer descriptions.

The goals that are used for evalua-tion are derived from an organisational context. That
means that they are situationally appli-cable, which means that they act like specific
business goals.

The basic strategy of this approach is to measure if predefined goals are fulfilled or not;
to what extent and in what ways. The approach is deductive. What is measured depends
on the

37
character of the goals and a quantitative approach as well as qualitative approach could
be used. In this paper we adopt the concept of goal-based evaluation from Patton (1990)
in order to identify this approach.

2.2 Goal-free evaluation

The second identified approach is a more interpretative approach (e.g. Remenyi, 1999;
Wal-sham, 1993). The interpretative perspective views IT-systems as social systems that
have in-formation technology embedded into it (Goldkuhl & Lyytinen, 1982). The aim of
interpretive evaluation is to gain a deeper understanding of the nature of what is to be
evaluated and to generate motivation and commitment (Hirschheim & Smithson, 1988).

The involvement of a wide range of stakeholder groups is essential to this approach of

evaluation. This can also be a practical obstacle where time or resources for the
evaluation are short. Patton (1990) uses the term goal-free evaluation. Goal-free
evaluation is defined as gathering data on a broad array of actual effects and evaluating
the importance of these effects in meeting demonstrated needs (Patton, 1990, Scriven,
1972). The evaluator makes a deliberate attempt to avoid all rhetoric related to program
goals; no discussion about goals is held with staff; no program brochures or proposals are
read; only the program’s outcomes and measurable effects are studied. The aim of goal-
free evaluation is to (Patton, 1990):

1) avoid the risk of narrowly studying stated program objectives and thereby missing
important unanticipated outcomes

2) remove the negative connotations attached to discovery of unanticipated effect:

“The hole language of side-effected or secondary effect or even unanticipated
effect tended to be a put-down of what might well be a crucial achievement,
especially in terms of new priorities.”

3) eliminate the perceptual biases introduced into an evaluation by knowledge of

goals; and

4) maintain evaluator objectivity and independence through goal-free conditions

2.3 Criteria-based evaluation

The third identified approach is a criteria-based approach. There are lot of criteria-based
ap-proaches around such as checklists, heuristics, principles or quality ideals. In the area
of Human-Computer Interaction you can find different checklists or heuristics (e.g.
Nielsen, 1994; Nielsen, 1993, Shneiderman, 1998). What is typical for these approaches
is that the IT-systems interface and/or the interaction between users and IT-systems acts
as a basis for the evaluation together with a set of predefined criteria. More action
oriented quality ideals and principles for evaluation can be found in Cronholm &

38
Goldkuhl (2002) and in Ågerfalk et al (2002). The basis for these action oriented ideals is
to understand if and how the IT-system support the actions performed in the business (see
discussion of IT-systems in section 3.1)

The criteria used are grounded in and derived from one or more specific perspectives or
theo-ries. For example, the criteria in Nielsen’s (1994) checklist are derived from
cognitive and computer science. The action oriented ideals are mainly derived from
language action theory but also inspired by usability issues. Using criteria means to set
focus on certain qualities that according to the perspective is important to evaluate. At the
same time the attention accord-

39
 ing to the criteria also de-emphasize other qualities. The criteria chosen governs the
evalua-tor’s attention and thereby the kind of knowledge the evaluator achieves.
Another difference in comparison to goal-based evaluation is that the criteria that
are used are not derived from a specific organisational context. That means that they are
more general ap-plicable (see section 2.1). Ideal typically, the basic strategy of criteria-
based evaluation is pure deductive. The word criteria is often used in relation to
preordinate designs, and the use of this term has a ‘hard’ scientific feel which supports
the tendency to prioritize technical and quantitative data

40
 Different Units topics

a) MIS Approach

MIS is generally used by medium and larger scale organizations. However, small
organizations are yet to understand its application. There is dire need to build up
computer culture by properly disseminating information about computer applications and
its benefits.

Implementation of MIS can be achieved by using any of the methods such as direct,
parallel, modular or phase in.

• Direct Approach

Direct installation of the new system with immediate discontinuance of the old existing
system is reffered as “cold turnkey” approach. This approach becomes useful when these
factors are considered.

1. The new system does no replace the existing system.

2. Old system is regarded absolutely of no value
3. New system is compact and simple.
4. The design of the new system is inexpensive with more advantages and less risk
involved.

• Parallel Approach

The selected new system is installed and operated with current system. This method is
expensive because of duplicating facilities and personal to maintain both the systems. In
this approach a target date must be fixed when the operations of old system cease and
new one will operate on its own.

• Modular Approach

This is generally recognized as “Pilot approach”, means the implementation of a system

in the Organization on a piece-meal basis.

This has few advantages / merits

1. The risk of systems failure is localized

2. The major problem can be easily identified and corrected before further
implementation.
3. Operating personal can be trained before system is installed in a location.

41
• Phase-in-Implementation

This approach is similar to modular method but it differs because of segmentation of

system, however, not the organization. It has advantages that the rate of changes in a
given Organization can be totally minimized and the data processing resource can be
acquired gradually over a period of time. System exhibits certain disadvantages such as
limited applicability, more costs incurred to develop interface with old system and a
feeling in the Organization that system is never completed.

b) Implementation Procedures

• Planning the Implementation

After designing the MIS it is essential that the organization should plan carefully for
implementation. The planning stage should invariably include the following:

1. Identification of tasks of Implementation

Planning the implementation activities, acquisition of facilities, procedures development,

generating files and forms, testing the system and evaluating and maintenance of the
system.

2. Relationship establishment among the activity

Network diagram must be prepared to correlate concurrent and sequential activities.

3. Establishing of MIS

For monitoring the progress of implementation and for proper control of activities,
efficient information system should be developed.

4. Acquisition of Facilities

For installation of new system or to replace current system the manager should prepare a
proposal for approval from the management by considering space requirement movement
of personal and location for utility outlets and controls.

5. Procedure Development

This is an important stop for implementation of the system including various activities
such as evaluation selection of hardware, purchase or development of software, testing

42
and implementation strategies.

6. Generating Files and Forms

The MIS manager should generate files and formats for storing actual date. This requires
checklist data, format date storage forms and other remarks in data base.

7. Testing of the System

Test should be performed in accordance with the specifications at the implementation

stage consisting of component test sub system test and total system test.
• Evaluation and maintenance of system

The performance should e evaluated in order to find out cost effectiveness and efficacy of
the system with minimum errors due to designs environmental changes or services.

c) Software Maintenances

The proper maintenance is the enigma of the system development and it holds software
industry captive lying up programming resources. There are some problems in
maintenance such as regarding it as non rewarding non availability of technicians and
tools no cognizance of users about maintenance problem and cost lack of standard
procedures and guidelines. Most programmers feel maintenance as low level drudgery. If
proper attentions is paid over a period of time eventually less maintenance is required.

Types of Maintenance

The maintenance of system are classified into corrective/adaptive/perfective. Corrective

maintenance means repairing process or performance failures. Adaptive maintenance
means changing the programming function whereas perfective maintenance deals with
enhancing the performance or modifying the program.

Primary Activities of a Maintenance Procedure

Documentation is major part of maintenance in system development. Maintenance staff

receives requests from the authorized user. Programming library should be maintained.

Reduction in Maintenance Costs

Several organizations having MIS generally go in for reducing maintenance costs and it
consists of three major phases.

1. Maintenance management audit through questionnaires and interviews.

2. Software system audit.
3. Software modification.

43
Evaluation Methods

Evaluation of the MIS in an organization is integral part of the control processes. There
are several evaluation approaches such as quality assurance review compliance of audits
budget performance review computer personal productivity assessment computer
performance evaluation service level monitoring user audit survey post installation
review and cost benefit analysis.

Evaluation performance measurement can be classified into two classes as effectiveness

and efficiency. The relationship between effectiveness and efficiency is that the format is
a measure of goodness of out put and the latter is a measure of the resources required to
achieve the output.

44