ASSESSING THE RELATIONSHIPS BETWEEN PROJECT SUCCESS,
AND SYSTEM ENGINEERING PROCESSES
Paul J . Componation, The Univer sity of Alabama in Huntsville
Alisha D. Youngblood, The Univer sity of Alabama in Huntsville
Dawn R. Utley, The Univer sity of Alabama in Huntsville
Phillip A. Far r ington, The Univer sity of Alabama in Huntsville
Abstract
NASAs Procedural Requirements for systems
engineering - NPR 7123.1A NASA Systems
Engineering Processes and Requirements was released
to provide the agency with a standard system
engineering model to support the continued
development of manned and unmanned space flight
systems. The NPR draws on both industry best
practices and NASA experience to identify a core set of
system engineering processes. The NPR is
comprehensive, however, managers are still faced with
the competing demands of balancing cost, schedule and
technical requirements on ever more complex system
development projects. Additional guidance on
effective tailoring of system engineering to meet
individual project needs may be of help. This paper
reports on an initial study into the relationship between
the system engineering processes identified in NPR
7123.1 and the technical, schedule and budget success
of past and currently NASA space flight hardware
projects. The results do show correlations; however
not all the processes identified had the same effects on
success. The unique characteristics of individual
projects do influence which system engineering
processes should be focused on.
Keywor ds: Project Success, System engineering,
Project Management
Introduction
NASAs Marshall Space Flight Center (MSFC) has
been a primary benefactor of the systems engineering
approach. They have been part of such complex and
overarching projects as the Apollo program, the space
shuttle, and more recently have become involved with
the redirected mission to return to the Moon in
anticipation to go beyond to Mars (ref). While some of
these complex projects have been very successful using
the systems engineering (SE) approach, others have
become unduly constrained and overburdened with the
full detailed use of the SE approach, and still others
have been successful with only a limited application of
the formal SE approach. As part of SE Skill Lead
responsibility an effort has been initiated to address
personnel training specific to systems engineering and
integration (SE&I), and develop effective ways to
tailor standard SE&I processes. The research reported
in this paper is part of a larger study in which an
assessment of historical and current MSFC projects
will be made to determine the correlation between the
extent of the formal SE&I process and organizational
implementation with programmatic and technical
success. Specifically the researchers were interested in
evaluating the Systems Engineering Approach and
Teaming Effectiveness as related to project success.
An SE&I Skill Board was established to provide
overarching guidance and to identify and rate
successful MSFC projects. Seven projects were
identified to begin the study. Results from the data
collection and analysis efforts will be used to develop
tailoring approaches to standard SE&I processes. The
ultimate goal is to apply lessons learned to on-going
MSFC projects, namely the Constellation project
(travel to Mars).
This paper is a summary of the key findings
reports the Beta project used to assess the methodology
and test the analysis and resulting conclusions for
appropriateness. The project used for the Beta test was
the International Space Welding Experiment (ISWE)
project whose mission was to incorporate Russian
technology of welding in space in the space station.
While this project never flew, it was considered
extremely successful in overcoming obstacles and
working toward a viable plan to safely weld in space.
Literature Review
Systems Engineering Approach.
NASA defines systems engineering as a robust
approach to the design, creation, and operation of
systems. In simple terms, the approach consists of
identification and quantification of system goals,
creation of alternative system design concepts,
performance of design trades, selection and
implementation of the best design, verification that the
design is properly built and integrated, and post-
implementation assessment of how well the system
meets (or met) the goals. (NASA SP-610S, 1995) This
definition, while developed and used to specifically
support NASA systems, does reflect similar
components as other definitions from the International
Council on Systems Engineering (INCOSE) and
researchers in the field. This definition is further
developed and put into practice with the publication of
NASA NPR 7123.1 Systems Engineering
1
Processes and Requirements in 2006. This new
Procedural Requirement refines and operationalizes
NASAs approach to deploying system engineering in
agency programs and projects. It is this approach that
is used as a basis for analyzing system engineering
processes in past and current projects.
Methodology
This research into the relationship between project
success and system engineering processes was
conducted in four phases: Phase I was the identification
and assessment of a sample of aerospace flight
projects; Phase II was the collection of data on the use
of system engineering processes; Phase III was to
conduct interviews with project personal to verify data;
and, Phase IV was to assess relationships between
project success and system engineering processes. The
artifacts used to evaluate the relationship between
success and system engineering processes were NASA
space flight hardware projects. Characteristics on the
individual projects selected for study were also
collected. This research was conducted from June,
2006 through December 2007.
A SE&I Skill Board was established to provide
oversight and guidance for the research effort. The
SE&I Skill Board members for this project were all
senior managers at NASAs Marshall Space Fight
Center (MSFC) and included:
Deputy Director, Constellation Systems
Engineering & Integration Office
Deputy Director, Exploration Launch Office
Deputy Manager, Shuttle Propulsion Office
Chief, Systems Engineering Division
Phase I: Identification and Assessment of Sample
Aerospace Flight Projects
The first phase of the research effort included two
activities. First was the identification of an appropriate
method to assess the success of the projects and second
was the .selection of a sample set of aerospace flight
hardware projects.
The modified list of project success assessment
questions looked at the three common measures of
project success, technical performance, schedule
performance, and budget performance. The research
team tried to account for the unique culture and
operating environment at MSFC by asking for success
to be assessed based on both the initial project planning
documents for each project, and success to be assessed
based on the projects performance compared to similar
projects. The project success assessment also looked at
the overall satisfaction with the project management
project and asked for an overall assessment of project
success. The final success assessment questionnaire
totaled eight questions (Exhibit 1).
Exhibit 1. Project Success Assessment Questionnaire
How would you rate the technical success of
this project relative to the initial set of system
requirements?
How would you rate the technical success of
this project relative to other similar projects?
How well did the project stay on schedule in
relation to the original project plan?
How well did the project stay on schedule in
relation to other similar projects?
How well did the project stay within budget in
relation to the original project plan?
How well did the project stay within budget in
relation to other similar projects?
What is your level of satisfaction with the
process by which the project was managed?
What is your overall assessment of the project
success?
The assessment of the projects was done by the
Skill Board members. The members were asked to
only assess those projects with which they had closely
worked; therefore, not all projects were assessed by all
members. The research team elected to use a 5-point
scale (Exhibit 2) for the assessment. A short
discussion was held with the Skill Board members and
they confirmed that they were comfortable with this
scoring system. A comparison between each
individual Skill Board member project scores was done
as a validation for this approach
Exhibit 2. Project Success Assessment Scoring Key
Key
5 Excellent
4 Very Good
3 Good
2 Fair
1 Poor
The second activity was to select a sample of the
aerospace flight projects that had been managed by
MSFC. The selection of projects was limited to this
site to help improve access to project team personnel
and materials.
The projects identified for the research effort included:
Space Shuttle Main Engines (SSME)
X-37 Experimental unpiloted vehicle
Chandra - Orbiting observatory
EXPRESS Rack - provides interfaces for payloads
on the shuttle
2
 Gravity Probe B - is the relativity gyroscope
experiment
International Space Welding Experiment (ISWE)
Gravity Probe B (GP-B)
ECLSS
Shuttle External Tank
SME Advanced Health Monitoring System
AXAF
Saturn Booster
IML
Phase II: Collection of Data on the use of System
Engineering Processes
The second phase of this research included three
activities. First was to identify the system engineering
model that were to be used as the basis for comparison
in the research, second was to identify a scoring
scheme to quantify the level of the processes planned
on each project, and third was to conduct the individual
project assessments.
This research was focused primarily on NASA, so
the systems engineering model selected was the NASA
Exhibit 3. NPR 7123.1A System Engineering Processes
Procedural Requirements (NPR) for systems
engineering - NPR 7123.1A NASA Systems
Engineering Processes and Requirements (Exhibit 3).
This NPR "establishes a core set of common Agency-
level technical processes and requirements needed to
define, develop, realize, and integrate the quality of the
system products created and acquired by or for
NASA." The processes described in the NPR are
intended to "clearly delineate a successful model to
complete comprehensive technical work, reduce
program and technical risk, and improve mission
success." (NASA, 2007).
NPR 7123 identifies 17 common technical
processes fall into three groups: systems design
processes (4 each), product realization processes (5
each), and technical management processes (8 each).
Within the NPR, each technical process is described by
giving its purpose, inputs and sources, outputs and
destinations, typical activities, and a process flow
diagram.
3
 The second task was project scoring. The
presence or absence of these system engineering
processes was scored by reviewing the data archives
for each project. The data reviewed was limited to
approved planning documents. The level of
documentation on individual projects was variable, but
all projects included in this research included at a
minimum the project plan, the systems engineering
management plan, the risk management plan, the
systems engineering requirements document, and
design review planning documents. The research team
did work to keep the data reviewed for all the projects
the same.
The third task, assessment of the data, was done
by reviewing the project documentation and noting the
presence or absence of references to the system
engineering processes identified in NPR 7123. The
research team elected to use a 4-point scale (Exhibit 4)
for the assessment that was modeled on previous
research done at MSFC to assess system engineering
processes in relation to CMMi.
Exhibit 4. Project System Engineering Processes
Assessment Scoring Key
Key
0 None: No direct references located for this
system engineering process
1 Low: Direct reference to this system
engineering process is evident in the document.
The reference is in standard form and could be
viewed as a boiler-plate reference.
2 Medium: Direct reference to this system
engineering process is evident in the
documentation. The reference shows evidence
that the process has been cross-referenced and is
tailored to meet the needs of the project.
3 High: Direct references to this systems
engineering process is evident in multiple
documents. The documents show evidence that
the process has been cross-referenced, is tailored
to the specific project, and there are monitoring
and control procedures identified.
The assessment of project data was done
independently by two research team members.
Variances between the two assessments were then
resolved. A cross-section of the total project team was
asked to participate in the study.
Phase III: Project Team Member Interviews to Verify
Data
The third phase of this research effort was to interview
project team members to verify the data that was
collected and assessed. Missing documents and
difficulties with data interpretation were also
addressed. The project team members included in the
interviews included the project manager, the chief
engineer, and a representative of the functional leads.
The majority of the interviews included two
researchers and one project team member. A limited
number of interviews included multiple team members.
It is should be noted again that this research effort,
assessing the relationships between project success,
and system engineering process, was part of larger
effort that also looked at project success and team
organization. The interviews collected data for both
efforts. For the purposes of this research the data
assessment verification was the primary focus of the
interviews.
Phase IV: Assessing the Relationships between Project
Success and System Engineering Processes.
The final phase of this research was to assess the
relationship between the project success and system
engineering processes. This assessment was done both
qualitatively and quantitatively. The quantitative
assessment was done first and was limited to a
correlation analysis due to the limited size of the data
set. A preliminary review of the data was done to
identify correlations of .5. The qualitative assessment
was then conducted using the correlations found in the
initial analysis as a starting point. The qualitative
reviews looked at specific data to further explore the
relationships that may exist between the unique
characteristics of the individual projects, system
engineering processes along with level of technical,
schedule, budget and overall managerial success.
Results
The research methodology was initially tested on the
International Space Welding Experiment (ISWE). Two
changes were made to the methodology based on this
beta test. First, the interviews with project team
members were initially planned for one researcher and
one team member. Since the interviews on the beta test
took much longer than planned, 60 minutes on average
rather than an initial plan of 30 minutes, two
researchers were used for each interview. This helped
with note taking during the interviews. Second, the
document reviews for each project were found to take
approximately 20 hours for this relatively small
project. To reduce the labor hours required a system
engineering process check sheet was developed. Two
reviewers also looked at each document for
consistency. The document review process was still
the most resource intensive task in the research effort.
These modifications were made before the remaining
projects were reviewed.
The first analysis concerned the data collected
from the SE&I Skill Board members associated with
4
the project success criteria questionnaire. To check for On average, and in each individual project, the
validity in the approach a comparison was made technical success scores were consistently higher than
between the rankings of individual members. The two the success scores for other factors. This observation
key observations of the questionnaire results were was raised during interviews with project team
noted. First was the observation that the rank ordering members. Team members who commented on the data
of the projects by the Skill Board members very noted that the results do support NASAs emphasis on
consistent (Exhibit 5). The only exception was the technical performance over programmatic
ordering of SRB/RSRM and SSME by three of the requirements. Also, the scores for technical, schedule
members. Upon closer examination of the raw data, and budget success compared to other similar projects
there did not seem to be a strong difference between was consistently higher than the scores for technical,
the individuals scores for these projects, so the schedule and budget success in relation to other similar
discrepancy is not considered to be problematic. projects. This observation does support the emphasis
on selecting projects that had successfully flown during
Exhibit 5. Rank Order of Pilot Study Projects this initial phase of the research effort.
The third analysis concerned the comparison of
Rater Rater Rater Rater the project success assessments with the project system
Average Rank engineering processes assessment scoring. A
1 2 3 4
Chandra 1 1 - - correlation was done to determine if there were
relationships between the success metrics and the
SRB/RSRM 3 - 1 1
system engineering processes. The analysis looked for
EXPRESS Rack - 2 - - both positive and negative correlations.
SSME 2 3 2 2 An initial assumption going into the research was
X-37 4 - 3 - that the primary purpose of system engineering was to
GP-B 5 - - - assist in meeting technical requirements for system
development; therefore we would assume that system
engineering processes would have a positive
Second was the observation that the comparison
correlation with the technical success measures for a
between technical, schedule, and budget success
project. This was not always the case. A look at
(Exhibit 6).
positive correlations between project success measures
and system engineering processes (exhibit 7) showed
Exhibit 6. Breakout by Project of Average Response
that product verification, product integration, product
for Criteria
implementation, configuration management, and
technical risk management were correlated with the
Criteria Average per Project Average Average technical success measures. These processes, along
with technical assessment were also correlated with the
Technical success relative to overall project success measure. Technical
4.19
initial requirements requirements definition and technical planning were
4.32
Technical success relative to correlated with relative schedule success. Stakeholder
4.44 expectation definition, technical requirements
similar projects
definition, technical planning, and technical assessment
On schedule relative to
2.21 were all positively correlated with relative budget
original project plan
2.62 success. Of interest is the observation that system
On schedule relative to similar engineering processes that positively correlated with
3.03
projects technical success relative to initial requirements also
On budget relative to original correlated with technical success relative to similar
2.51 projects. However, the system engineering processes
project plan
2.89 that positively correlated with schedule and budget
On budget relative to similar relative to similar projects did not positively correlate
3.26
projects with schedule and budget relative to original project
Satisfaction with project plan. Also of note is that several of the system
3.50
management process engineering processes, such as logical decomposition
and design solution, did not show any positive
Overall project success 4.22
correlation with any success measures.

5
Exhibit 7. Positive Correlations between Project Success Assessments and System Engineering Processes

Project Success and System

1. Stakeholder Expectations Definition

2. Technical Requirements Definition

3. Logical Decomposition

4. Design Solution

5. Product Implementation

6. Product Integration

7. Product Verification

8. Product Validation

9. Product Transition

10. Technical Planning

11. Requirements Management

12. Interface Management

13. Technical Risk Management

14. Configuration Management

15. Technical Data Management

16. Technical Assessment

17. Decision Analysis

Engineering Processes

Observed Data

Technical success relative to initial req.

Technical success relative to similar proj
Schedule relative to original project plan
Schedule relative to similar projects
Budget relative to original project plan
Budget relative to similar projects
Satisfaction with project management
Overall project success

Exhibit 8. Negative Correlations between Project Success Assessments and System Engineering Processes

Project Success and System

1. Stakeholder Expectations Definition

2. Technical Requirements Definition

3. Logical Decomposition

4. Design Solution

5. Product Implementation

6. Product Integration

7. Product Verification

8. Product Validation

9. Product Transition

10. Technical Planning

11. Requirements Management

12. Interface Management

13. Technical Risk Management

14. Configuration Management

15. Technical Data Management

16. Technical Assessment

17. Decision Analysis

Engineering Processes

Observed Data

Technical success relative to initial req.

Technical success relative to similar proj
Schedule relative to original project plan
Schedule relative to similar projects
Budget relative to original project plan
Budget relative to similar projects
Satisfaction with project management
Overall project success

6
Conclusions
This study was designed to explore the
relationship between project success and system
engineering processes and was driven by the
introduction of the new NASA NPR 7123.1 Systems
Engineering Processes and Requirements. If this new
NPR would indeed improve project success then there
should be a relationship between the system
engineering processes identified in the NPR and
common measures of project success.
The research did show some correlations, however
the correlations were not all to project technical
success. Some correlations were found to be with
project schedule and budget success, as well as some
correlations with overall project success. In addition,
some system engineering measures did not show a
positive, but rather a negative correlation with some
success measures implying that while there presence
did not help support project success their absence
would support failure in some success measures.
There were two limitations on this work which
should be noted. The first was the relatively limited
number of projects that were reviewed. The thirteen
projects selected do present a reasonable sample of
NASA projects, but more projects would be needed to
determine if the observations from this work were of
statistical significance. The second was the selection
process for the projects in the study. Since the
objective of this work was to determine if the system
engineering processes noted in NASA NPR 7123.1
were present in successful projects, only relatively
successful projects were selected. If would be of
benefit to see if the processes were present or absent
in relatively unsuccessful projects.
This study has several ramifications for the
engineering manager. First, the success of the project
was found to be influenced by the use of system
engineering processes. The influences of those
processes on success are varied and does not influence
all type of project success the same. Second, not all
system engineering processes have the same influence
on project success, so tailoring of the processes should
be considered based on the individual characteristics of
the project being undertaken.
References
NASA NPR 7123.1A NASA Procedural Requirements,
NASA Systems Engineering Processes and
Requirements, (2007).
NASA Systems Engineering Handbook. SP-610S
(1995).
About the Authors
Paul J. Componation, Ph.D. is an Associate professor
in the Industrial and Systems Engineering and
Engineering Management Department at The
University of Alabama in Huntsville. He holds a
doctorate and bachelor degree in industrial engineering
from West Virginia University, and a masters degree
in management from Troy State University. His
research interests are in the development and
management of complex systems. Dr. Componation is
an active member of the American Society for
Engineering Management (ASEM), the International
Council on Systems Engineering (INCOSE), and a
senior member of the Institute of Industrial Engineers
(IIE).
Alisha D. Youngblood, Ph.D. is an Assistant Professor
in the Industrial and Systems Engineering and
Engineering Management Department at The
University of Alabama in Huntsville. She earned her
B.S., M.S, and Ph.D. in Industrial Engineering from the
University of Arkansas. Her research interests include
engineering management, performance measurement,
economic decision analysis, and logistics. She is an
active member of the American Society for
Engineering Management (ASEM) and the Institute of
Industrial Engineers (IIE) and.
Dawn R. Utley, Ph.D., P.E., is an Associate Professor
in the Industrial and Systems Engineering and
Engineering Management Department at The
University of Alabama in Huntsville. Her research
interests include the engineering management, the
motivation of the technical workforce, organization
structure and team building, and organizational
metrics. She is an active member of the Institute of
Industrial Engineers (IIE) and the American Society for
Engineering Management (ASEM).
Phillip A. Farrington, Ph.D. is an Associate Professor
in the Department of Industrial and Systems
Engineering and Engineering Management at The
University of Alabama in Huntsville. His research
interests are collaborative systems engineering,
integrated product development, quality engineering,
and rapid simulation model development. He is an
active member of American Society of Engineering
Management (ASEM), American Society of Quality
Control (ASQC), and the Institute of Industrial
Engineers (IIE).