FLYING QUALITIES DEMONSTRATION MANEUVERS
David B. Leggett* and Thomas J . Cord**
Wright Laboratory
Wright-Patterson AFB, OH
ABSTRACT
As a result of handling qualities
problems encountered in recent development
programs, and a subsequent review of the
development process, a number of changes to
that process have been recommended. Among
these is a recommendation to add high-gain,
closed-loop tasks as verification requirements
in the military fixed-wing flying qualities
standard, MIL-STD-1797A. This paper
discusses the effort to incorporate such
requirements into MILSTD-1797A. Among
Archive Set 539 downloaded from arc.aiaa.org
the issues discussed are the reasons for
including such requirements, some of the
problems involved with this kind of
requirement, and the kinds of tasks proposed.
The paper also includes a review of some of the
projects Wright Laboratory is pursuing to
develop data for these new requirements.
BACKGROUND
In the spring of 1992, the YF-22
experienced a pilot-induced oscillation (PIO)
which resulted in the loss of the aircraft. In
the summer of 1992, the C-17 also encountered
some P I 0 problems which resulted in
restrictions being placed on that aircraft.
Following these two incidents, the Commander
of Air Force Material Command, General
Yates, chartered a review team to examine the
development process and determine why P I 0
problems continued to occur. The team was
also to identify the "best practices" used in the
development processes of the various programs
that they reviewed.
* Aerospace Engineer, Member AIAA
** Aerospace Engineer
This paper is declared a work of the U.S. Government
and is not subject to copyright protection in the United
States.
The charter of the team was to focus on
the P I 0 problem, assess the current
development process, and document the "best
practices". A multi-agency, multi-disciplinary
team was established with members from Air
Force Material CommancVEngineering
(AFMCIEN), Wright LaboratoryIFlight
Dynamics Directorate (WLfFIG),Aeronautical
Systems CenterIEngineering (ASCEN) (both
engineers and managers were represented),
and Air Force Flight Test Center (AFFTC)
(both engineers and pilots were represented).
The team conducted the review in July and
August of 1992 and prepared the final report in
August and September of 1992 and briefed Gen
Yates in March 1993.
The team reviewed all major USAF
aircraft programs with digital flight controls,
specifically, the F-15E, F-16C/D, F-22, F-111,
C-17, and B-2. The team interviewed
contractor, System Program Office (SPO), and
Combined Test Force (CTF) personnel on these
programs. The team also went to NAS
Patuxent River to interview USN personnel
about the FIA-18 program. The team also
reviewed experimental USAF' and NASA
systems with digital flight control systems: the
X-29, X-31, F-15 STOL and Maneuver
Technology Demonstrator (SIMTD), and the
Variable In-Flight Stability Test Aircraft
(VISTA). Discussions were also held with
other experts in the field of flight control and
flying qualities, including Ralph Smith and
personnel from Calspan.
The review team made a number of
recommendations to change the development
process. Although the focus of the review was
PIO, the recommendations involved the wider
issues of flight control, flying qualities,
simulation, and system testing. Two of these
recommendations involved changes to the
 military standard for flying qualities of fixed-
wing aircraft, MIGSTD-1797. The first of
these recommendations was to incorporate
more quantitative requirements for PIO. The
second was to add explicit verification
requirements in the form of high-gain, closed-
loop (pilot-in-the-loop)evaluation maneuvers to
demonstrate good flying qualities.
Incorporation of high-gain, closed-loop
tasks as requirements in the flying qualities
standard is somewhat controversial. Use of
this kind of requirement has been very limited
in previous flying qualities specifications. One
philosophical reason for this is that the
specification is supposed to place requirements
on the delivered product; a requirement on a
closed-loop task involves another element that
the contractor has no control over, namely, the
pilot. A more compelling technical reason has
been the difficulty of determining compliance
Archive Set 539 downloaded from arc.aiaa.org
with this sort of requirement. Pilot evaluation
of flying qualities is basically a qualitative
assessment. Although pilot rating scales have
been developed to try to quantify this
assessment, the problem of variability in pilot
ratings has remained a drawback to this sort of
requirement. Consequently, the flying
qualities specifications and standards have
placed more emphasis on objective, open-loop
(pilot-out-of-the-loop)requirements.
But these objective, open-loop
requirements also have drawbacks. The
primary drawback is that they are inferential
in nature, that is, meeting these requirements
will probably result in good flying qualities, but
not necessarily. Therefore, evaluation of flying
qualities with high-gain, closed-loop tasks is
still necessary, even if the aircraft meets all of
the flying qualities requirements (which it
never does).
However, the review team found that, in
many development programs, the handling
qualities evaluation consisted mainly of open-
loop tasks . steps, ramps, doublets,
frequency sweeps, etc.) which were used for
parameter identification to compare with the
objective, open-loop requirements of the flying
qualities specification. Quite often there was
very little evaluation with closed-loop tasks.
There were three reasons for this that were
mentioned time and time again: the program
was already behind schedule, the program was
already over cost, and closed-loop tasks were
not required by the specification. Thus, the
recommendation from the review team was to
add closed-loop evaluation tasks to MIL-STD-
1797.
1797 UPDATE
In 1993, another team was established
in order to incorporate the recommend changes
into MIGSTD-1797. This team was known as
the 1797 Update Team and consisted of
members from ASCBN, WWIGC, and
AFFTC. Draft versions of changes to seven
requirements in MIGSTD-1797 were written
and sent out to other government and
contractor organizations to review. Their
comments have been received and are being
incorporated into the Update. The Update will
go through one further Tri-Service Review
before being published as Change Pages to
MIL-STD-1797A later this year.
One of the important changes, and the
subject of this paper, was the inclusion of
several high-gain, closed-loop handling
qualities evaluation maneuvers as
demonstration requirements. These evaluation
maneuvers were incorporated into four
requirements:
5.1.6.1 Allowable Levels for Aircraft Normal
States - verification
5.1.6.2 Flight outside the Service Flight
Envelope - verification
5.1.6.3 Ground operation - verification
5.1.11.6 Pilot-in-the-loop oscillation -
verification
These four requirements are verification
requirements. A verification requirement
states how compliance with a corresponding
system requirement is to be verified. The
corresponding system requirements for 5.1.6.1
 through 5.1.6.3 are Primary requirements.
Primary requirements specify the Level of
flying qualities required under certain
conditions, for example, 5.1.6.1 requires Level
1 flying qualities for Aircraft Normal States in
the Operational Flight Envelope and Level 2
flying qualities for Aircraft Normal States in
the Service Flight Envelope. The system
requirement associated with 5.1.11.6 is a
qualitative requirement that there shall be no
tendency for PIO.
Requirements 5.1.6.1 through 5.1.6.3
now specify that closed-loop tasks will be used
to verify that the aircraft has the appropriate
Level of flying qualities. Requirement
guidance provides the SPO with a list of
possible tasks to use and recommends some
performance objectives for these tasks. The
guidance also discusses the problem of pilot
rating variability and ways to reduce it.
Archive Set 539 downloaded from arc.aiaa.org
Recommended tasks include: capture
tasks, such as pitch captures or bank angle
captures; air-to-air and air-to-ground tracking
tasks, including gross acquisition and fine
tracking; aerial refuelling tasks (both boom and
probe-and-drogue),close formation flying, offset
precision landings, carrier landings, takeoffs,
and catapult takeoffs. Other tasks which are
critical to the mission that the aircraft is
intended to perform may also be used as
evaluation tasks, such as weapons delivery or
Low-Altitude Parachute Extraction (LAPES).
Proof of compliance i n these
demonstration tasks will consist of pilot
comments and Cooper-Harper (C-H) pilot
ratings. Actual task performance is not
recommended for use as proof of compliance
because i t is even more subject to pilot
variability than pilot ratings. The performance
objectives recommended for these tasks are not
intended for use as proof of compliance, but,
rather, for use with the C-H scale. Pilot
comments should be considered the most
important data. A C-H rating is only a
summary of the observed flying qualities
characteristics into a single number. Pilot
comments identify the specific deficiencies, if
any, that must be corrected.
The guidance recommends several ways
in which to reduce pilot variability. First of all,
it is absolutely necessary that more than one
evaluation be made for each test condition.
Studies of inter-pilot C-H rating variability
have indicated that three pilots is the
minimum for a n adequate evaluation.12 More
pilots will increase confidence in the results,
but Reference 4 further demonstrated that the
point of diminishing returns was reached at
about six. Therefore, the recommended
number of pilots per test condition is three to
six. Careful selection of the evaluation pilots
will also reduce the variability in results. For
proof of compliance tests, all of the evaluation
pilots should be test pilots trained in the use of
the C-H scale and they should all be
experienced in the Class of aircraft under
evaluation.
In order to insure that all of the pilots
attempt to achieve the same level of
performance, and thus insure consistency and
reduce the effects of pilot variability, it is
extremely important to explicitly define the
desired and adequate performance objectives to
be used for the C-H ratings. Best results are
achieved with task performance defined in
terms of quantifiable objectives which the pilot
can readily observe himself in real time.
Furthermore, consideration must be given to
defining objectives that can also be recorded on
some medium so that the flight test engineer
can confirm that pilot ratings are reasonably
consistent with task performance.
Another method to reduce the effects of
variability is to use the "long-look technique.
In this technique the pilot repeats the task
until he is confident of his evaluation before he
assigns a C-H rating. The "long-look approach
allows the pilot a more extensive appraisal of
the test condition, it allows him to weed out
the effects of unique events in a single run, it
allows him to get over the learning curve, and
i t allows him to clear his memory of
characteristics he may have observed in
evaluation of a preceding condition.
Requirement 5.1.11.6 requires a closed-
loop task to verify that the aircraft does not
 have P I 0 tendencies. The recommended tasks
for this requirement are similar to those in the
other three requirements except that the
HQDT technique is recommended. The key
element in the HQDT technique is that the
pilot must attempt to totally eliminate any
error in the performance of the task. Adequate
and desired performance are not defined and
Cooper-Harper ratings are not recommended.
The HQDT technique does not allow the pilot
to compensate for aircraft deficiencies, thus it
can be considered a "stress test" of handling
qualities.
There are many handling qualities
evaluation tasks which are well-known and
widely used for research purposes. For the
purpose of specification compliance the most
difficult part will be determining appropriate
performance objectives for these tasks. To
date there has been little in the way of rigorous
Archive Set 539 downloaded from arc.aiaa.org
testing of what kind of task performance
should be expected in these tasks. The
following sections discuss several recent Wright
Laboratory projects to define handling qualities
tasks and performance objectives suitable for
use in MIGSTD-1797 as demonstration
requirements.
STANDARD EVALUATION MANEUVER SET
(STEMS)
The STEMS project was actually begun
prior to the incidents of 1992. The objective
was to develop a set of standard maneuvers to
evaluate agility and flying qualities at high
angles of attack. A maneuver development
process, shown schematically in Figure 1,
involved a group of engineers and pilots with
experience in specifications, design, flight test,
and agility. They developed a set of
preliminary maneuvers which were then
combined into a series of nearly 200 potential
evaluation maneuvers. A qualitative screening
was performed to identify maneuvers which
were flight testable and would relate to both
the design and the operational communities.
This process defined 30 maneuvers for further
consideration.
A manned simulation was conducted to:
1) establish flight techniques and initial
geometries, 2) assure flyability and
repeatability, 3) generate high quality data,
and 4) maintain operational relevance of the
dynamic environment used by the maneuver.
Performance criteria for Cooper-Harper pilot
ratings were also defined. A Design of
Experiments technique was used to develop
maneuvers which were sensitive to design
parameters and relatively insensitive to pilot
variation.
Simulation data and pilot commentary
on the flight techniques and flying qualities of
each maneuver were examined and the
maneuver set was revised. Potential human
factors, such as g-induced loss of consciousness
and disorientation were also taken into
account.
The final simulation entry was
conducted to verify that the maneuvers
developed could identify aircraft characteristics
in addition to those addressed by development
simulations. This included transport-class
models. Limited in-flight validation of the
flyability of some maneuvers was also done by
USAF Test Pilot School students in two
separate class projects.
This process identified 20 preferred
maneuvers, listed in Table 1. These
maneuvers were shown to be repeatable as well
as capable of providing data useful in the
design process. The maneuvers are loosely
categorized as individual maneuvers, maneuver
sequences, or freestyle maneuvers. Individual
maneuvers, such as a full stick pitch pull, are
the most basic elements of a maneuver and
cannot be broken down further. Maneuver
sequences are combinations of individual
maneuvers. For example, a pop-up ground
attack maneuver is classified as a maneuver
sequence because the pilot performs several
tasks: 1) pull to a desired pitch attitude, 2)
climb to a given altitude, 3) roll inverted, 4)
pull to and capture the target, and 5) roll back
to wings level while tracking the target. In
freestyle maneuvers only the start and end
conditions are specified, thus giving the pilot a
great deal of freedom to choose the technique
 to transition from one state to another.
The results of the STEMS program are
documented in References 3 through 5.
Reference 3 discusses the maneuver
development process and the original 200
maneuvers. Reference 4 describes the final 20
maneuvers. Reference 5 presents data from
the simulations.
The results from some of the more
conventional STEMS maneuvers, such as
Offset Approach to Landing, will be added to
the 1797 Update. More unconventional
maneuvers dealing with agility and control at
high angles-of-attack will be validated in flight
test before being incorporated in future
revisions. A partial evaluation of some of these
maneuvers has already been accomplished on
the X-31 and the Multi-Axis Thrust Vectoring
(MAW) F-16 aircraft. Further validation will
Archive Set 539 downloaded from arc.aiaa.org
be continued in the near future with the High-
Angle-of-Attack Research Vehicle (HARV)F-18,
and the Variable In-flight Simulation Test
Aircraft (VISTA) F-16.
Reference 4 is intended to be a "living"
document. It will be revised from time to time
to add new tasks and revise existing ones as
more experience and data becomes available.
CLASS I11 CLOSED-LOOP MANEUVERS
The 4950th Test Wing (at that time
based a t Wright-Patterson AFB, OH)
performed a flight test project in the Spring of
1993to develop evaluation maneuvers for Class
I11 aircraft (large, heavy aircraft). This project
looked a t evaluation maneuvers for two tasks:
landing and aerial refuelling. Each task was
performed with three different levels of desired
performance objectives. The tasks were
evaluated with three different aircraft: a (2-141,
a KC-135A, and a (3-18 (an ex-airline Boeing
707 converted for military use).
The landing evaluation maneuver was
an offset, precision landing task. The offset
was 1 dot on the CDI, corrected at 200 feet
AGL. The performance objectives for adequate
performance were constant throughout the test.
Adequate performance was to touchdown
within +I500 to -500 feet of the aimpoint
longitudinally, and within 40 feet of the
centerline, with a vertical velocity no greater
than 400 feet per minute. Four different levels
of desired performance objectives were tested.
These are shown in Table 2. Additional
objectives for desired performance were no
bounce and no "hard landing".
Of these performance objectives, the one
the test pilots selected as the best was to
touchdown within 200 feet of the aimpoint
longitudinally and within 20 feet of the
centerline.
The aerial refuelling evaluation
maneuver was a boom tracking task. The
performance objective was the cumulative time
the boom nozzle could be kept within a
specified zone during two minutes of tracking.
The adequate and desired zones were defined
as shown in Figure 2. Adequate performance
was defined as keeping the boom nozzle within
the adequate zone for the full two minutes.
Again, four different levels of desired
performance were tested. These performance
objectives are shown in Table 3.
Of the four performance objectives, the
one the test pilots selected as the best was to
keep the boom nozzle in the desired zone for a
cumulative time of 60 seconds.
CLASS IV CLOSED-LOOP MANEUVERS
The USAF Test Pilot School has
conducted a series of projects to test evaluation
maneuvers for Class IV aircraft (fighters and
attack aircraft). The first of these was called
Closed Quality I and was conducted in the
Spring of 1993. Closed Quality I looked at
offset precision landing and air-to-air tracking
tasks for Class IV aircraft. The aircraft they
used to evaluate their tasks were the F-16, F-
15, and FIA-18.
Not surprisingly, the offset precision
landing task defined by Closed Quality I was
very similar to that of the 4950th Test Wing
except that they used an offset of 150 feet
 corrected a t 150 feet AGL. The landing zone
for adequate performance was a 100x1000 foot
box painted on the runway. The three landing
zones tested for desired performance were
75x600 feet, 50x400 feet, and 25x200 feet.
These landing zones are shown in Figure 3 as
they were painted on the runway.
Interestingly, the landing zone that
Closed Quality I chose was the 50x400 foot box.
A landing zone almost identical to that chosen
by the 4950th Test Wing in their project.
The air-to-air tracking task consisted of
three phases: a gross acquisition phase and two
fine tracking phases. The target aircraft
performed an S-turn with two different load
factors in each half of the S-turn. A diagram of
the initial set-up and target maneuver is
shown in Figure 4.
Archive Set 539 downloaded from arc.aiaa.org
Adequate performance for gross
acquisition was to acquire the target aircraft
exhaust nozzles within a 25-mil diameter
reticle within 8 seconds. Two different desired
performance criteria were tested. One was to
acquire the target nozzles within the 25-mil
diameter reticle within 7 seconds, the other
was to acquire the target nozzles within the 25-
mil diameter reticle within 6 seconds.
Adequate performance for fine tracking
was to keep the pipper on the target fuselage
ahead of the horizontal stabilizer continuously
for 0.5 seconds. Two different desired criteria
were tested. One was to keep the pipper on
the target fuselage ahead of the horizontal
stabilizer continuously for 2 seconds. The
other was to keep the pipper on the target
canopy continuously for 2 seconds. This
criteria was different from that frequently used
in handling qualities research in that the
performance was defined displacement was
defined in terms of target aircraft dimension
rather than mil displacement. The reason for
this was to make the criteria useful regardless
of the reticle display used.
The air-to-air tracking task devised in
Closed Quality I proved unsatisfactory for
various reasons. First of all, the same fixed
reticle was used for evaluation of gross
acquisition and fine tracking during the target
maneuver, but the fixed reticle could not be set
a t a depression angle optimized for both. Thus
pendulum effect caused lateral errors in the
gross acquisition that did not reflect what
would be encountered operationally.
Furthermore, the use of time to acquire as the
sole criteria for gross acquisition resulted in
disproportionately adverse ratings of the F-15,
whose roll performance was more sluggish than
that of the other two aircraft. The resulting
Cooper-Harper ratings, driven by time to
acquire, showed a preference for the F-16 over
the F-15. But qualitatively, the pilots
preferred the F-15 over the F-16 because it was
more predictable. There were two problems
with fine tracking: defining the performance
objectives in terms of displacement on the
aircraft proved to be excessively affected by
range to the target, and the 120 deg turns used
by the target during the S-turn proved to be
too short to allow adequate time to get on the
target and track it.
Closed Quality I recommended some
modifications to their proposed task. First, for
gross acquisition, the fixed-reticle should be
positioned so as to minimize pendulum effect.
For fine tracking, the performance objective
should be measured in terms of mil dispersion
from a specific point on the target and the
target maneuver should be altered to allow
more time for tracking.
Closed Quality I was just the first in a
series of projects to generate data on handling
qualities demonstration maneuvers. Closed
Quality 11, the next project in the series, is
continuing the development of the air-to-air
tracking task and also investigating an air-to-
ground tracking task. These projects are
generating valuable data to provide guidance to
users of MIGSTD-1797 on the sort of
demonstration tasks and the kinds of
performance objectives to use in future
specifications.
SUMMARY
The 1797 Update is not meant to be the
 final development of these tasks. It is an
interim change to MIL-STD-1797A that
incorporates the best recommendations and
data that we have available to date. Further
development of tasks and performance
objectives will continue. Future results will be
incorporated into the STEMS document and
will be included in the next major revision of
the flying qualities standard, MIL-STD-1797B,
which should be published in 1997.
REFERENCES
1. Kidd, E.A and Bull, G., "Handling Qualities
Requirements as Influenced by Pilot
Evaluation Time and Sample Size", CAL
Report No. TB-1444-F-1, February 1963.
2. Dukes, Theodor A., "Guidelines for
Designing Flying Qualities Experiments",
Archive Set 539 downloaded from arc.aiaa.org
3. Wilson, D.J., Riley, D.R., and Citurs, K.D.,
"Aircraft Maneuvers for the Evaluation of
Flying Qualities and Agility, Vol 1: Maneuver
Development Process and Initial Maneuver
Set", AFWAL-TR-93-3081, August 1993.
4. Wilson, D.J., Riley, D.R., and Citurs, KID.,
"Aircraft Maneuvers for the Evaluation of
Flying Qualities and Agility, Vol 2: Maneuver
Descriptions and Selection Guide", AFWAL-TR-
93-3082, August 1993.
5. Wilson, D.J., Riley, D.R., and Citurs, K.D.,
"Aircraft Maneuvers for the Evaluation of
Flying Qualities and Agility, Vol3: Simulation
Data", AFWAL-TR-93-3083, August 1993.
6. DeWitt, B.R., "Closed Loop Handling Tasks
Testing and Verification for Class 111-L
Aircraft", 4950-FTR-93-05, October 1993.

NADC-85130-60,l J u n 1985.

T a b l e 1. G e n e r a l C h a r a c t e r i s t i c s of the I n i t i a l STEMS Maneuvers

( F r o m R e f e r e n c e 3)
-
Env.

1. Trackii Durinp High AOA Sweep

2. High AOA Tracking
3. High AOA Lateral Gross Acquisition
4. Dual Attack
5. Rolling Defense
6. Maximum Pith Pull
7. -Up Pitch Angle Capture
8. Crossing Target Acq. and Trackiq
9. Pitch Rate ReSe~e
10. High AOA Longtudinal Gross Acq.
11. Sharkenhausen
12. High AOA Roll Reversal
13. High AOA ROWand Capture
14. Minlmum Speed Full Slick Loop
15. Minimum Time 180' Heading Ch-e
16. 1p Stabilized Pushover
17. J-Turn
18. Tanker Boom Trackicg
19. Trackiq in Power Approach
20. Otrset Approach lo Landing
 T a b l e 2. Desired and Adequate C r i t e r i a Tested i n Landing Task
(From Reference 6)
TOUCHDOWN POINT
PARAMETER DESIRED ADEQUATE
SET (from 1000 ft. marker
/center line point)
1A +/- 200 ft length +1500/- 500 ft length
+/-20 ft width +/- 40 ft width
Max VVI = 200 fprn Max VVI = 400 fprn
1B

I +/- 100 ft length

+/- 10 ft width
M a x VVI = 200 fprn
+/- 400 f t length
+1500/- 500 ft length
+/- 40 ft width
Max VVI = 400 fpm
+1500/- 500 ft length
+/- 30 ft width +/- 40 ft width
Archive Set 539 downloaded from arc.aiaa.org

Max VVI =200 fprn Max VVI = 400 fprn

+/- 500 ft length +1500/- 500 ft length
+I- 40 ft width +/- 40 ft width
Max VVI = 200 fprn Max VVI = 400 fprn

T a b l e 3. Desired and Adequate C r i t e r i a Tested i n A e r i a l

R e f u e l l i n g Task - ( ~ r o mReference 6)

PARAMETER DESIRED TIME ADEQUATE TIME

SET (within desired box) (in adequate box)

30 secs 1 2 mins
45 secs 1 2 mins
60 secs 1 2 mins
75 secs 1 2 mins
 and Documentation

7-
w
Final Evaluation
Maneuver
Archive Set 539 downloaded from arc.aiaa.org

Figure 1. Maneuver Development Process

(From Reference 3)

DESIRED
ZONE

Figure 2. Adequate and Desired Performance Zones for Aerial

Refuelling Task Based on KC-135 Tanker
(From Reference 6)
Archive Set 539 downloaded from arc.aiaa.org

Figure 3 . Landing Task Touchdown Zones

-
13,000 8,000 ft 350 MAS

Figure 4 . Air-to-Air Tracking Task

61