Operational Readiness Review Checklist

Overview and Purpose

The operational readiness review serves two primary purposes. It ensures that the
project is on schedule and that all the minor details required to release software has
been completed or a plan is in place to complete all those items. It also ensures
that all the deliverables associated with the release are properly consolidated and
available as needed.
The operational readiness review should be conducted by the Subcontractor Project
Manager of the key development effort. Below outlines the Process,
Responsibilities, and provide key checklists that should be completed to ensure that
the operational readiness review is complete and ensures the release will be smooth
and the deliverables are all in place.

Process
For every major software development release and at the end of any contract, there
shall be a formal review either in person or on a teleconference. This review should
happen about the time that software is being delivered to the quality assurance
software tier. For non-software projects, this should occur near the release of the
product. In all cases there should be a review near the end of a given contract to
ensure that all the deliverables, software and otherwise, are properly delivered.

Responsibilities
The below table outlines the main responsibilities for this IPR:
Responsibility
Schedule the meeting

Define Format

Build the Presentation

Present the Technical

Material
Provide Feedback
Correlate and document

Comment
COTR needs to provide key
stakeholders availability,
but the PM needs to
schedule the meeting
Initially this checklist can
provide insight into the
format. Long term having a
format template would be
helpful
Using any defined format if
available. Presentation
should be distributed prior
to the meeting.

Person
Subcontractors Program
Manager
COTR

Subcontractors Program
Manager
Subcontractors Program
Manager or Lead
Technical Person
All critical stakeholders
COTR

the feedback
Present consolidated
information to senior
SAIC-F Management

COTR

Attendees
Ensure that the following people are present at the Design Reviews:
(Note: the following table should be filled out as part of the documentation of the
meeting)
Role
Subcontractor Project
Manager
Subcontract Key Personnel
SAIC-F COTR
SAIC-F Functional Manager
SAIC-F Lead Engineer
NCI Program Lane (Govt
Sponsor)
NCI Engineering Lane (PO)
QA and/or Test Leads

NCI Systems Team

Key Identified User
Advocates
Key External Agency
Liaisons
Others

ORR Responsibility
Present all material

Person

Provide backup support as

necessary
Organize the meeting,
ensure key attendees are
present
Ensure there is
consistency in the
processes
Ensure system capabilities
meets needs of the users
Understand schedule and
risks, coordinate with NCI
resources
Ensure product test
requirements understood.
Provide detail on the test
plan
Ensure deployment
process understood
Ensure the system meets
the needs of the users

Checklists
The below lists should be completed. If there are any concerns, comments, or any
further information provided, it should be included here or in a separate attached
document.
Program (Government Sponsor)
Make sure that the following information is covered

Information
The key stakeholders understand their responsibilities
There is an opportunity for User Acceptance Testing and the UA
Testers are identified and know their responsibility
The usage of the product matches the design
The product capabilities match the expectataion and can be traced to
the design and the requirements
All the teams all understand their responsibility
The remaining schedule is consistent with the users/sponsors/key
stakeholders expectations.
Has the subcontractor defined and validated all key technical
assumptions with the stakeholders
The remaining programmatic risks are well understood and plans are
in place to mitigate as required
All external programmatic dependencies have been coordinated

Covered?

Additional Information

Technical
Make sure that the following information is covered
Information
The subcontractor has a good understanding of the CBIIT
environment and that they know how to deploy/patch/update their
tools in this environment
The subcontractor is working with the QA and Test teams
The subcontractor is conforming with CBIITs approach to the Open
Source development Initiative
Has the subcontractor defined all key technical assumptions
Have the appropriate stakeholders validated those assumptions
Has the subcontractor identified all the key technical risks
For each risk is there and understanding of the likelihood, the impact,
and any plans needed to mitigate that risk
Have the appropriate stakeholders seen and commented on the risks
Has the subcontractor enumerated all the technical dependencies
For each dependency what is the status? Is the owner of the issue
aware of the impact to this effort?

Covered?

Additional Information
Security
Make sure that the following information is covered
Information

Covered?

Has the product undergone an initial security application scan

Is there plans to do app scans in the staging environment well before
the release
Is there plans for the production app scan
Does this project need a C&A, if so, is it on schedule
Is there any FISMA or other security requirements? Is the project on
schedule to meet them?
Additional Information
Quality
Make sure that the following information is covered
Information
The subcontractor has a good understanding of the key technical
stakeholders and users of the system
Is there User Acceptance Testing planned to address the needs of the
users?
Is the project using the right CBIIT Tech Stack?
Is the user documentation sufficient?
Is the technical documentation sufficient?
Is there a good test plan that can be traced to the requirements?

Covered?

Additional Information

Assumptions, Risks and Dependencies

There needs to be a shared understanding of all assumptions, risks and
dependencies. It is important to call out this in the meeting and to ensure that the
subcontractor is identifying and communication this information on a regular basis.
The design review is a good gate for checking on this.

Overall Health Assessments

The overall health of a contract is sometimes very difficult to determine objectively.
This is where the skill of the COTR is very important. Frequently, from the material
presented above combined with the monthly status reports and status and other
calls, the COTR will be able to build a subjective picture of the health of a contract.
It is important that those contracts that need additional attention be identified.
Note: one significant indication of the health of the contract is the in-process
deliverables (status reports, designs, software, etc). The COTR is responsible for
ensuring that all the right people review each and every deliverable (This may be
the COTR, but the COTR can reach out to anyone to review these deliverables as

needed). The technical quality of these deliverables shall be evaluated subjectively,

but also, objectively. As an example, design documentation should be evaluated by
other developers especially those people that will use the design. If they cannot
interpret/use the deliverables, then the project needs to correct this as soon as
possible.
Unfortunately, it is often very difficult to quantify into objective measures that
subjective feel that a COTR has over the contract. For projects that need attention,
it is critical that the COTR go that extra level and generate and provide an objective
accounting of the contract. This may involve some additional work including, but
not limited to: Interviewing with the key stakeholders, measuring the progress
directly against the plan, doing a deeper investigation into the risks and likelihoods.
Understanding how the contract may have to change to fit into a changing CBIIT
ecosystem.
By the time of the Operational Readiness Review, the project quality should be well
understood. It is up to the COTR to use his or her best judgment to determine the
level of involvement of SAIC-F, CBIIT, and other external people. The COTR has a lot
of responsibility for the overall success of the project, especially as it nears its
completion.
Overall Health