Construction Management System (CMS) Deliverable Review Process

State of California Department of Transportation Division of Construction

December 21, 2009 Version 1.0

Approvals

Name: Title:

Mark Leja Project Sponsor

Name: Title:

TABLE OF CONTENTS
1 INTRODUCTION ........................................................................................................... 1 1.1 BACKGROUND ..................................................................................................... 1 1.2 DOCUMENT OBJECTIVES ...................................................................................... 1 1.3 RESOLUTION OF CONFLICTS OR INCONSISTENCIES ................................................. 2 2 SUMMARY OF THE REVIEW PROCESS ........................................................................... 3 2.1 PROCESS SCOPE ................................................................................................ 3 2.2 IMPORTANT FEATURES OF THE DELIVERABLE REVIEW PROCESS ............................. 3 2.3 IDENTIFICATION AND NOTIFICATION OF DELIVERABLE REVIEWERS ........................... 4 2.4 DELIVERABLE REVIEW PERIOD ............................................................................. 4 2.5 COMMENTS ON DRAFT DELIVERABLES .................................................................. 5 2.6 FINALIZING DELIVERABLES ................................................................................... 5 2.7 DELIVERABLE ACCEPTANCE CRITERIA ................................................................... 5 2.8 FORMAL DELIVERABLE ACCEPTANCE .................................................................... 6 3 NEXT STEPS .............................................................................................................. 8

APPENDICES
APPENDIX A: APPENDIX B: APPENDIX C: APPENDIX D: RFP CONTRACTOR DELIVERABLE REQUIREMENTS LIST OF DELIVERABLES SUBJECT TO THE DELIVERABLE REVIEW PROCESS DELIVERABLE REVIEW PROCESS FLOW CHART DELIVERABLE COMMENTS MATRIX

CMS Project

Page i

Deliverable Review Process

REVISION HISTORY
Version Number 1.0 2.0 3.0 Date 12/21/09 Description Final Author Jane Zhang

CMS Project

Page ii

Deliverable Review Process

1 INTRODUCTION
1.1 Background

The Department of Transportation (Department), Division of Construction develops, implements, and maintains enterprise construction support Information Technology (IT) systems related to construction engineering and contract administration. Currently the Departments construction contracts are being managed by various systems. The primary enterprise construction system is the Contract Administration System (CAS). CAS is a legacy mainframe computer system using COBOL programming language, and has been used by the Department since 1968. Due to the age and technology constraints of CAS, it has become increasingly difficult to keep pace with legislative changes and significant growth of construction programs. CAS is not able to support the informational needs of the construction programs, and there is an increasing risk of system failure. In addition, the Department does not have a single, reliable system with which to manage all construction project activities. Information is in multiple systems, including small desktop databases. In some cases, the most current data is contained in paper files within the districts. Since CAS does not store all project records, the Resident engineers (REs) at project sites are responsible for manual record keeping and researching volumes of paper when post-construction claims, arbitration or audits occur. Manual handling, ever-increasing paper documents, and files are labor intensive, error prone, and difficult to access, thus causing additional program cost and time with less product quality. To address the business problems, the Division of Construction has begun an effort to replace its current legacy CAS with a new Construction Management System (CMS). The proposed CMS solution is a Web-based centralized construction management application that will be accessible by all Department construction staff, contactor staff, and internal and external stakeholders. The total one-time project cost as planned over an 8-year period is approximately $20.5 million with full implementation expected to be completed in September 2014. The CMS Feasibility Study Report (FSR) was approved by the Department of Finance (now The Office of the State Chief Information Officer (OCIO)) on January 10, 2006. The first CMS Special Project Report (SPR) was approved on April 1, 2008 by the OCIO, and the latest CMS SPR is currently in the review and approval process. 1.2 Document Objectives

This document presents the Deliverable Review Process for the CMS project. It specifically documents the process that will be used to submit, track, review, update, finalize, and approve deliverable documents provided by the CMS prime contractor (contractor).

CMS Project

Page 1

Deliverable Review Process

1.3

Resolution of Conflicts or Inconsistencies

The CMS Request for Proposal (RFP) and the subsequent contract contain contractor deliverable requirements for the contractor to submit deliverables. The process described in this document is meant to supplement and/or provide further detail on the requirements provided in the CMS RFP (RFP DOT 2660-415) and any subsequent contracts, amendments, or agreements with the contractor. Should any inconsistencies or conflicts exist between this document and either the RFP or contract, amendments, or agreements, the RFP or contract, amendments, or agreements will take precedence. Refer to Appendix A for the Contractor Deliverable Requirements excerpted from the RFP.

CMS Project

Page 2

Deliverable Review Process

2 SUMMARY OF THE REVIEW PROCESS

The CMS project deliverable submittal and review process is provided in the following subsections. 2.1 Process Scope

This process pertains to the submittal of contractor deliverables which are documents, rather than automated tools or project software. A preliminary list of the deliverables subject to this process is identified in Appendix B. It is important to note that the list is provided for informational purposes only and may be revised based on changes in the scope, RFP, contract, or other aspects of the CMS project. It is also important to note that this document provides general guidelines that will be followed for the submittal and approval of CMS project deliverables. Refer to the RFP for detailed and specific requirements pertaining to the content, format, and other requirements relative to the CMS project deliverables. An excerpt of these requirements is provided in Appendix A for reference purposes. 2.2 Important Features of the Deliverable Review Process

The deliverable review process is presented in the flow charts in Appendix C of this document. The flow charts illustrate the specific steps in the deliverable review process and are followed by a matrix that provides a narrative description of each step. In summary, the process features: Completion of a Deliverable Expectation Document (DED) for each deliverable to help ensure that the State and the contractor possess a mutual understanding of the scope, objectives, content, and format of each deliverable. The contractor must complete the DED prior to commencing work on the deliverable. A template for the DED is provided as an attachment (in a separate file, rather than an Appendix) to this document. Contractor walkthroughs of each deliverable (if requested by the CMS Project Manager) to clarify content, answer questions, or to familiarize the project team with the deliverable. Formal review and written comments on deliverables by key project stakeholders, with the use of a standardized matrix to document comments. Refer to Appendix D for a template of this matrix.

CMS Project

Page 3

Deliverable Review Process

Contractor submittal of draft and final versions of each deliverable to accommodate appropriate revisions, corrections, or other changes requested by the Department. Formal written acceptance of project deliverables by the CMS Project Manager, IT Technical Program Manager, and the CMS Project Sponsor. Archiving of all project deliverables in the CMS project library. Identification and Notification of Deliverable Reviewers

2.3

After the CMS Project Manager approves the DED, the CMS Project Manager will identify the individuals who will review and comment on the deliverable. These individuals may vary depending on the scope and subject matter of each deliverable. They may specifically include: CMS project team members, Executives, managers, and staff within the Division of Construction and the Division of IT, The CMS project Independent Verification and Validation (IV&V), and; Other project stakeholders

The CMS Project Manager will develop, maintain, and publish a schedule of contractor project deliverables. The schedule will be disseminated to all potential deliverable reviewers to inform them of the timing of all potential deliverables. Once the DED for a deliverable is complete, the CMS Project Manager will identify the individuals who will review and comment on the draft and final deliverables. The CMS Project Manager will notify the reviewers of the key milestones in the review process for the deliverable, including the date that the draft deliverable is scheduled for delivery, due date for comments, and the due date for a final deliverable. The CMS Project Manager will also disseminate a copy of the DED to the deliverable reviewers so that they are aware of the scope, objectives, and content of the deliverable, and are prepared to comment on it once it is complete. 2.4 Deliverable Review Period

While reviewing the DED, the CMS Project Manager will determine the amount of time required to review and comment on the draft and final deliverables. The baseline time period for review and comment will be:

CMS Project

Page 4

Deliverable Review Process

Not less than ten (10) State business days for review and comment on draft deliverables; Not less than five (5) State business days for review and comment on final deliverables; and, The State will have up to 30 business days to review a draft or a final deliverable

The actual review and comment period specific for each deliverable will be included in the project schedule. The project schedule timeframes should be used by the contractor to establish the completion dates for draft and final deliverables. These dates should be included in the DED. During the review of the DED, the CMS Project Manager will validate the timeframes for review and determine whether any changes to the due date of the deliverable or the review and comment period are necessary. The contractor must make changes to the date/period that the CMS Project Manager determines. The ultimate goal, however, will be to change the dates preceding the due date of the final deliverable, but not the final deliverable due date in the project schedule. 2.5 Comments on Draft Deliverables

Deliverable reviewers will be responsible for documenting their requested changes in a matrix presented in Appendix D. The CMS Project Manager will compile and review comments to draft deliverables prior to forwarding them to the contractor for incorporation into the final draft. During this review process, the CMS Project Manager may consolidate, revise, or eliminate comments as necessary. The CMS Project Manager will be responsible for communicating any of these changes to the reviewer who initially made the comment. 2.6 Finalizing Deliverables

The contractor will be responsible for updating the draft deliverable and producing the final deliverable. The contractor will clearly identify changes to the draft document when the final document is submitted (this may be accomplished by tracking changes in the draft document, although other methods may be appropriate and accepted). In addition, the contractor must use the matrix in Appendix D to document how the comment was addressed in the final version of the deliverable. 2.7 Deliverable Acceptance Criteria

The RFP identifies general acceptance criteria for deliverables. Refer to Appendix A for a description of these criteria. These criteria, in addition to any other agreed upon criteria, should be documented by the contractor in the draft version of the DED.

CMS Project

Page 5

Deliverable Review Process

In addition to the criteria in the RFP, the CMS Project Manager may identify additional criteria specific to each individual deliverable. The CMS Project Manager may direct the contractor to add these criteria to the final version of the DED. In addition to the acceptance criteria in the RFP, examples of acceptance criteria may include: Does the deliverable address all components required by the contract and any areas agreed upon subsequently through meetings and planning sessions (these agreements should be reflected in the DED)? Is the deliverable consistent with the content and format agreed upon in the DED? Does the deliverable meet applicable industry and State of California standards? Does the deliverable meet the project and the Departments business and technical needs? Is the deliverable easy to understand and effectively formatted?

2.8

Formal Deliverable Acceptance

Deliverables will not be considered approved by the Department until they have been accepted by the CMS Project Manager. Acceptance will be documented in writing on a notice of deliverable acceptance. The CMS Project Manager will send the contractor a copy of this form as a written notice of deliverable acceptance. Unless otherwise stated in writing by the CMS Project Manager, the following individuals have the authority to formally accept a deliverable: CMS Project Sponsor IT Program Manager CMS Project Manager

The deliverable will not be considered accepted unless the notice of deliverable acceptance has been signed by each of these individuals. Payment for contractor deliverables will not be made unless the deliverable is formally approved. It is important to distinguish between the roles of deliverable reviewers and deliverable approvers as shown in the flowcharts in Appendix C. Deliverable reviewers are project team members, subject matter experts, stakeholders, and managers and staff within the Division of Construction or the Division of IT who will be asked to review and comment on deliverables for quality control purposes. Their feedback may or may not

CMS Project

Page 6

Deliverable Review Process

be forwarded to the contractor for incorporation into the final version of a contractor deliverable, subject to the discretion of the deliverable approvers. Deliverable approvers have the ultimate authority to approve and formally accept a deliverable. In this capacity, they must sign the notice of deliverable acceptance as indicated above. They may also provide comments on contractor deliverables as part of the deliverable review process.

CMS Project

Page 7

Deliverable Review Process

3 NEXT STEPS
This process will be implemented once the contractor has been selected and commences work. As a first step to implementing this process, the CMS Project Manager will work with the contractors project manager and the project team to ensure that the milestone due dates for all deliverables including DEDs, draft and final versions, and the review and comment period are clearly identified in the project schedule. After project schedule is complete, the CMS Project Manager will develop the list of contractor deliverables and their due dates, and disseminate the list to key stakeholders and the potential deliverable reviewers. The process described in this document should be implemented when the contractor begins developing the deliverables.

CMS Project

Page 8

Deliverable Review Process

APPENDIX A RFP CONTRACTOR DELIVERABLE REQUIREMENTS

CMS PROJECT Contactor Deliverable Requirements

The contractor deliverable requirements below are extracted from RFP DOT 2660-415, as of October 23, 2009. It is important to note that all contractor deliverable requirements have been included in this Appendix for reference purposes only. The contractor must adhere to the requirements applicable to the submittal of document deliverables as part of the process included in this document.
Development and Implementation Deliverable Minimum Content Requirements

Reqmt Deliverable Title # CD-1a The Project Management Plan must adhere to the Office of the State Chief Information

CD-1b

Officer California Project Management Methodology described in the Statewide Information Management Manual, Section 17 and the Department of Transportation Information Technology Project Management Offices standards and requirements. The Project Management Plan must include: 1- Scope Management Plan, including how scope issues will be resolved 2- Human Resources Plan, including: a) Organizational charts b) Depiction and description of authority and reporting structure c) Staff roles and responsibilities d) Approach to managing contractor and subcontractor staff e) Coordination of staff at multiple, remote locations including meeting and work status coordination f) How resource shortages will be resolved g) Contractors management structure and its methods for oversight and executive direction for the contract 3- Communications Plan that describes how the Contractor team will interact and communicate with State personnel including the Project Manager, project team, stakeholders, and internal and external department oversight entities 4- Schedule Management Plan that discusses how schedule delays will be resolved 5- Quality Management Plan 6- Risk Management Plan 7- Issue Management Plan 8- Conflict Resolution Plan that includes an escalation process for resolving issues and ensuring adequate executive direction and oversight 9- Document Management Plan including a) Method and location for document storage b) Standards for document format and content c) Approach to preparing documents and deliverables d) Approach to assigning and controlling document versions

CD-1b (contd)

CMS Project

Page A-1

Deliverable Review Process

Reqmt #

Deliverable Title
10-Product Development Lifecycle Plan (PDLC), describing the configuration / development approach (e.g., RUP, waterfall), including a) System Design Phase 1-Definition and review of system, software and hardware detailed requirements 2-Implementation of a Change Control Process 3-Development of physical, logical, interface and server diagrams 4-Development of a data dictionary 5-Business process mapping to technical requirements b) System Development Phase 1-Site preparation activities 2-Data conversion activities 3-Coding / configuration activities 4-Interface development activities 5-Report development activities c) System Testing Phase 1-Code and unit testing 2-Integration testing 3-System integration testing 4-Performance and load / stress testing 5-Field testing 6- Regression testing 7-User acceptance testing

CD-1b (contd)

d) System Training Phase 1-Administrator training 2-Train-the-Trainer training 3-Project staff training e) System Implementation Phase 1-System roll-out 2-Adminsitrative documentation, including disaster recovery procedures f) Production Support and Transition Lifecycle 1-Help desk services 2-Problem fixes 3-Change / enhancements process 4-Backups and disaster recovery

CMS Project

Page A-2

Deliverable Review Process

Reqmt #
5-Knowledge transfer 6-Issue resolution

Deliverable Title

g) Contract Closeout Lifecycle 1-Engagement completion 2-Documented outstanding contractual deficiencies

CD-2a CD-2b CD-2c

CD-2d CD-3 CD-4

CD-5

The Project Schedule must conform to the Departments high-level phase Work Breakdown Structure: Initiation, Planning, Execution, and Closeout. The Project Schedule must include the contractors proposed Project Development Lifecycle (PDLC) which is detailed in the Execution Phase. The Project Schedule must include: 1- Work Breakdown Structure 2- Key Milestones 3- Deliverables 4- Detailed Tasks 5- Task Dependencies 6- Durations for each Task 7- Planned and Actual Start and End Dates 8- Resources for each Task 9- Task Ownership, including Contract and Caltrans Tasks 10- Percent of Project Complete 11 Critical Path The Project Schedule shall contain tasks that are, at a minimum, defined such that the lowest level tasks are no greater than twenty (20) calendar days in duration. The Contractor shall use the Caltrans Weekly PMO Status Report template for creation of the Biweekly Status Report. The template is available from the Bidders Library. The Steering Committee Status Reports / Presentations must specify: 1- Current progress against milestones 2- Coordination needs, such as participation from the districts and / or other divisions 3- Potential risks or opportunities and mitigation / contingency plans 4- Pros / Cons and analyses of key issues These status reports and presentations will be delivered when requested by the State. The System Development / Configuration Plan must specify: 1- Description of the Software Development Methodology and Processes, including Development / configuration standards and tools to be used for: --Information modeling --Networking and transport protocols --Security and encryption --Directory access --Web services protocol stack Programming languages to be used (if applicable) Development / configuration, test and support tools to be used Approach to using and configuring third-party tools and software Approach to reusable software (if applicable)

CMS Project

Page A-3

Deliverable Review Process

Reqmt #

Deliverable Title
Approach to validating and refining the functional requirements Approach to managing requirements and traceability Design, code and testing standards Approach to design, code and test walkthroughs / reviews Approach to development and testing of CMS interfaces Approach to designing and validating system security throughout the Software Development Life Cycle (SDLC)

2- Approach to Testing and QA for the CMS Approach to product and process reviews Approach to deliverable quality reviews Approach to testing and test management Approach to integration and testing of interfaces and third-party software Approach to managing defects and anomalies, including how the fixes will be incorporated back into the main code line

CD-5 (contd)

3- Approach to Development / Configuration and Deployment of the Phased Releases Approach to configuration management of the system environments and software (Configuration Management Plan) Change Control Management Plan Approach to version and release management How the system will be audited and turned over to the State upon final system acceptance Approach to delivering and managing the phased releases Approach and sequencing of the releases (i.e., what order, how overlapping releases are addressed, how coordinated with users and stakeholders) Approach to verifying readiness to proceed to the next release Approach to training for each release Approach to deploying each release Approach to field test for Function 3 (daily diaries and mobile devices) Risks to the phased release deployment based on the proposed solution and approach

CD-6a

The System Requirements Specification must contain: Summary Of The Results Of The Requirements Analysis And Validation Baseline Requirements Set For The System, including any additional derived requirements and expanded definitions of each base requirement in order to demonstrate Contractor understanding Listing Of New Business Processes Required And / Or Impacts To State Policies An Updated Requirements Traceability Matrix, showing traceability from the RFP requirements to the detailed requirements which refine and further elaborate the requirement and associated business rules.

CD-6b

The functional requirements in Exhibit VI-A, Mandatory Functional and Technical Requirements, and Exhibit VI-B, Mandatory Optional and Desirable Optional Requirements

CMS Project

Page A-4

Deliverable Review Process

Reqmt #

Deliverable Title
(if the State exercises any of the M/O or Desirable options) shall serve as the baseline requirements that the Contractor shall use as the basis for validating and refining the requirements after contract execution. Modifications to the baseline requirement set are to be expected and will be processed through the System Change Control Process and approved by the State. The System Fit / Gap Analysis must specify: Gaps between the Contractors proposed solution and the CMS functional and technical requirements Strategies to address the gaps between the solution and the CMS requirements (e.g., configuration, customization, additional products) Impacts and risks of the proposed strategies Identification of State responsibilities related to addressing the gaps (e.g., policy changes) The High-Level Design must contain: Description of proposed hardware, for each environment including storage Description of proposed software and the purpose of each item Physical Architecture, including --Location of the environments (if hosted solution) --Description of the facilities (if hosted solution) --System architecture for each environment (i.e., development, test, training and production environments) --Proposed network design, including firewalls --Physical and logical system diagrams Physical and logical integration of the products (technical view) including --Approach to software integration, as applicable --Logical integration diagram or technical diagram showing proposed software integration --List of all required software integration points, including which software, versions, etc. will be integrated and for what purpose --Technical description of each software integration point, including discussing how the integration is proposed to be done --Description of data groups that will be shared through integration points --Approach to integration with mobile devices Application architecture (functional / user view) --Approach to navigation and user interface --Approach to reports and ad hoc queries --Approach to forms and correspondence Description of proposed system security features --User account management for internal and external users (e.g., types of roles, managing roles and access to contracts)

CD-7

CD-8

CMS Project

Page A-5

Deliverable Review Process

Reqmt # CD-8 (contd)

Approach to interfaces

Deliverable Title
--Security for transmission and uploading of files

Approach to capacity management --Performance modeling calculations and assumptions --Calculations used to size the CMS hardware and solution --Approach to addressing growth during and after the contract period --Approach to estimating, verifying and managing system performance --Archive strategy for management of data

Approach to meeting system performance and system availability requirements

CD-9

CD-10

The Detailed Design Document and Database Schema must contain: Capabilities, functions and features of the system Description of the overall design Description of software modules, interfaces, forms, correspondence, and reports Software design using narrative, pseudo code, tables, flow charts, data flow diagrams, screen formats / layout, etc. to describe or depict the inputs, outputs, data flows, control flows, manual and automated processing, and exception and error handling (as applicable to the type of solution) Description and design of the system workflow processes, including decision points and approvals, sequential / parallel routing, exceptions, and relationship to new / modified business processes Description of the systems role-based user access / authority features and tools Report layout for standard reports Form and correspondence layouts Data model and relationships Database(s) design and schema System interface design Specific formulas and calculations used to establish the system storage and processing needs (capacity and performance modeling) Description of the systems security features for protecting the system, data transmissions, and data from unauthorized access, including, as appropriate, encryption, auditing and logging An updated requirements traceability matrix, showing traceability to the specific areas of the design which describe how the requirement will be implemented The Interface Specification Document must specify: Discussion of each interfaces and its purpose Transmission methods and tools The approach to implementing each of the interfaces, including --Integrating with real-time systems (e.g., E-FIS) --Interfacing with stand-alone DBs / batch systems For each interface, a description of: --Interface method and protocol --Frequency of transmission

CMS Project

Page A-6

Deliverable Review Process

Reqmt #

Deliverable Title
--Estimated data volume and growth patterns --Interface format, including any control and error formats --Error handling and recovery methods, including restart procedures --Dependencies or constraints for the interface (e.g., timing of the transmission) --Interface security considerations --Interface testing considerations and approach --Operational considerations, such as scheduling, log file monitoring and operator notifications The Detailed Data Extract / Mappings (for loading Caltrans-converted data) must specify: Database Schema for the new databases Data mapping (legacy data elements to new data elements) Input and output record layouts Location of source and target data File / table sizing information File / table volume information The Test Plan for each testing phase (i.e., unit, integration, system, performance (including load / stress testing), field, regression, and user acceptance) must contain: Overall test strategy Description of each test phase Test objective and test strategy Approach to creating and maintaining the test data and test scripts High-level test design and test cases Approach to testing interfaces, as applicable Criteria for exiting the test phase Test documentation to be produced before, during and at the close of the testing activities Approach to managing, controlling, coordinating and communicating changes to the test environment and code during the test phase Test suspension activities and test resumption activities Approach to analyzing and resolving test anomalies and errors Roles and responsibilities of Contractor and State staff for the testing process The Test Scripts / Procedures for each test must contain: Specific test objectives, including requirements to be verified Specific test preparation activities, including setup of the test data and test environment Specific test activities with expected results and actions to be taken Specific analysis of the test results to verify correctness Specific criteria used to determine if the test has passed or failed The Contractor shall provide all test data for the unit, integration, system, performance, and regression test phases. The State will provide the test data for the field and acceptance tests. The Test Results Report for each testing phase must contain: A summary of all tests executed and the test results A detailed summary of all anomalies and defects found during testing and their resolution or status A list of the system configuration prior to the start of testing and at the end of testing (e.g., versions of software modules, system settings, etc.)

CD-11

CD-12

CD-13

Cd-14 CD-15

CMS Project

Page A-7

Deliverable Review Process

Reqmt #

Deliverable Title
An updated requirements traceability matrix showing traceability to the specific test(s) which prove satisfaction of the requirement. A substantiated recommendation for either resolution of problems (and further testing) or progression to the next phase The Training Plan must specify: Approach to train-the-trainer training to State trainers Approach to assessing user competencies and evaluations Approach to addressing refresher and remedial training Strategy for delivering training to Caltrans internal users and the CMS external users (i.e., contractors) that will be used by the Caltrans trainers Specific course curriculums and descriptions Approach to Computer-Based Training (CBT) for internal and external users (if CBT is optioned) Proposed training schedule Roles and responsibilities of Contractor and State staff for the training process The Contractor must provide all training materials for the Train-the-Trainers training, UAT training, and the System Administration / Technical training. The Train-the-Trainer Training Materials must include: Specific goals of the training class Methods for training end users Description of system features and functions, including workflows Description of new / modified business processes / policies Exercises to help reinforce concepts for end users Training Instructor Manuals for End User Training, for each release must include: Specific goals of the training class Topics and features to be covered in the course(s) How to setup and use the training environment for the course(s) Description, instructions, and answers for the class exercises Key concepts that must be conveyed Tips and tricks for using the system The Internal and External End User Training Materials must include: Course outlines for each end user training session Course content for each end user training session, including as appropriate: Description of each system function / module / feature Description of the data elements and their range of values and business rules Description of system workflows and Day-in-the-Life scenarios Hands-on and workbook exercises to reinforce system concepts Tips and key references for navigating the system Use of standard and ad-hoc reporting tools and features Creating queries and reports Specialized report writing training for advanced users Student materials used during the training course(s) Worksheets or exercises that can be used to evaluate training effectiveness and student proficiency

CD16a

CD16b CD-17

CD-18

CD19a

CMS Project

Page A-8

Deliverable Review Process

Reqmt #

Deliverable Title
Data to support the training exercises Note: Caltrans will be responsible for making copies of the materials for each end user training session. The External End User Training Materials shall be designed as self-paced training materials. If the State elects to purchase the mandatory optional CBT component, this requirement will be deleted in favor of the CBT for External Users. The UAT Training shall consist of the following topics: 1- Overview of the UAT process and purpose (i.e., how UAT will be conducted, roles and responsibilities); 2- Specific UAT management processes, including test preparation, test execution, recording and analyzing individual test results, test closeout, reporting bugs and anomalies, and test status reporting; 3- System basics, including logging in, navigation, workflow, and reporting; and 4- Overview of business processes that are applicable to the Function(s) being tested (e.g., for Function 1, how to setup a contract, how to update the contact list, how to generate a payment estimate). The Knowledge Transfer Training Plan (for technical training) must include: Approach to direct delivery training to Caltrans super-users and technical staff (e.g., job shadowing, classes, etc.) Training delivery methods Specific course curriculums and descriptions Strategy for knowledge transfer to technical staff Proposed knowledge transfer methods Proposed activities where State staff would participate Proposed technical training and knowledge transfer schedule Roles and responsibilities of Contractor and State staff for the training process The Training Materials for System Administrator / Technical Training must describe: Course outlines for each technical training session Course content for each technical training session, including as appropriate: Description of each system environment / component Hands-on exercises to reinforce system concepts Tips and key references for operating, maintaining and troubleshooting the system Use of appropriate administration, monitoring and reporting tools and features Security considerations Student materials used during the training course(s) Worksheets or exercises that can be used to evaluate training effectiveness and student proficiency Data to support the training exercises This plan may not be applicable to a hosted solution, depending on the amount of configuration vs. out-of-the-box functionality. The State shall have the final decision as to whether the plan is applicable. The User Manual(s) must describe: Features and functionality of the system System workflows, including mandatory data elements, expected actions for each workflow step, range of values, specific validations and business rules at each workflow step and tips to assist the users

CD19b CD-20

CD-21

CD-22

CD-23

CMS Project

Page A-9

Deliverable Review Process

Reqmt #

Deliverable Title
Procedures for creating ad hoc reports and correspondence Supervisory controls and special features only available to managers and supervisors, including reassignment of their staffs work, handling exceptions, and overriding workflow steps Description of codes values Description of online help features Expanded description of error messages, as appropriate The System Administration Manual(s) must describe specific procedures for: Management of internal and external user accounts and security Error and exception diagnosis and handling, including checklists Changing system constants and parameters (e.g., rates, deadlines, etc.) Reviewing and managing system and database logs for general activities, problems and trends Adding and modifying data elements and user-defined fields Creating and modifying standard reports, correspondence / templates and forms Creating and modifying system workflows Creating and modifying business rules and field validations Archival storage and retrieval Monitoring and managing interfaces and data transmissions Updating online help contents Activities for system monitoring and administration Routine periodic maintenance to be performed The Deployment and Transition Plan must specify: Deployment strategy (i.e., preparation, who, when, how, verification) Proposed criteria for Go / No-Go decision to commence system deployment Approach to loading converted data Proposed detailed deployment schedule Approach to managing the deployment for each system release, including Process for coordination and timing of training Process for establishing / expanding the help desk Process for tracking and resolving anomalies and errors during deployment Process for communicating with users during the deployment Criteria used to ensure the CMS release is functioning correctly Proposed decision points for shutting down the legacy system(s), as applicable Roles and responsibilities of Contractor and State staff during the deployment period Approach to transitioning operational responsibilities to State staff upon system acceptance Specific risks associated with the proposed deployment approach and specific mitigation / contingency plans for the identified risks The System Configuration Report must contain: Description of the hardware and network configurations for each environment The current versions of all COTS and custom software modules, including the date and time stamp of the version Any hardware settings or configurations made to the system production environment The database schema and database configuration information including table descriptions

CD-24

CD-25

CD26a

CMS Project

Page A-10

Deliverable Review Process

Reqmt #

Deliverable Title
and sizing Any supporting files that must be in place for correct system operations (e.g., environmental files, help files) Note: The purpose of this report is to ensure the State has sufficient information to rebuild and restore the system in the event of catastrophic failure. The level of detail provided in this report may vary depending on the type of solution proposed by the Contractor. The Final System Configuration Report must describe the final state of the production environment when the system is turned over to the State at the end of the CMS Production Acceptance Period. Any and all custom-developed software code created in support of the CMS must be delivered to the State. This code includes all items necessary to build and deploy the application (i.e., modify, enhance, compile and deploy), including but not limited to: Source code and code libraries, including appropriate comments and source code headers Software utilities, build scripts, or tools developed Configuration and environmental files (e.g., .DLL, .INI, etc.) Reports / queries, correspondence / templates, and style sheets The Contractor must provide documentation for all custom-developed software code including any Contractor-developed pre-defined queries and reports. Description of the security and user access / authority features Purpose, capabilities, functions, and features of the code Description of any dependencies or relationships with other modules, code or interfaces Charts and accompanying descriptions depicting inputs, outputs, data flows, control flows, and manual and automated processes All software development and testing tools must be off-the-shelf. Open source and freeware is acceptable for development and testing tools only. The Contractor must provide all of the software tools necessary to develop, build, test, and deploy the CMS, including automated testing tools. The Contractor must provide the off-the-shelf development / configuration tools necessary to design, develop, and implement the CMS. The Contractor must deliver at least one (1) copy of any and all development / configuration, testing, training, configuration, system monitoring, and maintenance / operations tools used in support of CMS with licenses to support five (5) concurrent users for each tool provided. The Contractor must provide an off-the-shelf automated testing tool(s) for CMS to perform regression, performance and load / stress testing. The Contractor must provide off-the-shelf system monitoring tools for CMS. The Contractor must provide COTS tools necessary for the maintenance and operation of the system. If third-party COTS software is being proposed as part of the production systems (i.e., if the software is necessary for the operation of the production system), the Contractor must provide software licenses to support 2,300 named internal users and 1,700 named external users.

CD26b CD28a

CD28b Cd-28c The software documentation for any custom-developed software must include:

CD29a CD29b CD29c CD29d CD-30 CD-31 CD-32 CD-33

CMS Project

Page A-11

Deliverable Review Process

APPENDIX B LIST OF DELIVERABLES SUBJECT TO THE DELIVERABLE REVIEW PROCESS

CMS PROJECT List of Contractor Deliverables Subject to the Deliverable Review Process
This process contained in this document applies to the submittal and review of document deliverables by the contractor. A list of document deliverables, extracted from RFP DOT 2660-415, as of October 23, 2009, is presented below.
Contract Deliverable List and Due Dates

Deliverable Deliverable Title # CD-1 Project Management Plan,

including Project Schedule

Update Frequency
At Least Quarterly

Due Date
Thirty (30) State Business Days after Contract Execution Thirty (30) State Business Days after Contract Execution Sixty (60) State Business Days after Contract Execution Six (6) months after Contract Execution

Associated Services
A.4.2

CD-2

Updates to the Project Schedule / Work Plan System Development / Configuration Plan System Requirements Specification

Every two (2) weeks As needed

A.4.2

CD-5

A.4.2

CD-6

As needed, based on approved change requests No updates Every release

A.4.3

CD-7 CD-8

System Fit / Gap Analysis High-Level Design

Eight (8) months after Contract Execution In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule

A.4.3 A.4.3

CD-9

Detailed Design Document with Database Schema Interface Specifications Document

Every release

A.4.3

CD-10

As needed, based on approved change requests As needed

A.4.3

CD-11

Detailed Data Extract Mappings (for loading Caltrans-converted data)

A.4.4

CMS Project

Page B-1

Deliverable Review Process

Deliverable Deliverable Title # CD-12 Test Plan for each phase of

testing: a) Unit (if custom code is required) b) Integration (if custom code is required) c) System d) Field e) Performance f) Regression g) User Acceptance

Update Frequency
Every release

Due Date
In accordance with the State-approved Project Schedule

Associated Services
A.4.5

CD-13

Test Scripts / Procedures for each phase of testing: a) Unit (if custom code is required) b) Integration (if custom code is required) c) System d) Field e) Performance f) Regression g) User Acceptance

Every release

In accordance with the State-approved Project Schedule

A.4.5

CD-14

Test Data for each phase of testing: a) Unit (if custom code is required) b) Integration (if custom code is required) c) System d) Field e) Performance

Every release

In accordance with the State-approved Project Schedule

A.4.5

CMS Project

Page B-2

Deliverable Review Process

Deliverable #

Deliverable Title
f) Regression g) User Acceptance

Update Frequency

Due Date

Associated Services

CD-15

Test Results Report for each phase of testing: a) Unit (if custom code is required) b) Integration (if custom code is required) c) System d) Field e) Performance f) Regression g) User Acceptance

Every release

In accordance with the State-approved Project Schedule

A.4.5

CD-16

Training Plan

Every release

In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule

A.4.6

CD-17

Train-the-Trainer Training Materials Training Instructor Manuals for End User Training Training Materials for Internal and External End Users Training Materials for UAT Training Knowledge Transfer Training Plan (for technical training) Training Materials for System Administrator / Technical Training

Every release

A.4.6.

CD-18

Every release

A.4.6.

CD-19

Every release

A.4.6.

CD-20

Every release

A.4.6.

CD-21

As needed

A.4.6.

CD-22

Every release

A.4.6.

CMS Project

Page B-3

Deliverable Review Process

Deliverable Deliverable Title # CD-23 User Manual(s)

Update Frequency
Every release

Due Date
In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule

Associated Services
A.4.6

CD-24

System Administration Manual Deployment and Transition Plan System Configuration Report

Every release

A.4.6

CD-25

Every release

A.4.7

CD-26

Every release and at the end of the CMS Production Acceptance Period No updates

A.4.7

CD-27

Coding Standards, if applicable to the solution

At least thirty (30) calendar days prior to the start of coding activities Five (5) State Business Days after System Acceptance Five (5) State Business Days after System Acceptance Five (5) State Business Days after System Acceptance Five (5) State Business Days after System Acceptance Five (5) State Business Days after System Acceptance

A.4.3

CD-28

Custom Developed Software Code and Documentation, if applicable to the solution Software Development Tools, including software, manuals, manufacturer warranties and licenses Software Testing Tools, including software, manuals, manufacturer warranties and licenses System Monitoring Tools, including software, manuals, manufacturer warranties and licenses Software Maintenance and Operations Tools, including software, manuals, manufacturer warranties and licenses, if applicable to the solution

Every release

A.4.3

CD-29

No updates

N/A

CD-30

No updates

N/A

CD-31

No updates

N/A

CD-32

No updates

N/A

CMS Project

Page B-4

Deliverable Review Process

Deliverable Deliverable Title # CD-33 Third-Party Software

required for system operation, including software, manuals, manufacturer warranties and licenses, if applicable to the solution

Update Frequency
No updates

Due Date
Five (5) State Business Days after System Acceptance

Associated Services N/A

CD-34

Computer-Based Training (CBT) for Internal Users (Mandatory Optional) Computer-Based Training (CBT) for External Users (Mandatory Optional)

Every release

In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule

N/A

CD-35

Every release

N/A

CMS Project

Page B-5

Deliverable Review Process

APPENDIX C DELIVERABLE REVIEW PROCESS FLOW CHARTS

CMS PROJECT Deliverable Review Process Flow Charts

The pages which follow present flow charts depicting the CMS deliverable review process. The flow charts are followed by a matrix that presents the narrative description of each task depicted on the flow chart. While following this process, the contractor must adhere to the RFP Contractor Deliverable Requirements presented in Appendix A of this document. Should any inconsistencies or conflicts exist between this document and either the RFP or contract, amendments, or agreements, the RFP or contract, amendments, or agreements will take precedence.

CMS Project

Page C-1

Deliverable Review Process

CMS PROJECT DELIVERABLE REVIEW PROCESS

Flowchart #1: Submittal, Review, and Approval of Deliverable Expectation Document Contractor
(1) Review Deliverable Review Process Document (2) Complete and Submit Draft Deliverable Expectation Document (DED) (7) Finalize DED and Submit for Approval

Start

CMS Project Manager

(3) Document Receipt Of Draft DED, Disseminate for Review

(5) Compile Comments Matrices and Review Comments

(6) Finalize and Submit Comments Matrix to Contractor

(8) Corrections Made As Agreed? YES

NO

(9) Determine Remediation Steps

(10) Approve / Sign Off On DED

(11) Send DED to Deliverable Reviewers

(12) Archive DED in Project Library

To Flowchart #2

Deliverable Reviewers

(4) Review DED and Complete Comments Matrix

CMS Project

Deliverable Approvers

Page C-2

Deliverable Review Process

CMS PROJECT DELIVERABLE REVIEW PROCESS

Flowchart #2: Submittal and Review of Draft Deliverables Contractor
From Flowchart #1 To Flowchart #3

(13) Complete and Submit Draft Deliverable

(15) Conduct Deliverable Walkthrough

(20) Conduct Walkthrough of Comments Matrix

CMS Project Manager

(14) Document Receipt of Deliverable

(16) Distribute Deliverable for Review

(18) Compile Comments Matrices and Review Comments

(19) Finalize and Submit Comments Matrix to Contractor

Deliverable Reviewers

CMS Project

Deliverable Approvers

(17) Review Deliverable and Complete Comments Matrix

Page C-3

Deliverable Review Process

CMS PROJECT DELIVERABLE REVIEW PROCESS

Flowchart #3: Submittal, Review, and Approval of Final Deliverables Contractor

From Flowchart #2

(21) Finalize and Submit Deliverable

(23) Conduct Deliverable Walkthrough

CMS Project Manager

(22) Document Receipt Of Deliverable

(24) Corrections Made as Agreed?

(25) Determine Remediation Steps

(26) Submit to Deliverable Approvers for Acceptance / Signature

(28) Archive and Disseminate Final Deliverable

End Process

Deliverable Approvers

Deliverable Reviewers

(27) Approve / Sign Off On Deliverable

CMS Project

Page C-4

Deliverable Review Process

CMS PROJECT Deliverable Review Process Flow Chart Narrative

TASK TASK TASK OWNER TASK DESCRIPTION # Flowchart #1: Submittal, Review, and Approval of Deliverable Expectation Document 1 Contractor The contractor will review the deliverable review process Review Deliverable Review document to gain a thorough understanding of the deliverable Process Document review process and its associated requirements. 2 Complete and Submit Draft Deliverable Expectation Document (DED) Contractor The contractor will use the CMS project DED template (separate attachment to this document) to complete a DED for the deliverable. Utilizing the DED template, the contractor will ensure that the DED: 3 Document Receipt of Draft DED, Disseminate for Review CMS Project Manager Documents the purpose and objectives of the deliverable; Provides a format and annotated outline for the deliverable; and, Summarizes the key content of the deliverable, including (where appropriate) key figures, diagrams, and tables.

The CMS Project Manager (or designee) will maintain a deliverable tracking log listing all contractor deliverables and their associated due dates. This includes the due dates for the DED, drafts, and final deliverables. Upon receipt of the DED for a particular deliverable, the CMS Project Manager will document the date and time the DED was received and disseminate the DED to Document Reviewers and Document Approvers for project team and stakeholder review. The CMS Project Manager will establish the timeframe and appropriate deadlines for the review and comment period.

Review DED and Complete Comments Matrix

Deliverable Reviewers and Deliverable Approvers

Deliverable reviewers and approvers will review and comment on the DED, document comments in the Deliverable Comments Matrix (refer to Appendix D), and forward the matrix to the CMS Project Manager.

CMS Project

Page C-5

Deliverable Review Process

Compile Comments Matrices and Review Comments Finalize and Submit Comments Matrix to Contractor Finalize DED and Submit for Approval

CMS Project Manager

The CMS Project Manager will compile and review all comments, follow up with Document Reviewers or Approvers as required, and eliminate or consolidate comments as necessary. The CMS Project Manager will complete a final comments matrix and forward it to the contractor. The contractor will review the final comments matrix and follow up with the CMS Project Manager as required to clarify issues or questions. The CMS Project Manager will modify the comments matrix to reflect any changes to the comments based on this process. The contractor will finalize the DED and submit it to the CMS Project Manager for approval. The CMS Project Manager will review the revised / final version of the DED and determine whether the agreed upon changes were made to the DED. If the agreed upon changes were not made to the DED, the CMS Project Manager will identify steps to remediate the deficiency. This may include further revisions to the DED or other remediation steps identified in the RFP, contract, or subsequent amendments or agreements. If the agreed upon changes have been made, the CMS Project Manager will approve and sign off on the DED. Acceptance of the DED will be in writing, on a letter signed by the CMS Project Manager. The CMS Project Manager will send the completed DED to the deliverable reviewers so that they are notified of the schedule for reviewing the deliverable and are prepared to review it when the draft is disseminated. The CMS Project Manager will ensure that the approved DED and signed acceptance letter are filed in the project library.

6 7

CMS Project Manager Contractor

Corrections Made As Agreed?

CMS Project Manager

Determine Remediation Steps

CMS Project Manager

10

Approve / Sign Off on DED

CMS Project Manager

11

Send DED to Deliverable Reviewers

CMS Project Manager

12

Archive DED in Project Library

CMS Project Manager

CMS Project

Page C-6

Deliverable Review Process

Flowchart #2: Submittal and Review of Draft Deliverables 13 Contractor Complete and Submit Draft Deliverable

Upon approval of the DED, the contractor will complete and submit the draft deliverable. The draft deliverable will be submitted to the CMS Project Manager (or a designee). Key requirements for the submittal of draft deliverables include: The contractor project manager must certify that each deliverable is complete. The contractor must submit a copy of each deliverable. Must be complete, clear, explicit, and precise in manner, without any errors, omissions, or inconsistencies. The contractor must submit a copy of each deliverable electronically via email. A signed transmittal letter and a copy of the approved annotated outline (as documented in the DED) must accompany all deliverable submissions. The deliverable transmittal letter must contain contractor certification that the deliverable has been prepared in accordance with all relevant terms and conditions of the contract. Each deliverable, both draft and final, must be complete, clear, explicit and precise in manner, without any errors, omissions, or inconsistencies. The deliverables must include additional materials such as flow charts, diagrams, project management charts and explanatory tables to assist the Department in thoroughly understanding and evaluating the deliverables.

Refer to the Appendix A for a detailed listing of RFP deliverable submittal requirements. 14 Document Receipt of Deliverable CMS Project Manager When the draft deliverable is submitted, the CMS Project Manager will date stamp the deliverable and record the time and date in the deliverable tracking log.

CMS Project

Page C-7

Deliverable Review Process

15

Conduct Deliverable Walkthrough

Contractor

If requested by the CMS Project Manager, the contractor may conduct a walkthrough of the deliverable. The walkthrough will include an overview of the structure, organization, and major components of the deliverable, along with an explanation of the contents in each section. The CMS Project Manager will distribute the deliverable to deliverable approvers and reviewers for review and comment. The Deliverable Reviewers and Deliverable Approvers will review the deliverable and document comments, questions, or issues on the comments matrix. They will forward the comments matrix to the CMS Project Manager. The review period will be negotiated with the contractor. Unless agreed upon otherwise, the Department review time shall not be less than ten (10) State business days.

16 17

Distribute Deliverable For Review Review Deliverable and Complete Comments Matrix

CMS Project Manager Deliverable Reviewers and Deliverable Approvers

18

Compile Comments Matrices and Review Comments Finalize and Submit Comments Matrix to Contractor Conduct Walkthrough of Comments Matrix

CMS Project Manager

The CMS Project Manager will compile and review all comments, follow up with Document Reviewers or Approvers as required, and eliminate or consolidate comments as necessary. The CMS Project Manager will complete a final comments matrix and forward it to the contractor. If requested by the contractor, the CMS Project Manager will conduct a walkthrough of the comments matrix to clarify questions or address issues related to the comments. The CMS Project Manager will modify the comments matrix to reflect any changes to the comments based on this process.

19 20

CMS Project Manager Contractor

CMS Project

Page C-8

Deliverable Review Process

Flowchart #3: Submittal., Review, and Approval of Final Deliverables 21 Finalize and Submit Deliverable Contractor The contractor will finalize the deliverable and submit it to the CMS Project Manager for approval. Unless otherwise agreed upon with the Department, the contractor will reflect changes in the deliverable within five (5) State business days of receiving the comments matrix. Upon submittal of the final draft, the contractor must clearly identify changes to the revised deliverable. The key requirements for the submittal of final deliverables are the same as those listed in Task #13 Key requirements for the submittal of draft deliverables except that at least one copy in paper hardcopy must be submitted in addition to an electronic copy via email. 22 Document Receipt of Deliverable CMS Project Manager When the final deliverable is submitted, the CMS Project Manager will date stamp the deliverable and record the time and date in the deliverable tracking log. If requested by the CMS Project Manager, the contractor may conduct a walkthrough of the final deliverable. The walkthrough will focus on the final changes made to the document. The CMS Project Manager will review the revised / final version of the deliverable and determine whether the agreed upon changes were made. Unless agreed upon otherwise, Departments review time after receipt of the revised deliverable shall not be less than five (5) State business days. 25 Determine Remediation Steps CMS Project Manager If the agreed upon changes were not made to the deliverable, the CMS Project Manager will identify steps to remediate the deficiency. This may include further revisions to the deliverable or other remediation steps identified in the RFP, contract, or subsequent amendments or agreements.

23

Conduct Deliverable Walkthrough Corrections Made as Agreed?

Contractor

24

CMS Project Manager

CMS Project

Page C-9

Deliverable Review Process

26

Submit Deliverable to Approvers for Acceptance / Signature

CMS Project Manager

When the CMS Project Manager agrees that the agreed upon changes have been made and that the deliverable meets the Departments acceptance criteria, the CMS Project Manager will forward the deliverable to the deliverable approvers for acceptance. Deliverable approvers have the ultimate authority to approve and formally accept a deliverable. Acceptance will be documented in writing on a notice of deliverable acceptance. The contractor will receive a copy of this form as a written notice of deliverable acceptance. Unless otherwise stated in writing by the CMS Project Manager, the deliverable approvers will be: CMS Project Sponsor IT Program Manager CMS Project Manager

27

Approve / Sign Off on Deliverable

Deliverable Approvers

The deliverable is not considered accepted unless the notice of deliverable acceptance has been signed by each of these individuals. In the event that the Department agrees to accept on an interim basis a deliverable document with one or more parts left incomplete, the contractor must provide the Department with a document indicating the date for completion, the basis for the incomplete portions, and the impact any of the incomplete portions of the document may have on contract milestones. 28 Archive and Disseminate Final Deliverable CMS Project Manager The CMS Project Manager will ensure that the approved deliverable and signed acceptance letter are filed in the project library. The final deliverable will be disseminated to project team members and stakeholders for reference. The contractor may not change any deliverables that have been accepted by the Department without the approval of the Department.

CMS Project

Page C-10

Deliverable Review Process

APPENDIX D DELIVERABLE COMMENTS MATRIX

CMS PROJECT Deliverable Comments Matrix

DELIVERABLE PAGE # SECTION HEADING / TEXT EXCERPT REVIEWER NAME COMMENT CONTRACTOR RESPONSE / NEW WORDING

CMS Project

Page D-1

Deliverable Review Process