Source: http://www.cic.gc.ca/english/resources/audit/gcms.asp
System-Under-Development Audit of the Global Case Management System
Final Report
Internal Audit and Disclosures Branch
Citizenship and Immigration Canada
November 2005
Executive Summary
1. Introduction
1.1. Background
2. Audit Objectives
3. Audit Scope
4. Audit Criteria
5. Methodology
6. Conclusions
7. Observations and Recommendations
7.1. Project management processes
7.2. Controls over data integrity and Sensitive information
8. GCMS Project Action Plan
Appendix: GCMS Audit Interviews List
Top of Page
Executive Summary
The Global Case Management System (GCMS) is a multi-year project that will replace several existing systems at Citizenship and Immigration Canada (CIC) and the Canada Border Services Agency (CBSA). The goal of the project, which began in fiscal 2000–01, is to provide CIC with an automated, highly integrated case management tool to support its global business network and provide enhanced end-to-end client services. When complete, GCMS will support more than 7,000 users at CIC and the CBSA in some 125 points of service in Canada (offices, call centres, processing centres, citizenship courts and ports of entry) and more than 100 missions abroad.
In September 2004, the first phase of the GCMS project was implemented with the “go live” of the Citizenship component (Deployment 1). Although this component was successful from a technological standpoint, some initial operational difficulties were encountered, but they have largely been resolved. Consequently, when the project sought to amend the Effective Project Approval (EPA) in the spring of 2005, an audit of the GCMS project was requested as part of the process.
Our audit sought to examine the progress made since GCMS’ Deployment 1 and the next phase of GCMS. We examined the project management processes and controls for ensuring both data integrity and the protection of sensitive information. Our observations cover the period from September 2004 to October 2005.
Our overall conclusions are that the GCMS project management processes are sound and that they will increase the likelihood of success by ensuring the existence of an effective management control framework. Furthermore, the audit team found that adequate controls are in place to support data integrity and protect sensitive information assets.
The following report provides the audit team’s observations and recommendations, as well as the GCMS project team’s responses and action plan to address these recommendations.
Top of Page
1. Introduction
The Internal Audit and Disclosures (IAD) Branch of Citizenship and Immigration Canada (CIC) was asked to conduct a focused system-under-development audit of the Global Case Management System (GCMS) project. The IAD audited the project management processes and the control measures in place for data integrity and sensitive information assets, to assess their adequacy for the successful implementation of the project.
The audit was conducted in accordance with the Government of Canada’s policy on internal audits, as well as auditing standards prescribed by the Institute of Internal Auditors.
1.1 Background
GCMS is a multi-year project that will replace several of the business systems of CIC and the Canada Border Services Agency (CBSA) with an integrated, case management-based set of applications and components that will support both organizations’ client operations.
The GCMS project received Preliminary Project Approval on March 1, 2001, with an initial budget of approximately $195 million (T.B. 828697). In view of the assessed level of risk and the scale of the project, CIC was instructed to manage GCMS as a Major Crown Project and to confirm the viability of using a “commercial off-the-shelf” (COTS), Client Relationship Management package to provide much of the required case management functionality.
Because of a variety of factors, many of which were outside the control of CIC, the selection of a vendor and the awarding and signing of the contract were delayed by nine months. As a result of this delay, it was April 2003 before the project team had access to the COTS product and technical experts required to build and deploy GCMS. Later in 2003, an amended Effective Project Approval was approved by Treasury Board ministers (T.B. 830875) and the budget increased to approximately $200 million.
Since the project was granted preliminary approval, it has had to deal with a number of external and internal events that have directly affected the project and that have necessitated adjustments to the project implementation strategy. These events include the following:
the events of September 2001, which forced countries to reconsider the tools they were using to deal with threats while, at the same time, not unduly impeding the legitimate movement of people;
the continued and growing interest in Canada as a destination for immigrants and temporary residents (tourists, foreign students, temporary workers);
the implementation of the most comprehensive reforms to the immigration legislation in more than 30 years, including the introduction of the secure permanent resident card; and
the creation of the Canada Border Services Agency.
The project has conducted several independent readiness reviews (July 2000, June 2001 and December 2002) as well as two Independent Verification and Validation reviews (August 2002 and July 2003) to address various issues and concerns about the project. The results of these reviews were provided to the Treasury Board Secretariat (TBS) through formal reporting (September 2002) and in regular monthly reports.
Treasury Board ministers also requested that the Chief Information Officer Branch (CIOB) of TBS engage the services of an independent third-party monitor to report on the progress of the project, with particular attention to the vendor relationship. Two monitors were engaged in October 2003. In January 2004, they provided their report to TBS. A subsequent follow-up report was provided in March 2005.
Reports prepared by the external Treasury Board monitor highlighted the fact that senior management at both CIC and the CBSA enthusiastically and unequivocally supported the GCMS project. The report also noted that there was a high degree of awareness and engagement within CIC and the CBSA user communities, as demonstrated in periodic surveys designed to “take the pulse” of the two organizations.
The project has also served as a catalyst for business transformation. In identifying, defining and formalizing business requirements, the project has had to re-examine existing business processes and to develop consensus among departmental users. The result is a more consistent, standardized set of practices aimed at promoting greater uniformity, increased efficiency and improved client-centred service across both organizations.
CIC’s business includes over 1.5 million client transactions a year, approximately two-thirds of which occur overseas in environments that can be characterized as challenging in terms of their complexity and the local infrastructure.
When completed, GCMS will support more than 7,000 users at CIC and the CBSA in some 125 points of service in Canada (offices, call centres, processing centres, citizenship courts and ports of entry) and more than 100 missions abroad.
GCMS is a critical element of a larger, client-centred service and business transformation vision that seeks to maximize the benefits associated with its citizenship and immigration programs in the years to come. By integrating the business functionality and information that now reside within more than a dozen of CIC’s legacy systems built over the last three decades, the project aims to eliminate redundant data, improve the consistency and quality of that data and contribute to streamlining operations. It should also allow both CIC and the CBSA to further improve the delivery of their programs and services.
In September 2004, the first phase of the GCMS project was implemented with the “go live” of the Citizenship component (Deployment 1). The project continues to meet the expectations of both users and IT experts and has proven to be a powerful tool that can be customized to meet the specific requirements of CIC and the CBSA.
Although the Citizenship component was successful from a technological standpoint, some initial operational difficulties were encountered. These were attributable to data conversion problems, a steep learning curve for users, and a higher than anticipated number of system defects. These problems are being addressed as the project moves toward the next phase of implementation. The system has become more stable, and production is approaching predeployment levels.
As part of Deployment 1, the project team engaged in several lessons-learned exercises. These exercises identified and documented areas for improvement such as accountabilities, requirements development, communications, project planning and scheduling, resource planning, and project management control and tracking. Learning from these exercises, the GCMS project team prepared detailed action plans and began to adopt new practices.
In the spring of 2005, the GCMS project team began preparing a submission to Treasury Board ministers through TBS for amendments to the Effective Project Approval (EPA) for the GCMS project. As part of this process, the TBS expressed a need for an audit of the project.
The type of audit that TBS considered most suitable was a system-under-development (SUD) audit. In general, TBS was looking for a focused SUD audit that would provide reasonable assurance that the GCMS project was on a solid foundation as it moved forward.
In June 2005, the terms of reference were developed in preparation for this audit, which began in July 2005.
Top of Page
2. Audit Objectives
The audit had two objectives:
to assess the soundness of the project management processes; and
to determine the appropriateness of controls to ensure data integrity and protect sensitive information assets.
3. Audit Scope
The audit scope covered the progress made since GCMS’ Deployment 1. We examined the project management processes and controls for ensuring both data integrity and the protection of sensitive information. Our observations cover the period from September 2004 to October 2005.
We did not examine processes or controls in effect prior to September 2004. Furthermore, an examination of the contract between the Government of Canada and the principal vendor for the GCMS project was outside the scope of this audit.
4. Audit Criteria
We derived our assessment criteria from a number of reasonable and credible standards of performance and control relevant to this engagement. The sources of criteria for this audit included the following:
Project Management Body of Knowledge (PMBOK) – the Project Management Institute’s (PMI) standards document for project management knowledge and practices;
Control Objectives for Information and Related Technology (COBIT) – the IT Governance Institute’s framework for IT practices;
Various Treasury Board Secretariat policies (e.g., TBS Enhanced Management Framework);
Various CIC departmental policies; and
The Management Control Framework for CIC Audit Activities.
The criteria for the two audit objectives were as follows:
The project management processes increase the likelihood that the project will succeed by ensuring that an effective management control framework exists.
Project planning techniques are sound.
Project monitoring, tracking and control techniques are appropriate.
There are processes to enable the project to achieve its overall business objectives.
There are improvements to the project management processes applied for Release 2.
The contract and subcontract management processes are adequate.
Project requirements management techniques are adequate.
Change management procedures are adequate.
Techniques for integration management applied with respect to the project management office’s (PMO) linkages to project authorities (e.g., Treasury Board) are sound.
The controls to ensure data integrity and the protection of sensitive information are adequate.
Procedures to support data conversion and to verify and maintain the integrity of converted data are adequate.
Procedures for protecting sensitive information have been incorporated into the project and are adequate.
Top of Page
5. Methodology
The audit began in July 2005, and we carried out our examination from July to October 2005. This involved the following activities.
interviews;
reviewing documentation;
reviewing best practices against the practices of recognized organizations such as the PMI ’s PMBOK and the IT Governance Institute’s COBIT;
a detailed examination of selected project management processes; and
audit testing.
As part of the audit, we interviewed 44 people, including key executives, GCMS project members and other key stakeholders. We also obtained relevant documentation to validate the information gathered during interviews (see appendix A for a list of interviewees).
The audit team also examined documentation and conducted detailed testing to determine whether specific processes or controls were in place. In addition, the team conducted a literature review of readily available best practices to identify any gaps in the project management processes or controls under review.
Detailed testing included the following identified areas:
Detailed master schedule;
Exception reporting;
Change management;
Risk management;
Dashboard reporting;
Task authorization/task authorization requests validation;
NON-COTS contracts;
Asset purchases; and
Security clearances.
6. Conclusions
We have concluded that the GCMS project management processes are sound, and that they will increase the likelihood of the project’s success by ensuring that an effective management control framework exists. Furthermore, the audit team found that adequate controls were in place to ensure data integrity and to protect sensitive information.
Top of Page
7. Observations and Recommendations
7.1 Project management processes
The audit objective was to assess the soundness of the project management processes in place and their contribution to the project’s success by ensuring that an effective management control framework exists. The project management processes that we examined as part of this audit included the following areas: project planning; project monitoring, tracking and control; contract and subcontract management; requirements management; integration management; and change management. The following paragraphs present our observations and, as applicable, recommendations for each area.
7.1.1 Project planning
In the context of project planning, the audit team reviewed a number of elements. Our observations in this area are as follows.
The business case
Under project planning, we expected to find that the GCMS project team had developed a business case that was comprehensive and complete, and that identified and justified the GCMS project and indicated how it related to CIC and government-wide priorities. At a minimum, a business case should identify:
the opportunities for improvement;
the benefits of such an undertaking;
the technical solution at a high level;
indicators against which to measure any improvement in program performance;
costs (up-front, direct ongoing, indirect and potential user/client costs); and
the risks, and steps to be taken to manage those risks.
The audit team found that the GCMS business case had covered all significant elements.
Project charter
We found that the GCMS project team had a project charter that defined the scope of the project. It also set out the overall project management framework and standards to be used. The project charter also clearly defined who the internal and external stakeholders were, described the roles and responsibilities of the members of the project team and established the overall project governance structure (e.g., accountabilities and project authorities). Furthermore, we found that the project charter had set out the management commitments (human and financial resources, time frame and deliverables).
The team found that the GCMS project charter had been continuously updated throughout the life cycle of the project to reflect various project changes, and that all significant elements had been addressed in the project charter.
Project approvals
The audit team expected to find that the project had received the appropriate approvals from not only CIC, but TBS as well. The TBS preliminary project approval was granted on March 1, 2001 (T.B. 828697). Effective project approval was granted on January 31, 2002 (T.B. 829502) and amended on October 9, 2003 (T.B. 830875). During this audit, CIC sought and obtained a further amendment to the EPA. Finally, we noted that the GCMS project senior management within CIC has demonstrated, and continues to demonstrate, strong support for the project.
Top of Page
Project plan
The audit team expected to find a project plan that included:
how system requirements would be managed;
how negotiation of resource commitments would be handled;
a software life cycle with predefined stages;
the key processes and deliverables for the project;
authorization to start work and key team members assigned;
estimate of the size and complexity of the software;
estimate of product resource requirements;
detailed schedules;
commitment and approvals from within CIC; and
reporting requirements.
The master project schedule (MPS) is the primary mechanism used for project planning within GCMS. Our examination showed that the MPS contained all the details listed above and that it reflected the GCMS project charter.
The MPS is maintained by the PMO with input from individual project teams who are responsible for preparing subteam schedules that feed into the MPS. Weekly meetings are also held with the project team’s project control officers to discuss the project’s progress as it applies to their sub-schedules.
To validate the information contained within the MPS, the audit team conducted the following audit tests (using the MPS dated July 15):
tests to validate the internal consistency of reported dates in subteam schedules; and
tests to map tasks contained in the MPS back to the subteam schedules and compare reported information.
We noted that the level of detail and consistency varied between the different subteam schedules. Consequently, the audit team had difficulty mapping tasks contained in the MPS back to subteam schedules and found that the level of difficulty in mapping tasks varied between sub-schedules. Without a better link between the MPS and the sub-schedules, the project cannot be sure that all material items are included in the MPS. Given the MPS’ importance in planning, we are concerned that stronger links do not exist. The project is aware of these facts and was in the process of revalidating all schedules at the time of the audit. We were informed that efforts are also being made to realign the schedules to improve the links between the MPS and its sub-schedules.
In our view, the project needs to ensure that all project team members commit to uniformly applying the project management planning standards. We believe that doing so will improve the quality and consistency of the information feeding into the MPS. Strengthening of the MPS should also strengthen the timeliness of information and decision making within the project.
Recommendation 1
The GCMS project should ensure the consistent application of project management planning standards (GCMS Schedule Management and Development Standards) developed by the PMO.
GCMS management response
Agreed.
Project reviews
The audit team expected to find a process for reviewing and revalidating the major planning deliverables (the business case, project charter and plan) for the project. The project team has completed a thorough review and analysis of Deployment 1 activities and has applied lessons learned to develop a revised approach and plan for completing the GCMS.
The project has carried out the following reviews:
Independent Readiness Reviews (July 2000, June 2001 and December 2002);
Independent Verification and Validation Reviews (August 2002 and July 2003);
Independent TBS Monitor Reviews (February 2004 and March 2005);
GCMS Project Team—Gates 1 and 2: Lessons Learned (November 2003 and December 2004); and
GCMS Review of Project Response to Deployment 1: Lessons Learned (June 2005).
The audit team found that the level of project review was generally adequate for the size and scope of the project.
Top of Page
Governance and organizational structure
The project has the following governance committees:
Change Control Board (CCB)
Design Review Board (DRB)
Design Review Team (DRT)
Executive Committee (ExCOM)
Executive Management Team (EMT)
Group of Approvers (GA)
Integrated Management Committee (IMC)
Operational Validation and Implementation Team (OVIT)
Project Management Board (PMB)
Risk Management Board (RMB)
Senior Project Advisory Council (SPAC)
Senior Level Advisory Committee (SLAC)
We examined the terms of reference for each of the governance committees and found them to be complete. A review of agendas, supporting materials and records of decisions indicated that they had met the requirements of government policies and CIC standards. The audit team attended (at random) various governance meetings (e.g., SLAC, RMB, CCB, PMB and EMT) and found that each was chaired at the right level and that the composition of the committees was appropriate, given their terms of reference.
We also found that the project had an appropriate organizational structure in place. Key positions have been identified with clearly documented and communicated roles and responsibilities. As one of its primary objectives, the IMC is dedicated to identifying and resolving staffing issues for the project in a timely fashion.
Overall, we found the governance and organizational structure of GCMS to be comprehensive and in compliance with applicable government directives (e.g., the Major Crown Projects Policy and the TBS Enhanced Management Framework).
Process plans
The audit team expected to find process plans for the following:
quality assurance;
risk management;
communications;
performance management;
HR management;
procurement management;
requirements management;
IM/IT;
testing;
roll-out; and
training.
Overall, the audit team found that plans had been completed for each of the areas noted above. Process plans are continually being improved as demonstrated by the incorporation of lessons learned from Deployment 1. Significant strides have been made in bolstering areas identified in lessons-learned exercises. In particular, these include:
a formal accountability framework (approved April 14, 2005);
the appointment of a deputy project manager;
tightened project management processes (e.g., for HR management);
improved management reporting (e.g., improvements to the GCMS dashboard); and
stronger control over project resourcing (e.g., IMC process).
Top of Page
7.1.2 Project monitoring, tracking and control
In the area of project monitoring, tracking and control, the audit team reviewed a number of elements, as noted below.
Data collection and tracking requirements
The audit team expected to find that baseline estimates for activities set out in the project plan (including milestones) had been established and accepted by the project team. The GCMS project team has identified, defined and documented major baseline activities, milestones and data. Updates to the project plan approach and associated deliverables reflect the intent of the business case. The evolving project charter is consistent with these adjustments to baseline activities.
The audit team found the project tracking and oversight process to be consistent with TBS Enhanced Management Framework principles.
Planned vs. actual performance
Processes have been identified to:
track actual performance;
compare actual with planned performance (e.g., track deviations from plan);
identify when and what adjustments will be needed; and
determine the impacts of any adjustments and the subsequent changes needed (e.g., additional resources) to close the gap.
The audit team completed a review of the minutes of the steering committee (discussed below) and other meetings to determine if performance was reviewed regularly and adjustments made as required. The audit team found that planned vs. actual performance was reviewed regularly and that the reasons for any deviations from planned performance were documented and explained concisely. However, ownership of these deviations is weak. This is discussed in greater detail in section 7.1.3 (Monitoring of service delivery by a third party).
Financial tracking
The audit team expected to find that financial tracking was being conducted and that the results were reported. In particular, we expected the following:
Budgeted vs. actual and forecast is analysed, documented and reported on a regular basis.
Financial data are accurate, complete and produced in a timely fashion.
High-level financial tracking information from the PMO is consistent with information at the project team level.
Financial coding structure is consistent with CIC standards.
Financial authorities as established in the project charter and CIC are applied.
Dedicated PMO staff who work in collaboration with the CIC Finance Branch track and control GCMS financial information. The audit team conducted a sample testing of asset acquisitions to test for compliance with the Financial Administration Act. Our examination in this area found that the processes were functioning as intended. Furthermore, a review of detailed financial data and supporting documentation for the month of July 2005 found that internal financial controls were adequate.
Tracking performance and results
The audit team expected to find that performance and results were documented and communicated within the PMO and were continuously adjusted to reflect project improvements and the tracking of all issues through to completion. In particular, we expected to find the following:
Project logs and status reports are prepared and updated regularly.
A distribution list of concerned parties is maintained.
Issues and problems are identified, documented and tracked to closure.
There is a clearly understood process for resolving problems when they arise.
The audit team reviewed various logs (e.g., issue, decision, action and risk) and found them to be adequate.
In addition, as part of its audit, the audit team tested one period’s GCMS dashboard (May 2005) to assess the accuracy of the information it contained. Our examination revealed that the dashboard reflected the information available within the project.
Top of Page
Risk management
The audit team tested the risk management tracking processes (May–June 2005) to ensure that the process worked as intended and was in accordance with generally accepted practices.
The audit team expected to find a process for ensuring that project risks were identified, assessed and documented and, in particular, that the following elements were in place for the GCMS project:
a systematic risk assessment framework;
a risk assessment approach that allows for regular updates;
risk assessment procedures that identify both external and internal risk factors;
procedures for monitoring changes in risks;
risk assessment documentation;
risk mitigation strategies identified and documented; and
involvement of stakeholders in identifying project risks.
We found that the GCMS project had mechanisms that identified, documented and tracked risks. An initial risk assessment was completed in 2001 and has been regularly updated throughout the life of the project. A comprehensive Threat and Risk Management Assessment was completed in December 2004. Overall, we found that the monitoring of risks associated with the project was well documented and tracked.
Meetings
The audit team expected to find that meetings were held on a regular basis to make decisions and to review the project’s status, performance and risks.
The audit team found that since Deployment 1, team meetings have been focusing on providing a better understanding of the status, costs, time frames and risks associated with the project. A review of agendas, minutes and records of decisions (ROD) for ExCOM, SLAC, SPAC, PMB and EMT conducted by the audit team found meetings were taking place, major activities were being updated and risks were being tracked and followed up on.
Steering committee
The audit team expected to find that a steering committee had been established and was operating and, in particular, that individuals at the appropriate levels had been identified and were participating in the committee.
Our audit found that the steering committee was functioning as intended. ExCOM is the senior departmental forum for monitoring progress, reviewing issues and making decisions regarding alternative approaches and priorities for the GCMS project. As such, it serves as GCMS’ steering committee. ExCOM meets on a monthly basis. Steering committee members include:
Deputy Minister (Chair)
ADM, Policy and Program Development
ADM, Centralized Services Delivery and Corporate Services
ADM, Operations
ADM, Strategic Directions and Communications
Assistant Deputy Attorney General
DG, Human Resources
The GCMS project team has also formed the Senior Level Advisory Committee to provide additional perspective and practical advice to the GCMS Executive Management Team. SLAC meets on a monthly basis. Its members are:
Project Sponsor – ADM, Policy and Program Development
ADM, Centralized Services Delivery and Corporate Services
ADM, Operations
Project Leader, DG, Business Solutions
DG, IMTB
Executive Director, IM/IT
Executive Director, PMO
Executive Director, Business Requirements and Application Functional Design
Executive Director, Deployment, User Acceptance Testing (UAT) and Training
Delivery Manager
Chief, GCMS Finance
IndependentmMonitor, TBS
VP, Enforcement, CBSA
Director, Science Innovation and Technology, CBSA
The audit team reviewed the minutes and presentations of ExCOM and SLAC and found that they had documented key decisions and directions. Roles and responsibilities, together with terms of reference, have been clearly defined and documented for each committee.
Top of Page
Communication strategy/plan
We expected to find that a communication strategy/plan had been developed and implemented to communicate the project’s results and progress to external stakeholders.
We found that the GCMS project team used the following key communication tools to reach its various audiences:
the GCMS electronic newsletter – “Spotlight on GCMS”;
Taking the Pulse – employee surveys;
CIC’s internal website – “CIC Explore”;
GCMS portal;
regional visits;
executive staff meeting: weekly GCMS topical briefings;
regular status updates to ExCOM, SLAC, SPAC;
TBS Monthly Operational Reports (MOR); and
GCMS dashboard.
The project regularly provides timely updates to all stakeholders (internal and external). We understand that the GCMS project team, as part of its activities for Release 2, has incorporated specific communication tasks into the project management plan to prepare users for Release 2. The goal of these tasks is to bring users to a stage where they are committed to the project and ready for the changes it will bring.
Process for managing documents
The audit team expected that the GCMS project team would have a sound document management process. We found that the GCMS project maintained records as part of the project’s historical records. Examples include:
versions of documents (e.g., Business Case, Project Charter);
logs (decision logs, issue logs, risk logs, change logs);
risk assessments; and
minutes, RODs and presentations.
The project team uses electronic tools (LiveLink – GCMS Portal and ClearCase) to electronically house documents relating to the project. The audit team found that the GCMS project team had a sound document management process and that it was functioning as intended.
Top of Page
7.1.3 Management of contracts and subcontracts
In examining this area, the audit team reviewed software acquisition plans, whether the experience of staff was adequate to support software acquisitions, the control of consultants, monitoring of service delivery by a third party, the independence of the PMO and principal vendor, and the extent to which the use of subcontractors was documented and controlled. Our observations in these areas are as follows.
Software acquisition plans
Plans for contracting and acquiring software include:
the acquisition strategy;
the method for soliciting bids;
constraints related to acquisition;
acquisition resource requirements;
roles and responsibilities;
cost and schedule estimates;
contract tracking, oversight and evaluation plans;
acquisition risk management; and
life cycle support.
The audit team found that the software acquisition plans for the GCMS project followed CIC’s and Public Works and Government Services Canada’s (PWGSC) standard procurement policies. In particular, CIC has a number of software licences that it has made available to the GCMS project. Documented procedures also exist for acquiring additional software. The approval process includes the project as well as the Information Management and Technologies Branch within CIC.
In addition, most of the procurement of hardware and software to establish the production environment had been completed, and the domestic production environment was operating in a high-availability, fail-over configuration mode. Upgrades to Foreign Affairs Canada’s (FAC) international network (MITNET) and server/desktop environment (SIGNET) were proceeding within the parameters required by GCMS. Project team members continue to work closely with FAC in efforts to minimize the amount of infrastructure required for the GCMS Release 2.
The original procurement strategy (November 2001) has been updated to reflect the project’s software requirements. In general, the audit team found that the software acquisition plan included all significant elements.
Staff experience
The audit team expected to find that experienced acquisition, contracting and management staff were supporting the software acquisition for the project team. In particular, we expected that key staff would have appropriate knowledge and expertise in:
evaluating the technical aspects of the proposed system;
CIC information management and information technology (IMIT) requirements (architecture, interfaces);
federal and CIC contracting requirements, costing methodologies and tools; and
end-user requirements.
The GCMS project team had a contracting expert from PWGSC assigned to provide advice. In addition, the project team had support from CIC internal contracting resources to facilitate the contracting process. IMTB staff with experience in evaluating and assessing technical solutions have been assigned to the project. Accordingly, we found that the experience of the acquisition, contracting and management staff assigned to the GCMS project to support software acquisitions was adequate.
Top of Page
Control of consultants
The audit team expected to find policies and procedures for controlling the activities of consultants assigned to the GCMS project in order to protect the organization’s information assets. In particular, we expected that the GCMS project team would ensure that:
contractual requirements are developed, managed and maintained;
terms and conditions are standardized and reviewed by legal counsel;
contractual commitments are traceable and verifiable;
cost and schedule estimates are independently reviewed; and
security screening of key contractors is performed in conformity with CIC policy.
The audit team found that over time, the project team had instituted processes (e.g., time reporting) to better manage its contractors. In particular, the audit team tested the exception-reporting process to ensure that overtime reporting for contractors was complete and that appropriate authorities had approved them. Our examination in this area indicated that this process was functioning as intended.
Monitoring of service delivery by a third party
The audit team expected to find that a process for monitoring service delivery by a third party had been set up to ensure that the terms of contractual agreements continued to be met. In particular, we looked at whether:
actual costs and schedules were compared to planned;
issues were documented and tracked until closure;
services or products were reviewed prior to financial settlements;
acceptance criteria were documented;
periodic reviews were carried out;
an independent audit of contractor operations was done; and
monitoring was consistent with the type of contract awarded.
The audit team tested the contracting process to ensure that it was functioning according to CIC/PWGSC policies by examining a sample of seven NON-COTS contracts to determine whether they were sufficiently detailed. Our examination in this area showed that the process was functioning as intended, but that some areas needed improvement. For example, we noted instances in which particular goods or services were to be delivered, but there was no reference to a delivery date beyond the contract end date. Including delivery dates for deliverables in the contract will allow the project to better understand and monitor progress toward targets. We advised the GCMS project team of these weaknesses and were assured that they were being addressed.
The audit team also expected to find that performance measures would be in place to assess the performance of the principal vendor. We noted that the main reporting tool, the GCMS dashboard, had been refocused since Deployment 1 to provide more information on the status of the project. However, it does not report on one key aspect of the project: vendor performance. We acknowledge that the project team is working with the vendor to identify and define appropriate criteria or indicators for measuring the vendor’s performance. We support these efforts and make the following recommendation.
Recommendation 2
Objective and quantifiable performance criteria related to the principal vendor should be identified and discussed with the principal vendor. These metrics should be chosen so that they adequately reflect contractor performance.
GCMS management response
Agreed.
Independence of the project management office and principal vendor
We expected to see the appropriate degree of independence needed to ensure an arm’s-length relationship between the PMO and the principal vendor.
Our review of a sample of seven NON-COTS contracts indicated that declarations of conflict of interest had been completed by all. We also reviewed the roles and responsibilities of key staff in the PMO to determine whether they were at arm’s length with the vendor. The audit team found that the GCMS project management office was staffed with individuals who were independent of the principal vendor.
Documentation and control of the use of subcontractors
The audit team expected to find the use of subcontractors to be documented and controlled and, in particular, that documented guidelines or procedures existed for the evaluation and use of subcontractors for the project.
We carried out a test of task authorization (TA)/task authorization requests (TARs) to determine whether it was working according to policy. An examination of 65 TA/TARs showed that COTS/NON-COTS were being monitored for start and end dates, and that clearly defined roles and responsibilities were set out for each person who joins the project. Appropriate authorities requested and approved these resources. Overall, the audit team found that the documentation and control of subcontractors was functioning as intended.
Top of Page
7.1.4 Requirements management
In examining this area, the audit team reviewed a number of elements, as discussed below.
System requirements and specifications
The audit team expected to find that the PMO had defined procedures and plans to ensure close liaison with system users in developing the design specifications, and to ensure that these specifications would meet the users’ requirements. In particular, we looked at whether:
project management had developed and approved policies regarding requirements management;
policies and procedures ensured that requirements had been documented;
the process for developing requirements had been defined and had involved users, project management and team members and key IMIT personnel;
business requirements had been clearly defined prior to acquisition;
plans were in place to link requirements to design components, test plans and contract management procedures; and
requirements had been reviewed and approved by project management, user groups and senior management.
The audit team found that the GCMS project team had a documented requirements management plan that had been signed off by the appropriate project authorities. The user community has been actively involved in developing the business requirements. In particular, the GCMS project team has actively engaged representatives from across the Department and the CBSA to define business requirements and design specifications (e.g., Group of Approvers, OVIT, Design Review Team, and Subject Matter Experts).
In trying to identify, define and formalize business requirements, it has been necessary to re-examine and rationalize processes in three major business domains (i.e., immigration facilitation, immigration control and integration), and to gain consensus among users in the Department. This approach has resulted in delays in finalizing the business requirements and obtaining final approvals and sign-offs. The project team is aware of this and believes that this approach is warranted. It believes that this approach will result in a more consistent and standardized set of business practices, including greater uniformity, increased efficiency and improved service across CIC.
Processes to ensure that requirements are met
Requirements are documented and available to all project members. In addition, requirements are linked to the overall design of the GCMS solution, including components, testing and contract management. The audit team expected to find that requirements had been analysed for completeness, understandability, feasibility and consistency and, more importantly, that specifications had considered:
the design of the process for importing data from existing systems to GCMS;
the definition and documentation of input requirements;
the definition of interfaces;
user-machine interfaces;
the definition of processing requirements;
the definition of output requirements;
internal control and security requirements; and
functional and operational requirements.
The audit team found that documentation on the requirements was available. However, it is controlled to ensure that information is safeguarded from unwanted changes. The final approved documents are provided to team members through the ClearCase document management tool. The audit team did not determine if the requirements existing at the time of our audit had been adequately analysed and linked to design components or test plans. We understand that the GCMS project team, as part of its activities for Release 2, will develop test plans that will cover those requirements.
Processes to manage changes to requirements
The audit team expected to find that changes to requirements were handled through a controlled process whereby user representatives and project team members negotiated and agreed upon any changes. The audit team found that changes to requirements had been documented and reviewed by the appropriate functional areas and user community representatives for completeness and operational approval. The process for managing changes to requirements was functioning as intended.
Top of Page
7.1.5 Integration management
In the context of integration management, the audit team reviewed several areas as discussed below.
Linkage to project authorities
The audit team expected to find that the project management office had formal reporting structures in place that would provide key decision makers with the information they need to make informed decisions. We also expected that the project would have mechanisms to incorporate feedback from key stakeholders such as other departments and TBS. In particular, we noted the following:
MOR and GCMS dashboards provide details on the technical status of the project to the Chief Information Officer Branch of the TBS.
An independent TBS monitor provides regular updates on the status of the project to Treasury Board and CIC executives.
Meetings are held every two weeks with the TBS analyst on the program side.
Quarterly meetings of the SPAC involve cross-departmental representation to address GCMS issues that may affect other government departments. SPAC consists of representatives from 16 federal departments and agencies.
7.1.6 Change management
The audit team reviewed a number of elements relating to change management, as follows.
Change management procedures
Effective change management is a vital aspect of the overall governance process and is critical to controlling the scope of a project. In the case of GCMS, the change management process consists of two types of requests:
Change requests – changes that result in a change to approved project or product deliverables; and
Problem reports – issues that address a defect or deviation from the approved functional baseline.
The project team had a clearly defined process for approving change requests that involved higher levels of management authorization than problem reports. The procedures for change requests for the GCMS project include:
categorizing and prioritizing requests;
detailing how urgent requests are to be handled;
updating procedures for keeping requestors informed;
allowing for assessment of the effects of the change; and
identifying the authorities required before the changes can be approved.
The audit team found that the process, procedures and change log were functioning as intended. Documentation indicates that the process is controlled and centralized.
Requests for changes
The audit team expected to find that change requests were monitored and recorded. In particular, we expected that change-control logs would be used to record all change requests.
The audit team found that two people managed the process for the GCMS project team. They are responsible for monitoring the life cycle of all change requests (i.e., opening, logging, tracking and closing them). The project team had recently improved the process to address the closing of change requests within acceptable time frames. In particular, the project team has focused on identifying high-priority change requests with an emphasis on addressing and closing these issues.
Documentation of approved changes
Change requests are used as a means of documenting or recording decisions and ensuring that representatives from not only the GCMS project team but, as appropriate, other key stakeholders, understand the impact of the potential change. Such tracking enables GCMS to:
provide members with a way to record the history of changes made to the GCMS product;
control proposed changes that may affect the scope, schedule or budget;
ensure that all changes adhere to the Configuration Management and Document Management processes; and
communicate approved changes to the GCMS team as a whole.
The audit team found that the project team documented and controlled change requests centrally. As part of our testing, the audit team examined the change request process by examining a random sample of 42 change requests from all change requests processed as part of Release 2. Our objective was to determine the degree of compliance with the change request policy. The audit team’s examination revealed some minor errors, but no systematic or material problems. We brought them to the attention of the GCMS project team and they are currently being addressed.
Top of Page
7.2 Controls over data integrity and sensitive information
The objective of the audit for this area was to assess the appropriateness of controls over data integrity, and those for ensuring that sensitive information is protected. The areas examined as part of this audit included the appropriateness of procedures and plans in the following areas: data conversion and integrity of converted data, and procedures to protect sensitive information. We present our observations and, as appropriate, recommendations for each area, as follows.
7.2.1 Data conversion and integrity of converted data
With respect to data conversion and the integrity of converted data, the audit team reviewed a number of elements, as discussed below.
Procedures for ensuring the accuracy of data
During Deployment 1, 18 percent of Citizenship data were converted. As a direct result of the lessons learned from Deployment 1, the data conversion strategy for the remaining Citizenship data was revisited and refocused to better determine which information is to be converted, the amount to be converted and the accessibility of data not converted, and to ensure that the approach is well integrated with data conversion plans for Release 2. The project has also indicated that CIC and the CBSA have embarked on a “data cleansing” exercise to improve the consistency of converted data.
The remaining 82 percent of Citizenship data will undergo further analysis to determine which data elements will be selected for conversion “as is” to the new GCMS system. The CIC IMIT maintenance group has assumed overall responsibility for this activity. The time lines for converting these select summary Citizenship data are expected to mirror those for Release 2. We understand that any Citizenship data that are not converted will be accessible through a data warehouse, with summarized details contained in the new GCMS “summary tab.”
For Release 2, a revised strategy that both minimizes the amount of data that will be transferred to GCMS and provides users with alternative mechanisms to access unconverted legacy data has been developed. Specifically, key data elements for all clients, and all data related to active cases, will be sourced from those legacy systems considered to contain the most accurate and up-to-date information, and imported into GCMS. Enough summary data stemming from inactive cases will also be imported into GCMS to handle reports, interfaces and real-time decisions. The general approach is to determine which information to convert, how much to convert, and how best to access data not converted. As with the Citizenship data noted above, the intent is to utilize the “summary tab” format for data elements not converted.
The remaining legacy system data will be moved to a data warehouse and made directly accessible to well-trained and mentored power users through ad hoc query tools. The information needs of other users will be met with custom reports. At this point in time, no conversion is scheduled to occur for Release 2 data until early 2006.
The plans and controls we reviewed in this area should assist the GCMS project team as it prepares for Release 2.
Control of access to data
The audit team expected to find that access to data was controlled to avoid risks of unauthorized changes. Access controls for sensitive information are authorized by delegated managers on a system-by-system basis (e.g., SAP, SMS, LiveLink and ClearCase). The audit team did not find clearly documented access procedures or protocols with respect to GCMS project-sensitive information assets as a whole. Interviews indicated that the sensitive information was the data. Thus, it is our understanding that the project team will be developing access controls and detailed procedures as part of the data conversion process for Release 2 in early 2006. These activities should assist the GCMS project team in protecting the GCMS data.
As part of our testing, the audit team examined the GCMS project team security clearance process by examining a random sample of 15 security clearances for individuals on the project to determine: 1) whether the individuals had the appropriate security clearance when they began working on the project; and 2) whether the individuals were still with the project and whether their security clearance was still valid. Our examination found the process was functioning as intended.
7.2.2 Protection of sensitive information
In the context of requirements management, the audit team reviewed the following elements.
Safeguarding of information
A preliminary Privacy Impact Assessment (PIA) for GCMS was completed as part of the original Effective Project Approval submission of January 2002, and a full PIA was submitted to the Office of the Privacy Commissioner (OPC) in February 2004. The GCMS project team had provided three periodic status updates to the OPC: one in August 2004, another in January 2005, and the third in March 2005.
In addition to the above activities, the project team has clearly defined and documented the procedures for records management, instructing project team members on what to keep and what to dispose of. Furthermore, the project team is actively engaged in discussions with CIC and Archives Canada over policies relating to the retention and destruction of data.
The audit team found that the GCMS project team’s measures for safeguarding information were adequate.
Top of Page
8. GCMS Project Action Plan
Recommendation Response Action Plan Responsibility Completion Date
1. The GCMS project should ensure the consistent application of project management planning standards (GCMS Schedule Management and Development Standards) developed by the PMO. Agreed.
Reinvigorate project management practices and enforce consistent execution and application of inputs, outputs and standards:
Integration activities initiated to confirm plan linkages and close gaps
Project management planning and quality assurance practices being enhanced to include validation, monitoring and escalation
Project Executive GCMS January 2006
2. Objective and quantifiable performance criteria related to the principal vendor should be identified and discussed with the principal vendor. These metrics should be chosen so that they adequately reflect contractor performance. Agreed.
Improve vendor management at the macro level by implementing qualitative measures to monitor overall performance, and at the micro level by strengthening contract management practices:
In discussions with the principal vendor to confirm the criteria to provide an overall qualitative assessment of vendor performance
In discussions with PWGSC to refine task authorizations to include identification of specific conditions of satisfaction and performance
Train managers on the use of task authorizations and conditions of satisfaction and performance to enhance performance management
Update executive dashboard reporting to report on principal vendor relationship and contract performance
Project Executive GCMS January 2006
Top of Page
Appendix: GCMS Audit Interviews List
Assistant Deputy Minister, Policy and Program Development
Assistant Deputy Minister, Centralized Services Delivery and Corporate Services
GCMS Project Leader
GCMS Associate Director
Executive Director, GCMS IMIT
Executive Director, Business Requirements and Functional Design
Executive Director, GCMS/National Case Management System
R2 Delivery Manager
Executive Director, PMO
GCMS Finance
Project Control Officer, PMO
Engineering Process Manager
Risk and Issue Manager
GCMS Training
Principal vendor, Subcontract Management
CIC, DG, Finance Branch
CIC, Chief Information Officer
Director, B.C./Yukon Region
Vice-President, CBSA
Manager, Information Management, CBSA
Analyst, TBS
Chief Information Officer Branch, TBS
GCMS Independent TBS Monitor
Consultant, GCMS Audit Program
Policy Officer, GCMS OVIT, CPC Sydney
Team 3 Leader, CPC Sydney
Program Support Officer, CPC Sydney
PMO, GCMS Master Project Scheduler
GCMS Change/Problem Management Project Leader
Project Control Officer, UAT
Project Control Officer, IMIT
Procurement and Contracting Officer, CIC Contracting
Personnel Security Supervisor, CIC Security
GCMS Project Planner
Manager, GCMS Data Conversion
GCMS Project Lead, Data Conversion
GCMS COTS Data Transformation Services Specialist, Data Conversion
CIC Information Management, Archiving
GCMS Project Office Manager, PMO
Officer, CIC St. Clair, Toronto Region (D1)
Supervisor, CIC St. Clair, Toronto Region (D1)
A/Operations Manager, – GTA West, Toronto Region (R2)
PRRA Coordinator, Toronto Region (R2)
System-Under-Development Audit of the Global Case Management System
Final Report
Internal Audit and Disclosures Branch
Citizenship and Immigration Canada
November 2005
Executive Summary
1. Introduction
1.1. Background
2. Audit Objectives
3. Audit Scope
4. Audit Criteria
5. Methodology
6. Conclusions
7. Observations and Recommendations
7.1. Project management processes
7.2. Controls over data integrity and Sensitive information
8. GCMS Project Action Plan
Appendix: GCMS Audit Interviews List
Top of Page
Executive Summary
The Global Case Management System (GCMS) is a multi-year project that will replace several existing systems at Citizenship and Immigration Canada (CIC) and the Canada Border Services Agency (CBSA). The goal of the project, which began in fiscal 2000–01, is to provide CIC with an automated, highly integrated case management tool to support its global business network and provide enhanced end-to-end client services. When complete, GCMS will support more than 7,000 users at CIC and the CBSA in some 125 points of service in Canada (offices, call centres, processing centres, citizenship courts and ports of entry) and more than 100 missions abroad.
In September 2004, the first phase of the GCMS project was implemented with the “go live” of the Citizenship component (Deployment 1). Although this component was successful from a technological standpoint, some initial operational difficulties were encountered, but they have largely been resolved. Consequently, when the project sought to amend the Effective Project Approval (EPA) in the spring of 2005, an audit of the GCMS project was requested as part of the process.
Our audit sought to examine the progress made since GCMS’ Deployment 1 and the next phase of GCMS. We examined the project management processes and controls for ensuring both data integrity and the protection of sensitive information. Our observations cover the period from September 2004 to October 2005.
Our overall conclusions are that the GCMS project management processes are sound and that they will increase the likelihood of success by ensuring the existence of an effective management control framework. Furthermore, the audit team found that adequate controls are in place to support data integrity and protect sensitive information assets.
The following report provides the audit team’s observations and recommendations, as well as the GCMS project team’s responses and action plan to address these recommendations.
Top of Page
1. Introduction
The Internal Audit and Disclosures (IAD) Branch of Citizenship and Immigration Canada (CIC) was asked to conduct a focused system-under-development audit of the Global Case Management System (GCMS) project. The IAD audited the project management processes and the control measures in place for data integrity and sensitive information assets, to assess their adequacy for the successful implementation of the project.
The audit was conducted in accordance with the Government of Canada’s policy on internal audits, as well as auditing standards prescribed by the Institute of Internal Auditors.
1.1 Background
GCMS is a multi-year project that will replace several of the business systems of CIC and the Canada Border Services Agency (CBSA) with an integrated, case management-based set of applications and components that will support both organizations’ client operations.
The GCMS project received Preliminary Project Approval on March 1, 2001, with an initial budget of approximately $195 million (T.B. 828697). In view of the assessed level of risk and the scale of the project, CIC was instructed to manage GCMS as a Major Crown Project and to confirm the viability of using a “commercial off-the-shelf” (COTS), Client Relationship Management package to provide much of the required case management functionality.
Because of a variety of factors, many of which were outside the control of CIC, the selection of a vendor and the awarding and signing of the contract were delayed by nine months. As a result of this delay, it was April 2003 before the project team had access to the COTS product and technical experts required to build and deploy GCMS. Later in 2003, an amended Effective Project Approval was approved by Treasury Board ministers (T.B. 830875) and the budget increased to approximately $200 million.
Since the project was granted preliminary approval, it has had to deal with a number of external and internal events that have directly affected the project and that have necessitated adjustments to the project implementation strategy. These events include the following:
the events of September 2001, which forced countries to reconsider the tools they were using to deal with threats while, at the same time, not unduly impeding the legitimate movement of people;
the continued and growing interest in Canada as a destination for immigrants and temporary residents (tourists, foreign students, temporary workers);
the implementation of the most comprehensive reforms to the immigration legislation in more than 30 years, including the introduction of the secure permanent resident card; and
the creation of the Canada Border Services Agency.
The project has conducted several independent readiness reviews (July 2000, June 2001 and December 2002) as well as two Independent Verification and Validation reviews (August 2002 and July 2003) to address various issues and concerns about the project. The results of these reviews were provided to the Treasury Board Secretariat (TBS) through formal reporting (September 2002) and in regular monthly reports.
Treasury Board ministers also requested that the Chief Information Officer Branch (CIOB) of TBS engage the services of an independent third-party monitor to report on the progress of the project, with particular attention to the vendor relationship. Two monitors were engaged in October 2003. In January 2004, they provided their report to TBS. A subsequent follow-up report was provided in March 2005.
Reports prepared by the external Treasury Board monitor highlighted the fact that senior management at both CIC and the CBSA enthusiastically and unequivocally supported the GCMS project. The report also noted that there was a high degree of awareness and engagement within CIC and the CBSA user communities, as demonstrated in periodic surveys designed to “take the pulse” of the two organizations.
The project has also served as a catalyst for business transformation. In identifying, defining and formalizing business requirements, the project has had to re-examine existing business processes and to develop consensus among departmental users. The result is a more consistent, standardized set of practices aimed at promoting greater uniformity, increased efficiency and improved client-centred service across both organizations.
CIC’s business includes over 1.5 million client transactions a year, approximately two-thirds of which occur overseas in environments that can be characterized as challenging in terms of their complexity and the local infrastructure.
When completed, GCMS will support more than 7,000 users at CIC and the CBSA in some 125 points of service in Canada (offices, call centres, processing centres, citizenship courts and ports of entry) and more than 100 missions abroad.
GCMS is a critical element of a larger, client-centred service and business transformation vision that seeks to maximize the benefits associated with its citizenship and immigration programs in the years to come. By integrating the business functionality and information that now reside within more than a dozen of CIC’s legacy systems built over the last three decades, the project aims to eliminate redundant data, improve the consistency and quality of that data and contribute to streamlining operations. It should also allow both CIC and the CBSA to further improve the delivery of their programs and services.
In September 2004, the first phase of the GCMS project was implemented with the “go live” of the Citizenship component (Deployment 1). The project continues to meet the expectations of both users and IT experts and has proven to be a powerful tool that can be customized to meet the specific requirements of CIC and the CBSA.
Although the Citizenship component was successful from a technological standpoint, some initial operational difficulties were encountered. These were attributable to data conversion problems, a steep learning curve for users, and a higher than anticipated number of system defects. These problems are being addressed as the project moves toward the next phase of implementation. The system has become more stable, and production is approaching predeployment levels.
As part of Deployment 1, the project team engaged in several lessons-learned exercises. These exercises identified and documented areas for improvement such as accountabilities, requirements development, communications, project planning and scheduling, resource planning, and project management control and tracking. Learning from these exercises, the GCMS project team prepared detailed action plans and began to adopt new practices.
In the spring of 2005, the GCMS project team began preparing a submission to Treasury Board ministers through TBS for amendments to the Effective Project Approval (EPA) for the GCMS project. As part of this process, the TBS expressed a need for an audit of the project.
The type of audit that TBS considered most suitable was a system-under-development (SUD) audit. In general, TBS was looking for a focused SUD audit that would provide reasonable assurance that the GCMS project was on a solid foundation as it moved forward.
In June 2005, the terms of reference were developed in preparation for this audit, which began in July 2005.
Top of Page
2. Audit Objectives
The audit had two objectives:
to assess the soundness of the project management processes; and
to determine the appropriateness of controls to ensure data integrity and protect sensitive information assets.
3. Audit Scope
The audit scope covered the progress made since GCMS’ Deployment 1. We examined the project management processes and controls for ensuring both data integrity and the protection of sensitive information. Our observations cover the period from September 2004 to October 2005.
We did not examine processes or controls in effect prior to September 2004. Furthermore, an examination of the contract between the Government of Canada and the principal vendor for the GCMS project was outside the scope of this audit.
4. Audit Criteria
We derived our assessment criteria from a number of reasonable and credible standards of performance and control relevant to this engagement. The sources of criteria for this audit included the following:
Project Management Body of Knowledge (PMBOK) – the Project Management Institute’s (PMI) standards document for project management knowledge and practices;
Control Objectives for Information and Related Technology (COBIT) – the IT Governance Institute’s framework for IT practices;
Various Treasury Board Secretariat policies (e.g., TBS Enhanced Management Framework);
Various CIC departmental policies; and
The Management Control Framework for CIC Audit Activities.
The criteria for the two audit objectives were as follows:
The project management processes increase the likelihood that the project will succeed by ensuring that an effective management control framework exists.
Project planning techniques are sound.
Project monitoring, tracking and control techniques are appropriate.
There are processes to enable the project to achieve its overall business objectives.
There are improvements to the project management processes applied for Release 2.
The contract and subcontract management processes are adequate.
Project requirements management techniques are adequate.
Change management procedures are adequate.
Techniques for integration management applied with respect to the project management office’s (PMO) linkages to project authorities (e.g., Treasury Board) are sound.
The controls to ensure data integrity and the protection of sensitive information are adequate.
Procedures to support data conversion and to verify and maintain the integrity of converted data are adequate.
Procedures for protecting sensitive information have been incorporated into the project and are adequate.
Top of Page
5. Methodology
The audit began in July 2005, and we carried out our examination from July to October 2005. This involved the following activities.
interviews;
reviewing documentation;
reviewing best practices against the practices of recognized organizations such as the PMI ’s PMBOK and the IT Governance Institute’s COBIT;
a detailed examination of selected project management processes; and
audit testing.
As part of the audit, we interviewed 44 people, including key executives, GCMS project members and other key stakeholders. We also obtained relevant documentation to validate the information gathered during interviews (see appendix A for a list of interviewees).
The audit team also examined documentation and conducted detailed testing to determine whether specific processes or controls were in place. In addition, the team conducted a literature review of readily available best practices to identify any gaps in the project management processes or controls under review.
Detailed testing included the following identified areas:
Detailed master schedule;
Exception reporting;
Change management;
Risk management;
Dashboard reporting;
Task authorization/task authorization requests validation;
NON-COTS contracts;
Asset purchases; and
Security clearances.
6. Conclusions
We have concluded that the GCMS project management processes are sound, and that they will increase the likelihood of the project’s success by ensuring that an effective management control framework exists. Furthermore, the audit team found that adequate controls were in place to ensure data integrity and to protect sensitive information.
Top of Page
7. Observations and Recommendations
7.1 Project management processes
The audit objective was to assess the soundness of the project management processes in place and their contribution to the project’s success by ensuring that an effective management control framework exists. The project management processes that we examined as part of this audit included the following areas: project planning; project monitoring, tracking and control; contract and subcontract management; requirements management; integration management; and change management. The following paragraphs present our observations and, as applicable, recommendations for each area.
7.1.1 Project planning
In the context of project planning, the audit team reviewed a number of elements. Our observations in this area are as follows.
The business case
Under project planning, we expected to find that the GCMS project team had developed a business case that was comprehensive and complete, and that identified and justified the GCMS project and indicated how it related to CIC and government-wide priorities. At a minimum, a business case should identify:
the opportunities for improvement;
the benefits of such an undertaking;
the technical solution at a high level;
indicators against which to measure any improvement in program performance;
costs (up-front, direct ongoing, indirect and potential user/client costs); and
the risks, and steps to be taken to manage those risks.
The audit team found that the GCMS business case had covered all significant elements.
Project charter
We found that the GCMS project team had a project charter that defined the scope of the project. It also set out the overall project management framework and standards to be used. The project charter also clearly defined who the internal and external stakeholders were, described the roles and responsibilities of the members of the project team and established the overall project governance structure (e.g., accountabilities and project authorities). Furthermore, we found that the project charter had set out the management commitments (human and financial resources, time frame and deliverables).
The team found that the GCMS project charter had been continuously updated throughout the life cycle of the project to reflect various project changes, and that all significant elements had been addressed in the project charter.
Project approvals
The audit team expected to find that the project had received the appropriate approvals from not only CIC, but TBS as well. The TBS preliminary project approval was granted on March 1, 2001 (T.B. 828697). Effective project approval was granted on January 31, 2002 (T.B. 829502) and amended on October 9, 2003 (T.B. 830875). During this audit, CIC sought and obtained a further amendment to the EPA. Finally, we noted that the GCMS project senior management within CIC has demonstrated, and continues to demonstrate, strong support for the project.
Top of Page
Project plan
The audit team expected to find a project plan that included:
how system requirements would be managed;
how negotiation of resource commitments would be handled;
a software life cycle with predefined stages;
the key processes and deliverables for the project;
authorization to start work and key team members assigned;
estimate of the size and complexity of the software;
estimate of product resource requirements;
detailed schedules;
commitment and approvals from within CIC; and
reporting requirements.
The master project schedule (MPS) is the primary mechanism used for project planning within GCMS. Our examination showed that the MPS contained all the details listed above and that it reflected the GCMS project charter.
The MPS is maintained by the PMO with input from individual project teams who are responsible for preparing subteam schedules that feed into the MPS. Weekly meetings are also held with the project team’s project control officers to discuss the project’s progress as it applies to their sub-schedules.
To validate the information contained within the MPS, the audit team conducted the following audit tests (using the MPS dated July 15):
tests to validate the internal consistency of reported dates in subteam schedules; and
tests to map tasks contained in the MPS back to the subteam schedules and compare reported information.
We noted that the level of detail and consistency varied between the different subteam schedules. Consequently, the audit team had difficulty mapping tasks contained in the MPS back to subteam schedules and found that the level of difficulty in mapping tasks varied between sub-schedules. Without a better link between the MPS and the sub-schedules, the project cannot be sure that all material items are included in the MPS. Given the MPS’ importance in planning, we are concerned that stronger links do not exist. The project is aware of these facts and was in the process of revalidating all schedules at the time of the audit. We were informed that efforts are also being made to realign the schedules to improve the links between the MPS and its sub-schedules.
In our view, the project needs to ensure that all project team members commit to uniformly applying the project management planning standards. We believe that doing so will improve the quality and consistency of the information feeding into the MPS. Strengthening of the MPS should also strengthen the timeliness of information and decision making within the project.
Recommendation 1
The GCMS project should ensure the consistent application of project management planning standards (GCMS Schedule Management and Development Standards) developed by the PMO.
GCMS management response
Agreed.
Project reviews
The audit team expected to find a process for reviewing and revalidating the major planning deliverables (the business case, project charter and plan) for the project. The project team has completed a thorough review and analysis of Deployment 1 activities and has applied lessons learned to develop a revised approach and plan for completing the GCMS.
The project has carried out the following reviews:
Independent Readiness Reviews (July 2000, June 2001 and December 2002);
Independent Verification and Validation Reviews (August 2002 and July 2003);
Independent TBS Monitor Reviews (February 2004 and March 2005);
GCMS Project Team—Gates 1 and 2: Lessons Learned (November 2003 and December 2004); and
GCMS Review of Project Response to Deployment 1: Lessons Learned (June 2005).
The audit team found that the level of project review was generally adequate for the size and scope of the project.
Top of Page
Governance and organizational structure
The project has the following governance committees:
Change Control Board (CCB)
Design Review Board (DRB)
Design Review Team (DRT)
Executive Committee (ExCOM)
Executive Management Team (EMT)
Group of Approvers (GA)
Integrated Management Committee (IMC)
Operational Validation and Implementation Team (OVIT)
Project Management Board (PMB)
Risk Management Board (RMB)
Senior Project Advisory Council (SPAC)
Senior Level Advisory Committee (SLAC)
We examined the terms of reference for each of the governance committees and found them to be complete. A review of agendas, supporting materials and records of decisions indicated that they had met the requirements of government policies and CIC standards. The audit team attended (at random) various governance meetings (e.g., SLAC, RMB, CCB, PMB and EMT) and found that each was chaired at the right level and that the composition of the committees was appropriate, given their terms of reference.
We also found that the project had an appropriate organizational structure in place. Key positions have been identified with clearly documented and communicated roles and responsibilities. As one of its primary objectives, the IMC is dedicated to identifying and resolving staffing issues for the project in a timely fashion.
Overall, we found the governance and organizational structure of GCMS to be comprehensive and in compliance with applicable government directives (e.g., the Major Crown Projects Policy and the TBS Enhanced Management Framework).
Process plans
The audit team expected to find process plans for the following:
quality assurance;
risk management;
communications;
performance management;
HR management;
procurement management;
requirements management;
IM/IT;
testing;
roll-out; and
training.
Overall, the audit team found that plans had been completed for each of the areas noted above. Process plans are continually being improved as demonstrated by the incorporation of lessons learned from Deployment 1. Significant strides have been made in bolstering areas identified in lessons-learned exercises. In particular, these include:
a formal accountability framework (approved April 14, 2005);
the appointment of a deputy project manager;
tightened project management processes (e.g., for HR management);
improved management reporting (e.g., improvements to the GCMS dashboard); and
stronger control over project resourcing (e.g., IMC process).
Top of Page
7.1.2 Project monitoring, tracking and control
In the area of project monitoring, tracking and control, the audit team reviewed a number of elements, as noted below.
Data collection and tracking requirements
The audit team expected to find that baseline estimates for activities set out in the project plan (including milestones) had been established and accepted by the project team. The GCMS project team has identified, defined and documented major baseline activities, milestones and data. Updates to the project plan approach and associated deliverables reflect the intent of the business case. The evolving project charter is consistent with these adjustments to baseline activities.
The audit team found the project tracking and oversight process to be consistent with TBS Enhanced Management Framework principles.
Planned vs. actual performance
Processes have been identified to:
track actual performance;
compare actual with planned performance (e.g., track deviations from plan);
identify when and what adjustments will be needed; and
determine the impacts of any adjustments and the subsequent changes needed (e.g., additional resources) to close the gap.
The audit team completed a review of the minutes of the steering committee (discussed below) and other meetings to determine if performance was reviewed regularly and adjustments made as required. The audit team found that planned vs. actual performance was reviewed regularly and that the reasons for any deviations from planned performance were documented and explained concisely. However, ownership of these deviations is weak. This is discussed in greater detail in section 7.1.3 (Monitoring of service delivery by a third party).
Financial tracking
The audit team expected to find that financial tracking was being conducted and that the results were reported. In particular, we expected the following:
Budgeted vs. actual and forecast is analysed, documented and reported on a regular basis.
Financial data are accurate, complete and produced in a timely fashion.
High-level financial tracking information from the PMO is consistent with information at the project team level.
Financial coding structure is consistent with CIC standards.
Financial authorities as established in the project charter and CIC are applied.
Dedicated PMO staff who work in collaboration with the CIC Finance Branch track and control GCMS financial information. The audit team conducted a sample testing of asset acquisitions to test for compliance with the Financial Administration Act. Our examination in this area found that the processes were functioning as intended. Furthermore, a review of detailed financial data and supporting documentation for the month of July 2005 found that internal financial controls were adequate.
Tracking performance and results
The audit team expected to find that performance and results were documented and communicated within the PMO and were continuously adjusted to reflect project improvements and the tracking of all issues through to completion. In particular, we expected to find the following:
Project logs and status reports are prepared and updated regularly.
A distribution list of concerned parties is maintained.
Issues and problems are identified, documented and tracked to closure.
There is a clearly understood process for resolving problems when they arise.
The audit team reviewed various logs (e.g., issue, decision, action and risk) and found them to be adequate.
In addition, as part of its audit, the audit team tested one period’s GCMS dashboard (May 2005) to assess the accuracy of the information it contained. Our examination revealed that the dashboard reflected the information available within the project.
Top of Page
Risk management
The audit team tested the risk management tracking processes (May–June 2005) to ensure that the process worked as intended and was in accordance with generally accepted practices.
The audit team expected to find a process for ensuring that project risks were identified, assessed and documented and, in particular, that the following elements were in place for the GCMS project:
a systematic risk assessment framework;
a risk assessment approach that allows for regular updates;
risk assessment procedures that identify both external and internal risk factors;
procedures for monitoring changes in risks;
risk assessment documentation;
risk mitigation strategies identified and documented; and
involvement of stakeholders in identifying project risks.
We found that the GCMS project had mechanisms that identified, documented and tracked risks. An initial risk assessment was completed in 2001 and has been regularly updated throughout the life of the project. A comprehensive Threat and Risk Management Assessment was completed in December 2004. Overall, we found that the monitoring of risks associated with the project was well documented and tracked.
Meetings
The audit team expected to find that meetings were held on a regular basis to make decisions and to review the project’s status, performance and risks.
The audit team found that since Deployment 1, team meetings have been focusing on providing a better understanding of the status, costs, time frames and risks associated with the project. A review of agendas, minutes and records of decisions (ROD) for ExCOM, SLAC, SPAC, PMB and EMT conducted by the audit team found meetings were taking place, major activities were being updated and risks were being tracked and followed up on.
Steering committee
The audit team expected to find that a steering committee had been established and was operating and, in particular, that individuals at the appropriate levels had been identified and were participating in the committee.
Our audit found that the steering committee was functioning as intended. ExCOM is the senior departmental forum for monitoring progress, reviewing issues and making decisions regarding alternative approaches and priorities for the GCMS project. As such, it serves as GCMS’ steering committee. ExCOM meets on a monthly basis. Steering committee members include:
Deputy Minister (Chair)
ADM, Policy and Program Development
ADM, Centralized Services Delivery and Corporate Services
ADM, Operations
ADM, Strategic Directions and Communications
Assistant Deputy Attorney General
DG, Human Resources
The GCMS project team has also formed the Senior Level Advisory Committee to provide additional perspective and practical advice to the GCMS Executive Management Team. SLAC meets on a monthly basis. Its members are:
Project Sponsor – ADM, Policy and Program Development
ADM, Centralized Services Delivery and Corporate Services
ADM, Operations
Project Leader, DG, Business Solutions
DG, IMTB
Executive Director, IM/IT
Executive Director, PMO
Executive Director, Business Requirements and Application Functional Design
Executive Director, Deployment, User Acceptance Testing (UAT) and Training
Delivery Manager
Chief, GCMS Finance
IndependentmMonitor, TBS
VP, Enforcement, CBSA
Director, Science Innovation and Technology, CBSA
The audit team reviewed the minutes and presentations of ExCOM and SLAC and found that they had documented key decisions and directions. Roles and responsibilities, together with terms of reference, have been clearly defined and documented for each committee.
Top of Page
Communication strategy/plan
We expected to find that a communication strategy/plan had been developed and implemented to communicate the project’s results and progress to external stakeholders.
We found that the GCMS project team used the following key communication tools to reach its various audiences:
the GCMS electronic newsletter – “Spotlight on GCMS”;
Taking the Pulse – employee surveys;
CIC’s internal website – “CIC Explore”;
GCMS portal;
regional visits;
executive staff meeting: weekly GCMS topical briefings;
regular status updates to ExCOM, SLAC, SPAC;
TBS Monthly Operational Reports (MOR); and
GCMS dashboard.
The project regularly provides timely updates to all stakeholders (internal and external). We understand that the GCMS project team, as part of its activities for Release 2, has incorporated specific communication tasks into the project management plan to prepare users for Release 2. The goal of these tasks is to bring users to a stage where they are committed to the project and ready for the changes it will bring.
Process for managing documents
The audit team expected that the GCMS project team would have a sound document management process. We found that the GCMS project maintained records as part of the project’s historical records. Examples include:
versions of documents (e.g., Business Case, Project Charter);
logs (decision logs, issue logs, risk logs, change logs);
risk assessments; and
minutes, RODs and presentations.
The project team uses electronic tools (LiveLink – GCMS Portal and ClearCase) to electronically house documents relating to the project. The audit team found that the GCMS project team had a sound document management process and that it was functioning as intended.
Top of Page
7.1.3 Management of contracts and subcontracts
In examining this area, the audit team reviewed software acquisition plans, whether the experience of staff was adequate to support software acquisitions, the control of consultants, monitoring of service delivery by a third party, the independence of the PMO and principal vendor, and the extent to which the use of subcontractors was documented and controlled. Our observations in these areas are as follows.
Software acquisition plans
Plans for contracting and acquiring software include:
the acquisition strategy;
the method for soliciting bids;
constraints related to acquisition;
acquisition resource requirements;
roles and responsibilities;
cost and schedule estimates;
contract tracking, oversight and evaluation plans;
acquisition risk management; and
life cycle support.
The audit team found that the software acquisition plans for the GCMS project followed CIC’s and Public Works and Government Services Canada’s (PWGSC) standard procurement policies. In particular, CIC has a number of software licences that it has made available to the GCMS project. Documented procedures also exist for acquiring additional software. The approval process includes the project as well as the Information Management and Technologies Branch within CIC.
In addition, most of the procurement of hardware and software to establish the production environment had been completed, and the domestic production environment was operating in a high-availability, fail-over configuration mode. Upgrades to Foreign Affairs Canada’s (FAC) international network (MITNET) and server/desktop environment (SIGNET) were proceeding within the parameters required by GCMS. Project team members continue to work closely with FAC in efforts to minimize the amount of infrastructure required for the GCMS Release 2.
The original procurement strategy (November 2001) has been updated to reflect the project’s software requirements. In general, the audit team found that the software acquisition plan included all significant elements.
Staff experience
The audit team expected to find that experienced acquisition, contracting and management staff were supporting the software acquisition for the project team. In particular, we expected that key staff would have appropriate knowledge and expertise in:
evaluating the technical aspects of the proposed system;
CIC information management and information technology (IMIT) requirements (architecture, interfaces);
federal and CIC contracting requirements, costing methodologies and tools; and
end-user requirements.
The GCMS project team had a contracting expert from PWGSC assigned to provide advice. In addition, the project team had support from CIC internal contracting resources to facilitate the contracting process. IMTB staff with experience in evaluating and assessing technical solutions have been assigned to the project. Accordingly, we found that the experience of the acquisition, contracting and management staff assigned to the GCMS project to support software acquisitions was adequate.
Top of Page
Control of consultants
The audit team expected to find policies and procedures for controlling the activities of consultants assigned to the GCMS project in order to protect the organization’s information assets. In particular, we expected that the GCMS project team would ensure that:
contractual requirements are developed, managed and maintained;
terms and conditions are standardized and reviewed by legal counsel;
contractual commitments are traceable and verifiable;
cost and schedule estimates are independently reviewed; and
security screening of key contractors is performed in conformity with CIC policy.
The audit team found that over time, the project team had instituted processes (e.g., time reporting) to better manage its contractors. In particular, the audit team tested the exception-reporting process to ensure that overtime reporting for contractors was complete and that appropriate authorities had approved them. Our examination in this area indicated that this process was functioning as intended.
Monitoring of service delivery by a third party
The audit team expected to find that a process for monitoring service delivery by a third party had been set up to ensure that the terms of contractual agreements continued to be met. In particular, we looked at whether:
actual costs and schedules were compared to planned;
issues were documented and tracked until closure;
services or products were reviewed prior to financial settlements;
acceptance criteria were documented;
periodic reviews were carried out;
an independent audit of contractor operations was done; and
monitoring was consistent with the type of contract awarded.
The audit team tested the contracting process to ensure that it was functioning according to CIC/PWGSC policies by examining a sample of seven NON-COTS contracts to determine whether they were sufficiently detailed. Our examination in this area showed that the process was functioning as intended, but that some areas needed improvement. For example, we noted instances in which particular goods or services were to be delivered, but there was no reference to a delivery date beyond the contract end date. Including delivery dates for deliverables in the contract will allow the project to better understand and monitor progress toward targets. We advised the GCMS project team of these weaknesses and were assured that they were being addressed.
The audit team also expected to find that performance measures would be in place to assess the performance of the principal vendor. We noted that the main reporting tool, the GCMS dashboard, had been refocused since Deployment 1 to provide more information on the status of the project. However, it does not report on one key aspect of the project: vendor performance. We acknowledge that the project team is working with the vendor to identify and define appropriate criteria or indicators for measuring the vendor’s performance. We support these efforts and make the following recommendation.
Recommendation 2
Objective and quantifiable performance criteria related to the principal vendor should be identified and discussed with the principal vendor. These metrics should be chosen so that they adequately reflect contractor performance.
GCMS management response
Agreed.
Independence of the project management office and principal vendor
We expected to see the appropriate degree of independence needed to ensure an arm’s-length relationship between the PMO and the principal vendor.
Our review of a sample of seven NON-COTS contracts indicated that declarations of conflict of interest had been completed by all. We also reviewed the roles and responsibilities of key staff in the PMO to determine whether they were at arm’s length with the vendor. The audit team found that the GCMS project management office was staffed with individuals who were independent of the principal vendor.
Documentation and control of the use of subcontractors
The audit team expected to find the use of subcontractors to be documented and controlled and, in particular, that documented guidelines or procedures existed for the evaluation and use of subcontractors for the project.
We carried out a test of task authorization (TA)/task authorization requests (TARs) to determine whether it was working according to policy. An examination of 65 TA/TARs showed that COTS/NON-COTS were being monitored for start and end dates, and that clearly defined roles and responsibilities were set out for each person who joins the project. Appropriate authorities requested and approved these resources. Overall, the audit team found that the documentation and control of subcontractors was functioning as intended.
Top of Page
7.1.4 Requirements management
In examining this area, the audit team reviewed a number of elements, as discussed below.
System requirements and specifications
The audit team expected to find that the PMO had defined procedures and plans to ensure close liaison with system users in developing the design specifications, and to ensure that these specifications would meet the users’ requirements. In particular, we looked at whether:
project management had developed and approved policies regarding requirements management;
policies and procedures ensured that requirements had been documented;
the process for developing requirements had been defined and had involved users, project management and team members and key IMIT personnel;
business requirements had been clearly defined prior to acquisition;
plans were in place to link requirements to design components, test plans and contract management procedures; and
requirements had been reviewed and approved by project management, user groups and senior management.
The audit team found that the GCMS project team had a documented requirements management plan that had been signed off by the appropriate project authorities. The user community has been actively involved in developing the business requirements. In particular, the GCMS project team has actively engaged representatives from across the Department and the CBSA to define business requirements and design specifications (e.g., Group of Approvers, OVIT, Design Review Team, and Subject Matter Experts).
In trying to identify, define and formalize business requirements, it has been necessary to re-examine and rationalize processes in three major business domains (i.e., immigration facilitation, immigration control and integration), and to gain consensus among users in the Department. This approach has resulted in delays in finalizing the business requirements and obtaining final approvals and sign-offs. The project team is aware of this and believes that this approach is warranted. It believes that this approach will result in a more consistent and standardized set of business practices, including greater uniformity, increased efficiency and improved service across CIC.
Processes to ensure that requirements are met
Requirements are documented and available to all project members. In addition, requirements are linked to the overall design of the GCMS solution, including components, testing and contract management. The audit team expected to find that requirements had been analysed for completeness, understandability, feasibility and consistency and, more importantly, that specifications had considered:
the design of the process for importing data from existing systems to GCMS;
the definition and documentation of input requirements;
the definition of interfaces;
user-machine interfaces;
the definition of processing requirements;
the definition of output requirements;
internal control and security requirements; and
functional and operational requirements.
The audit team found that documentation on the requirements was available. However, it is controlled to ensure that information is safeguarded from unwanted changes. The final approved documents are provided to team members through the ClearCase document management tool. The audit team did not determine if the requirements existing at the time of our audit had been adequately analysed and linked to design components or test plans. We understand that the GCMS project team, as part of its activities for Release 2, will develop test plans that will cover those requirements.
Processes to manage changes to requirements
The audit team expected to find that changes to requirements were handled through a controlled process whereby user representatives and project team members negotiated and agreed upon any changes. The audit team found that changes to requirements had been documented and reviewed by the appropriate functional areas and user community representatives for completeness and operational approval. The process for managing changes to requirements was functioning as intended.
Top of Page
7.1.5 Integration management
In the context of integration management, the audit team reviewed several areas as discussed below.
Linkage to project authorities
The audit team expected to find that the project management office had formal reporting structures in place that would provide key decision makers with the information they need to make informed decisions. We also expected that the project would have mechanisms to incorporate feedback from key stakeholders such as other departments and TBS. In particular, we noted the following:
MOR and GCMS dashboards provide details on the technical status of the project to the Chief Information Officer Branch of the TBS.
An independent TBS monitor provides regular updates on the status of the project to Treasury Board and CIC executives.
Meetings are held every two weeks with the TBS analyst on the program side.
Quarterly meetings of the SPAC involve cross-departmental representation to address GCMS issues that may affect other government departments. SPAC consists of representatives from 16 federal departments and agencies.
7.1.6 Change management
The audit team reviewed a number of elements relating to change management, as follows.
Change management procedures
Effective change management is a vital aspect of the overall governance process and is critical to controlling the scope of a project. In the case of GCMS, the change management process consists of two types of requests:
Change requests – changes that result in a change to approved project or product deliverables; and
Problem reports – issues that address a defect or deviation from the approved functional baseline.
The project team had a clearly defined process for approving change requests that involved higher levels of management authorization than problem reports. The procedures for change requests for the GCMS project include:
categorizing and prioritizing requests;
detailing how urgent requests are to be handled;
updating procedures for keeping requestors informed;
allowing for assessment of the effects of the change; and
identifying the authorities required before the changes can be approved.
The audit team found that the process, procedures and change log were functioning as intended. Documentation indicates that the process is controlled and centralized.
Requests for changes
The audit team expected to find that change requests were monitored and recorded. In particular, we expected that change-control logs would be used to record all change requests.
The audit team found that two people managed the process for the GCMS project team. They are responsible for monitoring the life cycle of all change requests (i.e., opening, logging, tracking and closing them). The project team had recently improved the process to address the closing of change requests within acceptable time frames. In particular, the project team has focused on identifying high-priority change requests with an emphasis on addressing and closing these issues.
Documentation of approved changes
Change requests are used as a means of documenting or recording decisions and ensuring that representatives from not only the GCMS project team but, as appropriate, other key stakeholders, understand the impact of the potential change. Such tracking enables GCMS to:
provide members with a way to record the history of changes made to the GCMS product;
control proposed changes that may affect the scope, schedule or budget;
ensure that all changes adhere to the Configuration Management and Document Management processes; and
communicate approved changes to the GCMS team as a whole.
The audit team found that the project team documented and controlled change requests centrally. As part of our testing, the audit team examined the change request process by examining a random sample of 42 change requests from all change requests processed as part of Release 2. Our objective was to determine the degree of compliance with the change request policy. The audit team’s examination revealed some minor errors, but no systematic or material problems. We brought them to the attention of the GCMS project team and they are currently being addressed.
Top of Page
7.2 Controls over data integrity and sensitive information
The objective of the audit for this area was to assess the appropriateness of controls over data integrity, and those for ensuring that sensitive information is protected. The areas examined as part of this audit included the appropriateness of procedures and plans in the following areas: data conversion and integrity of converted data, and procedures to protect sensitive information. We present our observations and, as appropriate, recommendations for each area, as follows.
7.2.1 Data conversion and integrity of converted data
With respect to data conversion and the integrity of converted data, the audit team reviewed a number of elements, as discussed below.
Procedures for ensuring the accuracy of data
During Deployment 1, 18 percent of Citizenship data were converted. As a direct result of the lessons learned from Deployment 1, the data conversion strategy for the remaining Citizenship data was revisited and refocused to better determine which information is to be converted, the amount to be converted and the accessibility of data not converted, and to ensure that the approach is well integrated with data conversion plans for Release 2. The project has also indicated that CIC and the CBSA have embarked on a “data cleansing” exercise to improve the consistency of converted data.
The remaining 82 percent of Citizenship data will undergo further analysis to determine which data elements will be selected for conversion “as is” to the new GCMS system. The CIC IMIT maintenance group has assumed overall responsibility for this activity. The time lines for converting these select summary Citizenship data are expected to mirror those for Release 2. We understand that any Citizenship data that are not converted will be accessible through a data warehouse, with summarized details contained in the new GCMS “summary tab.”
For Release 2, a revised strategy that both minimizes the amount of data that will be transferred to GCMS and provides users with alternative mechanisms to access unconverted legacy data has been developed. Specifically, key data elements for all clients, and all data related to active cases, will be sourced from those legacy systems considered to contain the most accurate and up-to-date information, and imported into GCMS. Enough summary data stemming from inactive cases will also be imported into GCMS to handle reports, interfaces and real-time decisions. The general approach is to determine which information to convert, how much to convert, and how best to access data not converted. As with the Citizenship data noted above, the intent is to utilize the “summary tab” format for data elements not converted.
The remaining legacy system data will be moved to a data warehouse and made directly accessible to well-trained and mentored power users through ad hoc query tools. The information needs of other users will be met with custom reports. At this point in time, no conversion is scheduled to occur for Release 2 data until early 2006.
The plans and controls we reviewed in this area should assist the GCMS project team as it prepares for Release 2.
Control of access to data
The audit team expected to find that access to data was controlled to avoid risks of unauthorized changes. Access controls for sensitive information are authorized by delegated managers on a system-by-system basis (e.g., SAP, SMS, LiveLink and ClearCase). The audit team did not find clearly documented access procedures or protocols with respect to GCMS project-sensitive information assets as a whole. Interviews indicated that the sensitive information was the data. Thus, it is our understanding that the project team will be developing access controls and detailed procedures as part of the data conversion process for Release 2 in early 2006. These activities should assist the GCMS project team in protecting the GCMS data.
As part of our testing, the audit team examined the GCMS project team security clearance process by examining a random sample of 15 security clearances for individuals on the project to determine: 1) whether the individuals had the appropriate security clearance when they began working on the project; and 2) whether the individuals were still with the project and whether their security clearance was still valid. Our examination found the process was functioning as intended.
7.2.2 Protection of sensitive information
In the context of requirements management, the audit team reviewed the following elements.
Safeguarding of information
A preliminary Privacy Impact Assessment (PIA) for GCMS was completed as part of the original Effective Project Approval submission of January 2002, and a full PIA was submitted to the Office of the Privacy Commissioner (OPC) in February 2004. The GCMS project team had provided three periodic status updates to the OPC: one in August 2004, another in January 2005, and the third in March 2005.
In addition to the above activities, the project team has clearly defined and documented the procedures for records management, instructing project team members on what to keep and what to dispose of. Furthermore, the project team is actively engaged in discussions with CIC and Archives Canada over policies relating to the retention and destruction of data.
The audit team found that the GCMS project team’s measures for safeguarding information were adequate.
Top of Page
8. GCMS Project Action Plan
Recommendation Response Action Plan Responsibility Completion Date
1. The GCMS project should ensure the consistent application of project management planning standards (GCMS Schedule Management and Development Standards) developed by the PMO. Agreed.
Reinvigorate project management practices and enforce consistent execution and application of inputs, outputs and standards:
Integration activities initiated to confirm plan linkages and close gaps
Project management planning and quality assurance practices being enhanced to include validation, monitoring and escalation
Project Executive GCMS January 2006
2. Objective and quantifiable performance criteria related to the principal vendor should be identified and discussed with the principal vendor. These metrics should be chosen so that they adequately reflect contractor performance. Agreed.
Improve vendor management at the macro level by implementing qualitative measures to monitor overall performance, and at the micro level by strengthening contract management practices:
In discussions with the principal vendor to confirm the criteria to provide an overall qualitative assessment of vendor performance
In discussions with PWGSC to refine task authorizations to include identification of specific conditions of satisfaction and performance
Train managers on the use of task authorizations and conditions of satisfaction and performance to enhance performance management
Update executive dashboard reporting to report on principal vendor relationship and contract performance
Project Executive GCMS January 2006
Top of Page
Appendix: GCMS Audit Interviews List
Assistant Deputy Minister, Policy and Program Development
Assistant Deputy Minister, Centralized Services Delivery and Corporate Services
GCMS Project Leader
GCMS Associate Director
Executive Director, GCMS IMIT
Executive Director, Business Requirements and Functional Design
Executive Director, GCMS/National Case Management System
R2 Delivery Manager
Executive Director, PMO
GCMS Finance
Project Control Officer, PMO
Engineering Process Manager
Risk and Issue Manager
GCMS Training
Principal vendor, Subcontract Management
CIC, DG, Finance Branch
CIC, Chief Information Officer
Director, B.C./Yukon Region
Vice-President, CBSA
Manager, Information Management, CBSA
Analyst, TBS
Chief Information Officer Branch, TBS
GCMS Independent TBS Monitor
Consultant, GCMS Audit Program
Policy Officer, GCMS OVIT, CPC Sydney
Team 3 Leader, CPC Sydney
Program Support Officer, CPC Sydney
PMO, GCMS Master Project Scheduler
GCMS Change/Problem Management Project Leader
Project Control Officer, UAT
Project Control Officer, IMIT
Procurement and Contracting Officer, CIC Contracting
Personnel Security Supervisor, CIC Security
GCMS Project Planner
Manager, GCMS Data Conversion
GCMS Project Lead, Data Conversion
GCMS COTS Data Transformation Services Specialist, Data Conversion
CIC Information Management, Archiving
GCMS Project Office Manager, PMO
Officer, CIC St. Clair, Toronto Region (D1)
Supervisor, CIC St. Clair, Toronto Region (D1)
A/Operations Manager, – GTA West, Toronto Region (R2)
PRRA Coordinator, Toronto Region (R2)
