The Standard

Change Management

Introduction

Change Management Purpose and Objectives from ITIL:

“The purpose of the change management process is to control the lifecycle of all changes, enabling beneficial changes to be made with minimum disruption to IT services.

The objectives of change management are to:

  • Respond to the customer’s changing business requirements while maximizing value and reducing incidents, disruption and re-work.
  • Respond to the business and IT requests for change that will align the services with the business needs.
  • Ensure that changes are recorded and evaluated, and that authorized changes are prioritized, planned, tested, implemented, documented and reviewed in a controlled manner.
  • Ensure that all changes to configuration items are recorded in the configuration management system
  • Optimize overall business risk – it is often correct to minimize business risk, but sometimes it is appropriate to knowingly accept a risk because of the potential benefit.”

Due to the fact that ITIL has not defined, in Version 2011, a definitive change process flow, but instead listed examples of it, we like to recommend a general flow based on our operational experience.

Change Approvals

ITIL is unspecific about the planning state of a change when it comes to approvals. Our experience shows that there are two types of changes:

  • Changes of minor nature which may or may not need oversight by a change management team.
  • Important, risky and complex changes which absolutely require the full attention of an operational change management team. The change management team must be aware of the planning of such changes and so it is our recommendation that no CAB (Change Advisory Board) approval is granted for changes without a specific plan, including dates, times and teams. Essentially the change must be fully planned prior to approval and release of the change. After approval only minor changes may be applied without revoking the approval.

Vendor Inclusion in the Approvals Process:

As technology implementations get more and more complex we recommend the involvement of key relevant vendors in change validations prior to the approval of a change. There are generally two types of involvements possible:

  • Vendor reviews a planned change and provides feedback for improvements to the plan
  • Vendors create the change plan for a service provider/customer.

Run Books:

For Major, Significant or Special Management Focus Changes we recommend writing a so-called “Run Book” for the change. This is a document which outlines the activities of the change against a time-line. Additionally, it outlines at which points decisions are taken like Start, Point of no return, Back out, etc. It also outlines which teams or team members (if these are unique critical resources) perform which task.

Usually Service Management Tools have the capability of inputting all of this data, but it can become very difficult to read and review by upper management, which is required to provide approval and accept any documented risks.

Build and Test Environments

ITIL recommends that regular Change Management also reviews changes for build and test environments. Our recommendation is that:

where there is a specific requirement in build & test environments, organizations should perform regular change management i.e. legal requirements, certification requirements, etc.

Where there is no specific requirement, our recommended best practice is to perform full change management for productive or production related systems/services only. In this case the development and test organization must perform change oversight in these environments themselves.

For production related changes the Change Manager and CAB generally need to request documentation and/or declarations that the change has been successfully tested in build or test environments. Where changes are required on production systems, to enable development to continue their work, a production change is always required.

Example: A test environment requires a new port to be opened on the production firewall. The firewall is in production and therefore opening the port requires a change, although the test environment is non-productive.

Change Management Process Overview

 

 

Central Change Advisory Boards (CCAB)

The Central Change Advisory Board is the central authority regarding all changes within the service provider’s organization. It approves all major and significant changes and supervises the implementation. It sets global and company-wide standards for successful changes and implementations, and ensures that those standards are being observed. In an international organization all local business units and subsidiaries need to work to the same standards to ensure a similar quality level globally.

Change Advisory Board (CAB)

The Central Change Advisory Board is supported by local Change Advisory Boards in the local business units which manage standard and less critical changes as part of their daily operations. It operates under the same standards and procedures as the Central Change Advisory Board. The knowledge transfer between CCAB and CAB is ensured by a virtual community of all local CABs and the CCAB. The community also works on the continuous improvement of the agreed change management process and discusses and agrees on necessary adaptations.

ITIL outlines 5 levels of change authorities. We suggest combining Levels 1 and 2 and giving responsibility for both to the outlined CCAB organization:

  • No authorization, eventually release through CAB – Level 5
  • CAB – Level 4
  • CAB – Level 3
  • CCAB after CAB approval – Level 2
  • CCAB after Safeguarding approval – Level 1

Emergency Changes (E-Changes) are approved by the assigned Manager on Duty .

Activities:

  • Check if RFC is reasonable, feasible and compliant.
  • Identify correct entry point, which operations unit is responsible.
  • Ask for prepared change models.
  • Provide information about cause/target of RfC.
  • Consider preparation time of the change.
  • Identify responsible change manager.

Points to consider:

  • Operational teams need to be involved.
  • Configuration items /service chain to be altered.
  • Customers affected by the change request.
  • Avoid requests on short notice.
  • A single request might raise several changes.

Activities:

  • Use change models (pre-approved, pre-released).
  • Be aware of the risks related to the change; identify and plan counter measures.
  • Assign activities to necessary teams (tasking), consider using a 4-eyes-principle.
  • Identify change type and risk type.
  • Consider mandatory approver and lead times.
  • Identify configuration items.

Important change parameter:

  • Criticality of the changed configuration item (CI) and service impact will determine the change type.
  • Change type and complexity will determine the risk type.
  • Major changes might require a hyper-care phase after the change to detect side effects, or previously undetected instabilities, early. During hyper-care, systems must be closely observed to detect issues immediately.

Suggested lead times:

  • Preapproved Changes: No lead time
  • CAB: minor changes (depends on organisation, usually between 1- 4 days).
  • CCAB: major &significant changes 10 days
    40 day announcement in advance for high changes.

Make sure that all important change information is available before entering the next phase.

The Service Provider and its vendors/suppliers must agree on a joint run book for complex or high risk changes. The accountability lies with the service provider, even though the partners can be engaged to the required level to create or modify the run book,

Verification by partners (strongly recommended for complex or high-risk changes):

  • Complexity of the changes determines the required lead time to prepare the run book, but in general expect 5 -10 working days (verify your specific contract status with relevant suppliers/partners).

Run book creation by partners (some partners offer this service)

  • Agree with your supplier’s preparation timelines (incl. procurement & delivery) and reflect agreed lead times within change planning.
  • Accountability cannot be outsourced! The requester of the runbook has full accountability and therefore should verify the created runbook for validity/errors/fit for purpose.

During the planning phase all possible outcomes and effects of the change need to be considered. Risks are to be taken into consideration and actions for risk mitigation should be defined. The enhanced planning phase with risk management and optimized timing is one of the main aspects of change management according to the Zero Outage Industry Standard.

The change execution must be planned in detail prior to authorisation, which means:

  • The exact time and resource plan must exist prior to the request for change approval.
  • No updates to the plan after the approval, except for very minor updates.
  • The individual responsible for the change planning should be the change coordinator and not the change manager.
  • The service provider and his vendors/suppliers should agree on a joint run book that is mandatory for complex or high-risk changes.
  • An adequate back out plan must be defined for significant, major and special focus changes. This plan must include Go/No Go decision points prior to execution.
  • The “point of no return” must be clearly defined and no action shall be performed prior to a formal “Go” decision prior the “point of no return”. The responsible Manager needs to be named prior to the Go / No Go decision.
  • The 4-Eyes Principle must be included and visible for the change tasks.

Provide a change calendar for all organisational units with a six month forecast, where frozen zones, blocked maintenance windows and changes with their criticality are listed. Additionally, there might be a central forecast list of special focus changes.

Additionally, a maintenance window forecast for the next year should exist at least during Q3 of the current year.

Improved Change Forecast

Implementing those tools and guidelines will ensure resource availability and long-term resource planning. It will reduce conflicts within different changes in the same environment and create enhanced management attention for critical and important changes. Additionally, the service provider can enable prioritization of incident solution and security enhancement.

Activities:

  • Change coordinator:
    • ensures mandatory approvals.
    • Present the changes to mandatory change advisory boards.
    • Get Service Delivery Manager/customer approval. If necessary, consider both general commitment and formal approval.

Points to consider:

  • Major and significant changes should be reviewed and approved by the CCAB.
  • Minor changes must be approved by (regular) the CAB.
  • Standard changes do not need any approval.
  • Emergency changes and short lead time changes should be approved by the Manager on Duty (MoD) or similar responsibility.
  • MoD approval is the default escalation procedure for exceptional requirements (short term, outside business hours) while Change Management should handle everything inside business hours.
  • Special focus changes (usually on management request) must be safeguarded and approved at top management level.

Change Types

We recommend that the following change types and the corresponding review and approval rules are used for larger organizations:

Special Focus

Special Focus changes are special in every way. There is no formal definition. Selected management is allowed to name a change as “Special Focus” change to ensure the highest attention level for an organization.

The reasons can vary from extremely high risk changes, never been done before, to touching a critical system at an inconvenient time which may result in severe damages or issues if it is going wrong. It may also be simply at the heart of top management through political reasons that nothing shall go wrong with this change or any other reason.

A Special Focus change will be reviewed and safeguarded with highest organizational attention to:

  • Focussing on known hot spot areas or areas core to the customer or overall company.
  • Risk-oriented safeguarding of changes.
  • Extended supplier support in planning & implementation.
  • Extended management support (top management).
Major

Authorization for Major Changes which is provided by the Central Change Advisory Board (CCAB), together with the business owner. There is always a Safeguarding for Major changes.

Significant

Authorization for Significant Changes is provided by the Central Change Advisory Board (CCAB) after the corresponding local CAB has approved the change. There might be a Safeguarding for Significant changes.

Minor

Authorization for minor changes is provided by the Change Advisory Board (CAB). In large organisations there are usually multiple “local” CAB which approve minor changes within their responsibility.

Standard

Standard changes are pre-authorized and therefore need no CAB approval. We suggest two types of standard changes.

  1. Pre-approved Standard Change
    These Changes are pre-approved, but not pre-released. A local CAB still need to review and release the standard change prior execution. Therefore, these changes cannot be performed immediately.
  2. Pre-released Standard change
    These changes can be performed immediately, no further activity is required by a CAB.

 

Change Safeguarding is a detailed review for high risk changes or special focus changes. An organisation must define who is entitled to request special focus changes. All further calls and organisational activities to review the change and gain defined management approvals is done by the assigned Change Manager. The review of such changes is performed by a CCAB with special participants, due to mandatory top management participation. The entitled requester of the special focus change must at least attend the change review.

Change safeguarding should align to the following recommended process:

Initial Safeguarding Call (mandatory)

  • Review the safeguarding documentation.
  • Check, that the run book is complete, that all relevant milestones are listed and that the milestones follow-on correctly.
  • Check, that all the people responsible are named in the document.

Follow-up Safeguarding Calls (optional)

  • Check the settlement of the conditions from the previous call.
  • Review of the changes made to the change and run book.
  • Confirm the participants invited to final safeguarding call.

Final Safeguarding Call (mandatory)

  • Final presentation of the change to top management and CCAB.
  • Final decision for the change (denial/approval) by top management and CCAB.

Our recommendation is that a change coordinator should ensure the change implementation.

The coordinator is a person close to the technical/business topic that is being changed.

Activities:

  • Pre- and post-health checks.
  • Ensure that all involved people are available (supplier, customer, …) prior to the start.
  • Document test results and obtain customer sign-off, if needed.
  • React to issues raised by the test result (e.g. back-out, …).
  • Do not pass the “point of no return” without MoD involvement.
  • Secure the implementation of the defined 4-Eyes Principle for critical tasks.
  • Secure CMDB update.

Points to note:

  • Only start in a healthy environment.
  • Stick to the (CAB approved) plan.
  • In case of a critical deviation react and escalate appropriately and early.
  • Consider a hyper-care phase after the change execution, if there have been unexpected outcomes.
  • Perform adequate tests following a change back-out to ensure the environment(s) work properly.

4-Eye-Principle (4EP)

This is a core component of error free services and must be implemented during the planning of changes as well as the change reviews and execution. This means that important change activities are only performed if a second pair of eyes have reviewed it to mitigate the “human error” factors. For example, if a key DNS change is to be performed at 03:00 am, which, if completed incorrectly, may impact all of your communications – it makes sense to ensure that the command submitted is absolutely correct, having been verified by a second independent person prior submission.

Change Planning

  • Review of the run book using the 4-eyes-principle (4EP).
    • Within the change team
    • With the relevant supplier
  • 4EP tasks need to be included in the planned change tasks.
  • Confirmation of the 4EP task must be provided by an individual other than the main implementer.
  • The additional colleague must not be sitting next to the implementer. Experience shows that if colleagues sit together in front of the screen, they often make the same mistake
  • The person should verify the command, activity result, prior submission or after execution to be fully correct and complete.

Change Implementation

  • Mandatory for all major and significant changes.
  • Critical steps within change implementation need be verified by a second person with similar skills.
  • The success of each critical step must be verified with the 4EP implementer.
  • This is valid for the back-out as well.

 

Activities:

  • Update CI in CMDB.
  • Closure comments must provide information about problems, deviations, reduced scope and potentially more.
  • Transfer results back to project plan.
  • Define follow-up activities, lessons learned, best practice.
  • Provisioning of implementation feedback to change requester that his desired functionality is now in operation.

Points to consider:

  • In case of deviation to change planning perform a Post Implementation Review (PIR).
  • RfC – Closing must not be done before completion of enhanced monitoring phase.