process training industry best practice change management
agenda 1 2 3 4 5 6 7 our goal purpose of change management general remarks general role description change management flow activities, responsibilities & entities best practices: change forecast 8 9 10 11 12 13 14 best practices: create & review rfc best practices: assess & evaluate rfc best practices: change planning best practices: vendor change review best practices: authorize change best practices: change implementation best practices: review & close change record
1 our goal itil is the de facto standard for the whole it industry. however the itil release cycles vary from 5-10 years and do not reflect the speed of business changed today. our association focuses on cross-company collaboration to define best practices based on our design and operational experiences. our goal is to provide practical industry best practices beyond the itil standards. our chapter for today : change management!
2 purpose of change management based on itil change management purpose and objectives from itil: “the purpose of the change management process is to control the lifecycle of all changes, enabling beneficial changes to be made with minimum disruption to it services. the objectives of change management are to: respond to the customer’s changing business requirements while maximizing value and reducing incidents, disruption and re-work. respond to the business and it requests for change that will align the services with the business needs. ensure that changes are recorded and evaluated, and that authorized changes are prioritized, planned, tested, implemented, documented and reviewed in a controlled manner. ensure that all changes to configuration items are recorded in the configuration management system. optimize overall business risk – it is often correct to minimize business risk, but sometimes it is appropriate to knowingly accept a risk because of the potential benefit.”
3 general remarks change approval & vendor inclusion general change approval remark: itil is unspecific about the planning state of a change when it comes to approvals. our experience shows that there are on one side changes of minor nature which may or may not need supervising by a change management team. on the other side there are important, dangerous and complex changes which absolutely require full attention of an operational change management team securing these. to enable adequate work, the change management team must know the planning of the changes. therefore our recommendation is, that no cab approval is granted on changes without a specific plan including dates, times and used resources. essentially the change must be fully planned prior approval and re- lease of the change. after approval only minor adjustments may be applied without revoking the approval. general vendor inclusion remark: as technology implementations get more and more complex, we recommend to involve key vendors into important change validations prior approval of a change. there are generally two types of involvements possible: vendor reviews a planned change and provides feedback for an improved planning vendors create the change plan for a service provider/customer
3 general remarks run book & change models general run book remark: at least for major and significant changes we recommend to write a so called run book for the change. this is a document which outlines the activities of the change in a time overview. additionally it outlines at which points decisions are taken like start, point of no return, backout, etc. it also outlines which teams or team members (if these are a unique critical resource) perform which task. usually service management tools also provide the capability to input all this data, but usually larger, more complex changes become very difficult to read, review and execute general change model remark: there are two types of change models: pre-approved change models these models are validated by the change management team and cab for use. however changes created from that model still require a one by one release from change management or cab depending on their change type. pre-release change models these models are pre-approved and pre-released. this means these are so safe to be performed that no issues and impacts should happen for years to come. such changes can be performed immediately. incidents based on these change models must be reviewed very carefully.
3 general remarks build and test environments general build- and test-environments remark: itil defines that regular change management also reviews changes for build and test environments. our recommendation is: where there is a specific requirement in build & test environments, organizations should perform regular change management i.e. legal requirements, certification requirements, etc. where there is no specific requirement, our recommended best practice is to perform full change manage- ment for productive or production related systems/services only. in this case the development and test organization must perform change oversight in these environments themselves. for production related changes the change manager and cab generally need to request documentation and/ or declarations that the change has been successfully tested in build or test environments. where changes are required on production systems, to enable development to continue their work, a production change is always required. example: a test environment requires a new port to be opened on the production firewall. the firewall is in production and therefore opening the port requires a change, although the test environment is non-productive. changes on non-productive environments require a different handling within the development organization, but the interfaces of productive- and test- changes have to be considered.
4 general role description service delivery manager (sdm) operations manager problem manager problem analyst main interface to customer allocates all customer requests into the internal organization. responsible for contract fulfill- ment regarding delivery and commercial aspects plans, controls and is respon- sible for the adequate provision of one or a number of services has operative responsibility for the whole lifecycle of the problems main contact for the sdm provides the technical inventory and service data for the part of the service ensures required approvals (root cause analysis results, resolution actions, etc..) leads the root cause analysis including detailed documen- tation and involvement of external parties ensures of all necessary resources and of all required information sources change manager change coordinator change approver change requester analyzes and reviews rfc and change planning coordinates changes and is responsible for change planning is responsible for safeguarding of complex and high risk changes approves changes; leads cab ensures mandatory approvals (e.g. presents the changes to man- datory change advisory boards ) is responsible that change implementation is compliant with approved change planning e.g. sdm, cabs, customer, etc. reviews the change planning approves or denies the change e.g. customer, product manager, project manager, sdm, etc. requests a change with a de- scription of the cause and the target as a rfc (request for change) change implementer incident manager lead incident manager (lim) manager on duty (mod) is responsible for activities as part of a change implementation ensures processing of incidents during their entire life cycle within service level or operational level agreements § overall responsibility for the execution of the inm process responsible to handle escalations steers as a project manager the incident solution is responsible for the document- ation and handover to problem management supports incident solution with dedicated customer or service line knowledge ensures that incident solution will be pushed forward 24x7
5 change management flow based on itil standard deployment request (where the deployment process is tried and tested) role generic change request (from a template) create and review rfc initiator requested asses and evaluate rfc workorders change management ready for decision zero outage recommendation: exact time plan and team must exist prior approval and not during change implementation change planning change planning vendor change review authorize and schedule change change authority scheduled coordinate change implementation change management implemented review and close change record closed zero outage recommendation: change coordinator is responsible for change implementation *includes build and test the change adapted from itil framework: example of a process flow for standard deployment request s m c n i n o i t a m r o f n i n o i t a r u g fi n o c d n a e g n a h c e t a d p u zero outage recommendati- on: no updates of planning after authorisation except minor updates
6 activities, responsibilities & entities overview within change management process open rfc create and plan change vendor change review authorize change implement change change record and rfc review and closing, ci update change coordinator analyses the rfc plans all activities needed for the change implementation describes the implementation tasks aligned with the change implementer coordinates the change planning for complex changes change manager checks and analyses the rfc finds a change coordinator reviews the change planning • specifies needed approvals change requester e.g. customer, project manager, sdm, operations responsible, etc. resquets a change with a description of the cause and the target as a rfc (request for change) vendor review change planning checks depen- dencies verify technical solution verify sequence of steps change approver e.g. sdm, cab´s, customer, etc... reviews the change planning checks dependencies approves or denies the change change coordinator coordinates the previously planned activities during change implementation triggers ci updates change implementer implements change validates expected results • confirms successful implementation of assigned activities change coordinator checks result of the change • confirms successful change implementation change requester checks change result against the rfc • confirms correct result
6 activities, responsibilities & entities cab & ccab central change advisory boards (ccab): the central change advisory board is the central authority regarding all changes within the service providers organization. it approves all major and significant changes and supervises the implementation. it sets global and company wide standards for successful changes and observes these standards. in an international organization all local business units and subsidiaries need to act with the same standards to ensure a similar quality level globally. change advisory board (cab): a central change advisory board is supported by a number of local change advisory boards in the local business units which manage minor and standard changes within the daily operations. it works with the same standards and procedures as the central change advisory board. the knowledge transfer between ccab and cab is ensured by a virtual community of all local cabs and the ccab. the community works on the continuous improvement of the agreed change management process, discusses and agrees on necessary adjustments. cab and ccab are the key entities within the change management process.
6 activities, responsibilities & entities cab & ccab authority change advisory board (cab) authority: itil outlines 5 levels of change authorities. we suggest to combine level 1 and 2 and give responsibility for both to be organized by the outlined ccab organization: ccab – level 1 + 2 cab – level 3 change manager – level 4 local authorization – level 5 special case emergency change: emergency changes (e-change) have to be approved by the assigned mod. communications, decisions and actions communications, escalation for rfcs, risks, issues change authority level of risk / impact level 1 central cab + business executive board high cost / risk change – requires decision from executives level 2 central cab + it management board or it steering group change impacts multiple services or organizational divisions level 3 cab change which only affects local or service group level 4 change manager low-risk change level 5 local authorization standard change
7 best practices: change forecast during planning phase all possible outcomes and effects of the change need to be considered. risks are to be taken under consideration and actions for risk mitigation should be defined. the enhanced planning phase with risk management and optimized timing is one of the main aspects of change management according to the zero outage industry standard: tools for improved change forecasts. provide a change calendar for all organizational units with 6 months forecast, where frozen zones, blocked main- tenance windows and changes with their criticality are listed. additionally there might be a central forecast list of special focus changes. additionally a maintenance window forecast for the next year should exist at least during q3 of the previous year. implementing those tools and guidelines will ensure resource availability and long term resource planning. it will reduce conflicts within different changes on the same environment and create enhanced management attention on critical and important changes. additionally the service provider can enable prioritization of incident solution and security enhancement. a solid change calendar is the foundation for change forecasts and planning.
8 best practices: create & review rfc standard deployment request (where the deployment process is tried and tested) role generic change request (from a template) create and review rfc initiator requested asses and evaluate rfc workorders change management ready for decision zero outage recommendation: exact time plan and team must exist prior approval and not during change implementation change planning change planning vendor change review authorize and schedule change change authority scheduled coordinate change implementation change management implemented review and close change record closed zero outage recommendation: change coordinator is responsible for change implementation *includes build and test the change adapted from itil framework: example of a process flow for standard deployment request s m c n i n o i t a m r o f n i n o i t a r u g fi n o c d n a e g n a h c e t a d p u zero outage recommendati- on: no updates of planning after authorisation except minor updates
8 best practices: create & review rfc important points for rfc creation 7 r‘s of change management : 1. who raised the change? 2. what is the reason for the change? 3. what is the return required from the change? 4. what are the risks involved in the change? 5. what resources are required to deliver the change? 6. who is responsible for the build, test and 7. what is the relationship between this change and other changes? implementation of the change? activities: document all needed information in a rfc record (7 r’s of change management). identify correct entry point, which operations. unit is responsible. ask for prepared change models. provide information about cause/target of rfc. consider preparation time of the change. check if rfc is reasonable, feasible and compliant. identify responsible change manager. points to consider: operational teams to be involved. configuration items /service chain to be altered. affected customers of the change request. avoid requests on short notice. a request might raise several changes.
9 best practices: assess & evaluate rfc standard deployment request (where the deployment process is tried and tested) role generic change request (from a template) create and review rfc initiator requested asses and evaluate rfc workorders change management ready for decision zero outage recommendation: exact time plan and team must exist prior approval and not during change implementation change planning change planning vendor change review authorize and schedule change change authority scheduled coordinate change implementation change management implemented review and close change record closed zero outage recommendation: change coordinator is responsible for change implementation *includes build and test the change adapted from itil framework: example of a process flow for standard deployment request s m c n i n o i t a m r o f n i n o i t a r u g fi n o c d n a e g n a h c e t a d p u zero outage recommendati- on: no updates of planning after authorisation except minor updates
9 best practices: assess & evaluate rfc activities and parameter to consider activities: use change models (pre-approved, pre-released). be aware of the risks related to the change; identify and plan counter measures. assign activities to necessary teams (tasking), consider using a 4-eye-principle. identify change type and risk type. consider mandatory approver and lead times. identify config items. important change parameter to be considered for the evaluation: criticality of the changed config item (ci) and service impact shall determine the change type. change type and complexity to determine the risk type. urgency. major and significant changes might require a hypercare phase after the change to detect side effects or previously undetected instabilities. during hypercare systems are watched to detect potential issues immediately. suggested lead times for approval preapproved changes: no lead times. cab: minor changes (depends on organization, usually between 1- 4 days). ccabs: major & significant changes 10 days and 40 days announcement in advance. make sure that all important change information is available before entering the next phase. .
9 best practices: assess & evaluate rfc important change information overall change description what to enter critical question 1 2 3 4 5 6 7 8 description and documents details and comprehensible description of the action. attach all necessary documents is the information described in a comprehensible way (for an outsider as well)? • are important documents attached? cause / target of change description about the cause and the target of the change. is the information described in a comprehensible way? • does also non-experts understand it? service impact impact: description how does implementing the change affect the customer´s service? • is the change window agreed with the customer? • is there downtime necessary (online / offline procedure)? configuration items (ci) • if more than one ci is affected as part of the change, all cis are to be entered. are the cis marked as valid and is the criticality of the cis considered in priority calculation? • does the number of cis match the complexity? risk of omission / implementation what risk exist, if the change may not be carried out and which risk exists during the implementation. (stating “no risk” is not valid!) • if no risk of omission, why we should implement? • does the description match with specified risk type? back-out short and precise description of how the previous status can be reconstituted in the event of an error. (entries like “none” are not valid) is the needed time for all activities and a possible back-out included in the change window? • when / who will decide on and do the back-out? tasks and validation • enter tasks for actions and implementers. description, how to test the successful implementation of the change. • are tasks valid with the overall time planning? • who will validate the implementation? • is there a validation by the customer planned? approvals add all necessary approval groups. add customer approval. • add “dedicated cab” approval if priority > minor in which way the customer approval is documented?
9 best practices: assess & evaluate rfc urgency – definition of emergency – and short lead time change emergency change short lead time change a short lead time change cannot fulfill the regular the process due to several valid reasons. lead time as defined in an emergency change has to be carried out immediately in order to resolve an incident (without an incident there cannot be an emergency change). emergency changes are meant to avoid massive damages and have to be authorized at least by an mod (manager on duty) and the emergency change advisory board. late change planning does not justify an emergency change. changes without a relation to a incident situation emergency changes have to be approved by mod and in case customers are affected by the impacted customer. short lead time changes have to be approved by mod and in case customers are affected by the impacted customer. the detailed documentation in an emergency change ticket may be done retrospectively when time is of the essence (latest at the following business day). however, a change number and short abstract of the change must exist for gaining approval. the emergency change must be related with the incident a short lead time change is always completely documented before change implementation. the change requires a documented justification for the short lead time. definition not in scope approval documentation
10 best practices: change planning standard deployment request (where the deployment process is tried and tested) role generic change request (from a template) create and review rfc initiator requested asses and evaluate rfc workorders change management ready for decision zero outage recommendation: exact time plan and team must exist prior approval and not during change implementation change planning change planning vendor change review authorize and schedule change change authority scheduled coordinate change implementation change management implemented review and close change record closed zero outage recommendation: change coordinator is responsible for change implementation *includes build and test the change adapted from itil framework: example of a process flow for standard deployment request s m c n i n o i t a m r o f n i n o i t a r u g fi n o c d n a e g n a h c e t a d p u zero outage recommendati- on: no updates of planning after authorisation except minor updates
10 best practices: change planning in addition to itil we recommend to complete the full change planning prior to change approval. that means: the exact time and resource plan must exist prior request for change approval. no updates of planning after the approval except minor updates. responsible for the change planning should be the change coordinator and not the change management. the service provider and his vendors/suppliers should agree on a joint run book that is at least mandatory for complex or high risk changes. an adequate backout plan must be defined for at least significant, major and special focus changes. this plan must include go/no go decision points prior execution. if available, a point of no return must be defined. activities after the point of no return must not be started without a formal go decision. the responsible owner for making the go/no go decision must be defined during change planning. this might be a standard team or group within the company. 4-eyes principle must be included and visible from the change tasks.
11 best practices: vendor change review standard deployment request (where the deployment process is tried and tested) role generic change request (from a template) create and review rfc initiator requested asses and evaluate rfc workorders change management ready for decision zero outage recommendation: exact time plan and team must exist prior approval and not during change implementation change planning change planning vendor change review authorize and schedule change change authority scheduled coordinate change implementation change management implemented review and close change record closed zero outage recommendation: change coordinator is responsible for change implementation *includes build and test the change adapted from itil framework: example of a process flow for standard deployment request s m c n i n o i t a m r o f n i n o i t a r u g fi n o c d n a e g n a h c e t a d p u zero outage recommendati- on: no updates of planning after authorisation except minor updates
11 best practices: vendor change review the inclusion of vendors and suppliers is also an important part of change planning. the responsibility lies with the service provider, even though the partners can be engaged to the required level to create or modify the run book. that means that there are generally two types of involvements possible: run book verification by partners (strongly recommended for complex or high risk changes): complexity decides over required lead time to prepare the run book, but in general expect 5 -10 working days (verify your specific contract status with involved suppliers/partners). run book creation by partners (some partners offer this service) agree with your suppliers preparation timelines (incl. procurement & delivery) and reflect agreed lead times within change planning. responsibility cannot be outsourced! the requester of the run book has full accountability and therefore should verify the created run book for validity/errors/fit for purpose.
12 best practices: authorize change standard deployment request (where the deployment process is tried and tested) role generic change request (from a template) create and review rfc initiator requested asses and evaluate rfc workorders change management ready for decision zero outage recommendation: exact time plan and team must exist prior approval and not during change implementation change planning change planning vendor change review authorize and schedule change change authority scheduled coordinate change implementation change management implemented review and close change record closed zero outage recommendation: change coordinator is responsible for change implementation *includes build and test the change adapted from itil framework: example of a process flow for standard deployment request s m c n i n o i t a m r o f n i n o i t a r u g fi n o c d n a e g n a h c e t a d p u zero outage recommendati- on: no updates of planning after authorisation except minor updates
12 best practices: vendor change review activities, categories and approval governance activities: change coordinator ensures mandatory approvals. presents the changes to mandatory change advisory boards. gets service delivery manager/customer approval if necessary, considers both: general commitment and formal approval. change categories & approval governance major and significant changes should be reviewed and approved by ccab. minor changes have to be approved by (regular) cab. standard changes do not need any approval. emergency changes and short lead time changes should be approved by manager on duty or similar responsibility. mod approval is the default escalation procedure for exceptional requirements short term outside business hours while change management is handling everything inside business hours. special focus changes (usually on management request) have to be safeguarded and approved on top management level.
12 best practices: authorize change activities, categories and approval governance lead-time planning/approval special focus min 40/10 days few cases per year top management, ccab, operation senior management, ccab, operation major management, ccab, operation significant operation, cab minor & standard --/0-4 days rest 40/10 days 40/10 days ) y t i l a c i t i r c n o d n e p e d ( e m t i max 1% up to 4% g n i t r o p e r , t r o p u s r o n o i t a l a c s e f o l e v e l
12 best practices: vendor change review safeguarding of major changes change safeguarding: change safeguarding is a detailed review for high risk changes and special focus changes. safeguarding is organized by the responsible change manager. the review of such changes is performed by ccab with additional mandatory top management participation. this review shall secure at least: known internal hot spot areas are reviewed. core customer areas are reviewed. extended supplier support is included in planning & implementation. extended management support is secured.
12 best practices: vendor change review suggested safeguarding set-up initial safeguarding call first call to review the prepared change documentation. is the run book ok, are all relevant milestones listed? are all (management) responsible known and named? follow-up safeguarding calls optional calls to check the settlement of the conditions from the previous call. review of the changes made in the change and the run book. check of the participants invited to final safeguarding call. final safeguarding call final presentation of the change to the management and ccab. final decision for the change (denial/approval) by management and ccab. we recommend a safeguarding structure based on this 3-step call governance.
13 best adapted change flow chart standard deployment request (where the deployment process is tried and tested) practices: change implementation role generic change request (from a template) create and review rfc initiator requested asses and evaluate rfc workorders change management ready for decision zero outage recommendation: exact time plan and team must exist prior approval and not during change implementation change planning change planning vendor change review authorize and schedule change change authority scheduled coordinate change implementation change management implemented review and close change record closed zero outage recommendation: change coordinator is responsible for change implementation *includes build and test the change adapted from itil framework: example of a process flow for standard deployment request s m c n i n o i t a m r o f n i n o i t a r u g fi n o c d n a e g n a h c e t a d p u zero outage recommendati- on: no updates of planning after authorisation except minor updates
13 best practices: change implementation activities & points to consider our recommendation is that a change coordinator should ensure the change implementation. the coordinator is a person close to the technical/business topic that is being changed. activities: pre- and post health checks ensuring that all involved people are available (supplier, customer, ...) prior to start documenting test results and customer sign-off if needed reacting on issues of test results (e.g. backout, ...) do not exceed point of no return without formal go decision securing performance of defined 4-eyes principle for critical tasks securing cmdb update points to consider: only start in a healthy environment stick to the (cab approved) plan in case of critical deviation react and escalate appropriately and early consider a hypercare phase after change execution when special occurrences have happened do adequate tests after a performed change backout to ensure the environment(s) work properly again
13 best practices: change implementation 4-eye-principle (4ep) the 4-eyes-principle (4ep) is a core component of error free services and must be implemented within the planning of changes as well as in the change reviews and execution. this means that important change activities are only performed if a second pair of eyes looking over it to secure human errors do not happen. think about it, if a key dns change is performed at 3 o`clock in the morning which, if done wrong, may bring down all your communication… wouldn´t it be smarter to ensure the command submitted is absolutely correct = verified by a second independent person prior submission?
13 best practices: change implementation 4-eye-principle (4ep) during change planning & implementation 4ep during change planning: review of run book in a 4-eyes-principle (4ep) - within the change team - together with a supplier 4ep tasks need to be planned as a separate change task 4ep during change planning: mandatory for all major and significant changes. critical steps within change implementation need be verified by a second person with similar skills. the additional colleague shall not sit next to the implementer. the person shall only verify the command, activity or result prior to submission or after execution to secure the correctness. experience shows that if colleagues sit together in front of the screen, they often make the same mistake. the success of each critical step has to be verified with the 4ep implementer. this is valid for the backout as well.
14 best practices: review & close change record standard deployment request (where the deployment process is tried and tested) role generic change request (from a template) create and review rfc initiator requested asses and evaluate rfc workorders change management ready for decision zero outage recommendation: exact time plan and team must exist prior approval and not during change implementation change planning change planning vendor change review authorize and schedule change change authority scheduled coordinate change implementation change management implemented review and close change record closed zero outage recommendation: change coordinator is responsible for change implementation *includes build and test the change adapted from itil framework: example of a process flow for standard deployment request s m c n i n o i t a m r o f n i n o i t a r u g fi n o c d n a e g n a h c e t a d p u zero outage recommendati- on: no updates of planning after authorisation except minor updates
14 best practices: review & close change record activities: update ci in cmdb. closure comments have to provide information about problems, deviations, reduced scope and potentially more. transfer results back to project plan. define follow-up activities, lessons learned, best practice. provisioning of implementation feedback to change requester that his desired functionality is now in operation. points to consider: in case of deviation to change planning perform a post implementation review (pir). rfc - closing must not be done before completion of enhanced monitoring phase.
abbreviations & terms abbreviation 4ep cab ccab cmdb e-change ci itil mi mod pir rfc sdm explanation 4-eyes-principle change advisory board central change advisory board configuration management database emergency change configuration item information technology infrastructure librarytm major incident manager on duty post implementation review request for change service delivery manager