Have you heard the saying: "If you can't measure it, you can't manage it"? That's a very powerful statement. Measurement is critical to the success of every organization, and a holistic approach to monitoring and measurement must be taken. Measurements provide metrics that can be processed and then analyzed in order to provide insight into every aspect of organizational health, especially IT departments.
Is IT achieving its operational, tactical, and strategic goals and objectives? Are the IT processes being carried out efficiently and effectively, and is the output of the processes providing the desired value? Are we able to achieve the level of service agreed upon in the service level agreements (SLAs)? Are customers happy with the support being provided by the service desk? The answers to these, and many other questions, can be answered if IT is gathering the right data and presenting it in the right manner.
One would think that there would be a well-defined set of measurements that can be implemented to provide all of the required information. Unfortunately, it's not that easy. With measurements, there is no one-size-fits-all formula, and it takes a significant amount of forethought, discussion, planning, time, and resources to determine the right set of measurements for an organization. It's not enough just to measure. It's critical to measure the right things at the right time. And to make it more complex, what might be right today, may not provide value tomorrow. It's necessary to continually align and realign the measurements with the needs of the organization.
In this paper we'll take a look at why we should measure, and define the basis for what is measured, the areas that need to be measured, and the various types of measurements that should be captured.
In the past, most IT organizations focused their measurements on the underlying technology that was in the data center. Those are the servers, network devices, and applications that support the delivery of services. When a device went down, and an application failed or performance was degraded, IT worried about getting that device back up and running, so they could move on to the next technical issue.
Although that is an important aspect of Service Measurement, it is just the tip of the iceberg. It's not enough to know that a hardware component failed, or that the performance of a database is degraded. What's critical is how those issues affect the services we deliver, the customers of that service, our bottom line, and how likely we are to achieve our objectives.
IT has come a long way in the past decade and has begun to understand that it's the impact a technical issue causes to the organization that should be the driver. IT also realizes that the impact can be hard (financial) or soft (the loss of reputation or customer dissatisfaction). Most IT departments now have a customer-centric view of issues. In fact, many of the monitoring tools now have consoles that display how a particular device is impacting the delivery of services. This is a huge paradigm change, and as a result, a greater focus has been placed on ensuring proactive comprehensive holistic service measurement.
Service measurement's objective is to collect metrics, which verify in every aspect, that IT is efficiently and effectively supporting the business and helping it meet its goals and objectives. At the same time, IT is using the measurements and metrics to ensure they are doing the right things to achieve their own goals and objectives. Finally, and just as importantly, these measurements must provide insight into risks, issues, and areas needing improvement.
Ask someone why we measure, and you'll hear things like "so we can catch issues," or "so we can create reports." Theoretically, those reasons are correct, but they are really a subset of one of the four major reasons that we monitor and measure.
There should be a well-thought out underlying purpose for measurement. A good basis sets the objectives, practices, parameters, and constraints that once defined and established, will help guide the measurement process. The result of which, is to provide both IT and the business with the information to understand the health of the organization from their own unique perspective. There are a couple of different schools of thought regarding the basis of the framework, but let's take a look at the two that are widely used—the Balanced Scorecard and the Quality Management System.
The Balanced Scorecard is a management tool that provides an overall view of the health of the organization.
Although it's gone through various iterations since its inception, the Balanced Scorecard, which was developed by Dr. Robert S. Kaplan and David P. Norton as part of a research project, was broken into four separate and distinct perspectives.
Figure 1 summarizes the approach for defining measurements utilizing the Balanced Scorecard approach.
Figure 1: Vision and Strategy
A Quality Management System (QMS) is defined as "a set of coordinated activities to direct and control an organization in order to continually improve the effectiveness and efficiency of its performance." The coordinated activities are carried out as the processes of an organization, and those processes should help to ensure that we are delivering quality services to internal and external customers.
A QMS enables an organization to achieve the goals and objectives set out in its policy and strategy. It provides consistency and ensures all of the activities of the organization, beginning with the identification of customer requirements and ending with the customers' satisfaction to provide consistency, quality, and value.
A fully documented QMS will ensure that:
Once you've decided the underlying basis for the framework, it's time to build the plan for gathering metrics, determine what should be monitored and at what intervals, where that information is to be stored, whether there are dependencies, the retention strategy, and so forth. Again, the overall goal is to provide both IT and the business with the information needed to understand the health of the organization from their own unique perspective.
Although there are others, there are three main types of metrics that support the development of the balanced scorecard report and the QMS.
Component metrics provide insight into the underlying technologies residing in the data center that support the processes and delivery of services. Components include devices such as servers, routers, switches, and firewalls. It's critical that we understand how these components are performing from both a standalone perspective and as part of the overall infrastructure that's been put into place to deliver a service. When we monitor a component, we are capturing data and as long as we are storing the data each and every time a monitor runs, we can then report on the trends for the capacity, availability, and performance of that component.
Much of the alert monitoring we do is at the component level. Component metrics are also used to verify that the technologies are working within defined parameters, and if not, can be used to report issues in the environment.
These metrics are collected as critical success factors (CSFs), and key performance indicators (KPIs). Before continuing, let's take a look at the meaning of each of these.
Critical Success Factor: For each process, there are elements that if achieved, translate to success. This success could be related to the activities of the process and how well they are carried out, or how well a process is achieving its intended purpose.
Key Performance Indicator: A KPI can help you determine whether the goal of the CSF has been achieved, or whether the organization is moving in the right direction. If not, the data can be used to help determine how to change course so that success can be achieved. KPIs are broken into four different categories:
It's important to note that KPIs can be either quantitative or qualitative, where qualitative measures are related to quality, and quantitative measures are related to quantity.
These metrics provide insight into the end-to-end performance of a service and its delivery. There are many organizational assets required for the delivery of services and to gain a holistic view of the end-to-end service all should be measured.
Since services need to provide value, let's take a deeper dive into what makes up value. From the service provider perspective, a service must meet the needs of the customer as defined in the requirements, and the service provider must deliver the service at the agreed target levels as defined in the SLAs. In addition, it's actually the service assets of the service provider that are used to provision and provide services to the customer, so it's important that we ensure these service assets are efficient and effective from both a cost and a productivity perspective.
The service assets of an organization are the resources and capabilities and must be included in the service measurements:
In reality though, it's the customer who determines the value of a service. This means no matter how good a service provider thinks their service is, unless the customer believes it provides value, it doesn't. Therefore, we must measure the satisfaction our customers have with all aspects of service delivery. That includes its ease of use, cost, the customer service and support provided, the sales team, the SLAs, and so on.
We've covered the basis for measurements, the reasons to measure, and the types of metrics, but we need to put those metrics into use across the organization, which is made up of many levels or layers. As Figure 3 shows, we should even have metrics to ensure we will achieve our vision, but it's almost impossible to go from vision to the correct measurements. We first have do some analysis to determine how we will achieve that vision over time and as we break that process down further and further, we can then determine the correct measurements to help us stay on track.
At the strategic level, there are the executives, who are responsible for creating the vision and mission of the organization, defining the strategy, setting policy and guidelines, and determining the services that will be offered. The tactical level takes direction from them, and develops plans of action to carry out the strategy within the constraints of the policies and guidelines. Finally, the operational level of the organization carries out the plans defined at the tactical level.
To support each of the levels, we need to gather metrics that will ensure each is achieving its goals and objectives, but how do we do that?
The following diagram shows what is required to break down the vision further and further to get to the point where we can define the metrics needed to compare our performance against desired accomplishments.
Figure 2: Connecting Your Vision and Measurements
In this hierarchy, for each objective at the strategic, tactical, and operational level, we will choose a set of CSFs and KPIs that enable verification that objectives are being achieved.
Note that in the description above we started by defining our vision and then worked downward to the operational level. As we're carrying out the activities, we then work our way back up the hierarchy. If we achieve all of the operational objectives, then the tactical objectives can be achieved. And if all of the tactical objectives are achieved, the strategic goals and objectives are realized which in turn, enable the organization to achieve its mission. Over time, if the mission is consistently realized, an organization will realize its vision.
By measuring at each of these three levels, an organization can proactively determine whether they are on track to achieve the desired outcomes, and if not, they can use the data and information to intervene and get back on track.
For an organization to stay competitive in the marketplace, it must continually seek to improve across the service delivery spectrum. Measurements play a key role in determining where improvements are needed. Remember where this all started: "If you can't measure it, you can't manage it." It's where we started, it's the guiding principle throughout all of the activities an organization carries out to deliver its goods and services to customers, and it's where we end up when things don't go as we envisioned.
What exactly can measurements give us insight into improving? Let's take a look, along with some examples for each.
People: Skills, knowledge, communication, technical capabilities, social interaction, and customer satisfaction.
Processes: Efficiency and effectiveness, tools, enhancing the output, and better integration with other processes.
Technology: Better performance, minimizing incidents, and improving technology.
Services: Functionality, performance, availability, capacity, security, achievement of SLAs, and customer satisfaction. Health of the organization: Alignment with goals and objectives, cost containment, and meeting ROI targets.
A structured approach to measuring for improvement should be embraced across the organization. The Seven Step Improvement Process is one such approach that can be utilized for improvement, regardless of the area. Figure 3 is a graphical representation of that process that includes the steps: identify, define gather, process, analyze, present, and implement. Let's take a look at each individual step.
Identify: The first step is to identify the areas on which to focus the measurements for improvement. Define the CSFs for the goals and objectives in each of the above areas. It's best to start with a small, manageable number, so due diligence is required to determine the most important areas to the organization. More CSFs can be added later, or current measures can be modified.
Define: Based upon the areas chosen, define what needs to be measured to help you understand where there are gaps between your desired state and the actual state, so that you can intervene, to get back on track.
Gather: Preferably, gathering the metrics will be automated through the use of monitoring tools. There may be some cases where automation is not possible with the current tools. In those cases, manually gather the data. For example, some methods of surveying customers to understand their level of satisfaction is done manually.
Process: Adding context to the gathered data will provide information. It's important to understand how often to process the data, the level of detail needed, who will be utilizing the data, and the format that will facilitate the organization's ability to use the information.
Analyze: We gain knowledge by analyzing information, and discussing it with others. The next time you're in a discussion, take notice of how each individual has their own ideas, perspectives, and experiences related to the information being discussed. When we analyze information, and take into account these different perspectives, we gain knowledge that we can utilize to determine whether we are achieving the CSFs. If we are, then a determination should be made regarding the viability and need for the particular measurements used for the analysis.
Present: In this step, the knowledge gained in the analyze step is used to create a business case or justification for improvement. The business case will include information about the proposed improvement including the reason, assumptions, benefits, business impacts, risks, and recommendation. The data and information that were created for this particular CSF will be used to support the business case to provide the appropriate stakeholders with enough knowledge to make an informed decision about the best way to address the situation.
Implement: Once the business case is approved, a Service Improvement Plan (SIP) is created, and the actions to make the needed improvement are implemented. It's critical that any changes to processes, activities, etc., are communicated throughout the organization and mechanisms are put in place to ensure the organization does not revert back to the old way of doing things.
It's critical that once the improvement is implemented, we continue to measure and gather associated data, so that we can verify that over time, the improvement has its desired effect.
Figure 3: Moving from Objectives to Improvement Implementation
Measurements and metrics provide a view into every aspect of the organization. From issuing alerts, to resource availability, and continual improvement areas, measurements are the key to successfully understanding how the organization is performing. Whether it's ensuring an SLA has been achieved or a customer is satisfied, every organization must have a well-thought out plan for service measurement. This paper has provided guidance on why measurement is important, how to get started, different types of metrics, what should be measured, and how to go about initiating improvement in an organization.
Michael Caruso is the founder and managing principal of ITSM Professional Consulting. With extensive experience in the information technology field, Mr. Caruso's practice offers IT service management, IT governance, and project management instruction and consulting. His background encompasses both technical and leadership roles in application development, infrastructure, service desk, and operations across many industries. Mr. Caruso is certified to teach all ITIL classes, as well as the COBIT Foundations, COBIT Implementation, Cloud Essentials, Virtualization Essentials, and Project Management courses.