Supply Chain / Value Chain Metrics - Getting Started (2) Benchmark Programs

The holy grail of business information for the executive, analyst, or consultant is benchmarking. When I was with the Supply-Chain Council, I estimated that about half of the organizations that joined were looking for benchmarking information. For over ten years, I probably spent some time at each face-to-face meeting or workshop answering questions about benchmarking. Things haven't changed much since 1996 and the dangers and pitfalls of working with benchmarking programs are much the same.

For some time I have been building and maintaining a list of benchmarking programs, on this blog, that are related to value chain operations. While this may provide a half way decent starting point to find resources, it is not intended as a blanket endorsement for the surveys on the list. It would also be very unusual for a company or organization to participate in all of the surveys.

So, how do you sort through list and find a program that might help you get the information you need? If you are working with a consultant who has a private benchmarking program, how do you qualify their list? What questions should you be asking?

About Benchmarking

While everyone wants benchmarking information, it is difficult and expensive to run a decent program. You might remember a college statistics course that spent an agonizing amount of time trying to teach you how to craft good survey questions, how to determine whether or not you are getting valid data, and how to determine what the data means. In the world of business, those questions can be pretty important but some companies are so eager for competitive information they don't invest the effort to qualify the source of their data. (For anyone who hasn't read it I highly recommend How to Lie with Statistics - a good, light read which helps remind people that seeing survey results, pie or bar charts on a website or in presentation is no guarantee that it is based on sound data).

Many "free" benchmarks reports are available to companies who are willing to respond to a survey. For example, you may be asked to complete a survey on the use of supply chain technology (RFID) for example. You will be referred to a website or online survey that collects your responses (usually takes 15 or 20 minutes). Your results are then used with other anonymous respondents to produce reports (which are sold and used as marketing collateral). A statistically significant number of the other respondents in the survey may have no investment or authority over supply chain technology but will enter data to receive the report - they make their best guess.

While it may take little time to enter data into an online survey, it may take a significant investment in time to collect and measure data. (The average time it takes for multi-national companies I work with to collect and report performance measurements for the first time is 3 months. I have seen companies spend months just deciding what KPIs or metrics are important. Most data is ready available somewhere in the company. Tracking information down, reformatting, and making sense of it takes time. After the first effort, most companies can measure and report their peformance in relatively short time periods. Of course, if you only collect information every 30 days, there is not much merit in demanding reporting faster than that.) Many companies are investing in business dashboards now which can collect and report performance information in "real time." More on business dashboards in another entry.

A careful purchaser of benchmark services understands that the benchmark service provider is spending resources to run the benchmark program and it will require an investment on the part of the purchaser to collect and report information.

Benchmark Scope

What are you trying to measure?

Probably the first question to address if you are considering a benchmarking program is: what are you trying to compare and to whom do you want to compare yourself? If time to market is an important operational measurement to you, it is important that the benchmarking program reports the metric or provides enough information to help you. If you intend that time to market includes product design, manufacturing, and distribution, a supply chain benchmarking program might not include enough information. A product design / development program probably won't have enough. A marketing and sales set of data might not either. You may have to use multiple programs or look for research programs that are specific to "time to market."

Identify the key metrics or KPIs you want to examine. You know you will probably be able to get general P&L and balance sheet numbers from public records or your marketing department. Pick high level operational questions you need to compare. While supply chain velocity may be too ambiguous for meaningful measurement, order fulfillment time or order fulfillment cycle time may be right on target.

I would also recommend that you pick multiple aspects or dimensions of the business problem to examine. There may be little merit in finding that a competitor has higher transportation costs without also realizing they have shorter delivery times and carry less inventory.

To whom are you trying to compare the company?

Most companies want to participate in benchmarking programs that will identify their weaknesses and strengths so they can become stronger in their market space. If you have worked with any competitive analysis you will be familiar with grouping companies by industry. There can be issues with relying on too easy a classification. If you visit the public sources of information that I identified in Part 1 of this series you will find that in many cases they don't group companies the same way or even use the same industry classifications.

Equally challenging, a large company may have multiple product lines that compete in what are intuitively different industries. For example, the largest provider of IT to the federal government is in what industry? Technology or information technology? No. Business services and solutions? No. The answer is Aerospace and Defense and the company is Lockheed Martin.

Even the best benchmarking programs (and consultants) may not group companies to allow you to compare apples and apples. There are two reasons.

First, if you are are comparing organizations instead of value chains (logistics vs. supply chains) you don't differentiate by products or product family. From a financial investment perspective it makes sense to compare how an organization manages their combined investments and businesses. If you are trying to improve your competitiveness in the marketplace, you are probably looking for information about well you generate demand, identify and satisfy demand and how efficiently you conduct the operations. It requires significantly more initial research and analysis to group by product and product lines.

The second reason benchmarking programs tend to group by industry and, indeed, very general industry categories is a very simple numbers game. If you are investing the time and money to participate in a benchmarking program you want to be able to compare yourself against as many of your competitors as possible. So, it is in the best interest of the benchmark program to have as few buckets as possible. How many Aerospace and Defense companies are there? How many aircraft manufacturers? How many manufacturers of large commercial aircraft? For many benchmark programs those questions are unimportant. For the business executive who understands that defense aircraft production is a cost-based market space (where the customer knows the cost of the product and negotiates the margin) and that commercial aircraft is a price-based market space (where the customer negotiates the price and may not know the cost) the difference is enormous. If you are a Boeing executive trying to benchmark your company it may be far less important to have a dozen companies to compare yourself to than one company, if that one company is Airbus. Of course, if you are selling or providing benchmarking services and you are trying to sell to Aerospace companies, you may be willing to group a manufacturer of aircraft landing gear gaskets and seals with the manufacturer of a helicoptor.

Besides determining who is in your industry (or product grouping) you may want to determine whether or not you want to limit yourself to your competitors. There is ample evidence that world-class organizations try to determine what the best-in-class organizations in other industries are doing. Why? Because breathroughs in competitive advantage may be developed by someone who is performing the same process for different customers and in support of different products. For example, most companies have to purchase goods and services. Best practices in purchasing like vendor managed inventory may be equally effective in different industries - but someone will do it first. Benchmarking may point to best practices or operational effiiciencies that have been captured in another industry and may be realized in yours. Of course, if you participate in a benchmarking program that is restricted to your industry you may not see the opportunities.

My recommendation is to participate in the broadest (cross-industry) program you can find that also gives you the detail that you need. If you are working with a consultant or benchmarking provider that support multiple industries, it may be possible to participate in a detailed study but have access to the summary information for other industries.


Qualifying the Benchmark Provider


After you have developed a short list of benchmark providers, there are probably a number of areas you should explore with the providers. I would qualify them just like you would any other major buy.


Price and Obligations


There is typically a set of conditions, including price, for participating in benchmark programs. These include:

• Price. Frequently benchmarking is a subscription service. You put your data into the pot and can extract data that is comparable within the same time horizon. Issue: What is the length of time data is "comparable?" To make sure that some surveys have enough participants sometimes the time period can be 18 months or two years. Economically, that can be the difference between boom and recession. Within a market, major competitors may have come and gone. Products that dominated the market may be obsolescent.


• Data Contribution. You must contribute data to receive data. If everyone could buy the data without contributing, programs would rapidly run out of results to share / sell. Theoretically this also means that consultants and vendors - who don't have data to contribute - don't have access to the benchmark results. Right.


• Confidentiality. You may be asked to sign a non-disclosure agreement which could limit your ability to share the information with your partners. This can be particularly problematic if you work with consultants, technology providers, or may need to work with them in the future.


• Anonymity. The contributor of individual data remains anonymous. There is some variation in programs but basically this means that if you report how long it takes for you to design a product, no one will be able to determine that the specific time belongs to you. (When I was a practitioner, I was invited to participate in a benchmarking program that promised anonymity. In my industry, there would have been two participants - my company and a major competitor. I might not be a mathematician but if there are two participants in a survey and I am one of them, I can figure out what data the other guy contributed.

Benchmark Population

While we have already talked about the issues with industry grouping, be very sensitive to benchmark programs that talk about 10 or 20 years of data and hundreds of participants.

• Current Population. Determine the current (active) number of subscribers for the service you wish to obtain results from. If you are going to participate in an industry survey get the number of current subscribers, verify that they participate in the same survey, and try to determine you are comparing the same products / product families / businesses. Ideally, get a list of the active subscribers. Verify that benchmark reports don't blend former participants with current participants. If there are a hundred companies in the study you may want to verify that these are companies that contributed data to participate in the current study as opposed to being "grandfathered" in from previous "similar" studies.


• Benchmark Study History. A number of long-term studies and reports change their survey questions from study to study. An annual benchmark report may not change its title but change its content on an annual basis. It is useful to compare questionairres for multi-year studies (and executive summary data).

Metrics, KPIs, and Test Question Validity

• KPIs and Metrics. I will talk about selecting and defining metrics in other blog entries. For now, let's say there is little agreement within a company, between companies, between analysts, well between anyone about what or how to measure just about anything. That's a problem if you are doing benchmarking. Issue: Are you measuring the same thing? For example, if you are interested in customer service levels one measure you might be interested in is On Time In Full Delivery (OTIF). OTIF is understood by some pracitioners as the percentage of orders that are delivered when specified (neither early or late) and are complete based on the customers order. Some practitioners don't capture receipt information and so they calculate OTIF as a percentage of orders that are shipped on time and complete. Others calculate OTIF as the percentage of orders that are shipped on time and have an estimated shipping time that would have the product arrive at the customer's site on time. Those are three very different calculations that could potentiatlly lead to very different results. On many occassions I have seen companies puzzled by customers who complain about on time delivery. The company looks at their measurements and see OTIF number of 95-98%. The customer is complaining that OTIF is less than 80%. Frequently what is happening is the customer is measuring when they are supposed to receive the order and the company is ordering when it was shipped. The mismatch in measurement leads to misunderstanding and a loss of business.

• Metric Standards
  
  Some of the benchmark programs will advertise that they use a "standard" set of metrics. Having spent over a decade working the members of the Supply-Chain Council in trying to get consistent, well-understood metric definitions I can tell you that these standards are elusive. I would certainly agree that a reference library is a useful "truth" source for measurement data. There may also be great value in having a company-neutral source of metric definitions.

• Data Collection and Measurement - Validity of the Results If you do a bit of research on how the benchmark providers "assist" companies in assembling and contributing data (try to find presentations or webinars that provide guidance to propsective participants in collecting and post data) you may get a sense for how long it takes and what assistance the benchmark providers offer. I have seen in multiple presentations benchmark providers relate how they will help a company "guess" on performance results if the measurement data is not available. This should raise a red flag for anyone planning on seriously using the benchmarking data for business planning. Issue: If a provider enables the entry of questionable data they are committing two sins. First, they are skewing the results of the benchmarking - you may not know what a "real" answer is. Second, they are effectively hiding what participants know and don't know about their operations. It may be as valuable for a company to know that no one in their industry knows how to predict the time it takes for a concept to transition to the manufacturing floor as it would be to guess the time plus or minus 100%.

Benchmarking programs and surveys can be an invaluable source of information when you are first starting to systematically use KPIs and Metrics for your value chain. Working with a reputable benchmarking provider will not only provide you competitive information but the provider can help educate you in how to collect and measure your value chain - what is meaningful and why it is meaningful. If you don't exercise due diligence, you may find that you are investing time and resources in an exercise that provides little value and can lead you to investments that provide little return.

While this entry dealt primarily with formal benchmark programs, many of the large consultants and integrators offer versions of benchmarking where information is shared among their active clients. The business information as advantageous (or meaningless) as well run (or poorly run) commercial and academic programs.

Many large practitioners operate large performance measurement databases in their purchasing departments (to review their suppliers) and in their marketing and sales departments (to review performance to their customers). These internal sources of information may be used by an enlightened company to provide an "internal" benchmarking program that may be better and more extensive that one that could be purchased.