For all the time and money spent designing and building ethics and compliance programs, the follow-up question compliance officers need to answer is simple: How can they be certain all that effort actually pays off?
Many compliance officers still measure hotline activity, employee training completion rates, and conduct audits of the compliance program—all good starting points, to be sure. The peril in analyzing these metrics in isolation, however, is that they only reveal whether a compliance department is busy. That’s not the same as effective.
Before compliance officers can begin to measure the effectiveness of their ethics and compliance programs, they must first understand what they want to accomplish. “You have to define your goals,” says Anne Harris, an independent ethics and compliance consultant and former chief ethics officer of General Dynamics.
“Try to put things into context for senior leaders so that they don’t jump to possibly mistaken conclusions.”
Anne Harris, Independent Ethics and Compliance Consultant
For example, if you are a spin-off company in the early stages of developing an ethics and compliance program, your goals and compliance metrics will differ from those of a large, global company with a mature compliance program, says Daniel Torpey, a partner with EY’s Fraud Investigations and Dispute Services practice. Understanding what the goals of the program are will help determine the compliance metrics you want to measure, he says.
Annual surveys that take the pulse of the corporate culture are “a gold mine of information and probably the most useful in terms of assessment effectiveness,” Harris says. If you have a history of employee survey data, and those surveys point to improvements or deteriorations in employee perception about the ethics and compliance program, the responses themselves are an effective data point, she says.
ARE CCOS USING THE RIGHT METRICS?
Below is an excerpt from the 2014 Compliance Trends Survey from Compliance Week and Deloitte, which explores what metrics chief compliance officers are using.
Compliance officers have long struggled to find the right way to measure the effectiveness of their compliance programs. Four years ago, in the first incarnation of this survey, 38 percent of respondents said they did not measure effectiveness at all. That number has since fallen to 23 percent today (down from 31 percent in 2013), but other evidence still suggests CCOs aren’t wholly comfortable with the metrics they use to get that sense of confidence.
What are the metrics compliance officers use to gauge effectiveness? The most popular ones tend to be internally focused: analyzing internal audit findings; analyzing hotline calls; tracking completion rates for required compliance training—each was cited by roughly 70 percent of respondents as a metric they use. Comparisons to external data (independent evaluations, benchmarking studies, analysis of regulatory reviews, and the like) were considerably less common, all notching only 45 percent or less.
“While compliance officers should take advantage of the Big Data within their own organizations, it would be a mistake to not look outside for new insights to make sure your program doesn’t become insular,” says Mohlenkamp. “When I worked as an in-house compliance professional, some of my best ideas came from time spent with my peers and external benchmarking.”
Another interesting dynamic emerges when you dive deeper into the respondents’ answers: Staff-level compliance professionals (managers, directors, vice presidents) are actually more confident in their companies’ measurement of effectiveness than C-level compliance executives. The total pool of respondents broke down almost 50-50 into those two groups, and their differences of opinion were striking:
• 64 percent of staff-level compliance professionals are “confident” or “very confident” that the metrics they use give them a true sense of how well the compliance function works, compared to only 52 percent of chief compliance officers;
• 26 percent of chief compliance officers are not confident in their IT systems’ ability to fulfill all compliance and reporting requirements, but only 11 percent of staff-level compliance officers say as much.
The differing opinions may be driven by a lack of adequate communication—perhaps the staff may have greater access to or understanding of the details behind the overall KPIs reported to the CCOs. Or it could mean that CCOs are looking at the more strategic or high-impact risks—the ones that represent significant reputational risks to the company—while the staff is focused on the more tactical risks in formulating their opinions about the adequacy of metrics.
“The survey underlines a continuing challenge many companies face in measuring compliance effectiveness, and demonstrating that to stakeholders,” says George Hanley, director in the financial services risk consulting group at Deloitte & Touche. “Compliance officers are using a mix of process-activity measures, and outcome- or results-based measures, which have different values.”
Ultimately, compliance departments are effective (and compliance officers are achieving their strategic goal) when the rest of the enterprise believes that corporate behavior aligns well with the company’s professed values—and the 2014 Compliance Trends report has good news on that front: 70 percent of respondents say their company’s culture and values align “very well” or “above average,” with little variation between C-level and staff-level compliance officers. Only 4 percent rated alignment as “poor” or “below average.”
Source: 2014 Compliance Trends Survey.
TE Connectivity is undertaking a culture survey right now. That survey will measure several aspects of employees’ perception about the compliance program, including, “Are they aware of resources? Are they comfortable calling the hotline?” says Christine Stickler, chief compliance officer at TE.
In addition to cultural surveys, seeing how you compare to industry peers is another way to measure compliance program effectiveness. “It’s always helpful to get industry-specific benchmark data,” Harris says. First understand how mature your compliance program is, and then figure out what data you can compare to your own to indicate whether your program is effective, she says.
Substantiation rates for hotline calls—that is, how many calls are raising a valid concern—can be another valuable metric. At TE, for example, roughly one-third of cases that come through the company’s hotline are substantiated, and nearly half result in some kind of corrective action plan. That shows the program is working because employees feel comfortable about coming forward to report issues or concerns, Stickler says.
Boards of directors and senior officers might not see high substantiation rates as a positive sign, because it could signal a wider compliance and ethics problem. “Interpreting that data can be a struggle,” Harris says. “There’s no right or wrong answer.”
Finding the Right Context
Compliance officers, then, must be deliberate about how they report compliance metrics to the board. “Try to put things into context for senior leaders so that they don’t jump to possibly mistaken conclusions,” says Harris.
For example, allegation rates might spike in the same year that the company goes through an acquisition. “Acquisitions always cause a spike,” Harris says. “So you have to be aware of what’s going on in the business, so that you can take that into account when you interpret the metrics.”
You also want to watch for anomalies that might occur in particular jurisdictions where the company operates. Maybe you want to look into why you’re not getting hotline calls from employees in a particular country, for example. “Those are some things that would be measured using analytics and year-over-year trend analysis,” Torpey says.
At General Dynamics, “we also looked at the data by business unit or facility,” Harris recalls. That data proved helpful, because as you start accumulating the types of allegations and concerns that surface over time, you’re then able to put together a graph and see, “‘We have a real spike in reports in this facility,’ or ‘We have a spike in reports regarding this subject matter.’”
One facility might have a consistently higher rate of allegations about a particular topic than other facilities. “That’s the kind of information you need for management decision making, because it will prompt somebody to say, ‘We better look into this,’” Harris says. “So you look for the outliers relative to your own data.”
When presenting to the board and senior management, rather than simply showing a graph saying, “We got this many calls from this factory and this many from the other,” a wiser course is to put statistics into proper proportion: for example, a graph showing over time how many allegations you got per 100 employees per quarter, Harris says. “If you see that this quarter, we have this number of allegations per 100 employees, and it’s significantly higher than past years, it tells you something is going on,” she says.
“The higher the level of leadership, the fewer details they want. They just want to get the information that they really need,” Harris says. “Reporting to senior leaders on data that doesn’t really tell them anything is not going to win you any points.”
It’s important to keep in mind the goals of the business leaders. The board of directors or the audit committee just wants to know whether any major problems are afoot, and how those problems are being addressed, Harris says. “Showing them data that indicates ‘This quarter we trained this number of employees,’ or ‘This quarter we had this number of hotline calls,’ for example, doesn’t inform decision making,” she says.
Compliance is just one of many issues that boards of directors have to deal with. “They want [the data] synthesized into something manageable,” Torpey says.
If you can show two or three pie charts over time, indicating where changes have occurred—or where no changes have occurred at all—the overall idea is to be able to use the data, Harris says, “to demonstrate that your initiatives are actually working and are having impact.”
No comments yet