Effectiveness is a cornerstone of modern corporate compliance. The U.S. Sentencing Guidelines expect it, and compliance officers spend substantial time and resources trying to create an effective program. And as reflected in recent studies and surveys, assessing compliance program effectiveness continues to be top-of-mind for senior compliance officers. On its face, the regular monitoring and measuring of the program can prove beneficial to company success.

Yet in spite of the Sentencing Guidelines for Organizations being in existence for over 20 years, and the recent focus on developing metrics, it remains challenging to demonstrate the effectiveness of the compliance function.

In recent columns I’ve raised the limitations on government pronouncements and so-called metrics, with the current lack of rigor in measuring effectiveness. Enforcers and regulators (typically lawyers) are not scientists, and the field of compliance can benefit from more empirical research. Surveys of compliance professions reveal that many are not confident that the metrics they use to assess compliance program effectiveness give them a true picture of program success.

Still, a company will want to be able to demonstrate that it is creditworthy under the sentencing guidelines to benefit from penalty reductions, and more importantly to avoid indictment altogether. The company’s other constituents, including shareholders, the board, and management, will also seek some level of assurance that the compliance program is effective and worthy of investments that have been made. Program “efficiency” is another consideration for evaluating performance. Without an agreed-upon methodology, and needing more than a qualitative description of “I know effectiveness when I see it,” how can a company approach this measurement challenge?

Regular reporting and meeting with the board

One way companies can respond to this concern is to compile regular (at least annual) compliance program reports that detail all key aspects of their respective programs at a particular time. The report can be a compilation of quarterly reporting with a summary of highlights for the fiscal year. If sufficiently comprehensive and persuasive, such reports may help a company surmount the evidentiary challenge of proving the effectiveness of its program at a given point of time in the relatively distant past.

Of course this begs the question of what goes into a program report. At a minimum you want to make sure your board is up to speed with your compliance program by summarizing the key changes and developments. The compliance framework essentially boils down to three basic questions which form the basis of an effectiveness evaluation and program audit methodology: 1) Is the compliance program well-designed? 2) Is it being applied in good faith? and 3) Does it work?

The science and mostly art of effectiveness assessments is still evolving. You need to carefully identify types of data available, apply insight to the data, and design metrics to create the story around effectiveness.

Compliance programs, particularly in highly regulated industries, have matured to the point where data for the first two bullets are periodically collected and reviewed. This is at least a good start with building the case for effectiveness. To evaluate effectiveness, compliance departments now analyze internal audit findings, track hotline calls, monitor training completion rates, review the disposition of internal investigations, perform self-assessments, survey employees, compare themselves against peer companies, retain outside professionals to review the compliance function, and track performance on regulatory reviews. When meeting with the board you can talk through progress, results, and challenges as they stand today, in relation to previous years, and benchmarked against other companies:

Implementation Process – Status of important compliance initiatives, any major program operational updates, and what work remains.

Risk Profile Changes – Any new, emerging risks or noteworthy changes to the likelihood or severity of your organizational profile, either due to business changes or environmental developments.

Policy Attestation and Training Certification – What percentage of employees have successfully completed training and policy requirements, including the results of any post-training tests and policy attestation rates? Are there consequences for those who have not completed?

Employee Feedback – Highlights of feedback received through employee focus groups, culture surveys, suggestion box, and how you are using this feedback to drive improvements.

Compliance Audit Findings – Results of internal or external audits, and what these findings mean for the organization and the compliance program.

Hotline/Internal Reporting Data – How many tips your hotline or other reporting channels have received, trends by type of incidents being reported, and any hotspots that have emerged in particular locations, departments, or business units.

Incidents and Investigations – The number and type of investigations that took place, the disposition of cases, and what ongoing investigations the board should be aware of.

Risk focus

A feature of an effective program is the regular performance of a compliance risk assessment. Regulators and enforcement agencies will be looking for correlation between the risk assessment measures and performance indicators that are being used to monitor those risks and compliance performance. These measures should consider the high-risk areas and what has been put in place to address those special risks; internal audit will undertake an annualized range of audits as part to identify compliance issues that provides a good source of measures and also indicate to the regulator that the company is operating cohesively.

The COSO Enterprise Risk Management and Risk Framework identifies the core elements of well-designed KRIs (Key Risk Indicators) to link business objectives to strategies to risk. The KRIs, if robust, should give you visibility into your most risky areas. Periodic risk assessment results should be used to determine whether compliance risks are increasing or decreasing.

The data and results from a compliance risk assessment provide an opportunity to support program effectiveness. An approach to consider is to incorporate risk ratings that are generated from the risk assessment into routine monitoring reports. The status of mitigation efforts can be tracked and the impact on the risk rating reported as part of regular compliance program updates to senior management and the board. Such reports can be trended to (hopefully) depict the impact of mitigation activities with risk ratings adjusting over time.

Dashboards

Some organizations use dashboards (or scorecards) as a shortcut to giving executives and board members information about what is being accomplished by the compliance program and where the organization is at risk. The challenge is figuring out what metrics will go on the dashboard. Your metrics need to be specific and unique to your company and what business it conducts along with what goals you’re trying to achieve as a whole and as a compliance program—there is no one-size list of these metrics. Best practice and regulatory standards call for risk-based program reviews to specifically account for an organization’s unique risk profile.

Given the lack of standard measurement techniques, how else can dashboard metrics be identified? A rigorous audit to evaluate a compliance program will analyze specific program elements. The auditor can start with tools utilized when conducting a review of the compliance environment under COSO. This includes techniques for evaluating entity-level controls, the control environment, and fraud-control activities. There are metrics around surveillance and testing but, in the end, do we know if we have an effective program? It’s still difficult to say. For purposes of the sentencing guidelines a company can stand-up a program that ticks all the boxes. One can engage independent consultants to come in and validate the existence and good faith effort being made. From benchmarking we know how our company compares to others. While metrics do not yet fully answer the crucial question of program efficacy, it can help build the case for effectiveness.

The science and mostly art of effectiveness assessments is still evolving. You need to carefully identify types of data available, apply insight to the data, and design metrics to create the story around effectiveness. Ultimately you want to create a report that tells the story of the compliance program to leadership, and if ever needed—to enforcement authorities and industry regulators.

Topics