Not long ago we held another of our Compliance Week editorial roundtables, in Atlanta with co-hosts Compliance 360, to talk about the challenges of measuring the effectiveness of your compliance program. Our full coverage of that discussion will appear in next week's newsletter, but for now let me recap some of the most interesting points—because the more I ponder how you measure “effective” compliance, the more I realize how elusive that idea really is.

Regulation helps. We ended up with an unusual mix in our group of 10 participants: four attendees from the healthcare industry, four from retail or consumer goods, and two from public utilities. In other words, we had a split between one group that is highly regulated (healthcare) and another that is regulated much less (retail and consumer products). The difference was enormous. The healthcare folks reported excellent lines of communication with their regulators, and much more sophisticated systems to peer into their compliance efforts at any given moment in time. The retailers, meanwhile, said they still sometimes struggle to know exactly who their regulators are—a mish-mash of state agencies, local authorities, and federal regulators who each worry about one piece of what the retailer is doing.

If you want just one example of how that situation can complicate your job, consider the question of whether to self-report misconduct. All our healthcare participants said they were confident they could self-disclose and benefit from that, because they know who their regulators are and have worked with them or their agencies for years. And the retailers? They all looked stricken when I asked them whether they'd self-disclose. As one said: “Good lord, I don't know that I'd ever do that. These people at the state agencies, they come and go all the time. I have no idea how they'd react or whether the next guy would do the same as the last.”

As much as we all might dislike strong, savvy regulators with large budgets, let's all admit the obvious: They are much better than the alternative of weak or disparate regulators struggling for the money to do their jobs.

Effectiveness and visibility. Whenever I ask, “How do you measure the effectiveness of your compliance program?” I always get some automatic answers. We measure hotline calls, people tell me; we track complaints from their filing to resolution, or certify that employee training has been completed. (An unsettling percentage of them also tell me they don't measure the effectiveness of their program at all, but we'll put that aside for now.)

The problem with all of these common metrics—hotline calls, employee training certification, disclosure of material weaknesses, and so forth—is that they only tell you whether your compliance program is busy. That's not the same as effective. An effective program catches risks as they emerge into something more serious, and alerts senior management that an intervention might be necessary. How do you measure your success at that?

A lot of the roundtable conversation dwelled on that point. We spoke at length about how you need the right metric, yes, but must also measure it with the right frequency. Think about it: If you conduct an ethics perception of your employees only every January, regardless of that new acquisition in Asia that closed in July—you've exposed yourself to significant risks there. Likewise, if you track the life-cycle of employee complaints from filing to resolution, but only give updates on those complaints every six months—well, that employee who blew the whistle internally might get tired of the delay and take his issue to the feds. 

When you combine metric with frequency of measurement, what you're really talking about is visibility into the compliance program—so that you can always determine exactly how some issue is being handled, and report that fact whenever someone (the board, regulators, shareholders) asks about it. That demonstrates that you, the chief compliance officer, are on top of your game. That's what effectiveness is.

Bringing it together. Our roundtable even pulled out a thread that connects my two points above: compliance departments have different degrees of visibility into different risks, based on the regulatory and business pressures they face.

Remember those healthcare participants, who had outstanding visibility into risks like fraud or off-label marketing? They were still struggling to develop risk and compliance programs around social media, because healthcare overall lags so far behind other industries in adopting social media. But those retailers? They were all over their social media risks, because that's increasingly how customers interact with retail outlets. The retailers all had social media policies, and could talk at length about how they try to implement and monitor them. The healthcare participants are still trying to explain to their board what Twitter is, and that yes, those nasty tweets about healthcare insurers are out there for all the world to see.