IT security may be a routine part of doing business, but it retains an unmistakable aura of a black art, where recovering hackers in white hats battle evil black-hats using various forms of digital sorcery.
However whimsical that imagery may sound, it’s a problem, says Elizabeth Nichols, partner in security consulting firm PlexLogic and a leading voice in what she calls the “metrics movement” in IT security.
Nichols
“We need to free security from the morass of black art,” she says. “As a discipline becomes more mature, it develops the kind of metrics and quantitative analysis that makes it less like a black art.”
Lack of metrics is precisely what drives compliance executives, auditors, senior managers, and even board members to distraction about IT. Nichols compiles and distributes IT security metrics (for free) via SecurityMetrics.org, a small step toward providing that necessary information about IT security and performance.
Sometime this quarter, however, the Center for Internet Security will take a much larger step forward: It will release a set of consensus metrics arrived at by 85 professionals from 70 companies in finance, health care, and utilities as well as government and academic organizations.
The group released its list in concept in September. The criteria a company should be monitoring include:
Mean time between security incidents;
Mean time to recover from security incidents;
Percent of systems configured to approved standards;
Percent of systems patched to stated IT policy;
Percent of systems with antivirus protection;
Percent of business applications that had an IT risk assessment;
Percent of business applications that had a penetration or vulnerability assessment;
Percent of application code that had a security assessment, threat model analysis, or code review prior to production deployment.
That list might sound simple (or just arcane) to those who lack administrator privileges on the corporate network. In reality, however, the list is a select few points from hundreds of possible criteria a company might track—including criteria that might not be the right ones for your particular company to study. According to Burt Muiccio, CEO of the Center for Internet Security, the whole point of the project is to give compliance and IT executives a reliable, cross-industry benchmark for IT security.
The ultimate objective, Muiccio says, is to help security professionals use hard data to develop better IT security strategies and, by extension, security-investment decisions.
Sebes
John Sebes, an information security consultant and chief technology officer of the Open Source Digital Voting Foundation, says the CIS’s preliminary list of metrics seems well-grounded, in particular in its focus on “breakage risk” rather than risk of being hacked, which he views as “a common but regrettable mistake.”
The CIS’s reputation for independence and thoroughness should make the fruits of the effort all the more valuable, says Heriot Prentice, director of standards and guidance at the Institute of Internal Auditors. He says the list’s scope from antivirus to risk assessment will make the resulting data valuable not only to IT professionals, but also to business managers and auditors.
Prentice
With such metrics in hand, Prentice says, “When you go in to audit an area, you already know what you’re looking for—so it’s very cost effective and a big help.”
Yet what’s really new about the CIS initiative has been the process of arriving at metrics via consensus, Nichols says. “This is something they are very good at.”
Setting Benchmarks
Indeed, the CIS has been best known for its benchmark security configurations for various operating systems, middleware, software applications, and network devices, which are downloaded a million times a year. All 40 of the existing benchmarks—as well as new ones coming online later this year for Microsoft Office, various Web browsers and other IT resources—were developed through similar consensus among IT security professionals, Muiccio says.
Reaching consensus in IT security remains a challenge. Ask 10 IT security professionals how to calculate “the meantime between security incidents” and you’ll get 10 different answers, Muiccio quips.
“I love the idea of cross-organizational metrics. If a whole group of organizations get together, you have some weight.”
— Alan Paller,
Director of Research,
SANS Technology Institute
Every organization has tried to do metrics, says Alan Paller, director of research for the SANS Technology Institute—but a chronic problem for compliance executives is whether those metrics are any good absent a broader context. Without benchmarking, who’s to say your company’s metrics are the right ones?
“I love the idea of cross-organizational metrics,” Paller says. “If a whole group of organizations get together, you have some weight. That’s the whole CIS thrust.”
Key to the success of any such IT metrics effort is whether it convinces senior executives to invest in IT security, Paller says. Cross-company metrics showing your firm to be sub-standard in a given area could help, he says.
John Kirkwood, chief information security officer of Dutch supermarket giant Ahold and former CISO of American Express, says Ahold developed its own IT security metrics based upon internal policy objectives, business requirements, and such goals as compatibility with PCI and ISO/IEC 27002 information security standards. Metrics, he says, are the bridge between one-dimensional data and knowledge and intelligence.
“For me, the kind of metrics the Center is talking about is very interesting. I can get internal metrics, but don’t have anything to relate them to industry,” says Kirkwood, who is not involved in the CIS effort. “A good example would be the number of administrators to servers. If there’s one administrator for 50 machines, is that good or bad? How does that compare to the industry among companies of our size and complexity?”
Kirkwood says he has an anecdotal sense of how his company compares to its peers on IT security, but has no real proof of it. The CIS metrics may not be the whole answer, “but they could provide an indication of magnitude, which is important. There’s potential for a great facility here,” he says.
It may benefit the field of IT security in other ways, Nichols says.
“Security as a discipline needs to get more specific about evaluating its performance and making decisions about how effective the strategies you’re implementing really are,” Nichols says. “I think metrics are absolutely critical if security people want to get the respect they deserve.”
No comments yet