Ask a CCO: How does your firm assess the efficacy of new tech?

Q: How does your firm assess the efficacy of new tech prior to use?

EMMA BREDIN: First and foremost, we draw on our internal experience and expertise to ensure we understand the problem to be solved. It is critical to have a clear end goal as to what the technology will do and how it will be utilized. That goal will act as the measuring stick in assessing whether the technologies are effective.

We then enter the research phase. Before selecting or designing new technologies, we conduct rigorous market reviews of service providers and existing products to determine whether an existing solution or internal build is our best option.

Once the technology is selected, prior to onboarding any ‘live’ data, we assess efficacy through our due diligence process and extensive testing program. BNY Mellon has a multifaceted approach for testing new technologies, which includes assessment and oversight across the three lines of defense and leverages existing mechanisms like risk control self assessments, quality assurance, and internal audits. We make sure to foster collaborative relationships with stakeholders across the company to ensure compliance has a seat at the table to oversee and challenge the technology build and proactively embed compliance controls within the architecture.

ILONA NIEMI: To remove bias in the selection process, any new technology onboarding must follow a holistic vendor management process. Several gates, including requirements documentation, test cases, diverse evaluation team creation involving key business stakeholders, and structured decision-making scorecards, are obligatory to enable a solid conclusion on the most efficient and effective solution that is in the best interest of our organization.

Patience is critical during this process, given the number of steps and documentation required, and will pay off when the best-fit compliance solution is onboarded.

SHELLEY SCHACHTER-CAHM: We have a series of tools and processes that help us ensure all new technological integrations are both vendor agnostic and outcome oriented while continuing to meet our unique regulatory and user-focused needs.

To determine efficacy, we operate from a standard of ‘trust but verify’ by testing our data sets across multiple programs or potential partners to see which yields the strongest results. This helps showcase the best performance in a ‘bake-off’ setting.

Once the results cool, we calculate the hard and soft costs associated with implementation—built infrastructure, training requirements, educational investments—alongside opportunity costs to get a complete picture.

ERIC WILSON: When evaluating new technologies, we look to the expertise of teams across the organization, such as our cybersecurity department. Once everyone is moving in the same direction, we then determine the additional needs to maintain that technology. In parallel, we look further down the road to identify what needs to happen to achieve long-term success with that new technology.