Don't Overlook Intangible Metrics When Evaluating A CMO

By Louis Garguilo, Chief Editor, Outsourced Pharma

Something Denise McDade of Capricor Therapeutics said during a session at Outsourced Pharma West in San Diego last year stuck in my mind: “Metrics are fundamentally important in measuring and managing any relationship, but there’s a paradox when it comes to outsourcing.”

That paradox stems from an awareness that while there is a rising criticality to establishing a tangible set of metrics for your outsourcing partners, those increasingly important metrics still don’t offer as complete a picture as many would believe. Even with more sophisticated performance measurements for items like cycle time, throughput, deviations per batch, time to resolve deviations, on-time delivery, surpassing quality or quantity goals … these results may lead a sponsor astray at times.

According to McDade, VP of quality, and whose career has spanned various positions at Genentech, Novartis, Amgen, and other biopharmas before joining Capricor, there are equally critical elements — such as trust and the spirit of the supplier relationship — that defy even today’s finer metrics. Here’s a closer look at what McDade and others are saying.

DO YOU KNOW YOUR CMO EVEN IF YOU MEASURE?
McDade made her initial comments as a panelist at on OPW session titled, “How do you know you’re on target if you don’t measure your CRO/CMO?” Other panelists included Michele Johnson, director, supplier relationship management, Sandoz; Sylvie Sakata, sr. director, external research solutions west, Pfizer; and Nicholas Virca, president & CEO, HedgePath Pharmaceuticals, Inc. John Newsam, CEO, Tioga Research, was the moderator.

"On paper, performance can look similar, but from a relationship standpoint, performance can turn out to be completely different."

Denise McDade
VP of Quality, Capricor Therapeutics

 

No panelist or member of the audience was advocating less measurement of CMOs. The pendulum has certainly swung to the side of “the more measurement, the merrier,” away from a substantially less vigorous approach of even a few years ago. A separate panel at OPW dealt specifically with new FDA statements exploring the need to measure “culture” as a function of reliably deriving a quality product. This panel also included discussion on the FDA’s plan to receive quality data directly from CMOs.

But back to McDade. She has arrived at her convictions during a career that spans more than 25 years. During about half that time she has been focused on the outsourced world, while the other half she has been on the manufacturing owner-operator side.

“I’ve worked with CMOs where we had a robust and healthy management process, ample performance feedback, and a lot of measurement,” she explains. “However, there was still little trust in the supplier relationships. Ultimately, those types of relationships were not successful or healthy in an overall business sense. It’s difficult to measure exactly what the issue was with trust, but it was crucially important for the CMOs to try to maintain — or salvage — it as a part of their relationship management efforts with us. Therefore, I’ve seen that both sides — the measurable and the more intangible components — are equally important. In my experience, you can’t be successful unless you have both.”

THE 5 A.M. CALL BUILDS TRUST
At the conference, and in a follow-up discussion, McDade walked me through an example that draws out her feelings.

At one point in her career, she was outsourcing the same product to a CMO in the U.K. and one in the U.S. There were similar quality agreements in place stipulating that within 24 or 48 hours the CMOs inform McDade, as head of quality, of different levels of deviation that might occur at the facilities during the manufacture of the products. “The U.K.-based CMO had a facility contamination issue in an area of their facility that ultimately didn’t impact our product,” McDade explains. “However, I got a call at 5 a.m. my time from their QP [Qualified Person, as referred to in Europe] so I could be informed as soon as they found out. There were other times as well where the QP would pick up the phone to let me know immediately of anything that he thought I should know.”

There was also an issue at the U.S.-based CMO, this time a critical contamination in its WIFI (purified water) system. Again, the issue was not occurring in McDade’s part of the facility, so by a strict reading of the quality agreement the CMO was not obligated to inform McDade’s company. In fact, McDade only found out about the problem after the CMO was inspected by the FDA and received a warning letter.

“By strict measurement, both CMOs were within the language of the contract, but only one was within the spirit of the relationship. Which one did I trust and continue to work with? You could look at the numbers on the metrics reports and they might be similar, but the quality of the relationships was totally different,” she explains.

CAN YOU METRICIZE TRUST?
The OPW panel agreed that the trust factor would not diminish in importance because more and better metrics are put in place at the sponsor to measure provider performance. In fact, the same reason for an increased need for CMO metrics also bolsters the need to have the intangibles — like personal relationships — contributing positively to outsourcing outcomes. As one panelist said, “You can’t have a relationship with an organization; you have a relationship with individuals.” While systems and processes are set up so that projects, processes, and relationships are not “completely persondependent,” sponsors will always be “somewhat dependent upon the individuals on the other end.”

It’s important to have an open structure where the letter and spirit of the relationship can be transitioned as personnel changes — at either the sponsor or provider. One panelist talked about working with a supplier who had to replace project leaders a few times, but “how the transition was accomplished had a huge impact on how successful and seamless we continued to move forward.” Trust as an independent variable gets high marks for how well productivity, quality, and reliability — and the relationship itself — are sustained over time and as people change. That trust then becomes the yardstick for the entire organization as a fully reliable partner.

McDade has had outsourced supply chains find ways to document intangibles to head off any paradox cropping up in the more standard metrics. These broader measurements became an important part of an overall supplier scorecard.

“We measured items that were subjective by applying a numeric rating to them and then presenting these ratings as parts of our quarterly review meetings with the supplier. We actually spent a fair amount of time talking about these items with our CMOs so they understood the importance to us,” says McDade. She adds: “We measure and measure. On paper, performance can look similar, but from a relationship standpoint, performance can turn out to be completely different. For me there is always another equally important ingredient in that relationship, and that is trust.”