A Guide to AI in Pharma: Part II

In this second part of our series, we dive deeper into the essential steps for reaching AI's potential in the pharmaceutical industry. From aligning AI initiatives with business goals to ensuring data integrity and navigating the trade-offs between transparency and model performance, we provide a roadmap for developing AI solutions that deliver real-world value while adhering to ethical standards.

Defining Successful AI for Pharma

Artificial Intelligence (AI) is reshaping the pharmaceutical landscape, offering new opportunities to enhance decision-making, optimize operations, and improve patient outcomes. But effectively using AI goes beyond deploying advanced algorithms; it requires a strategic approach grounded in purpose and fortified by clarity.

Success in AI for pharma begins with defining the right business problems, selecting meaningful performance metrics, gathering high-quality data, and striking a balance between performance and explainability.

Establishing the Right Business Problem

When defining a business problem for an AI project, many organizations start with broad, ambiguous objectives like "improve patient outcomes" or "increase market share." While these goals may capture the over-arching aspiration, they lack the specificity needed for an effective machine learning (ML) initiative. Without a clear, focused problem statement, AI projects can quickly become misaligned with business needs, leading to wasted time and resources.

Instead, for a business problem to be suitable for an ML project, it needs to start with a specific question that can be answered with data. For example, instead of saying, "We want to improve patient outcomes," a more effective ML-focused problem statement would be: "Which healthcare professionals (HCPs) are most likely to adopt a new treatment that has shown better results for patients with chronic conditions?" This shifts the focus to a concrete, data-driven question that ML can address, allowing for models that provide actionable insights.

Steps to formulate a proper ML business problem:

1. Identify a Specific Challenge: Define the problem in a way that is narrow enough to be actionable. For example, "predict which HCPs are likely to increase prescribing of Drug Y following attendance at a virtual educational event."
2. Ensure It Is Measurable: Focus on a problem where success can be measured quantitatively, such as "predicting a 20% increase in prescription rates for Drug X among targeted HCPs."
3. Align with Strategic Goals: Ensure the problem directly supports the organization’s strategic objectives, like boosting sales for a specific product or reducing churn among prescribers.

By refining the business problem in this way, organizations set a strong foundation for developing effective ML solutions that drive meaningful business outcomes.

Selecting the Best Performance Metric for Your Model

Choosing the right metric to evaluate a machine learning model is essential to ensure that it delivers value to the business. Unknown to most, using "accuracy" as the default key metric can be misleading if not considered in the proper context because it only considers the overall proportion of correct predictions. But what does this mean in practice?

Assume a model is designed to detect cancer. If only 1% of the cases are positive, and the model that predicts "no cancer" for every case, it would achieve 99% accuracy—but this model would miss all actual cases of cancer, making it practically useless for real-world application.

Selecting the correct performance metric depends entirely on how the business problem is phrased:

• Classification Problems: If the goal is to categorize HCPs into groups (such as those likely to prescribe versus those who are not), you will need metrics like precision, recall, F1-score and accuracy to measure how well the model identifies the right targets.
• Regression Problems: If the objective is to predict a numerical outcome, like the number of scripts an HCP will write in the next quarter, regression metrics such as Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE) help evaluate how close the predicted values are to the actual outcomes.

• Precision: Measures how many of the positive predictions were actually correct, focusing on the "correctness" of these predictions. Precision is crucial when resources, like a sales rep's time, are limited or costly. In these cases, you want to target HCPs who are most likely to prescribe, minimizing wasted effort on less promising leads.
• Recall: Measures how many of the actual positive cases were correctly identified, focusing on capturing as many true positives as possible. Recall is important when the goal is broad outreach, like in marketing campaigns, where you want to ensure that all potential prescribers are reached, even if it includes some who may not prescribe.
• F1 Score: Combines precision and recall into a single measure, balancing both the correctness of positive predictions and the ability to identify all true positives. The F1 score is useful when both finding all potential prescribers and avoiding incorrect targets are important, offering a balanced view of model performance.

• Mean Absolute Error (MAE): Measures the average difference between predicted and actual values, showing how much, on average, predictions deviate from the true numbers. MAE is particularly useful when all errors are equally important, providing a straightforward view of overall prediction error.
• Mean Squared Error (MSE): Calculates the average of the squared differences between predicted and actual values, giving more weight to larger errors. MSE is helpful when you want to see the impact of errors and prioritize reducing significant mistakes, especially when large errors are more problematic.
• Root Mean Squared Error (RMSE): Represents the square root of MSE, providing the typical size of prediction errors in the same units as the target variable. RMSE is particularly helpful when you want to understand the error magnitude in familiar terms while still giving more weight to larger errors.

Choosing the right performance metric is critical to aligning your machine learning model's output with your business objectives. Whether dealing with classification or regression problems, selecting the appropriate metric ensures that the model's predictions are both meaningful and actionable. However, even the best-selected metrics cannot compensate for poor data quality. Next, we’ll discuss how to find the right data and ensure it is both high-quality and representative to build reliable and fair models.

Finding the Right Data: Ensuring Quality and Reducing Bias

Data is the lifeline of any AI initiative. To build effective models, it's essential to use high-quality data that is both diverse and representative of the real-world scenarios your organization faces. Poor data quality or biased datasets can lead to unreliable and potentially harmful insights. Here’s how to ensure your data meets these standards:

Bias occurs when certain groups, patterns, or behaviors are overrepresented or underrepresented in the training data, leading the model to learn patterns that do not accurately reflect the real world. For example, if your data is primarily sourced from a specific demographic or geographic area, like urban HCPs, your model may not perform well when predicting behavior for rural HCPs. In the pharmaceutical context, this means that a model trained on prescribing patterns from one setting might fail in another, limiting its usefulness.

Identifying bias involves carefully examining your data sources to ensure they are representative of the population or scenarios your model will face. Techniques like data stratification, sampling methods, and bias detection tools can help identify imbalances. Reducing bias often requires diversifying your data sources, such as including data from various demographics, regions, or practice types, to capture a wide range of behaviors and ensure fairer predictions.

A common misconception is that "more data is always better." While having a large dataset can help, it’s not just about quantity; the quality and relevance of the data are far more important. For instance, including every possible data feature in a model doesn’t necessarily improve performance; it can actually lead to overfitting, where the model becomes too tailored to the training data and performs poorly on new, unseen data.

Instead, focus on gathering clean, accurate, and relevant data directly related to the business problem. Note that high-quality data, even in smaller quantities, often yields better model performance.

In pharmaceuticals, data privacy is paramount, especially when dealing with patient data. Regulations like GDPR in Europe and HIPAA in the United States require strict controls over how personal data is collected, stored, and used. When using patient records or real-world evidence, ensure compliance by anonymizing data where necessary and obtaining explicit consent for its use. This not only adheres to legal standards but also builds trust with stakeholders.

Data in healthcare is constantly changing, and regularly updating datasets is critical to maintaining model relevance. Depending on the data strategy, using Type 1 SCD (overwriting old data with new) may simplify storage but can limit model performance by losing valuable historical trends needed for predicting long-term behaviors. Alternatively, strategies like Type 2 or 3 SCDs, which retain historical data, can help ensure your models remain robust and capable of capturing long-term patterns over time.

When data is scarce, especially for rare disease or infrequent treatments, data augmentation can generate synthetic data points to enrich the training set. Techniques like GANs (Generative Adversarial Networks) help balance datasets with few positive cases, such as rare adverse events, enhancing model robustness by providing a more balanced learning environment.

Maintaining different data versions is crucial for reproducibility and tracking model performance over time. For example, when new prescription data is added monthly, tracking each version helps assess how changes affect results. Tools like DVC (Data Version Control) or Delta Lake allow easy rollbacks, performance comparisons, and maintaining a clear change history.

Ensure your data has the right level of detail for your problem; granular HCP-level data works for individual predictions, while aggregated data may better suit market trends. Establish metrics for data quality—completeness, consistency, accuracy, timeliness—and regularly audit your data to maintain reliability and robustness.

With a solid foundation of high-quality, representative data, the next step is to address the trade-offs between performance and explainability to ensure models are both effective and trustworthy.

Balancing Performance with Explainability in AI Models

Building trust in AI solutions is crucial, especially in the pharmaceutical industry, where AI-driven decisions can significantly impact patient health and business outcomes. One of the primary challenges in achieving this trust lies in the trade-off between high-performing "black box" models and more interpretable "white box" models. Black box models, such as deep learning and reinforcement learning, often deliver superior performance due to their ability to learn from complex patterns in large datasets. However, these models come with a downside: a lack of transparency in how they arrive at their decisions, which can create hesitation or resistance among stakeholders who require clear reasoning for the AI's predictions.

Explainability becomes particularly important for stakeholders like experienced sales reps who need to trust and act on AI-generated insights. For these reps, knowing how a model reached its conclusion is not just about building trust but also about understanding how to effectively use the information to achieve the best outcomes.

If an AI model predicts that a specific doctor is likely to prescribe a new drug, the rep needs to know which factors— such as recent changes in prescribing behavior or interest in specific clinical studies —were most influential in that prediction. This transparency allows them to tailor their engagement strategies accordingly. In such cases, simpler, more interpretable models (like decision trees or logistic regression) are often preferred, even if they sacrifice some performance, because they provide the clear, actionable insights needed to build confidence and guide decision-making.

However, there are scenarios where the higher performance of black box models justifies their use. For instance, when using an LLM to generate responses for customer inquiries or create personalized marketing messages at scale, the focus is on whether the output is relevant, accurate, and compelling, rather than on understanding the model's internal decision-making process. In such cases, the business value lies in the efficiency and effectiveness of generating high-quality content quickly, regardless of how the model arrived at its answers. Here, the advantage of using a high-performing black box model outweighs the need for explainability, as long as the outputs meet the desired standards.

The decision between using high-performance black box models and more transparent white box models depends on the specific business context and the needs of the stakeholders involved. When trust, interpretability, and actionable insights are essential—such as in direct sales or treatment recommendations—explainable models are often preferred, even if they compromise some performance. On the other hand, when dealing with complex data patterns or large-scale tasks where the accuracy of outcomes is more important than understanding the reasoning behind them, black box models can provide a significant advantage. Striking the right balance between performance and explainability is key to developing AI solutions that are both effective and aligned with business goals and stakeholder expectations.

Building a Path to Success with AI in Pharma

AI is unlocking new possibilities in the pharmaceutical industry, from predicting patient needs to optimizing engagement strategies with healthcare professionals. Yet, with its immense potential comes the responsibility to ensure these tools are used thoughtfully and ethically. It’s not enough to deploy advanced algorithms.

Success with AI requires a deep understanding of the business challenges, careful selection of performance metrics, and a commitment to high-quality, unbiased data.

Balancing these elements with a focus on transparency and ethical considerations helps build trust in AI-driven solutions, ensuring they genuinely improve patient outcomes and drive meaningful business results. The challenge lies not just in building high-performing models but in doing so in a way that aligns with the needs of stakeholders and respects the complexities of healthcare.

Looking ahead, it’s clear that the future of AI in pharma is bright, but it also requires a commitment to continuous improvement and ethical considerations. In the final part of our series, we’ll delve into the factors making AI ready for primetime and how to effectively start your AI journey.