Poster Presentations

Objectives: Informing trial design assumptions with real-world data (RWD) often requires clinical detail—disease severity, pain level, symptom timing—that is absent from structured claims and EHR fields but routinely documented in free-text clinical notes. In a total knee arthroplasty (TKA) use case among knee osteoarthritis (OA) patients, we observed a substantial gap between key-opinion-leader-anticipated 1-year TKA conversion rates and the rate estimated from structured EHR data alone. This study leveraged large language models (LLMs) to extract trial-grade phenotypes—OA severity, pain severity, and pain at night—directly from unstructured clinical notes, aiming to close this gap and validate trial feasibility assumptions.

Methods: We developed a four-step LLM-based extraction pipeline—cohort query, LLM-based NLP phenotyping, result aggregation, and human review—using GPT-OSS-20B deployed on the nFerence platform. Prompt development was guided by four design principles: hyperparameter tuning for reproducibility, general prompt engineering for an auditable framework, fit-for-purpose engineering to anchor phenotypes to the indication, and explicit competing-etiology controls to reduce false positives. Pipeline performance was validated against expert chart review on 200 independent notes.

Key Contributions: This work demonstrates a reproducible, auditable framework for LLM-based phenotyping of severity and pain constructs from unstructured clinical text, with fit-for-purpose prompt design that generalizes across clinical indications. By recovering previously unmeasured clinical detail, the pipeline enables more accurate validation of trial feasibility assumptions from RWD and establishes a scalable approach to multi-modal RWD integration—including longitudinal unstructured clinical notes—for profiling heterogeneous disease progression trajectories. More broadly, this approach opens new avenues for population selection, optimized intervention timing, and biomarker-driven trial design across therapeutic areas.

High-dimensional prediction problems often exhibit structured sparsity: only a few predictors are relevant, and relevant predictors may belong to related groups. Existing grouped Bayesian Additive Regression Tree (BART) methods can exploit this structure, but they generally require the grouping information to be known in advance. In many real applications, however, such group structure is unavailable or poorly defined. Our work addresses this limitation by studying BART for high-dimensional prediction with unknown group structures, targeting both sparsity across groups and sparsity within groups.

Causal inference is central to scientific discovery, yet choosing appropriate methods remains challenging because of the complexity of both statistical methodology and real-world data. Inspired by the success of artificial intelligence in accelerating scientific discovery, we introduce InferenceEvolve, an evolutionary framework that uses large language models to discover and iteratively refine causal methods. Across widely used benchmarks, InferenceEvolve yields estimators that consistently outperform established baselines: against 58 human submissions in a recent community competition, our best evolved estimator lay on the Pareto frontier across two evaluation metrics. We also developed robust proxy objectives for settings without semi-synthetic outcomes, with competitive results. Analysis of the evolutionary trajectories shows that agents progressively discover sophisticated strategies tailored to unrevealed data-generating mechanisms. These findings suggest that language-model-guided evolution can optimize structured scientific programs such as causal inference, even when outcomes are only partially observed.

AI and statistics are transforming modern medicine by improving diagnosis, prognosis, treatment planning, and clinical decision-making. Statistical models, logistic regression, survival analysis, Bayesian models, time-series models, and ML techniques help predict disease risk, analyze patient outcomes, and support personalized healthcare. These methods offer major opportunities for early disease detection, patient monitoring, and efficient healthcare delivery. However, risks such as biased data, overfitting, lack of model transparency, privacy issues, and overreliance on automated systems remain important challenges. Therefore, AI and statistical models must be carefully validated using methods such as cross-validation, external validation, calibration, sensitivity analysis, and performance measures, including accuracy, sensitivity, specificity, and AUC. Responsible use of AI in medicine requires reliable data, transparent models, ethical regulation, and clinical expertise.

Generating regulatory-grade real-world evidence (RWE) from observational health data is critical for drug and device decisions, yet remains slow, error-prone, and dependent on scarce expertise across medicine, epidemiology, biostatistics, and causal inference. We present the Causal Roadmap Copilot, an agentic AI system that helps researchers produce high-quality causal analyses by operationalizing the Causal Roadmap (Petersen & van der Laan, 2014). It is a human-in-the-loop, multi-agent architecture mirroring the roadmap's stages: specifying a causal question and estimand, building a causal model, establishing identification, performing statistical estimation, and supporting interpretation and sensitivity analysis. An orchestration layer manages the iterative, long-horizon workflow as specialized agents guide users to draft the study protocol and statistical analysis plan. We demonstrate an end-to-end prototype with early pilot evaluations. By guiding researchers through the entire roadmap, the Copilot lowers the time and expertise barriers to high-quality RWE while guarding against common threats to validity such as ill-defined estimands. It elicits, justifies, and archives every assumption and analytic decision alongside prespecified analysis plans and reproducible code, strengthening the transparency, reproducibility, and regulatory defensibility of real-world evidence across diverse healthcare data sources.

The increasing adoption of consumer wearable devices, like wristbands and smartwatches, offers an unprecedented opportunity to capture longitudinal, real-world sleep and activity data at scale. However, their integration into pharmaceutical research and therapeutic optimization remains limited by challenges in validation, interpretability, and regulatory alignment.

We propose a framework to estimate treatment response in sleep disorders using wearable-derived digital sleep biomarkers. Temporal Convolutional Network (TCN) are applied directly to multi-sensor time-series data (triaxial accelerometer and heart rate) structured in sliding temporal windows. Dilated residual convolutional blocks capture short- and medium-range temporal dependencies, enabling improved sleep–wake classification and refined characterization of sleep continuity and fragmentation. The network outputs probabilistic sleep state sequences from which clinically interpretable biomarkers are derived and evaluated longitudinally relative to individualized baselines. Trained using class-weighted binary cross-entropy and cross-validation against expert sleep staging, the model achieved an F1 score of 0.72 on an independent holdout set for short sleep episodes and demonstrated robustness under conditions of minimal movement and subtle physiological variation, supporting its applicability to real-world ambulatory monitoring.

Within a pharmaceutical partnership setting, such a system enables objective, high-frequency monitoring of treatment response in sleep disorders, including hypersomnolence conditions. By linking medication exposure data with wearable-derived digital endpoints, the framework supports dynamic treatment optimization, early signal detection of therapeutic response, and real-world effectiveness assessment.

Future work will explore agentic AI systems capable of automated data quality control, generation of interpretable clinical summaries, enabling patient-centered decision support systems that accelerate drug development while promoting measurable improvements in public health.

siRNAs are an emerging therapeutic modality, with six FDA-approved drugs and a market projected to exceed $12 billion by 2033. However, designing effective siRNAs remains challenging due to complex interdependencies between sequence composition, thermodynamic properties, and inherent potency. While computational approaches like DSIR exist, the field lacks the diversity of methods available for adjacent domains such as small-molecule design. To address this gap, we formulate siRNA design as a sequential optimization problem rather than static prediction, enabling the application of reinforcement learning techniques previously unused for this task.

We implement two complementary RL approaches: local-window context-based Q-learning and a full-sequence Deep Q-Network (DQN). The local-window method enables tabular learning over fixed-size windows of the siRNA sequence. The DQN encodes 19-nucleotide sequences as 76-dimensional one-hot vectors and uses experience replay with Double-DQN to handle sparse reward signals. Both methods employ potential-based reward shaping to provide dense feedback on individual mutations. The reward function combines i-Score (intrinsic potency) and minimum free energy estimates, weighted equally and normalized to lie between 0 and 1. Episodes consist of 50 iterative single-nucleotide mutations starting from random sequences.

Both RL methods substantially outperformed random mutation baselines. Context-based Q-learning achieved a best normalized reward of 0.898 (vs. 0.624 baseline), while DQN achieved 0.885. The local-window method exhibited an inverse relationship between window size and performance, with single-position windows yielding the highest rewards. The DQN agent demonstrated non-uniform mutation behavior, concentrating edits around positions 3, 6, 11, and 18, with reward-improving substitutions showing interpretable position-base preferences. Temporal-difference loss remained stable throughout training, and PCA analysis of visited sequences revealed a smooth, continuous embedding with broad contiguous regions of high-reward solutions. Notably, the DQN-discovered best sequence exhibited known siRNA design principles: A/U-enrichment in the seed region (positions 2–8), thermodynamic asymmetry, and GC content within empirically recommended ranges.

This work demonstrates that reinforcement learning provides a structured and effective framework for siRNA design, with both local and global sequence representations supporting high-performing policies. The comparable performance of fundamentally different approaches suggests the underlying reward landscape is well-behaved and navigable via mutation-based optimization. The discovered sequences align with established empirical principles, lending credibility to the approach. Future directions include hybrid local-global policies, integration with RNA foundation models for richer embeddings, and evaluation on multiple target mRNAs with in vitro validation to assess the biological relevance of in silico discoveries.

Adaptive Targeted Maximum Likelihood Estimation (A-TMLE) provides a principled framework for targeting within data-adaptive working models to achieve efficient, debiased inference. The recently developed regHAL-ATMLE realizes this framework for Highly Adaptive Lasso (HAL) working models via regularized projection of influence functions onto the score space. We propose Targeted Deep Architectures (TDA), a framework that brings the reg-ATMLE idea to deep learning: the neural network itself serves as the parametric working model, and targeting proceeds via regularized projection of influence functions onto network score vectors (gradients of the loss). Concretely, TDA treats the full trained neural network as an offset---analogous to regHAL's super learner offset construction---then selects a targeting subset of weights and fluctuates them via targeting gradient descent until the projected efficient influence curve equation is approximately solved. Unlike sieve-based working models, the neural network architecture is typically expressive enough to well-approximate the efficient influence function direction without requiring structural augmentation. TDA extends naturally to multi-dimensional causal estimands (e.g., entire survival curves) by merging targeting gradients into a single universal update. Theoretically, TDA inherits A-TMLE guarantees including double robustness and semiparametric efficiency, and can be dually justified as a general regularized TMLE directly approximating the semiparametric efficient influence function. We additionally explore incorporating the targeting step during training (training-time targeting, TTR) and its combination with post-training TDA. Empirically, across three settings---the IHDP benchmark, a synthetic DGP with severe positivity violations, and survival curve estimation under concordant informative censoring---TDA matches classical TMLE for ATE debiasing, and the Combined estimator (TTR followed by post-training TDA) achieves the best overall ATE performance. However, training-time targeting does not improve the 50-dimensional survival target in our experiments, suggesting that naively blending predictive and targeting objectives faces challenges as the number of simultaneous EIF equations grows---though alternative training-time strategies remain to be explored.

Adverse drug reactions (ADRs) represent a significant public health challenge and are among the leading causes of preventable morbidity and mortality. Existing pharmacovigilance approaches often fail to fully leverage the rich temporal information available in longitudinal healthcare records. We propose a novel graph-based machine learning framework for ADR discovery using large-scale Medicare claims data. The dataset contains comprehensive patient-level information, including demographic characteristics, diagnoses, procedures, treatments, and healthcare service utilization.

Our approach models each patient's clinical trajectory as a Precision Disease Network (PDN), a temporal graph that captures the evolution of treatment exposures and subsequent clinical events. Unsupervised clustering is employed to identify cohorts of patients with similar adverse-event trajectories, while graph summarization techniques are used to uncover frequent subgraphs associated with specific treatments. By explicitly preserving temporal ordering and inter-event dependencies, the proposed framework captures patterns that are often lost in conventional feature-based representations.

Experimental results indicate that temporal graph-based representations significantly enhance ADR signal detection compared with static disease-state models. Furthermore, the approach reduces the impact of confounding and latent factors by leveraging the sequential structure of patient histories. These findings demonstrate the potential of patient-centric temporal networks as a powerful foundation for scalable, data-driven pharmacovigilance and drug safety surveillance.

Covariate adjustment is a general method for improving precision when estimating treatment effects in randomized trials and is recommended by the FDA in its 2023 guidance when baseline variables are prognostic for the primary outcome. We focus on a method highlighted in that guidance called ``standardization" (or ``g-computation") for estimating the marginal treatment effect. We address the question of how to reliably estimate variance for binary outcomes when marginal outcome probabilities are close to 0 or 1. We propose an influence function-based leave-one-out cross-validated (IF-LOO) variance estimator for the standardized difference-in-means average treatment effect. Through simulation studies, we show that this estimator provides appropriate type-I error control and performs reliably in challenging settings where existing methods can yield inflated type-I error or fail entirely, such as when outcome events are rare or sample sizes are small. In addition to having desirable statistical properties, we derive a closed-form expression for the proposed estimator, enabling straightforward and reliable implementation by study statisticians. The robust finite-sample performance and ease of implementation suggest the IF-LOO variance estimator is a prudent default choice for standardization in clinical trials.

Current healthcare AI evaluation is largely organized around benchmark leaderboards, where models are compared by aggregate accuracy on datasets such as MedQA, MedMCQA, PubMedQA, and MedXpertQA. Prior work has improved benchmark scale, difficulty, modality coverage, and clinical relevance, but most evaluations still treat benchmark scores as direct evidence of medical competence without sufficiently testing whether the benchmark actually measures the intended construct (Bean et al., 2026).

This study applies a psychometric framework to medical AI benchmarking. Using MedXpertQA as the main benchmark and comparing it with MedQA, MedMCQA, and PubMedQA, the study examines whether these benchmarks measure the same underlying construct of medical reasoning or instead capture different mixtures of exam knowledge, biomedical recall, clinical reasoning, and evidence-based inference. Item response theory is used to estimate model ability, item difficulty, and item discrimination across benchmarks, providing a more nuanced alternative to raw accuracy. Measurement invariance analysis is then test whether scores are meaningfully comparable across benchmarks. By evaluating both model performance and benchmark validity, this poster aims to develop a more rigorous framework for selecting appropriate medical AI models and for interpreting benchmark scores in healthcare contexts.

Objectives

The REBOOT trial suggests that beta blockers after a heart attack may not have the same effect for every patient, especially when we look at sex and heart function, measured by ejection fraction. This raises an important question: can Targeted Learning and machine-learning methods help identify differences in treatment response between women and men? The objective of this study is to estimate sex-specific beta-blocker treatment effects after heart attack and explore whether these effects also vary across ejection-fraction groups. The broader goal is to support more personalized treatment decisions in cardiovascular pharmaceutical medicine.

Methods

We will first use simulated patient data with characteristics such as age, sex, ejection fraction, diabetes, hypertension, smoking, and cardiovascular history. Beta-blocker treatment will depend on patient characteristics to create confounding similar to what may occur in observational real-world data. We will use a Targeted Learning framework, applying Super Learner to flexibly estimate treatment and outcome models and Targeted Maximum Likelihood Estimation (TMLE) to estimate causal treatment effects. Effects will be estimated overall and separately for women and men, with additional analysis across ejection-fraction groups. Because the data are simulated, the true treatment effects will be known and can be compared with the estimated effects. A later phase of this research is planned using real-world IQVIA data.

Key Contributions

This study provides a reproducible framework for studying sex-specific and personalized drug treatment effects using Targeted Learning and machine learning. It connects an important question in pharmaceutical treatment with modern causal inference methods and provides a methodological foundation for future evaluation using real-world data of beta-blocker treatment in cardiovascular pharmaceutical research.

Given the growing reliance on LLMs in generic workflows in science, we sought to understand how LLMs reason about causal inference and challenges that often go unresolved in real life sciences. Using linear probes, out-of-distribution transfer tests, residual-stream patching, and behavioral prompt batteries, we find the following. First, selectivity is decodable only when both the primary assay and a placebo-like control assay are available. In the absence of the null assay, the probe decreases in performance from 0.98 to 0.58 AUROC. Second, when the information is perturbed in two ways: a) reparameterization of units from pEC50 to nanomolar notation, and b) full dose-response curves rather than lossy summary statistics, the LLMs do not reason as accurately as expected, implying that they may over-rely on simple arithmetic difference scores rather than full nonlinear contrast of two experiments. Third, when the model is asked to interpret experimental results without a control assay, it only asks for a control assay 0% of the time in five of seven prompt framings. Asking the model to interpret a colleague’s answer elicited the need for control assays 44% of the time, at best, in one open-weight model. In short, novel scientific experiments with rich sets of orthogonal controls and validation offer a rich testbed for benchmarking LLM and agentic reasoning about scientific causal inference in real-world science. Our preliminary experiments also demonstrate that how LLMs represent and reason about scientific concepts relevant to measurement validity of scientific experiments is of independent interest.