Predicting neutralization susceptibility to combination HIV-1 monoclonal broadly neutralizing antibody regimens

Abstract

Combination monoclonal broadly neutralizing antibodies (bnAbs) are currently being developed for preventing HIV-1 acquisition. Recent work has focused on predicting in vitro neutralization potency of both individual bnAbs and combination regimens against HIV-1 pseudoviruses using Env sequence features. To predict in vitro combination regimen neutralization potency against a given HIV-1 pseudovirus, previous approaches have applied mathematical models to combine individual-bnAb neutralization and have predicted this combined neutralization value; we call this the combine-then-predict (CP) approach. However, prediction performance for some individual bnAbs has exceeded that for the combination, leading to another possibility: combining the individual-bnAb predicted values and using these to predict combination regimen neutralization; we call this the predict-then-combine (PC) approach. We explore both approaches in both simulated data and data from the Los Alamos National Laboratory’s Compile, Neutralize, and Tally NAb Panels repository. The CP approach is superior to the PC approach when the neutralization outcome of interest is binary (e.g., neutralization susceptibility, defined as inhibitory 80% concentration < 1 μg/mL). For continuous outcomes, the CP approach performs nearly as well as the PC approach when the individual-bnAb prediction algorithms have strong performance, and is superior to the PC approach when the individual-bnAb prediction algorithms have poor performance. This knowledge may be used when building prediction models for novel antibody combinations in the absence of in vitro neutralization data for the antibody combination; this, in turn, will aid in the evaluation and down-selection of these antibody combinations into prevention efficacy trials.

Introduction

Monoclonal broadly neutralizing antibody (bnAb) regimens for HIV-1 prevention have been researched extensively [1–4]. The Antibody Mediated Prevention (AMP) randomized efficacy trials of VRC01 versus placebo (HVTN 704/HPTN 085 and HVTN 703/HPTN 081, NCT02716675 and NCT02568215, respectively) provided evidence that a bnAb can prevent HIV-1 acquisition [5]. Defining a VRC01-susceptible strain as having 80% inhibitory concentration (IC₈₀) <1 μg/mL, the AMP trials showed that estimated prevention efficacy of VRC01 versus placebo for VRC01-susceptible strains was 75.4% [5]. Estimated prevention efficacy for strains with 50% inhibitory concentration (IC₅₀) <1 μg/mL was approximately 50% [5]. Combination regimens consisting of multiple bnAbs targeting several HIV-1 Env epitopes have greater in vitro neutralization potential than their constituent bnAbs [6–8], and as such are currently being prioritized for HIV-1 prevention [3].

Several methods have been developed for predicting individual [9–17] and both individual and combination [18] in vitro neutralization outcomes from Env sequence data. Many of these algorithms were trained using the Los Alamos National Laboratory’s Compile, Analyze, and Tally NAb Panels (CATNAP) database [19] and yield good prediction performance (small mean squared error or large area under the receiver operating characteristic curve [AUC]) for individual bnAbs, with AUC often much greater than 0.5 [11, 13, 16–18]. One limitation of CATNAP is that while sufficient numbers of pseudoviruses have been measured for neutralization against individual bnAbs, far fewer (and in many cases, zero) have measured neutralization against combination regimens. Additionally, many more pseudoviruses tend to have measurements of IC₅₀ than IC₈₀, likely due to the higher sensitivity of the former. To maximize the amount of information available when investigating combination regimens, combination neutralization is often defined as a function of the individual neutralization values, and then used as an outcome for training the prediction model [6, 18, 20]. We refer to this as the combine-then-predict (CP) approach. Other outcomes, including multiple susceptibility—defined as the binary indcator that k bnAbs in the regimen have IC₈₀ < 1 μg/mL—could be used instead. Each of these models results in a prediction function that takes as input Env sequence data and returns a predicted neutralization outcome. These prediction models may be used to compare bnAb regimens when determining which to pursue in further research [17, 21], since neutralization is associated with prevention efficacy [5, 22].

However, prediction performance for some individual bnAbs has exceeded that for the combination [21]; in some cases, predicting the individual-bnAb neutralization seems to be an easier task than predicting combination neutralization. This suggests a second possible method for predicting combination neutralization outcomes: combining the individual-bnAb predicted values and using these combinations to predict neutralization by the combination regimen. We refer to this as the predict-then-combine (PC) approach. In this article, we explore the PC approach and contrast it with the previous CP approach of directly predicting the combined neutralization. We study the performance of both approaches in simulated data and data from CATNAP, aiming to provide clarity on when each approach should be used. We use the term prediction performance to refer to a summary of the discrepancy between predictions from an algorithm and observed neutralization outcomes; possibilities include R-squared for continuous outcomes and AUC for binary outcomes, among others.

Materials and methods

Predicting neutralization values

Neutralization values can be predicted as a function of Env sequence AA data using standard prediction algorithms for high-dimensional data. It is important to account for the large number of features compared to the often smaller number of sequences with measured neutralization. Hake and Pfeifer [11] compared the lasso [23], random forests [24], and support vector machines [25] (also evaluated in [9]), finding good performance for each method. Neural networks [26] have been used in several studies [10, 14, 17], while Rawi et al. [13] used gradient boosted trees [27]. Finally, Magaret et al. [16] and Williamson et al. [21] used the Super Learner [28], an ensemble method. Each of these prediction approaches has been shown to perform well for predicting HIV neutralization in certain settings.

In our analyses described below (Results), we will focus on the lasso for two reasons. First, it is a commonly-used prediction algorithm that performs well in many settings. More importantly, our goal is to compare prediction performance using the same modeling approach between two methods for combining individual-bnAb neutralization values, rather than building the best possible prediction model for a given outcome.

Combining individual-bnAb neutralization values

The first method of predicting combination bnAb regimen susceptibility that we consider, which we refer to as pre-prediction combination or the combine-then-predict (CP) approach, involves applying a mathematical model to combine measured in vitro inhibitory concentration prior to training a prediction model. We consider two mathematical models defined by Wagh et al. [6] for combining the in vitro neutralization values from J constituent bnAbs in a bnAb combination regimen: an additive model,(1)and a Bliss-Hill model,(2)The Bliss-Hill solution is obtained using the method of Brent [29]. These models have only been validated for combinations where the constituent bnAbs target different HIV-1 Env epitopes [6, 20]. Combination susceptibility based on IC₈₀, a binary outcome, is defined as the binary indicator that combination IC₈₀ < 1 μg/mL. Combination IC₅₀ (the 50% inhibitory concentration) and combination susceptibility based on IC₅₀ are defined in a similar manner, with IC₅₀ replacing IC₈₀ in Eq (1) and h_j in Eq (2). Then, a prediction model can be trained by using the Env features to predict the combination neutralization outcome (either continuous or binary). Any prediction modelling approach can be used: for example, the lasso [23] or random forests [24]. This approach is laid out in the left-hand column of Fig 1.

[Figure omitted. See PDF.]

An illustration of the combine-then-predict (top) and predict-then-combine (bottom) approaches to predicting combination regimen in vitro neutralization, for a two-bnAb combination regimen.

One could instead perform post-prediction combination, which is the second method of predicting combination bnAb regimen susceptibility that we consider. We will refer to this as the predict-then-combine (PC) approach. This method involves first training J prediction models using the Env features to predict the continuous neutralization outcome (e.g., IC₈₀) of each individual bnAb in the combination regimen. As above, any prediction modelling technique can be used for these individual prediction models. We then obtain predicted neutralization and use either the additive or Bliss-Hill model to combine the predicted values, using the predictions g_j in place of the measured IC_80,j or IC_50,j. To predict susceptibility, we check if the combined predicted value is less than 1 μg/mL.

Creating synthetic datasets

We first consider synthetic datasets where we determine the relationship between Env sequence features and neutralization outcomes. This allows us to evaluate the CP and PC approaches in a controlled setting where we know the true neutralization outcomes. For sample size n ∈ {100, 200, …, 1000} and three bnAbs, we generated 1000 Env AA sequence features and IC₈₀ values for each bnAb and each of the n simulated pseudoviruses. We assumed that conditional on the Env AA sequence features, the IC₈₀ values were independent. We ranged the sample size to provide a comprehensive evaluation of the CP and PC approaches for different amounts of data, to see when results between the two approaches separate. In general, prediction performance should improve with increasing sample size. In our simulated setting, only ten Env AA sequence features impact the IC₈₀. This setup, where there are a large number of Env AA sequence features but only a small number of highly predictive features, is similar to the setting and results of analyses using CATNAP data [16].

We considered two scenarios: one where there was a weak relationship between these ten Env AA sequence features and IC₈₀, and one where there was a strong relationship. The proportion susceptible to each of the three bnAbs is 0.36, 0.29, and 0.42, respectively, in the weak-relationship scenario; and is 0.23, 0.21, and 0.27, respectively, in the strong-relationship scenario. We then obtained true combination IC₈₀ using the additive model described above. The proportion susceptible using combination IC₈₀ is 0.74 in the weak-relationship scenario and 0.38 in the strong-relationship scenario. We repeated the above data-generating process 2500 times for each sample size.

Creating CATNAP datasets

We consider several bnAb combination regimens with data available in CATNAP. These regimens consist of those undergoing HIV Vaccine Trials Network (HVTN) or HIV Prevention Trials Network (HPTN) clinical testing as of October 2022 [21] and those with at least 125 pseudoviruses with direct measurement of neutralization by the bnAb combination in CATNAP [18]. The full list of bnAb combination regimens is provided in Table 1. Each combination regimen consists of individual bnAbs targeting different HIV-1 Env epitopes: V1V2, V3, CD4 binding site (CD4bs), fusion peptide, or membrane proximal external region (MPER) [12, 13, 18] (Table 1).

[Figure omitted. See PDF.]

Broadly neutralizing antibody (bnAb) combination regimens either with at least 125 sequences in CATNAP with direct measurement of combination neutralization or undergoing HVTN/HPTN clinical testing as of October 2022. The HIV-1 Env epitopes targeted by each bnAb in the combination regimen are listed in order. Combination regimens highlighted in the Results section below are highlighted in bold.

We began by using the Super LeArner Prediction of NAb Panels (SLAPNAP) tool [18] to create an analysis dataset for each unique bnAb in Table 1 consisting of individual-bnAb IC₅₀ and IC₈₀ values, along with Env gp160 sequence features. We also obtained datasets of directly measured combination neutralization for those bnAb regimens in Table 1 with such information (Table 1 column 3 not equal to zero). The features for each pseudovirus consist of geographic information (binary indicator variables describing the geographic region of origin), viral geometry variables (length, number of sequons, and number of cysteines in the Env, gp120, V2, V3, and V5 regions), and amino acid sequence variables (binary indicators of residues containing amino acids, frameshifts, gaps, stops, or sequons at each HXB2-referenced site in gp160). Each dataset had approximately 6000 columns.

We next merged the individual-bnAb neutralization values and directly-observed combination neutralization values (if available) into a single dataset for each bnAb regimen in Table 1. We computed model-predicted combination neutralization using both the additive and Bliss-Hill models described above. The statistical learning approaches described below aimed to predict the following neutralization outcomes for each individual bnAb in the regimen: (1) the continuous log₁₀ IC₅₀; (2) the continuous log₁₀ IC₈₀; (3) the dichotomous outcome IC₅₀ susceptibility, defined as IC₅₀ < 1 μg/mL; and (4) the dichotomous outcome IC₈₀ susceptibility, defined as IC₈₀ < 1 μg/mL. We also aimed to predict the following outcomes for each combination regimen and both the additive and Bliss-Hill combination models: (5) the continuous log₁₀ combination IC₅₀; (6) the continuous log₁₀ combination IC₈₀; (7) the dichotomous outcome combination IC₅₀ susceptibility, defined as combination IC₅₀ < 1 μg/mL; and (8) the dichotomous outcome combination IC₈₀ susceptibility, defined as combination IC₈₀ < 1 μg/mL. For combination regimens with directly-observed combination neutralization, we further aimed to predict outcomes (1) through (4) but using the directly-observed combination neutralization values.

Obtaining predictions and assessing prediction performance

We took the same approach to obtaining predictions and assessing prediction performance in both the synthetic and CATNAP data settings. For each given dataset, we used the lasso [23] to (i) predict the individual-bnAb neutralization (both continuous and binary); (ii) predict the combination neutralization directly (both continuous and binary), i.e., use the CP approach; and (iii) combine the continuous-valued predictions from (i) using the additive model, i.e., use the PC approach. We used 10-fold cross-validation to obtain the optimal lasso tuning parameter in all cases. We then assessed prediction performance using an additional layer of 10-fold cross-validation, where the lasso was trained on nine-tenths of the data and prediction performance was estimated on the remaining tenth. We evaluated prediction performance using cross-validated (CV) R-squared for continuous outcomes and cross-validated area under the receiver operating characteristic curve (CV AUC) for binary outcomes. We also obtained a 95% confidence interval for the prediction performance [30, 31].

In the synthetic setting, we know the true combination neutralization values. Prediction performance is evaluated using these observed values and the predictions from either the CP or PC approach. We computed the average point estimate, empirical standard error of the point estimates, and average confidence interval width after applying the prediction performance estimation procedure described above to each of the 2500 datasets.

In the CATNAP setting, we only know the true combination neutralization values for those bnAb regimens in Table 1 with a nonzero number of sequences with IC₈₀ data measured against the bnAb combination. We evaluated individual-bnAb prediction performance as described above. Performance for predicting combination regimen neutralization was measured by either combining the individual-bnAb predictions or by directly predicting the combination neutralization. We considered both the additive and Bliss-Hill models described above for combining neutralization values. For a given combination regimen of interest, to evaluate both the CP and PC approaches fairly, we held out pseudoviruses with observed neutralization for the combination regimen as a test set.

Software and compiled datasets

The procedures described above (and the results described below) can be reproduced using code available on GitHub at https://github.com/bdwilliamson/hiv_neutralization_susceptibility_supplementary and at Zenodo (https://zenodo.org/doi/10.5281/zenodo.10372961). In addition to code, these repositories host compiled datasets following the procedure described in Methods (Creating CATNAP datasets). We used the latest compiled version of SLAPNAP [18] to compile CATNAP datasets, using Docker [32] version 24.0.0. All analyses were performed using R [33]; the simulations were performed in version 4.3.1, while the data analyses were performed in version 4.2.2. Specific R packages and versions used are: here version 1.0.1 [34], readr version 2.1.5 [35], janitor version 2.2.0 [36], glmnet version 4.1.8 [37, 38], data.table version 1.15.4 [39], dplyr version 1.1.4 [40], tidyr version 1.3.1 [41], parallel version 4.4.0 [33], foreach version 1.5.2 [42], doParallel version 1.0.17 [43], cvAUC version 1.1.4 [44], vimp version 2.3.3 [30, 45], ggplot2 version 3.5.1 [46], cowplot version 1.1.3 [47], and gridExtra version 2.3 [48].

Results

Performance of combine-then-predict and predict-then-combine on synthetic data

We display the results for the simulated data described in Methods (Creating synthetic datasets) in Figs 2 and 3. In Fig 2 (left-hand column), we see that the prediction performance varies across individual bnAbs in the combination regimen, with the second bnAb having uniformly highest prediction performance; the third bnAb having uniformly lowest prediction performance; and the first bnAb having prediction performance between the other two. Prediction performance is degraded in the weak-relationship case, which is expected. Confidence interval width decreases with increasing sample size for both outcomes, as expected (right-hand column of Fig 2), though width tends to be higher in the weak-relationship case than in the strong-relationship case. In Fig 3 (left-hand column), we see that prediction performance is similar across the two methods of predicting combination neutralization for the continuous outcome (nearly within Monte-Carlo error) in the strong-relationship case, but is uniformly higher when using the CP approach in the weak-relationship case, corresponding to a case where CV R-squared for all individual bnAbs was less than 0.5. Training prediction models using the CP approach results in higher prediction performance for susceptibility regardless of the strength of relationship between predictors and outcome. Confidence interval width is uniformly smaller for the CP method (Fig 3 right-hand column).

[Figure omitted. See PDF.]

Prediction performance (left-hand column) and 95% confidence interval width (right-hand column) versus sample size for predicting IC₈₀ (top row) and IC₈₀ < 1 μg/mL for each of three individual bnAbs averaged over 2500 simulated datasets. Prediction performance is evaluated against the observed IC₈₀ values for the given bnAb. Panels within rows denote a strong or weak relationship between the Env AA predictors and the outcome. Shapes denote the individual bnAbs, and Monte-Carlo error is displayed in error bars.

[Figure omitted. See PDF.]

Prediction performance (left-hand column) and 95% confidence interval width (right-hand column) versus sample size for predicting combination IC₈₀ (top row) and combination IC₈₀ < 1 μg/mL using both the CP and PC approaches, averaged over 2500 simulated datasets. Prediction performance is evaluated against the observed combination IC₈₀. Panels within rows denote a strong or weak relationship between the Env AA predictors and the outcome. Shapes denote the approach, and Monte-Carlo error is displayed in error bars.

Performance on bnAb combinations in CATNAP

We further explored the distinction between the CP and PC approaches by obtaining prediction performance for the bnAb combination regimens in Table 1. We followed the approach described in Methods (Creating CATNAP datasets) to define the dataset for each bnAb regimen.

For the bnAb regimens with direct lab-measured neutralization for the combination regimen (third column of Table 1), we further assessed the performance of both approaches for predicting these neutralization values. Below, we highlight the results for two bnAb regimens, VRC07–523-LS + 10–1074 (chosen because it is a regimen in clinical testing) and 10–1074 + 10E8 (chosen because it is a regimen with direct measurement of neutralization for the regimen). Since predicting the directly-measured neutralization against a combination regimen is an important goal, in cases where this direct measurement exists we evaluate prediction performance for this outcome as well as the model-combined outcome.

The proportion of pseudoviruses in CATNAP susceptible (measured using IC₈₀, defined as IC₈₀ < 1 μg/mL) to VRC07–523-LS, 10–1074, and 10E8 are 78.75%, 48.89%, and 21.34%, respectively. The proportion estimated to be sensitive to VRC07–523-LS + 10–1074 is 89.5% and 93.5% for the additive and Bliss-Hill models, respectively; estimated sensitivity is computed using the model based on the individual-bnAb neutralization data in CATNAP, as described in Methods (Combining individual-bnAb neutralization values). The proportion estimated to be sensitive to 10–1074 + 10E8 is 59.65% and 79.7% for the additive and Bliss-Hill models, respectively, while the true proportion susceptible is 64.8%. This true proportion is based on a direct measurement of IC₅₀ for the combination regimen in CATNAP (Table 1; see Methods (Creating CATNAP datasets)). Based on the number of pseudoviruses, the number of susceptible viruses is comparable to n ∈ {600, …, 1000} in the simulations above. Results for the remaining bnAb regimens from Table 1 can be found in S1 Appendix.

We display the results for predicting neutralization susceptibility to VRC07–523-LS + 10–1074 in Fig 4. In the left-hand column, we see that prediction performance is higher for 10–1074 for both continuous and binary neutralization outcomes. In the right-hand column, we see that for both continuous and binary outcomes, the CP approach leads to better prediction performance (the CV R-squared ranges from 0.07 to 0.41 lower for PC compared to CP; the CV AUC ranges from 0.05 to 0.21 lower). This improved performance of the CP approach is particularly striking for the Bliss-Hill model in the continuous-outcome setting. This combination regimen does not have direct measurements of neutralization by the combination regimen in CATNAP. As in the simulations, prediction performance for the combination tends to be less than the best individual-bnAb prediction performance. When using the CP approach, prediction performance for the combination tends to be greater than the worst individual-bnAb prediction performance.

[Figure omitted. See PDF.]

Prediction performance for continuous (top row, CV R-squared) and binary (bottom row, CV AUC) neutralization outcomes for individual bnAbs (left-hand column) and the combination (right-hand column) VRC07–523-LS + 10–1074. For individual bnAbs, prediction performance is evaluated against the observed IC₅₀ or IC₈₀ values for the given bnAb; shapes denote the bnAb. For combination bnAbs, prediction performance is evaluated against the calculated combination IC₅₀ or IC₈₀ values based on the observed bnAb-specific values using the additive or Bliss-Hill method; shapes denote the combination method (additive or Bliss-Hill) and color denotes the approach (CP or PC). Error bars reflect 95% confidence intervals.

We display the results for predicting neutralization susceptibility to 10–1074 + 10E8 in Fig 5. In the left-hand column, we see that prediction performance is again higher for 10–1074 for both continuous and binary neutralization outcomes. In the right-hand column, we see that prediction performance for continuous outcomes is similar between the CP and PC approaches (CV R-squared ranges from 0.07 lower for PC compared to CP [Bliss-Hill model, IC₅₀] to 0.32 higher [Bliss-Hill model, directly-observed combination IC₈₀]); for binary outcomes, performance is better when using the CP approach (CV AUC ranges from 0.03 higher for PC compared to CP [Bliss-Hill model, directly-observed IC₅₀ susceptibility] to 0.34 lower [additive model, directly-observed IC₅₀ susceptibility]). This antibody combination does have direct measurements of neutralization by the combination regimen in CATNAP, and the results follow a similar pattern to the combination neutralization results. These results are similar to the results from the simulations.

[Figure omitted. See PDF.]

Prediction performance for continuous (top row, CV R-squared) and binary (bottom row, CV AUC) neutralization outcomes for individual bnAbs (left-hand column) and the combination (right-hand column) 10–1074 + 10E8. For individual bnAbs, prediction performance is evaluated against the observed IC₅₀ or IC₈₀ values for the given bnAb; shapes denote the bnAb. For combination bnAbs, prediction performance is evaluated against both the observed IC₅₀ or IC₈₀ values based on the bnAb regimen (denoted by the prefix “observed”) and the calculated combination IC₅₀ or IC₈₀ values based on the observed bnAb-specific values using the additive or Bliss-Hill method; shapes denote the combination method (additive or Bliss-Hill) and color denotes the approach (CP or PC). Error bars reflect 95% confidence intervals.

Results for the remaining bnAb regimens from Table 1 followed similar patterns (S1 Appendix). We first compared the performance of the Bliss-Hill and additive models across all combination regimens with directly-observed combination neutralization. For IC₅₀, the mean difference in CV R-squared was −0.05 with interquartile range (IQR) (−0.08, 0.17) and the mean difference in CV AUC was −0.10 with IQR (−0.13, 0); for IC₈₀, the mean difference in CV R-squared was −0.02 with IQR (−0.22, 0.19) and the mean difference in CV AUC was −0.01 with IQR (−0.04, 0.04). We observed that prediction performance for binary neutralization outcomes (e.g., IC₈₀ < 1 μg/mL) was often better when using the CP approach than when using the PC approach [IQR of the difference in CV AUC = (−0.13, 0.05) when comparing PC to CP across all combination regimens]. The CP approach is not uniformly better than the PC approach for continuous outcomes, but is often at least as good as the PC approach [IQR of the difference in CV R-squared = (−0.46, −0.07) when comparing PC to CP across all combination regimens]. The results reflect what we saw in the simulations: when continuous-outcome prediction performance for the individual bnAbs is higher than a CV R-squared of 0.5 (e.g., for 10–1074 + 10E8 above, or 10–1074 + PG9, S14 Fig in S1 Appendix) then prediction performance is similar using either the CP or PC approach. For many combinations, however, the CP approach results in better prediction performance than the PC approach.

Discussion

When predicting virus neutralization susceptibility to a combination bnAb regimen, a key consideration is whether to directly predict the combination neutralization susceptibility (often combined using the individual-bnAb neutralization values due to a lack of data on the combination regimen), which we call the combine-then-predict (CP) approach, or to combine the predictions of individual-bnAb neutralization, which we call the predict-then-combine (PC) approach. In both simulated experiments and our data analysis, we found that prediction performance for binary neutralization outcomes (e.g., IC₈₀ < 1 μg/mL) was better using the CP approach compared to the PC approach. Results for continuous outcomes were mixed: in the simulations and the data analysis, we observed that in small samples, prediction performance for combination neutralization was similar between the two approaches. In larger samples (in the simulations), the PC approach seems to result in slightly better prediction performance, when individual-model prediction performance is high. This knowledge may be used when building prediction models for novel antibody combinations in the absence of in vitro neutralization data for the antibody combination; this, in turn, will aid in the evaluation and down-selection of these antibody combinations into prevention efficacy trials.

It is intuitive that increased individual-model prediction performance leads to increased PC performance, since as we noted above prediction errors may be compounded in this approach. The CP approach is similar to averaging IC₈₀ values over technical replicates, which generally improves the signal-to-noise ratio (SNR). This is similar to the phenomenon observed in Huang et al. [49], where the authors observed that averaging several immune response endpoints prior to ranking vaccine regimens resulted in better selection of vaccine regimens than ranking the immune response endpoints and then averaging the ranks. The method of Follmann et al. [50] has also been used to compare multiple immune response biomarkers [51], and considers the SNR for each immune response. Applying this method to the simulated examples yields that the SNR for the combination susceptibility is always higher than the SNR for the individual-bnAb susceptibilities, while the SNR for the combination IC₈₀ is only higher than the SNR for the individual-bnAb IC₈₀ values in the case where the individual-bnAb prediction performance is high; the PC approach performed better than the CP approach in this case. In cases with high SNR for the individual bnAb readouts, denoising through averaging (i.e., using the CP approach) may be less important. This suggests that the SNR can be used as a tool for evaluating whether the CP or PC approach may perform better in practice. However, in all cases, the CP approach led to narrower confidence intervals for prediction performance.

All comparisons between the combine-then-predict and predict-then-combine approaches appear to be more pronounced for the Bliss-Hill model than for the additive model. In particular, the prediction performance results differed between CP and PC more when using the Bliss-Hill model than when using the additive model. This may be due in part to the fact that the Bliss-Hill model uses both IC₅₀ and IC₈₀, so any prediction error in the individual-bnAb models is compounded; the additive model relies only on predictions of either IC₅₀ or IC₈₀. While these mathematical models have only been validated for combination regimens where the bnAbs target different Env epitopes, there may be utility in considering regimens that target overlapping epitopes [52].

The CP approach has been taken in prior work where the focus has been developing prediction models for combination neutralization [17, 18, 21]. The results presented here provide evidence that these prior findings are robust, and suggest that the PC approach would not have resulted in increased prediction performance. Thus, future work for predicting combination neutralization should focus on the CP approach, even if prediction performance for individual bnAbs is high.

This study has several limitations. The first is that we only used a single prediction method, the lasso, for our analysis of the CATNAP data. Several groups of authors have observed that either an ensemble of multiple prediction methods [18] or more flexible prediction methods [11, 13, 17] can improve prediction performance using data from CATNAP. While we could certainly observe increased prediction performance by using a more flexible prediction method, comparisons of the CP and PC approaches within a bnAb regimen are nonetheless valid since the predicted values from each approach are evaluated in the same manner regardless of the prediction model used to obtain these values. Additionally, as we saw in the simulations, a CV R-squared greater than approximately 0.5 for all bnAbs in a regimen is necessary for the PC approach to outperform the CP approach; in previous work using an ensemble prediction method, we observed very few CV R-squared values greater than 0.5 across a large number of individual bnAbs [18]. Second, all of the predictions in this work are based only on the HIV-1 Env sequence and do not account for the bnAb variable region amino acid sequence, which has been observed to increase prediction performance [17]. Third, in the simulations, the lasso model provides an unbiased estimator of the true data-generating model, so it is unlikely that prediction performance could be improved. A related limitation is that there is relatively little neutralization data in CATNAP on combination regimens. This limits our ability to either develop prediction models using the neutralization outcomes based on the combination regimen (here, we held these viruses out as an independent test set) or to obtain precise estimates of prediction performance. Fourth, all of the predictions in this work focuses primarily on parental form bnAbs. Common mutations such as methionine-to-leucine (L) and asparagine-to-serine (S) substitutions in the Fc region of bnAbs are expected to improve stability, half-life, and ease of manufacturing, but not neutralization. Therefore, we do not expect our results would change for such LS-formulated bnAbs and combinations. Finally, when direct neutralization data exist in CATNAP for a combination regimen, one could use a meta-learner or ensemble approach [53] to predict the combination neutralization, by predicting individual neutralization for each antibody and fitting a second-stage prediction model with the individual-antibody predictions as features. This approach, while not broadly applicable due to limited combination neutralization data in CATNAP, should be pursued in future research.

Conclusion

We investigated two approaches to predicting HIV-1 neutralization susceptibility to combination broadly neutralizing antibody (bnAb) regimens in cases where direct measurement of susceptibility is unavailable. We found that in most cases, combining neutralization values from the individual bnAbs and predicting this value resulted in better prediction performance than predicting the individual bnAb values and combining the predicted results.

Supporting information

S1 Appendix. Supplementary materials.

Links to the GitHub repository with code for replicating our results, and additional results from the other bnAb combinations in CATNAP (Table 1) that were not presented in the main manuscript.

https://doi.org/10.1371/journal.pone.0310042.s001

References

1. 1. Mahomed S, Garrett N, Baxter C, Abdool Karim Q, Abdool Karim SS. Clinical trials of broadly neutralizing monoclonal antibodies for human immunodeficiency virus prevention: a review. The Journal of Infectious Diseases. 2021;223(3):370–380. pmid:32604408

* View Article

* PubMed/NCBI

* Google Scholar

2. 2. Julg B, Barouch D. Broadly neutralizing antibodies for HIV-1 prevention and therapy. In: Seminars in Immunology. vol. 51; 2021. p. 101475.

3. 3. Karuna ST, Corey L. Broadly neutralizing antibodies for HIV prevention. Annual Review of Medicine. 2020;71:329–346. pmid:31986089

* View Article

* PubMed/NCBI

* Google Scholar

4. 4. Walsh SR, Seaman MS. Broadly neutralizing antibodies for HIV-1 prevention. Frontiers in Immunology. 2021;12:712122. pmid:34354713

* View Article

* PubMed/NCBI

* Google Scholar

5. 5. Corey L, Gilbert PB, Juraska M, Montefiori DC, Morris L, Karuna ST, et al. Two randomized trials of neutralizing antibodies to prevent HIV-1 acquisition. New England Journal of Medicine. 2021;384(11):1003–1014. pmid:33730454

* View Article

* PubMed/NCBI

* Google Scholar

6. 6. Wagh K, Bhattacharya T, Williamson C, Robles A, Bayne M, et al. Optimal combinations of broadly neutralizing antibodies for prevention and treatment of HIV-1 clade C infection. PLoS Pathogens. 2016;12(3):e1005520. pmid:27028935

* View Article

* PubMed/NCBI

* Google Scholar

7. 7. Doria-Rose NA, Louder MK, Yang Z, O’Dell S, Nason M, Schmidt SD, et al. HIV-1 neutralization coverage is improved by combining monoclonal antibodies that target independent epitopes. Journal of Virology. 2012;86(6):3393–3397. pmid:22258252

* View Article

* PubMed/NCBI

* Google Scholar

8. 8. Diskin R, Klein F, Horwitz JA, Halper-Stromberg A, Sather DN, Marcovecchio PM, et al. Restricting HIV-1 pathways for escape using rationally designed anti-HIV-1 antibodies. Journal of Experimental Medicine. 2013;210(6):1235–1249. pmid:23712429

* View Article

* PubMed/NCBI

* Google Scholar

9. 9. Hepler NL, Scheffler K, Weaver S, Murrell B, Richman DD, Burton DR, et al. IDEPI: rapid prediction of HIV-1 antibody epitopes and other phenotypic features from sequence data using a flexible machine learning platform. PLoS Computational Biology. 2014;10(9):e1003842. pmid:25254639

* View Article

* PubMed/NCBI

* Google Scholar

10. 10. Buiu C, Putz MV, Avram S. Learning the relationship between the primary structure of HIV envelope glycoproteins and neutralization activity of particular antibodies by using artificial neural networks. International Journal of Molecular Sciences. 2016;17(10):1710. pmid:27727189

* View Article

* PubMed/NCBI

* Google Scholar

11. 11. Hake A, Pfeifer N. Prediction of HIV-1 sensitivity to broadly neutralizing antibodies shows a trend towards resistance over time. PLoS Computational Biology. 2017;13(10):e1005789. pmid:29065122

* View Article

* PubMed/NCBI

* Google Scholar

12. 12. Bricault CA, Yusim K, Seaman MS, Yoon H, Theiler J, Giorgi EE, et al. HIV-1 neutralizing antibody signatures and application to epitope-targeted vaccine design. Cell Host & Microbe. 2019;25(1):59–72.

* View Article

* Google Scholar

13. 13. Rawi R, Mall R, Shen CH, Farney SK, Shiakolas A, Zhou J, et al. Accurate prediction for antibody resistance of clinical HIV-1 isolates. Scientific Reports. 2019;9(1):14696. pmid:31604961

* View Article

* PubMed/NCBI

* Google Scholar

14. 14. Conti S, Karplus M. Estimation of the breadth of CD4bs targeting HIV antibodies by molecular modeling and machine learning. PLoS Computational Biology. 2019;15(4):e1006954. pmid:30970017

* View Article

* PubMed/NCBI

* Google Scholar

15. 15. Yu WH, Su D, Torabi J, Fennessey CM, Shiakolas A, Lynch R, et al. Predicting the broadly neutralizing antibody susceptibility of the HIV reservoir. JCI Insight. 2019;4(17). pmid:31484826

* View Article

* PubMed/NCBI

* Google Scholar

16. 16. Magaret C, Benkeser D, Williamson B, Borate B, Carpp L, et al. Prediction of VRC01 neutralization sensitivity by HIV-1 gp160 sequence features. PLoS Computational Biology. 2019;15(4):e1006952. pmid:30933973

* View Article

* PubMed/NCBI

* Google Scholar

17. 17. Dănăilă VR, Buiu C. Prediction of HIV sensitivity to monoclonal antibodies using aminoacid sequences and deep learning. Bioinformatics. 2022;38(18):4278–4285. pmid:35876860

* View Article

* PubMed/NCBI

* Google Scholar

18. 18. Williamson BD, Magaret CA, Gilbert PB, Nizam S, Simmons C, Benkeser D. Super LeArner Prediction of NAb Panels (SLAPNAP): a containerized tool for predicting combination monoclonal broadly neutralizing antibody sensitivity. Bioinformatics. 2021;37(22):4187–4192. pmid:34021743

* View Article

* PubMed/NCBI

* Google Scholar

19. 19. Yoon H, Macke J, West AJ, Foley B, Bjorkman P, et al. CATNAP: a tool to compile, analyze, and tally neutralizing antibody panels. Nucleic Acids Research. 2015;43(W1):W213–219. pmid:26044712

* View Article

* PubMed/NCBI

* Google Scholar

20. 20. Wagh K, Seaman M, Zingg M, Fitzsimons T, Barouch D, et al. Potential of conventional & bispecific broadly neutralizing antibodies for prevention of HIV-1 subtype A, C & D infections. PLoS Pathogens. 2018;14(3):e1006860. pmid:29505593

* View Article

* PubMed/NCBI

* Google Scholar

21. 21. Williamson BD, Magaret CA, Karuna S, Carpp LN, Gelderblom HC, Huang Y, et al. Application of the SLAPNAP statistical learning tool to broadly neutralizing antibody HIV prevention research. iScience. 2023;26(9). pmid:37654470

* View Article

* PubMed/NCBI

* Google Scholar

22. 22. Gilbert PB, Huang Y, deCamp AC, Karuna S, Zhang Y, Magaret CA, et al. Neutralization titer biomarker for antibody-mediated prevention of HIV-1 acquisition. Nature Medicine. 2022;28(9):1924–1932. pmid:35995954

* View Article

* PubMed/NCBI

* Google Scholar

23. 23. Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 1996; p. 267–288.

* View Article

* Google Scholar

24. 24. Breiman L. Random forests. Machine Learning. 2001;45(1):5–32.

* View Article

* Google Scholar

25. 25. Platt J. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers. 1999;10(3):61–74.

* View Article

* Google Scholar

26. 26. Barron A. Statistical properties of artificial neural networks. In: Proceedings of the 28th IEEE Conference on Decision and Control. IEEE; 1989. p. 280–285.

27. 27. Breiman L. Statistical modeling: The two cultures. Statistical Science. 2001;16(3):199–231.

* View Article

* Google Scholar

28. 28. van der Laan M, Polley E, Hubbard A. Super Learner. Statistical Applications in Genetics and Molecular Biology. 2007;6(1):Online Article 25. pmid:17910531

* View Article

* PubMed/NCBI

* Google Scholar

29. 29. Brent RP. An algorithm with guaranteed convergence for finding a zero of a function. The Computer Journal. 1971;14(4):422–425.

* View Article

* Google Scholar

30. 30. Williamson B, Gilbert P, Simon N, Carone M. A general framework for inference on algorithm-agnostic variable importance. Journal of the American Statistical Association (Theory & Methods). 2021.

* View Article

* Google Scholar

31. 31. LeDell E, Petersen M, van der Laan M. Computationally efficient confidence intervals for cross-validated area under the ROC curve estimates. Electronic Journal of Statistics. 2015;. pmid:26279737

* View Article

* PubMed/NCBI

* Google Scholar

32. 32. Merkel D. Docker: lightweight linux containers for consistent development and deployment. Linux Journal. 2014;2014(239):2.

* View Article

* Google Scholar

33. 33. R Core Team. R: A Language and Environment for Statistical Computing; 2024. Available from: https://www.R-project.org/.

34. 34. Müller K. here: A Simpler Way to Find Your Files; 2020. Available from: https://CRAN.R-project.org/package=here.

35. 35. Wickham H, Hester J, Bryan J. readr: Read Rectangular Text Data; 2024. Available from: https://CRAN.R-project.org/package=readr.

36. 36. Firke S. janitor: Simple Tools for Examining and Cleaning Dirty Data; 2023. Available from: https://CRAN.R-project.org/package=janitor.

37. 37. Friedman J, Hastie T, Tibshirani R. Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of Statistical Software. 2010;33(1):1–22. pmid:20808728

* View Article

* PubMed/NCBI

* Google Scholar

38. 38. Tay JK, Narasimhan B, Hastie T. Elastic Net Regularization Paths for All Generalized Linear Models. Journal of Statistical Software. 2023;106(1):1–31. pmid:37138589

* View Article

* PubMed/NCBI

* Google Scholar

39. 39. Barrett T, Dowle M, Srinivasan A, Gorecki J, Chirico M, Hocking T. data.table: Extension of ‘data.frame’; 2024. Available from: https://CRAN.R-project.org/package=data.table.

40. 40. Wickham H, François R, Henry L, Müller K, Vaughan D. dplyr: A Grammar of Data Manipulation; 2023. Available from: https://CRAN.R-project.org/package=dplyr.

41. 41. Wickham H, Vaughan D, Girlich M. tidyr: Tidy Messy Data; 2024. Available from: https://CRAN.R-project.org/package=tidyr.

42. 42. Microsoft, Weston S. foreach: Provides Foreach Looping Construct; 2022. Available from: https://CRAN.R-project.org/package=foreach.

43. 43. Corporation M, Weston S. doParallel: Foreach Parallel Adaptor for the ‘parallel’ Package; 2022. Available from: https://CRAN.R-project.org/package=doParallel.

44. 44. LeDell E, Petersen M, van der Laan M. cvAUC: Cross-Validated Area Under the ROC Curve Confidence Intervals; 2022. Available from: https://CRAN.R-project.org/package=cvAUC.

45. 45. Williamson BD. vimp: Perform Inference on Algorithm-Agnostic Variable Importance; 2023. Available from: https://CRAN.R-project.org/package=vimp.

46. 46. Wickham H. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York; 2016. Available from: https://ggplot2.tidyverse.org.

47. 47. Wilke CO. cowplot: Streamlined Plot Theme and Plot Annotations for ‘ggplot2’; 2024. Available from: https://CRAN.R-project.org/package=cowplot.

48. 48. Auguie B. gridExtra: Miscellaneous Functions for “Grid” Graphics; 2017. Available from: https://CRAN.R-project.org/package=gridExtra.

49. 49. Huang Y, Gilbert PB, Fu R, Janes H. Statistical methods for down-selection of treatment regimens based on multiple endpoints, with application to HIV vaccine trials. Biostatistics. 2017;18(2):230–243. pmid:27649715

* View Article

* PubMed/NCBI

* Google Scholar

50. 50. Follmann D. Reliably picking the best endpoint. Statistics in Medicine. 2018;37(29):4374–4385. pmid:30091264

* View Article

* PubMed/NCBI

* Google Scholar

51. 51. Benkeser D, Montefiori DC, McDermott AB, Fong Y, Janes HE, Deng W, et al. Comparing antibody assays as correlates of protection against COVID-19 in the COVE mRNA-1273 vaccine efficacy trial. Science Translational Medicine. 2023;15(692):eade9078. pmid:37075127

* View Article

* PubMed/NCBI

* Google Scholar

52. 52. Sajadi MM, Dashti A, Tehrani ZR, Tolbert WD, Seaman MS, Ouyang X, et al. Identification of near-pan-neutralizing antibodies against HIV-1 by deconvolution of plasma humoral responses. Cell. 2018;173(7):1783–1795. pmid:29731169

* View Article

* PubMed/NCBI

* Google Scholar

53. 53. Wolpert D. Stacked generalization. Neural Networks. 1992;5(2):241–259.

* View Article

* Google Scholar

Citation: Williamson BD, Wu L, Huang Y, Hudson A, Gilbert PB (2024) Predicting neutralization susceptibility to combination HIV-1 monoclonal broadly neutralizing antibody regimens. PLoS ONE 19(9): e0310042. https://doi.org/10.1371/journal.pone.0310042

About the Authors:

Brian D. Williamson

Roles: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Supervision, Visualization, Writing – original draft, Writing – review & editing

E-mail: [email protected]

Affiliations: Biostatistics Division, Kaiser Permanente Washington Health Research Institute, Seattle, WA, United States of Amerrica, Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Center, Seattle, WA, United States of Amerrica, Department of Biostatistics, University of Washington, Seattle, WA, United States of Amerrica

ORICD: https://orcid.org/0000-0002-7024-548X

Liana Wu

Roles: Formal analysis, Writing – review & editing

Affiliation: Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Center, Seattle, WA, United States of Amerrica

Yunda Huang

Roles: Conceptualization, Writing – review & editing

Affiliations: Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Center, Seattle, WA, United States of Amerrica, Public Health Sciences Division, Fred Hutchinson Cancer Center, Seattle, WA, United States of Amerrica, Department of Global Health, University of Washington, Seattle, WA, United States of Amerrica

Aaron Hudson

Roles: Conceptualization, Writing – review & editing

Affiliations: Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Center, Seattle, WA, United States of Amerrica, Department of Biostatistics, University of Washington, Seattle, WA, United States of Amerrica, Public Health Sciences Division, Fred Hutchinson Cancer Center, Seattle, WA, United States of Amerrica

Peter B. Gilbert

Roles: Conceptualization, Funding acquisition, Writing – review & editing

Affiliations: Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Center, Seattle, WA, United States of Amerrica, Department of Biostatistics, University of Washington, Seattle, WA, United States of Amerrica, Public Health Sciences Division, Fred Hutchinson Cancer Center, Seattle, WA, United States of Amerrica

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

References

1. Mahomed S, Garrett N, Baxter C, Abdool Karim Q, Abdool Karim SS. Clinical trials of broadly neutralizing monoclonal antibodies for human immunodeficiency virus prevention: a review. The Journal of Infectious Diseases. 2021;223(3):370–380. pmid:32604408

2. Julg B, Barouch D. Broadly neutralizing antibodies for HIV-1 prevention and therapy. In: Seminars in Immunology. vol. 51; 2021. p. 101475.

3. Karuna ST, Corey L. Broadly neutralizing antibodies for HIV prevention. Annual Review of Medicine. 2020;71:329–346. pmid:31986089

4. Walsh SR, Seaman MS. Broadly neutralizing antibodies for HIV-1 prevention. Frontiers in Immunology. 2021;12:712122. pmid:34354713

5. Corey L, Gilbert PB, Juraska M, Montefiori DC, Morris L, Karuna ST, et al. Two randomized trials of neutralizing antibodies to prevent HIV-1 acquisition. New England Journal of Medicine. 2021;384(11):1003–1014. pmid:33730454

6. Wagh K, Bhattacharya T, Williamson C, Robles A, Bayne M, et al. Optimal combinations of broadly neutralizing antibodies for prevention and treatment of HIV-1 clade C infection. PLoS Pathogens. 2016;12(3):e1005520. pmid:27028935

7. Doria-Rose NA, Louder MK, Yang Z, O’Dell S, Nason M, Schmidt SD, et al. HIV-1 neutralization coverage is improved by combining monoclonal antibodies that target independent epitopes. Journal of Virology. 2012;86(6):3393–3397. pmid:22258252

8. Diskin R, Klein F, Horwitz JA, Halper-Stromberg A, Sather DN, Marcovecchio PM, et al. Restricting HIV-1 pathways for escape using rationally designed anti-HIV-1 antibodies. Journal of Experimental Medicine. 2013;210(6):1235–1249. pmid:23712429

9. Hepler NL, Scheffler K, Weaver S, Murrell B, Richman DD, Burton DR, et al. IDEPI: rapid prediction of HIV-1 antibody epitopes and other phenotypic features from sequence data using a flexible machine learning platform. PLoS Computational Biology. 2014;10(9):e1003842. pmid:25254639

10. Buiu C, Putz MV, Avram S. Learning the relationship between the primary structure of HIV envelope glycoproteins and neutralization activity of particular antibodies by using artificial neural networks. International Journal of Molecular Sciences. 2016;17(10):1710. pmid:27727189

11. Hake A, Pfeifer N. Prediction of HIV-1 sensitivity to broadly neutralizing antibodies shows a trend towards resistance over time. PLoS Computational Biology. 2017;13(10):e1005789. pmid:29065122

12. Bricault CA, Yusim K, Seaman MS, Yoon H, Theiler J, Giorgi EE, et al. HIV-1 neutralizing antibody signatures and application to epitope-targeted vaccine design. Cell Host & Microbe. 2019;25(1):59–72.

13. Rawi R, Mall R, Shen CH, Farney SK, Shiakolas A, Zhou J, et al. Accurate prediction for antibody resistance of clinical HIV-1 isolates. Scientific Reports. 2019;9(1):14696. pmid:31604961

14. Conti S, Karplus M. Estimation of the breadth of CD4bs targeting HIV antibodies by molecular modeling and machine learning. PLoS Computational Biology. 2019;15(4):e1006954. pmid:30970017

15. Yu WH, Su D, Torabi J, Fennessey CM, Shiakolas A, Lynch R, et al. Predicting the broadly neutralizing antibody susceptibility of the HIV reservoir. JCI Insight. 2019;4(17). pmid:31484826

16. Magaret C, Benkeser D, Williamson B, Borate B, Carpp L, et al. Prediction of VRC01 neutralization sensitivity by HIV-1 gp160 sequence features. PLoS Computational Biology. 2019;15(4):e1006952. pmid:30933973

17. Dănăilă VR, Buiu C. Prediction of HIV sensitivity to monoclonal antibodies using aminoacid sequences and deep learning. Bioinformatics. 2022;38(18):4278–4285. pmid:35876860

18. Williamson BD, Magaret CA, Gilbert PB, Nizam S, Simmons C, Benkeser D. Super LeArner Prediction of NAb Panels (SLAPNAP): a containerized tool for predicting combination monoclonal broadly neutralizing antibody sensitivity. Bioinformatics. 2021;37(22):4187–4192. pmid:34021743

19. Yoon H, Macke J, West AJ, Foley B, Bjorkman P, et al. CATNAP: a tool to compile, analyze, and tally neutralizing antibody panels. Nucleic Acids Research. 2015;43(W1):W213–219. pmid:26044712

20. Wagh K, Seaman M, Zingg M, Fitzsimons T, Barouch D, et al. Potential of conventional & bispecific broadly neutralizing antibodies for prevention of HIV-1 subtype A, C & D infections. PLoS Pathogens. 2018;14(3):e1006860. pmid:29505593

21. Williamson BD, Magaret CA, Karuna S, Carpp LN, Gelderblom HC, Huang Y, et al. Application of the SLAPNAP statistical learning tool to broadly neutralizing antibody HIV prevention research. iScience. 2023;26(9). pmid:37654470

22. Gilbert PB, Huang Y, deCamp AC, Karuna S, Zhang Y, Magaret CA, et al. Neutralization titer biomarker for antibody-mediated prevention of HIV-1 acquisition. Nature Medicine. 2022;28(9):1924–1932. pmid:35995954

23. Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 1996; p. 267–288.

24. Breiman L. Random forests. Machine Learning. 2001;45(1):5–32.

25. Platt J. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers. 1999;10(3):61–74.

26. Barron A. Statistical properties of artificial neural networks. In: Proceedings of the 28th IEEE Conference on Decision and Control. IEEE; 1989. p. 280–285.

27. Breiman L. Statistical modeling: The two cultures. Statistical Science. 2001;16(3):199–231.

28. van der Laan M, Polley E, Hubbard A. Super Learner. Statistical Applications in Genetics and Molecular Biology. 2007;6(1):Online Article 25. pmid:17910531

29. Brent RP. An algorithm with guaranteed convergence for finding a zero of a function. The Computer Journal. 1971;14(4):422–425.

30. Williamson B, Gilbert P, Simon N, Carone M. A general framework for inference on algorithm-agnostic variable importance. Journal of the American Statistical Association (Theory & Methods). 2021.

31. LeDell E, Petersen M, van der Laan M. Computationally efficient confidence intervals for cross-validated area under the ROC curve estimates. Electronic Journal of Statistics. 2015;. pmid:26279737

32. Merkel D. Docker: lightweight linux containers for consistent development and deployment. Linux Journal. 2014;2014(239):2.

33. R Core Team. R: A Language and Environment for Statistical Computing; 2024. Available from: https://www.R-project.org/.

34. Müller K. here: A Simpler Way to Find Your Files; 2020. Available from: https://CRAN.R-project.org/package=here.

35. Wickham H, Hester J, Bryan J. readr: Read Rectangular Text Data; 2024. Available from: https://CRAN.R-project.org/package=readr.

36. Firke S. janitor: Simple Tools for Examining and Cleaning Dirty Data; 2023. Available from: https://CRAN.R-project.org/package=janitor.

37. Friedman J, Hastie T, Tibshirani R. Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of Statistical Software. 2010;33(1):1–22. pmid:20808728

38. Tay JK, Narasimhan B, Hastie T. Elastic Net Regularization Paths for All Generalized Linear Models. Journal of Statistical Software. 2023;106(1):1–31. pmid:37138589

39. Barrett T, Dowle M, Srinivasan A, Gorecki J, Chirico M, Hocking T. data.table: Extension of ‘data.frame’; 2024. Available from: https://CRAN.R-project.org/package=data.table.

40. Wickham H, François R, Henry L, Müller K, Vaughan D. dplyr: A Grammar of Data Manipulation; 2023. Available from: https://CRAN.R-project.org/package=dplyr.

41. Wickham H, Vaughan D, Girlich M. tidyr: Tidy Messy Data; 2024. Available from: https://CRAN.R-project.org/package=tidyr.

42. Microsoft, Weston S. foreach: Provides Foreach Looping Construct; 2022. Available from: https://CRAN.R-project.org/package=foreach.

43. Corporation M, Weston S. doParallel: Foreach Parallel Adaptor for the ‘parallel’ Package; 2022. Available from: https://CRAN.R-project.org/package=doParallel.

44. LeDell E, Petersen M, van der Laan M. cvAUC: Cross-Validated Area Under the ROC Curve Confidence Intervals; 2022. Available from: https://CRAN.R-project.org/package=cvAUC.

45. Williamson BD. vimp: Perform Inference on Algorithm-Agnostic Variable Importance; 2023. Available from: https://CRAN.R-project.org/package=vimp.

46. Wickham H. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York; 2016. Available from: https://ggplot2.tidyverse.org.

47. Wilke CO. cowplot: Streamlined Plot Theme and Plot Annotations for ‘ggplot2’; 2024. Available from: https://CRAN.R-project.org/package=cowplot.

48. Auguie B. gridExtra: Miscellaneous Functions for “Grid” Graphics; 2017. Available from: https://CRAN.R-project.org/package=gridExtra.

49. Huang Y, Gilbert PB, Fu R, Janes H. Statistical methods for down-selection of treatment regimens based on multiple endpoints, with application to HIV vaccine trials. Biostatistics. 2017;18(2):230–243. pmid:27649715

50. Follmann D. Reliably picking the best endpoint. Statistics in Medicine. 2018;37(29):4374–4385. pmid:30091264

51. Benkeser D, Montefiori DC, McDermott AB, Fong Y, Janes HE, Deng W, et al. Comparing antibody assays as correlates of protection against COVID-19 in the COVE mRNA-1273 vaccine efficacy trial. Science Translational Medicine. 2023;15(692):eade9078. pmid:37075127

52. Sajadi MM, Dashti A, Tehrani ZR, Tolbert WD, Seaman MS, Ouyang X, et al. Identification of near-pan-neutralizing antibodies against HIV-1 by deconvolution of plasma humoral responses. Cell. 2018;173(7):1783–1795. pmid:29731169

53. Wolpert D. Stacked generalization. Neural Networks. 1992;5(2):241–259.