Many "advanced" player quality metrics for NBA players such as Value Over Replacement Player (VORP), Player Efficiency Rating (PER), and Win Shares (WS) have become popular in recent years. However, due to their complex formulations, little is known about how common performance measures influence these metrics. Using box-score statistics from the previous 15 seasons, we predict VORP, PER, and WS for each player using a variety of supervised learning approaches and evaluate those models on a test set of NBA players from the 2016-2017 season. Notably, we find that random forests can produce highly accurate predictions using only a small subset of intuitive covariates such as shooting percentage and minutes played. Making use of recent theoretical results for these black-box models, we demonstrate formally that conditional on those simple measures, including additional covariate information does not produce models with significantly more accurate predictions. Finally, we use this information to suggest an intuitive quality metric which we show performs well with respect to recently proposed measures for assessing sports metrics.