Despite recent growth in public football analytics, there is limited research on how effectively certain metrics can isolate player talent. For example, quarterback performance is generally analyzed by using box score summaries such as completion percentage and quarterback rating in combination with newer tools like win probability and expected points. However, it is still mostly unknown if and how these complex modern metrics offer an advantage. Using the meta-metric framework of Franks et al. (2017), we aim to quantify properties of quarterback metrics to better understand how performance measures vary over time, between players, and within players. Results using game and season level data from 2007 to 2018 suggest that expected-point based summaries tend to feature preferred statistical properties.