Bayesian latent variable models are commonly used to model network data, and the choice of prior on the latent variables is important to the estimation of the covariates. An inverse-gamma or inverse-Wishart prior is commonly used on the variance parameters of the latent variables due to ease of computation. However, there are two situations in which this approach can lead to biased estimates for the covariate effects and lower than nominal coverage of confidence intervals. The first is when the variation in a binary response variable is adequately explained by the covariates and latent variables are still estimated. The second is when there is correlation between unobserved and observed covariates. In this setting, we often hope that as much of the variation as possible would be attributed to the covariates and the latent effects will account for excess structure. However, when using standard priors this is not the case. To remedy these situations, we propose using a half-Cauchy prior on the variance parameters and restricting the latent variables to be orthogonal to the space of the covariates. We illustrate the key advantages of our proposed approach using simulation.