Human exposure to complex chemical mixtures is ubiquitous, and its relationship to health outcomes is of particular interest in environmental epidemiology. We focus on an application in assessing the associations between prenatal phthalates exposures and childhood obesity in a prospective cohort study. Part of our interest is in modeling how these associations change as children age. Because of moderate to strong correlations between phthalates metabolites, it is desirable to summarize exposure patterns in lower dimensions using tools such as factor analysis. We propose a Bayesian varying-coefficient factor regression to jointly model the latent structures in the mixtures and their associations to (one or more) measures of fat mass over time. The model adopts a hierarchical shrinkage prior on the regresssion coefficients. It also accounts for pair-wise interactions between phthalate metabolites, interactions between phthalates and demographics covariates, and potential unobserved confounders. We compare our estimates with results from past studies that employed two-staged or mixed-effects modeling approaches.