Diffusion tensor imaging (DTI) is a prevalent neuroimaging tool, and diffusion directions are one of the most important information derived from DTI as they depict the anatomical structures of fiber tracts. This work studies the possible association between diffusion directions and Alzheimer's disease progression via an image-on-scalar type regression model. The spatial dependence further motivates us to assume the residuals and the regression effects of age, gender, and disease status are correlated when they are both spatially close and on the same fiber tract. To characterize the diffusion directions, we propose a von-Mises Fisher distribution-based error model with mean direction, dependent on the subjects' covariates through a link function. The link function allows us to model the regression coefficients in the Euclidean space and eases our prior specification. This allows us to characterize their dependence along the fiber tract. The model's key statistical properties and a comprehensive toolbox for Bayesian inference of the directional data are provided. The numerical studies based on synthetic data and ADNI data demonstrate that our model performs overwhelmingly better.