In many regression settings the unknown coefficients may have some known structure, e.g.\ they may be ordered in space or correspond to a vectorized matrix or tensor. However, many commonly used priors and corresponding penalties used for regression analysis do not encourage structured estimates of the coefficients. In this paper we develop structured shrinkage priors that generalize multivariate normal, Laplace, exponential power and normal-gamma priors. These priors allow the regression coefficients to be correlated a priori without sacrificing sparsity and shrinkage. The primary challenges in working with these structured shrinkage priors are computational, as the corresponding penalties are intractable p-dimensional integrals and the full conditional distributions that are needed to approximate the posterior mode or simulate from the posterior distribution may be non-standard. We overcome these issues using a flexible elliptical slice sampling procedure, and demonstrate that these priors can be used to introduce structure while preserving sparsity of the corresponding penalized estimate given by the posterior mode.