In Bayesian inference, optimization is mostly used as a tool for point estimation or variational inference. In this talk, I will first introduce a new computational method named "Transport Monte Carlo" that targets the exact posterior distribution. This method uses fast optimization to learn a non-parametric mixture of one-to-one transformations, between the sample from the posterior and the one from an iid uniform distribution. I will demonstrate much-improved performances in often-encountered sampling problems, such as high-dimensional regression and combinatorial estimation in graph modeling. In the second part of the talk, I will introduce a new class of prior/likelihood that lacks closed-form but are defined through optimization. This expands Bayesian inference for uncertainty quantification in a much broader class of models such as linear trend filtering of multivariate time series.