In this work we study optimal estimation of conditional average treatment effects, which are essential for understanding how treatment effects vary across subject characteristics and for tailoring treatment regimes. Under unconfoundedness, the conditional effect can be represented as a difference of regression functions under different treatment levels. Despite numerous estimators being proposed for this quantity in recent years, to the best of our knowledge no one has derived a minimax lower bound. This is a critical element in this problem since it gives a benchmark for the best possible performance across all estimators, and indicates the fundamental inferential limits of effect estimation in a flexible nonparametric model. We give the first such bound in this setting, and construct new estimators that attain the bound under various conditions, yielding provable rate optimality. We also illustrate the methods in simulations and an illustrative data analysis.