It is well-known that a trial could end up with different conclusions due to difference in the selected clinical centers. However, the impact of center effects on trial endpoints is not studied adequately. We quantify the impact of center effects on Objective Response Rate (ORR) in early oncology single-arm trials, which are conducted at multiple small and heterogeneous centers. Based on the variance formula for ORR we derived after adjusting the center effects, we provide guidance on minimizing the center effects during the trial design stage, by considering how many centers to be selected, how to distribute patients among centers and how to set a center enrollment cap for a trial. The conclusion can be applied directly to the clinical trials with binary endpoints other than ORR and also shed light on clinical trials with different endpoints.