Distributed data are becoming increasingly available. Recently, various methods have been developed be effectively combine data stored at distributed centers. A focus has been on the estimation problems. While for the hypothesis testing and confidence set estimations, normal approximations with some variance estimations are considered as the natural approach. In this study, we investigate the impact on statistical inferences from the number the distributed centers. We find that even when the number of centers is moderate compared with the sample size at each center, normal approximation may perform poorly. Through our theoretical analysis, we quantity the approximation errors due to increasing number of data centers. A significant finding is that such an impact may not be negligible when the number of centers is not small. To rectify the problem, we propose a re-sampling method that is shown to be consistent under general settings. We confirm our analysis with numerical examples.