In December 2019, an unprecedented outbreak of the COVID-19 pandemic emerged. Profoundly affected by this pandemic, the scientific community disseminates research findings at a fast rate. As of April 10, 2021, there are more than 120,000 articles on COVID-19. Among these manuscripts, the preprints publicly accessible without peer review account for more than 20%. Despite the heterogeneous quality, the preprints benefit the authors with timely sharing of representative findings. To tackle the challenges of including preprints in a meta-analysis, we develop a rigorous evidence synthesis method that accounts for the heterogeneity of the preprints and corrects the potential publication bias. The proposed method has been validated with two case studies on two COVID-19 treatments. The relative risk of hydroxychloroquine effectiveness is 1.05 [95% CI: 0.71, 1.56], indicating that the hydroxychloroquine is not likely to benefit the reduction of mortality rate compared with the placebo. The relative risk of corticosteroids effectiveness is 0.90 [95% CI: 0.59, 1.36], which shows that the corticosteroids are likely to be beneficial in reducing the mortality rate of the COVID-19 patients.