Genome-wide association studies (GWAS) have discovered many variants that are associated with multiple, sometimes seemingly unrelated, traits. There are many underlying mechanisms that can create cross-phenotype associations including causal effects between observed traits, linkage disequilibrium, and unobserved common causes mediating variant effects. These mechanisms can only be distinguished by combining information across many variants simultaneously. I will present an empirical Bayes matrix factorization method which can infer unobserved common causes of large sets of traits using GWAS summary statistics. Our proposed method, genetic factor analysis (GFA), is able to account for overlapping samples between GWAS, which manifests as row correlation in the matrix of GWAS effect estimates. We demonstrate in simulations that this correction is critical to correctly recovering the underlying mediator network. We also demonstrate that GFA can recover arbitrary networks more accurately than alternative methods, even when there is no sample overlap between GWAS. We demonstrate an application of GFA using lipid phenotypes.