How the genome is organized in 3D is a subject of intense interest. The genome-wide assay of choice is Hi-C, which - for technical reasons - is usually performed in cell lines. As such we have little understanding of how 3D genome structure varies between individuals and how genetic variation shapes this variation. We have studied this question using dilution Hi-C in lymphoblastoid cell lines. We show extensive sample-to-sample variation in Hi-C data as well as the presence of substantial unwanted variation. We develop a method to remove this variation, while retaining signal, and use the corrected data to study the impact of genetic variation in 3D structure.