Using Record Linkage to Create Big Data? How Good Is It?

• Activity Number: 339
• Type: Contributed
• Date/Time: Tuesday, August 5, 2014 : 10:30 AM to 12:20 PM
• Sponsor: Government Statistics Section
• Author(s): K. Bradley Paxton*+
• Companies: ADI
• Keywords: Record Linkage ; Data Quality ; Census ; Testing

Abstract

Record Linkage is being increasingly used to create new "big data" collections or improve already existing data. If, for example, a Census data set is correctly linked to a Tax data set, a new or improved data set may result; if, however, the linkage is done incorrectly, then the data set may be made worse. It is important to know how good your record linkage system is performing to optimize it and get the best results for your investment. Record linkage systems, however, are difficult to test.

This paper presents two proven methods for testing record linkage systems: a synthetic data GAMUT (Great Automated Model Universe for Test) for use in the development phase and Production Data Quality (PDQ) in the production phase. These methods were used successfully in the 2010 Census for evaluating forms processing but they are extensible to record linkage and also apply to any IT classification system.

Both methods are cost-effective because they use automation to replace a lot of human effort. Together, the two methods provide "cradle-to-grave" testing and improvement capability. Future work includes trading-off precision and recall to achieve maximum record linkage system value.