Secure and scalable deduplication of horizontally partitioned health data for privacy-preserving distributed statistical computation

Yigzaw, KY., Michalas, A. and Bellika, JG. (2017) Secure and scalable deduplication of horizontally partitioned health data for privacy-preserving distributed statistical computation. BMC Medical Informatics and Decision Making. ISSN 1472-6947

2016-BMC - Secure_Scalable_Deduplication.pdf - Published Version

Download (1MB) | Preview
Official URL:


Background Techniques have been developed to compute statistics on distributed datasets without revealing private information except the statistical results. However, duplicate records in a distributed dataset may lead to incorrect statistical results. Therefore, to increase the accuracy of the statistical analysis of a distributed dataset, secure deduplication is an important preprocessing step. Methods We designed a secure protocol for the deduplication of horizontally partitioned datasets with deterministic record linkage algorithms. We provided a formal security analysis of the protocol in the presence of semi-honest adversaries. The protocol was implemented and deployed across three microbiology laboratories located in Norway, and we ran experiments on the datasets in which the number of records for each laboratory varied. Experiments were also performed on simulated microbiology datasets and data custodians connected through a local area network. Results The security analysis demonstrated that the protocol protects the privacy of individuals and data custodians under a semi-honest adversarial model. More precisely, the protocol remains secure with the collusion of up to N − 2 corrupt data custodians. The total runtime for the protocol scales linearly with the addition of data custodians and records. One million simulated records distributed across 20 data custodians were deduplicated within 45 s. The experimental results showed that the protocol is more efficient and scalable than previous protocols for the same problem. Conclusions The proposed deduplication protocol is efficient and scalable for practical uses while protecting the privacy of patients and data custodians.

Item Type: Article
Uncontrolled Keywords: Bloom Filter;Data Reuse;Deduplication;Distributed Statistical Computation;Data Linkage;Duplicate Record;Electronic Health Record;Privacy;Record Linkage;Set Intersection;
Subjects: University of Westminster > Science and Technology
SWORD Depositor:
Depositing User:
Date Deposited: 05 Jan 2017 12:08
Last Modified: 05 Jan 2017 12:08

Actions (login required)

Edit Item (Repository staff only) Edit Item (Repository staff only)