TITLE:
Fast Semantic Duplicate Detection Techniques in Databases
AUTHORS:
Ibrahim Moukouop Nguena, Amolo-Makama Ophélie Carmen Richeline
KEYWORDS:
Semantic Duplicate, Detection Technique, Detection Capability, Automatic Deduplication, Detection Rates and Error Rates
JOURNAL NAME:
Journal of Software Engineering and Applications,
Vol.10 No.6,
June
19,
2017
ABSTRACT: Semantic duplicates in databases represent today an important data quality challenge which leads to bad decisions. In large databases, we sometimes find ourselves with tens of thousands of duplicates, which necessitates an automatic deduplication. For this, it is necessary to detect duplicates, with a fairly reliable method to find as many duplicates as possible and powerful enough to run in a reasonable time. This paper proposes and compares on real data effective duplicates detection methods for automatic deduplication of files based on names, working with French texts or English texts, and the names of people or places, in Africa or in the West. After conducting a more complete classification of semantic duplicates than the usual classifications, we introduce several methods for detecting duplicates whose average complexity observed is less than O(2n). Through a simple model, we highlight a global efficacy rate, combining precision and recall. We propose a new metric distance between records, as well as rules for automatic duplicate detection. Analyses made on a database containing real data for an administration in Central Africa, and on a known standard database containing names of restaurants in the USA, have shown better results than those of known methods, with a lesser complexity.