Tutorial

Simple deduplication

import pandas as pd
from deduplipy.datasets import load_data

Load data

df_train = load_data(kind='voters')

Column names: 'name', 'suburb', 'postcode'

df_train.head(2)

	name	suburb	postcode
0	khimerc thomas	charlotte	2826g
1	lucille richardst	kannapolis	28o81

Import Deduplicator class

from deduplipy.deduplicator import Deduplicator

Instantiate Deduplicator class with the column names

myDedupliPy = Deduplicator(['name', 'suburb', 'postcode'])

Perform the fitting using active learning

myDedupliPy.fit(df_train)

Predict on new data

res = myDedupliPy.predict(df_train)
res.sort_values('deduplication_id').head(10)

	name	suburb	postcode	deduplication_id
252	kiera matthews	charlotte	28216	1
1380	kiea matthews	charlotte	28218	1
0	khimerc thomas	charlotte	2826g	2
1190	chimerc thomas	charlotte	28269	2
1302	chimerc thmas	chaflotte	28269	2
1255	kimberly craddock	charlotte	28214	6
15	kimbefly craddock	charlotte	28264	6
1313	kimberly craddoclc	charlotte	282|4	6
39	l douglas loujdin	charlotte	28225	9
1139	l douglas loudin	charlotte	28205	9

The Deduplicator instance can be saved as a pickle file and be applied on new data after training:

import pickle

with open('myDeduplipy.pkl', 'wb') as f:
    pickle.dump(myDedupliPy, f)

with open('myDeduplipy.pkl', 'rb') as f:
    loaded_obj = pickle.load(f)

res = loaded_obj.predict(df_train)
res.sort_values('deduplication_id').head(10)

	name	suburb	postcode	deduplication_id
252	kiera matthews	charlotte	28216	1
1380	kiea matthews	charlotte	28218	1
0	khimerc thomas	charlotte	2826g	2
1190	chimerc thomas	charlotte	28269	2
1302	chimerc thmas	chaflotte	28269	2
1255	kimberly craddock	charlotte	28214	6
15	kimbefly craddock	charlotte	28264	6
1313	kimberly craddoclc	charlotte	282|4	6
39	l douglas loujdin	charlotte	28225	9
1139	l douglas loudin	charlotte	28205	9

Advanced deduplication

Load your data. In this example we take a sample dataset that comes with DedupliPy:

from deduplipy.datasets import load_data

df = load_data(kind='voters')

Column names: 'name', 'suburb', 'postcode'

Create a Deduplicator instance and provide advanced settings

• The similarity metrics per field are entered in a dict. Similarity metric can be any function that takes two strings and output a number.

from deduplipy.deduplicator import Deduplicator
from fuzzywuzzy.fuzz import ratio, partial_ratio, token_set_ratio, token_sort_ratio

field_info = {'name':[ratio, partial_ratio], 'suburb':[token_set_ratio, token_sort_ratio], 'postcode':[ratio]}

• We choose our own set of rules for blocking which we define ourselves.

def first_two_characters(x):
    return x[:2]

• interaction=True makes the classifier include interaction features, e.g. ratio('name') * token_set_ratio('suburb'). When interaction features are included, the logistic regression classifier applies a L1 regularisation to prevent overfitting.

• We set verbose=1 to get information on the progress and a distribution of scores

myDedupliPy = Deduplicator(field_info=field_info, interaction=True, rules = [first_two_characters], verbose=1)

Fit the Deduplicator by active learning; enter whether a pair is a match (y) or not (n). When the training is converged, you will be notified and you can finish training by entering ‘f’.

myDedupliPy.fit(df)

Based on the histogram of scores, we decide to ignore all pairs with a similarity probability lower than 0.1 when predicting:

Apply the trained Deduplicator on (new) data. The column deduplication_id is the identifier for a cluster. Rows with the same deduplication_id are found to be the same real world entity.

res = myDedupliPy.predict(df, score_threshold=0.1)
res.sort_values('deduplication_id').head(10)

blocking started
blocking finished
Nr of pairs: 748767
scoring started
scoring finished
Nr of filtered pairs: 944
Clustering started
Clustering finished

	name	suburb	postcode	deduplication_id
0	khimerc thomas	charlotte	2826g	1
1190	chimerc thomas	charlotte	28269	1
1302	chimerc thmas	chaflotte	28269	1
1	lucille richardst	kannapolis	28o81	4
1194	lucille richards	kannapolis	28081	4
1449	darryl perry	fayetteville	28301	6
5	darr6l perry	fayetteville	28321	6
966	daryl perry	fayetteville	2830l	6
7	judith gerde5	sunset beach	28q68	9
1188	judith gerdes	sunset beach	28468	9