Tutorial¶
Simple deduplication¶
[1]:
import pandas as pd
from deduplipy.datasets import load_data
Load data
[2]:
df_train = load_data(kind='voters')
Column names: 'name', 'suburb', 'postcode'
[3]:
df_train.head(2)
[3]:
name | suburb | postcode | |
---|---|---|---|
0 | khimerc thomas | charlotte | 2826g |
1 | lucille richardst | kannapolis | 28o81 |
Import Deduplicator
class
[4]:
from deduplipy.deduplicator import Deduplicator
Instantiate Deduplicator
class with the column names
[5]:
myDedupliPy = Deduplicator(['name', 'suburb', 'postcode'])
Perform the fitting using active learning
[ ]:
myDedupliPy.fit(df_train)
Predict on new data
[7]:
res = myDedupliPy.predict(df_train)
res.sort_values('deduplication_id').head(10)
[7]:
name | suburb | postcode | deduplication_id | |
---|---|---|---|---|
252 | kiera matthews | charlotte | 28216 | 1 |
1380 | kiea matthews | charlotte | 28218 | 1 |
0 | khimerc thomas | charlotte | 2826g | 2 |
1190 | chimerc thomas | charlotte | 28269 | 2 |
1302 | chimerc thmas | chaflotte | 28269 | 2 |
1255 | kimberly craddock | charlotte | 28214 | 6 |
15 | kimbefly craddock | charlotte | 28264 | 6 |
1313 | kimberly craddoclc | charlotte | 282|4 | 6 |
39 | l douglas loujdin | charlotte | 28225 | 9 |
1139 | l douglas loudin | charlotte | 28205 | 9 |
The Deduplicator
instance can be saved as a pickle file and be applied on new data after training:
[8]:
import pickle
[9]:
with open('myDeduplipy.pkl', 'wb') as f:
pickle.dump(myDedupliPy, f)
[11]:
with open('myDeduplipy.pkl', 'rb') as f:
loaded_obj = pickle.load(f)
[13]:
res = loaded_obj.predict(df_train)
res.sort_values('deduplication_id').head(10)
[13]:
name | suburb | postcode | deduplication_id | |
---|---|---|---|---|
252 | kiera matthews | charlotte | 28216 | 1 |
1380 | kiea matthews | charlotte | 28218 | 1 |
0 | khimerc thomas | charlotte | 2826g | 2 |
1190 | chimerc thomas | charlotte | 28269 | 2 |
1302 | chimerc thmas | chaflotte | 28269 | 2 |
1255 | kimberly craddock | charlotte | 28214 | 6 |
15 | kimbefly craddock | charlotte | 28264 | 6 |
1313 | kimberly craddoclc | charlotte | 282|4 | 6 |
39 | l douglas loujdin | charlotte | 28225 | 9 |
1139 | l douglas loudin | charlotte | 28205 | 9 |
Advanced deduplication¶
Load your data. In this example we take a sample dataset that comes with DedupliPy:
[14]:
from deduplipy.datasets import load_data
[15]:
df = load_data(kind='voters')
Column names: 'name', 'suburb', 'postcode'
Create a Deduplicator
instance and provide advanced settings
The similarity metrics per field are entered in a dict. Similarity metric can be any function that takes two strings and output a number.
[16]:
from deduplipy.deduplicator import Deduplicator
from fuzzywuzzy.fuzz import ratio, partial_ratio, token_set_ratio, token_sort_ratio
[17]:
field_info = {'name':[ratio, partial_ratio], 'suburb':[token_set_ratio, token_sort_ratio], 'postcode':[ratio]}
We choose our own set of rules for blocking which we define ourselves.
[18]:
def first_two_characters(x):
return x[:2]
interaction=True
makes the classifier include interaction features, e.g.ratio('name') * token_set_ratio('suburb')
. When interaction features are included, the logistic regression classifier applies a L1 regularisation to prevent overfitting.We set
verbose=1
to get information on the progress and a distribution of scores
[19]:
myDedupliPy = Deduplicator(field_info=field_info, interaction=True, rules = [first_two_characters], verbose=1)
Fit the Deduplicator
by active learning; enter whether a pair is a match (y) or not (n). When the training is converged, you will be notified and you can finish training by entering ‘f’.
[ ]:
myDedupliPy.fit(df)
Based on the histogram of scores, we decide to ignore all pairs with a similarity probability lower than 0.1 when predicting:
Apply the trained Deduplicator
on (new) data. The column deduplication_id
is the identifier for a cluster. Rows with the same deduplication_id
are found to be the same real world entity.
[21]:
res = myDedupliPy.predict(df, score_threshold=0.1)
res.sort_values('deduplication_id').head(10)
blocking started
blocking finished
Nr of pairs: 748767
scoring started
scoring finished
Nr of filtered pairs: 944
Clustering started
Clustering finished
[21]:
name | suburb | postcode | deduplication_id | |
---|---|---|---|---|
0 | khimerc thomas | charlotte | 2826g | 1 |
1190 | chimerc thomas | charlotte | 28269 | 1 |
1302 | chimerc thmas | chaflotte | 28269 | 1 |
1 | lucille richardst | kannapolis | 28o81 | 4 |
1194 | lucille richards | kannapolis | 28081 | 4 |
1449 | darryl perry | fayetteville | 28301 | 6 |
5 | darr6l perry | fayetteville | 28321 | 6 |
966 | daryl perry | fayetteville | 2830l | 6 |
7 | judith gerde5 | sunset beach | 28q68 | 9 |
1188 | judith gerdes | sunset beach | 28468 | 9 |