deduplicator

class deduplipy.deduplicator.Deduplicator(col_names: Optional[List[str]] = None, field_info: Optional[Dict] = None, interaction: bool = False, rules: Optional[Union[List[Callable], Dict]] = None, recall=1.0, save_intermediate_steps: bool = False, verbose: Union[int, bool] = 0)

Bases: object

fit(X: pandas.core.frame.DataFrame, n_samples: int = 100000)deduplipy.deduplicator.deduplicator.Deduplicator

Fit the deduplicator instance

Args:

X: Pandas dataframe to be used for fitting n_samples: number of pairs to be created for active learning

Returns: trained deduplicator instance

predict(X: pandas.core.frame.DataFrame, score_threshold: float = 0.1, cluster_threshold: float = 0.5, fill_missing=True)pandas.core.frame.DataFrame

Predict on new data using the trained deduplicator.

Args:

X: Pandas dataframe with column as used when fitting deduplicator instance score_threshold: Classification threshold to use for filtering before starting hierarchical clustering cluster_threshold: threshold to apply in hierarchical clustering fill_missing: whether or not to apply missing value imputation on adjacency matrix

Returns: Pandas dataframe with a new column deduplication_id. Rows with the same deduplication_id are deduplicated.