deduplicator¶
-
class
deduplipy.deduplicator.
Deduplicator
(col_names: Optional[List[str]] = None, field_info: Optional[Dict] = None, interaction: bool = False, rules: Optional[List[Callable]] = None, recall=1.0, save_intermediate_steps: bool = False, verbose: Union[int, bool] = 0)¶ Bases:
object
-
fit
(X: pandas.core.frame.DataFrame, n_samples: int = 10000) → deduplipy.deduplicator.deduplicator.Deduplicator¶ Fit the deduplicator instance
- Args:
X: Pandas dataframe to be used for fitting n_samples: number of pairs to be created for active learning
Returns: trained deduplicator instance
-
predict
(X: pandas.core.frame.DataFrame, score_threshold: float = 0.1) → pandas.core.frame.DataFrame¶ Predict on new data using the trained deduplicator.
- Args:
X: Pandas dataframe with column as used when fitting deduplicator instance score_threshold: Classification threshold to use for filtering before starting hierarchical clustering
Returns: Pandas dataframe with a new column deduplication_id. Rows with the same deduplication_id are deduplicated.
-