oagdedupe

Dedupe is a Python library for scalable entity resolution, using active learning to learn blocking configurations and clasify matches. See Getting Started for installation and setup instructions. See User Guide for more detailed methodology.

The diagaram below shows an overview of the pipeline.

Note

This project is under active development.

Contents

Getting Started

• 1. Installation
  • 1.1. .dedupe
  • 1.2. docker
  • 1.3. postgres
  • 1.4. label-studio
• 2. Settings
• 3. Dedupe
  • 3.1. train model
  • 3.2. start fastAPI
  • 3.3. label-studio
  • 3.4. predictions
• 4. Record Linkage
  • 4.1. train model
  • 4.2. start fastAPI
  • 4.3. label-studio
  • 4.4. predictions

User Guide

• 1. Intro
  • 1.1. Scalability
  • 1.2. Active Learning
  • 1.3. Pipeline Overview
• 2. Key Terms
• 3. Generate Training Samples
• 4. Blocking
• 5. Distance
• 6. Classifier
• 7. Active Learning Loop
  • 7.1. Initialization and First Pass
  • 7.2. New Active Learning Loop
  • 7.3. Predictions on Full Data
• 8. Clustering

Examples

• Dedupe Example
• Record Linkage Example

API Reference

• API
• Database
• Blocker
• Distance
• Cluster