Understanding The LeadScale Engine Deduplication Process

Written by Wayne Kinsella on November 20 2023

Apparently, it’s again my turn to write a post for the company page, which anyone who knows me will know is not exactly my cup of tea or for our colleagues across the pond: “Not my 144oz pail of Mountain Dew Code Red”.   

I was racking my brain for what to write about, and it hit me. In the past few weeks, I have received many questions about the LeadScale Engine de-duplication and suppression modules. So, to make it easier for anyone who asks me again in the future, I thought, “Why not put it into a post?”. Now, anytime someone asks me about de-duplication, I will simply send them a link, and they will easily understand how the technology works.  

Here it goes a blog with a brief overview of the Engine dupes, suppression, and, for good measure, a little on exclusion and how it differs from suppression.

Understanding Duplication

Duplication: What do we mean by dupe?  

Here are a few examples:  

Exact Match 

    • Exact match – Replication
    • Exact match – Repeat   

Is the lead an exact match from the same source with the same details? vs Is it an exact match from the same source with differing opt-in data or from A.N.Other?   

Advanced Duplication (Capping)   

Contact plus content capping (Organic double touch)  

How many times will you allow the same contact to qualify with different content touches?  

Contacts from company cap 

How many contacts from an individual company will you allow?  

Duplicate suspects  

Are these the same person?  

    1. w.kinsella@leadscale.com
    2. wayne@leadscale.com
    3. wayne.kinsella@leadscale.com  

The assumption would be yes. However, the correct answer would be “don’t know.”   

LeadScale Engine Dedupe Process

During the Engine processing, we would ensure that Wayne is the only W. Kinsella at LeadScale and that Waynetta Kinsella isn’t also working with hubby. This would increase the confidence you could have in showing a dupe suspect. Bad example, I know, but try it with “J. Smith” in the NHS.  

Pre dupe vs. Post dupe

This is a good one, as we have many bionic (semi-automated) processes where the data is standardised and normalised. We realised a single dupe check didn’t cut the mustard.  

Dedupe count pre-processing will give different results than post-processing. For example, LeadScale Ltd would not count as a dupe with Leadscale Group on pre-dupe but would on post-dupe.

Capping groups

Carryover capping counts from a past job or jobs. Suppose the “Company Example Ltd” cap was 6 in Q2, and you only had two leads from them by grouping the cap across {Q2|Q3}. In that case, you will only allow 4 more in Q3 before capping out “Company Example Ltd”.  

Dynamic capping  

Cap contact/content or company by geo, region or lead type. Do you want 6 leads from “Company Example Ltd” in EMEA but will allow 12 from “Company Example Ltd” in North America? It is not an issue with the Engine.  

A Final note on capping  

It’s funny how wrong this logic can go; while writing this, we received a TAL (Target Account List) job with 300 accounts, which has been capped at 6 contacts from each company (account). It sounded fine until we realised they had just booked 2,500 leads in this job.  

I’m no Mystic Meg (showing my age and location here). Let’s try to be clairvoyant, but even I foresee issues here. 


Exclude company name or email domain. This isn’t as easy as it sounds; if you were to exclude “Company Example Ltd”, would “Company Example Limited” or “Company Example Group” be matched? Are they even the same company? With the Engine, yes. We enable fuzzy logic checks for our bionic processing, which allows for this.  

Domains are excluded on exact match only; therefore, if you have “CompanyExample.com” on your exclusion list and a lead from “Company Example Ltd” with the email domain of “CompanyExample.co.uk” comes in, it will qualify. 


The main point of suppression is to enable your provisioners to de-target specific individuals whether they are current customers, new or lapsed contacts, or those who have unsubscribed or requested not to be contacted—before broadcasting. 

The Engine allows for basic input suppression with raw or hashed email addresses. However, it also enables these suppressed contacts to be securely checked by your supplier network before they target their audience.     

That is enough for now. For the next post, we will dig deeper into Dynamic Suppression and Enablement and how these tools can improve your campaign performance, efficiency, and effectiveness.  

If you still have questions, please contact me. I will be more than happy to answer them.

Embrace the Future, Lead the Generation

LeadScale is revolutionizing the way businesses work with lead generation. Our experts are helping companies reach new heights by driving success and efficiency through data-driven insights. So, take the first step today. Let LeadScale empower you to unlock the true potential of your business and boldly navigate growth.

Contact our team to see how you can maximize your marketing budget with LeadScale.