Data Retention When Merging: Ensure You Keep Critical Data

While navigating the intricate landscape of data management, organizations often grapple with the task of merging duplicates in their CRM system. This process, while integral to maintaining a streamlined and accurate database, often comes with complexities that demand careful consideration.

Chief among these considerations is what data is kept after the records are merged. For example, if you were to merge five duplicate contacts and three of those records had a different phone number listed, how would you decide which to keep in the resulting master record?

Data retention plays a pivotal role in preserving unique information, upholding audit trails, and safeguarding record ownership and privacy. However, it is not without its challenges. Merging duplicates in a CRM system is akin to walking a tightrope. It's a process that, when executed correctly, enhances the accuracy and efficiency of the database. However, it also carries the risk of losing unique information, disrupting audit trails, and infringing on privacy. Therefore, it's essential to approach the duplicate merging process with a plan for data retention.

How Do Merge-Related Data Retention Issues Impact Businesses?

When duplicates are merged in a CRM system without proper data retention, it can lead to a variety of business issues that hurt operations and the experience of customers:

• Loss of crucial information: If the merging process isn't handled correctly, unique information from each record could be lost. This might include important customer interactions, different contact information, or varying data about customer preferences. Losing such data can negatively impact customer service and relationship management.
• Disrupted audit trails: Merging duplicates without maintaining a clear record of when and how the merging occurred can disrupt the audit trail. This can create issues for auditing or tracking changes over time, which is crucial for accountability and transparency.
• Record ownership confusion: If different duplicates are owned by or associated with different staff members, merging these records could lead to confusion or potential breaches of privacy. This could result in staff members losing access to records they were managing, reps stepping on each other’s toes, internal confusion, and customer dissatisfaction.
• Automated record routing issues. When fields are used in automation and the wrong record data is retained in that field, records could be routed to the wrong person, or cause other operational issues due to broken automation.
• Customer relationship challenges: If duplicate records contain conflicting data, such as different phone numbers or addresses for the same customer, incorrect merging can lead to inaccurate or misleading data in your CRM. This can affect your customer communication and relationship management efforts, leading to customer dissatisfaction.

To avoid these issues, it's essential to have a clear strategy for merging duplicates, including a thorough understanding of potential data retention concerns and a well-defined process for mitigating them.

Why Is It Difficult To Control Data Retention When Merging in Most CRMs?

When merging duplicate records in most CRM systems, controlling the retention of specific data can be a challenge due to several factors.

1. Many CRM systems have automated tools for identifying and merging duplicates. While these can save time and reduce errors, they often use predefined rules to decide which data to retain. If these rules don't align perfectly with your specific needs, important data may be overlooked or discarded.
2. In many CRM systems, when duplicates are merged, one record is typically chosen as the "master" and others are merged into it. The choice of master can have significant implications for data retention, and automated systems may not always select the optimal master record based on your specific needs.
3. Most CRM systems do not offer robust options for customizing the merge process. You might have limited control over defining the criteria for identifying duplicates and deciding how to handle different data fields during the merge. It is unlikely, however, that you will have complete control over your data retention without developing a custom solution.

These factors can make it difficult to control what data is retained when merging duplicates in a majority of CRM systems. Businesses typically deal with these data retention challenges in a variety of ways.

What Are the Options For Data Retention Control When Merging Duplicates in CRMs?

Handling data retention on a field-by-field basis when merging duplicates in a CRM can be complex, but many companies have developed strategies and adopted tools to manage this challenge. Here are some ways companies are dealing with this:

Manually Review and Merge

While this can be time-consuming, some companies choose to manually review and merge duplicate records to ensure control over what data is retained. This typically involves creating a shortlist of potential duplicates,  deciding which data to retain for each field, manually merging the records in Excel, importing the data into the CRM, and then deleting the other duplicate records.

Custom-Developed Solutions

Some companies develop custom-coded solutions to control the merge process and data retention automatically, using predefined rules. While this is often a great approach, it does have the downside of having to pull development resources off of other projects. Additionally, as bugs arise or the CRM makes changes, ongoing development will be required to maintain the solution.

Data Enrichment

CRM data enrichment tools can help by filling in missing data and updating outdated data before the merge process. By ensuring that records are as complete and up to date as possible, these tools can help to minimize conflicts and discrepancies when merging duplicates. However, data enrichment tools can be expensive. And there is no guarantee that, despite enriching the data, you won’t lose other critical information during the merge process.

Third-Party Integrated Apps With Data Retention Features

Companies also adopt specialized deduplication solutions that integrate with their existing CRM. These tools may have advanced capabilities to identify duplicates and customizable rules to retain data during the merge process, which provides companies with better control over their data.

Remember, the approach a company chooses can depend on several factors, including the scale of the data, the capabilities of the CRM system, and the company's specific data needs and policies.

Examples of Situations Where Data Retention Is Important

Until your company runs into a specific situation involving the need to retain specific data in a field, it may be difficult to fully understand just how critical this process can be.

Let’s look at some example situations where data retention would be important.

• Understanding recent interactions with customers: A company might have two duplicate records where each record indicates a different last contact date. One record might show that the last contact was a month ago, while the other might show that the last contact was three months ago. In this case, the company would likely want to retain the more recent contact date when merging the records to ensure they are aware of the most recent interaction.
• Sales rep record ownership: Suppose two sales reps have inadvertently been working with the same prospect because they have duplicate records. Each record indicates a different sales representative. When merging these records, the company would need to decide which rep should be retained as the owner. This might depend on factors like who has had more interactions with the customer, who contacted the customer first, or who has more capacity to manage the account.
• Recently updated contact information: Let's say a company has two duplicate records for the same customer with different contact information. One record has one email address, while the other has another. In this circumstance, they might want to retain the email address that was most recently updated.
• Multiple lead sources: If a company has two records for the same customer, but each record has a different lead source (for example, one is marked as "web form submission" and the other as "trade show lead"), it could be crucial to retain both sources. This information could be important for understanding the customer's journey and evaluating the effectiveness of different marketing strategies.
• Customer status: Imagine a company has duplicate customer records where one lists the customer as "Active" and the other as "Inactive." In this scenario, when merging these duplicates, it would typically be critical to retain the "Active" status.
• Associations: Records may have many associations in your CRM, and especially across duplicate records. A contact may be associated with several companies as an employee, board member, consultant, or other roles. A company may have associated parent or child companies in its hierarchy. Retaining all relevant associations is critical for gaining a complete understanding of each customer.

These are just a few of the many examples where data retention is a big concern when merging duplicates in your CRM.

We recently spoke with Matthias Werner, Head of Finance and Analytics at Data Virtuality, an Insycle customer who has used Insycle’s deduplication data retention features when merging duplicate HubSpot contacts. In our interview, he spoke about some of their biggest concerns for contact data retention when merging duplicates in their CRM:

“One of our main criteria is definitely the lead status. I created a hierarchy to identify the master record based on the lead status. That is the primary condition that we look at. Then, if no master record is identifiable based on this condition, then, for example, we look at the last activity date to ensure that we keep data from the record where the most recent activity happened.”

Matthias explained that once they have rules in place to identify the master record with the most up-to-date data, they then begin to set data retention rules for their most important contact fields.

“Once we have identified the master record, I set up field rules to define which values we want to retain. We have some internal considerations there. For example, we often want to keep the first conversion data from a duplicate group, or data from the first form a lead filled out,” he explained.

With Insycle, Data Virtuality was able to identify and merge duplicate contact records in HubSpot without worrying about potentially losing useful data and negatively impacting their sales and marketing processes.

Insycle Allows You To Set Data Retention Rules For Every Field

Insycle offers advanced CRM deduplication features with deep customization options for data retention.

Identify and Bulk-Merge Duplicate Records in Advanced Ways

To start, Insycle allows you to use any field in your CRM as a potential duplicate matching field, catching more duplicates than you otherwise would using standard CRM duplicate detection features. You can choose to use exact matching or similar (fuzzy) matching.

You also have advanced control over how the data contained within the field is assessed by Insycle. For example, you can compare phone numbers while ignoring symbols and whitespace, which allows you to identify duplicate phone numbers even if they are formatted differently. And you can do this for any matching field that you select.

For more control over how data is retained during the merge, Insycle offers rule-based master record selection.

Rule-Based Master Record Selection

Here, you can define rules for choosing the resulting master record after the merge. Insycle uses a process of elimination to go down the list of rules that you set and remove records that do not meet the requirements. It continues until only one record remains, and then chooses that one as the post-merge master record.

This gives you a lot of control over the data that you retain. In most fields, the data from the master record will be retained by default unless you set field-by-field data retention preferences, which we will cover next. In some fields where appending data from all records is the correct choice, such as a contact’s activity log, Insycle will automatically merge data from all of the duplicate records.

Insycle also offers other ways to control the data you retain while merging.

Rule-Based Data Retention Settings For Every Field

Insycle gives you unparalleled control over your data retention by allowing you to set rules for merging and retaining data on a field-by-field basis, for every field in your database.

For example, in the example below, we are instructing Insycle to:

• keep the email from the master record
• keep the contact owner for the record with the latest activity date
• combine and append all membership notes
• keep the lead source from the record that became a lead first
• keep the pipeline enrollment from any record in the sales pipeline

You can also collate data from multiple records, if you’d like to preserve a specific field from all of your duplicate records. For example, your duplicate contacts might have multiple listed email addresses that you wouldn’t want to lose during the merge. You can instruct Insycle to collect the email addresses of all records (except for the post-merge master record, whose email will stay in the primary email field) and collate them into an additional email addresses field.

With Insycle, you have unparalleled control over how your data is retained when bulk-merging.

Insycle Delivers Advanced Deduplication With Complete Data Retention Control

Insycle's rule-based master record selection and field-by-field data retention preferences provide unparalleled control over data retention. This makes Insycle an essential tool for businesses looking to maintain the accuracy and efficiency of their databases. It allows you to set data retention rules for every field, ensuring that no critical information is lost during the merging process.

However, Insycle is more than just a tool for managing duplicates. It is a complete customer data management platform that offers a range of features designed to streamline and enhance data management processes. Insycle integrates seamlessly with CRMs like HubSpot, Salesforce, Pipedrive, and Mailchimp.

Whether you're grappling with the complexities of merging duplicates, looking to streamline your data management processes, or seeking to clean and declutter your CRM, Insycle can help. Discover how Insycle can help you unlock the full potential of your data.