In Dynamics 365, the "Import Threshold Record Count" setting in the Data Management Framework (DMF) is a key player in how your data gets imported. It controls how many records are processed per thread, helping to split up imports into smaller chunks and potentially speeding up the process.
Why It Matters:
The "Import Threshold Record Count" setting determines how many records a single thread will process during an import.
How It Works:
When you set a threshold, the DMF breaks your import into several smaller tasks, each handling a portion of the records. So instead of one thread doing all the work, the load gets spread out.
Example:
Let’s say you’ve got a file with 10,000 records. If you set the threshold to 2,500 and choose to run 4 tasks, each of the 4 threads will process 2,500 records. Simple as that!
Where to Find It:
You can find this setting under the Entity Import Execution Parameters in the Data Management Framework (DMF) section of Dynamics 365.
Other Important Parameters to Know:
Import Task Count: This is the number of threads (or tasks) that will run during your import. It’s how you decide how many “workers” are helping with the job.
Entity: This is where you pick the data entity you want to configure these settings for.
Why Use the Threshold?
Boost Performance: By breaking the import into smaller tasks, you might see better performance, especially with larger datasets.
Manage Resources: You get more control over how resources are used during the import.
Flexibility: You can tweak the process to fit your specific needs by adjusting the threshold and task count.
Alternative
As an alternative, split your load files into smaller files and supply these via the standard DMF import routine. This gives greater granularity at the individual load level and does not require specific settings against the data entity in Dynamics.
Expected Performance Improvements
As an example, we loaded 4 million rows of pricing data into clients production environment using this method, splitting files into chunks of 2000 records, and running with 8 threads. Our loads took approximately 18 hours.
The equivalent loads using a single file and no settings on Import Threshold Record Count resulted in a load times of nearly 8 days.
** Timings will vary per entity and environment, so experiment with these setting per entity to determine your optimum configuration.
Konexxia Data Migration
Konexxia's data migration solution allow simply configuration of data output to facilitate splitting of files into specific sized outputs (1000, 2000, 5000 record etc.) or to split by one or many combined field values at the record level (for example, split by customer group, or VAT code etc.)