Pulling in large datasets is powerful, but if you are not careful, it can overwhelm your systems. At ZangaBee, we often help companies that struggle with performance issues caused by excessive data loads or unfiltered exports. The solution is not to stop importing data, but to import it wisely. That is where Celigo’s integrator.io platform comes in.
Here is how to configure it for efficient, large-scale data handling without slowing down your operations.
1. Adjust Concurrency Levels
Concurrency determines how many API calls Celigo can make at the same time. Setting this incorrectly can result in failed calls, throttling, or performance degradation.
Set the right level by aligning your concurrency setting with the limits of the external system. For example, if an API supports up to five concurrent requests, configure your connection to match that number.
If multiple connections are targeting the same endpoint, use the “Borrow concurrency from” feature. This allows your flows to share concurrency resources and stay within API governance limits.
2. Optimize Page Size and Batch Size
Managing how much data is sent or received at once is crucial for performance.
Page size controls how many records are processed per page during exports. While the default is 20, you can increase this depending on the capacity of the external system. Just make sure the total payload remains under five megabytes per page.
Batch size applies to imports, particularly when sending data to systems like NetSuite or a database. Sending too many records at once can cause timeouts. Finding the right batch size helps maintain stability and increases throughput. If the target system supports bulk api operations, use that endpoint to process a whole batch of records at once.
3. Use Bulk Operations for Database Imports
If your integration involves databases, avoid sending records one by one. Use bulk features for efficiency.
With systems like MySQL or Microsoft SQL, Celigo supports bulk insert, allowing you to import many records in a single step.
If you are working with Snowflake, enable the optimized bulk load feature. It groups and stages the data before loading, saving both time and compute resources.
4. Filter Data Using Delta Exports
You do not need to process every record every time. Celigo’s delta export option allows you to export only the data that has changed since the last flow run. This significantly reduces the number of records processed and improves performance.
5. Monitor and Troubleshoot Performance
Once your integration is live, make sure to keep an eye on how it performs.
Celigo provides monitoring tools to help you identify slow steps and bottlenecks. Use these insights to fine-tune your settings.
Also implement reliable error handling and retry logic to deal with temporary issues and protect data integrity.
Conclusion
Pulling in large volumes of data is often necessary, but processing everything without limits is not sustainable. By using smart filters, efficient batching, and scalable settings in Celigo, you can ensure your integrations stay fast, stable, and resilient.
At ZangaBee, we specialize in building high-performing, scalable integration flows. Need help setting yours up? Reach out — we are happy to take a look.
