Data Chunking

Overview

In the previous article, we covered streaming data to a destination with Alloy's Stream Data block. Now, let's take things a bit further.

By default, the Stream Data block will stream one item at a time to the webhook URL you've specified. Unfortunately, that can be pretty annoying sometimes if you're iterating through a list of hundreds, if not thousands of items. Fortunately, we've got you covered.

Chunking in Action

To remedy this, we've introduced a feature called "chunking." When enabled, our system automatically breaks an array down into groups, or chunks, of the size you've specified. Then, we only stream the chunk over to your API instead of sending hundreds of items at once.

Let's take an example:

When interacting with large data sets such as exporting a users's entire NetSuite order catalog, data is best handled in segments, or chunks.

Say you're looking to retrieve all sales orders from NetSuite. The store in question has over 10,000 orders. Rather than streaming each order over individually (and potentially overloading your own server's rate limits), you can tell Alloy to send over a specific chunk size. If you specify 50 as the chunk size, Alloy Embedded will send over 200 batches, or chunks, of 50 items.

Enabling this feature is simple. Simply click the "Optional Parameters" dropdown inside the Stream Data block to select the chunk size. That's it! Easy peasy.

Wrapping Up

Alloy Embedded makes it easy to chunk data using our stream data block. Additionally, you can customize the size of the chunk.