Azure Data Factory: Validation Activity

We looked at a specific scenario for branching the control flow based on some business rule or condition in the previous post. In this post, we are going to look at a different scenario.

Sometimes, during a scheduled data load, the source data store may not be ready i.e., the files in the source folder have not been generated when the pipeline gets triggered. By default, this would cause the Pipeline to fail. However, Data Factory provides an easy way to halt the Pipeline execution for the time being and retry later. This can be accomplished using the Validation Activity.

Azure Data Factory: Validation Activity

As we can see in the screenshot above, under the Settings tab, there are three configuration settings:

  1. Timeout: This is the time for which the validation activity is active, the default value for timeout is 7 days.
  2. Sleep: Time (in seconds) between retry attempts.

So, for example, if the pipeline trigger time is 2PM and we would like the Pipeline to keep checking for a file until 5 PM every 2 minutes, we can set these values as:

Timeout: 0:03:00:00 (Format D:HH:MM:SS)

Sleep: 120 (seconds)

This brings us to the third setting:

  1. Child Items: This setting is available only when the selected dataset points to a folder. It allows the users to control the Pipeline execution based on some checks the folder. There are three options here:
    • Ignore: Just checks the existence of the folder, irrespective of whether there are any objects in the folder
    • True: Checks if the folder exists and checks the folder for existence of object
    • False: This option will check if the folder exists, and it is empty.

Reference: https://docs.microsoft.com/en-us/azure/data-factory/control-flow-validation-activity

One thought on “Azure Data Factory: Validation Activity

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: