Azure Data Factory: Execute Pipeline Activity

We have discussed Data Factory Pipelines and Activities in a previous post. Some of these activities let you run other activities iteratively or based on condition evaluation. Let’s consider a scenario where we would like to run another Pipeline from within the current pipeline. This essentially means that we have a hierarchy of pipelines. The top level (Invoking) pipeline, i.e., Parent Pipeline and the lower level (Invoked) pipeline or the Child Pipeline.

There is a specific activity in Data Factory, the Execute Pipeline activity to enable this. Using the Execute Pipeline Activity, the underlying pipeline can be invoked simply within the control flow.

Under the settings tab, you can select the pipeline to be Invoked. The dropdown will show all available pipelines within the Data Factory.

If the selected pipeline has parameters configured, the settings area will show a list of those parameters. We can pass the parameter values while invoking the child pipeline here.

If no parameter values are passed, data factory will check if there are any default values for those parameters and execute the child pipeline with default parameter values. We will discuss this in detail in a future post about pipeline parameters.

Apart from simply using the execute pipeline activity in the control flow, (like above), the execute pipeline activity can also be used within an iterative/conditional activity. E.g. if we would like to execute a pipeline that can process multiple tables in the same database. We can create a child pipeline that accepts the target table name as parameter and pass the target table name to the child pipeline from a parent Pipeline.

To enable this, we first create a Lookup activity that returns the database table name and reference the output in the For Each activity, as shown below:

Next, we add an Execute Pipeline activity inside the For Each activity and pass the table name to the Pipeline parameter, using the code below:

@item.target_table_name

Please note, target_table_name is used as an example. Replace it with the name of the item containing the table name, returned by the Lookup activity

Reference: https://docs.microsoft.com/en-us/azure/data-factory/control-flow-execute-pipeline-activity