We have looked at the Set Variable activity and an example of how use Set Variable to increment a variable in a previous post. There is another activity which interacts with variables in Data Factory, the Append Variable activity. In this post, we will discuss the Append Variable activity and see how it is different from the Set Variable activity.
To understand how the Append Variable activity works in a better way, we must have a basic understanding of variables in Data Factory. Data Factory supports three variable types:
- String (Text)
- Boolean (Binary e.g. Y/N)
- Array (An Array Object)
If you know data structure basics, you must know that an Array is a collection of elements of similar data type and each position in the array can be addressed using a number.
Append Variable activity can only be used with an Array type variable. As the name suggests, Append variable activity appends new items to the next positions in the Array.
Let’s consider a scenario where we would like to get a list of filenames from a folder and load it into an array type variable for further processing.
Firstly, we must create an array type variable, that we will be using to load the File name list. This can be done in the Variables tab and selecting the type as Array.
Like the example in the previous post about loading filenames into an SQL table, we will use a Get Metadata activity to get the list of files from the folder. But instead of using a Stored Procedure Activity, we will use a For Each and Append Variable activities to process the file names.
In the Items setting for the For Each activity, use the following code to access the Get Metadata activity output.
@activity('Get Metadata1').output.valueNext, inside the For Each activity, we create an Append Variable activity and select the ADE1 array variable that we created above. Use the code below in the value field to append the Filename to the array variable.
@item().name
Please note, this example will just load the variable ADE1 with the Filenames from the Get Metadata activity. More activities can be added to the control flow to access the filenames using the array variable.
Reference: https://docs.microsoft.com/en-us/azure/data-factory/control-flow-append-variable-activity
Azure Data Engineering 