Azure Data Factory: Get Metadata Activity

We have looked at some Azure Data Factory activities such as, For Each Activity and Azure Function Activity in previous posts. In this post, lets have a look at another important activity, the Get Metadata Activity.

The purpose of the Get Metadata Activity, as the name suggests, is to get the metadata of any datastore or an individual file. The retrieved metadata can be any of the properties of the datastore/file such as, Item name, size, last modified date etc. The benefit of Get Metadata activity is that the activity output can be used in the subsequent steps in the Pipeline, e.g., to write conditional expression and perform further processing of the data based on the metadata value.

To use the Get Metadata activity in a Pipeline, the pre-requisite step is to have a dataset created, since the Get Metadata activity requires a dataset to connect to and retrieve the metadata information.

The other important setting under Dataset is the Filter by last modified setting. This setting is optional and only applies while retrieving metadata from a folder (not a file). e.g., getting childitems list from a folder. Get Metadata activity will only fetch the metadata information (list of files) that fall between the Start time and End time specified. E.g., it will return a list of files from a folder that have been created within the last month. Also, the Start time and End time values can be assigned dynamically with the help of expressions.

Depending on the type of dataset specified, the Get Metadata activity provides different types of metadata values that can be retrieved. The table below lists some of the important metadata types available:

Metadata Type	Metadata Retrieved	Applies to Folder	Applies to File
itemName	Name of Object	Y	Y
size	Object size	N	Y
created	Created datetime	Y	Y
lastModified	Last modified datetime	Y	Y
childItems	List of subfolders and files	Y	N
exists	Check if object exists; returns true/false	Y	Y

For a complete list of supported connectors and metadata options, please refer to the reference link below.

Reference: https://docs.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity