Azure Data Factory: Get Metadata Activity

We have looked at some Azure Data Factory activities such as, For Each Activity and Azure Function Activity in previous posts. In this post, lets have a look at another important activity, the Get Metadata Activity.

The purpose of the Get Metadata Activity, as the name suggests, is to get the metadata of any datastore or an individual file. The retrieved metadata can be any of the properties of the datastore/file such as, Item name, size, last modified date etc. The benefit of Get Metadata activity is that the activity output can be used in the subsequent steps in the Pipeline, e.g., to write conditional expression and perform further processing of the data based on the metadata value.

To use the Get Metadata activity in a Pipeline, the pre-requisite step is to have a dataset created, since the Get Metadata activity requires a dataset to connect to and retrieve the metadata information.

Azure Data Factory: Get Metadata Activity

The other important setting under Dataset is the Filter by last modified setting. This setting is optional and only applies while retrieving metadata from a folder (not a file). e.g., getting childitems list from a folder. Get Metadata activity will only fetch the metadata information (list of files) that fall between the Start time and End time specified. E.g., it will return a list of files from a folder that have been created within the last month. Also, the Start time and End time values can be assigned dynamically with the help of expressions.

Depending on the type of dataset specified, the Get Metadata activity provides different types of metadata values that can be retrieved. The table below lists some of the important metadata types available:

Metadata TypeMetadata RetrievedApplies to FolderApplies to File
itemNameName of ObjectYY
sizeObject sizeNY
createdCreated datetimeYY
lastModifiedLast modified datetimeYY
childItemsList of subfolders and filesYN
existsCheck if object exists; returns true/falseYY
Azure Data Factory: Get Metadata Activity Metadata Types

For a complete list of supported connectors and metadata options, please refer to the reference link below.

Reference: https://docs.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity

3 thoughts on “Azure Data Factory: Get Metadata Activity

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: