Azure Data Factory: Monitor Copy Data Activity

One of the most important features of any ETL application or service is the ability to monitor and analyse the performance of the data load. Data Factory comes equipped with robust monitoring solutions. Monitoring options in Data Factory are not only limited to visually monitoring using the UI but can be done programmatically as well.Continue reading “Azure Data Factory: Monitor Copy Data Activity”

Azure Data Factory: System Variables

We have discussed about Global parameters in Azure Data Factory in a previous post. In today’s post, let’s learn about System variables in Azure Data Factory. System variables are built-in variables available to be used within every Data Factory Pipeline. The main purpose of having system variables is to make it easy for users toContinue reading “Azure Data Factory: System Variables”

Azure Data Factory: Staged Copy

In the previous post about Delete activity, we discussed briefly about the staging area. In a Data Warehouse load scenario, we may choose to stage the data in a temporary storage area (such as Azure Blob storage or ADLS), instead of sending the data directly to the sink. Also, there could be other reasons suchContinue reading “Azure Data Factory: Staged Copy”

Azure Data Factory: Delete Activity

Sometimes during data load, there are scenarios where we would like to delete the existing files or folder and start with a clean slate. This will most likely be applicable to be a temporary storage area (such as Staging Area or Landing Zone), where data needs to be stored before it can be copied theContinue reading “Azure Data Factory: Delete Activity”

Azure Data Factory: Monitor Pipelines using Azure Monitor   

In the previous post, we discussed about creating an Elapsed Time metric to monitor long-running Pipelines in real time using Azure Monitor. In this post, lets discuss about Azure Monitor and how it can be used to monitor Data Factory Pipelines. The easiest way to monitor Pipelines visually, is to go the Azure Monitor UIContinue reading “Azure Data Factory: Monitor Pipelines using Azure Monitor   “

Azure Data Factory: Identify long running Pipelines during execution

Data Factory Pipelines are often used to run business critical data loads. Most of them are scheduled, using triggers. We have discussed options to automate the execution of Data Factory Pipelines using triggers in a previous post. While there are metrics available which can be captured using Azure Monitor. During runtime, sometimes an activity mightContinue reading “Azure Data Factory: Identify long running Pipelines during execution”

Azure Data Factory: Append Variable Activity

We have looked at the Set Variable activity and an example of how use Set Variable to increment a variable in a previous post. There is another activity which interacts with variables in Data Factory, the Append Variable activity. In this post, we will discuss the Append Variable activity and see how it is differentContinue reading “Azure Data Factory: Append Variable Activity”

Azure Data Factory: Lookup Activity

In the previous post, about Execute Pipeline Activity, we used Lookup Activity to return the table name and then passed it on to the Execute Pipeline Activity. In today’s post, let’s discuss the Lookup Activity in more detail. Lookup Activity is designed for use cases where we need to return a result set based onContinue reading “Azure Data Factory: Lookup Activity”

Azure Data Factory: Execute Pipeline Activity

We have discussed Data Factory Pipelines and Activities in a previous post. Some of these activities let you run other activities iteratively or based on condition evaluation. Let’s consider a scenario where we would like to run another Pipeline from within the current pipeline. This essentially means that we have a hierarchy of pipelines. TheContinue reading “Azure Data Factory: Execute Pipeline Activity”