Author Archives: ashish naik

We looked at a specific scenario for branching the control flow based on some business rule or condition in the previous post. In this post, we are going to look at a different scenario. Sometimes, during a scheduled data load, the source data store may not be ready i.e., the files in the source folderContinue reading “Azure Data Factory: Validation Activity”

While designing ETL data flows, we will come across some scenarios where we must branch the control flow based on a business rule or conditional expression. The easiest way to implement conditional branching, in Data Factory is by using the Switch Activity. It is very easy to understand and implement if you are familiar withContinue reading “Azure Data Factory: Switch Activity”

One of the most important features of any ETL application or service is the ability to monitor and analyse the performance of the data load. Data Factory comes equipped with robust monitoring solutions. Monitoring options in Data Factory are not only limited to visually monitoring using the UI but can be done programmatically as well.Continue reading “Azure Data Factory: Monitor Copy Data Activity”

We have discussed about Global parameters in Azure Data Factory in a previous post. In today’s post, let’s learn about System variables in Azure Data Factory. System variables are built-in variables available to be used within every Data Factory Pipeline. The main purpose of having system variables is to make it easy for users toContinue reading “Azure Data Factory: System Variables”

In the previous post about Delete activity, we discussed briefly about the staging area. In a Data Warehouse load scenario, we may choose to stage the data in a temporary storage area (such as Azure Blob storage or ADLS), instead of sending the data directly to the sink. Also, there could be other reasons suchContinue reading “Azure Data Factory: Staged Copy”

Sometimes during data load, there are scenarios where we would like to delete the existing files or folder and start with a clean slate. This will most likely be applicable to be a temporary storage area (such as Staging Area or Landing Zone), where data needs to be stored before it can be copied theContinue reading “Azure Data Factory: Delete Activity”

In the previous post, we discussed about creating an Elapsed Time metric to monitor long-running Pipelines in real time using Azure Monitor. In this post, lets discuss about Azure Monitor and how it can be used to monitor Data Factory Pipelines. The easiest way to monitor Pipelines visually, is to go the Azure Monitor UIContinue reading “Azure Data Factory: Monitor Pipelines using Azure Monitor   “

Data Factory Pipelines are often used to run business critical data loads. Most of them are scheduled, using triggers. We have discussed options to automate the execution of Data Factory Pipelines using triggers in a previous post. While there are metrics available which can be captured using Azure Monitor. During runtime, sometimes an activity mightContinue reading “Azure Data Factory: Identify long running Pipelines during execution”

We have looked at the Set Variable activity and an example of how use Set Variable to increment a variable in a previous post. There is another activity which interacts with variables in Data Factory, the Append Variable activity. In this post, we will discuss the Append Variable activity and see how it is differentContinue reading “Azure Data Factory: Append Variable Activity”

In the previous post, about Execute Pipeline Activity, we used Lookup Activity to return the table name and then passed it on to the Execute Pipeline Activity. In today’s post, let’s discuss the Lookup Activity in more detail. Lookup Activity is designed for use cases where we need to return a result set based onContinue reading “Azure Data Factory: Lookup Activity”