Azure Data Factory: New Monitoring View Features

It is very easy to visually monitor previous pipeline runs in Data Factory using the Monitor page in the Azure Data Factory, which we have already covered in a previous post. There have been some recent improvements to the monitoring view, we will go through these briefly in this post. Data from the Azure MonitorContinue reading “Azure Data Factory: New Monitoring View Features”

Azure Data Factory: Script Activity

While we have discussed various ways for running custom SQL code in Azure Data Factory in a previous post, recently, a new activity has been added to Azure Data Factory called Script Activity, which provides a more flexible way of running custom SQL scripts. As shown in the screenshot above, this activity supports execution ofContinue reading “Azure Data Factory: Script Activity”

Azure Data Factory: Monitor Self Hosted Integration Runtime Metrics

Self-hosted integration runtime in the context of Azure data factory is a gateway that connects the on-prem data sources to datastores in the cloud. To know more about Integration runtimes, please refer to the previous post. We have discussed how to check whether Integration Runtime is online or offline using PowerShell command in a previousContinue reading “Azure Data Factory: Monitor Self Hosted Integration Runtime Metrics”

Azure Data Factory: How to edit default parameter definition for ARM templates?

ARM or Azure Resource Manager templates make it easy to manage deployments for Data Factory. When we connect Data Factory to a source control repository (e.g. GitHub or Azure DevOps Git), the data factory along with all its artefacts (pipelines, datasets, linked services etc.) is saved in the repository in the form of ARM templates.Continue reading “Azure Data Factory: How to edit default parameter definition for ARM templates?”

Azure Data Factory: Stored Procedure Activity

When it comes to transforming structured data, (e.g., applying business logic, standardization etc.) stored in a database, SQL is the most convenient and fit-to-purpose option.  Stored procedures provide a way to store the transformation logic as a set of SQL statements that can be re-executed as pre-compiled code. The Stored Procedure Activity in Data FactoryContinue reading “Azure Data Factory: Stored Procedure Activity”

Azure Data Factory Linked Service: Advanced Authoring

We have discussed Linked Service parameterization through the UI, in a previous post. But not all Linked Service Types support parametrization using the UI.  In this post, we will discuss the Linked Services that can’t be parameterized using the UI. (i.e., they don’t have any option to add parameter). If you are familiar with AzureContinue reading “Azure Data Factory Linked Service: Advanced Authoring”

Azure Data Factory: Fail Activity

During some scenarios in Azure Data Factory, we may want to intentionally stop the execution of the pipeline. An example could be when we want to check the existence of a file or folder using Get Metadata activity. We may want to fail the pipeline if the file/folder does not exist. To achieve this, weContinue reading “Azure Data Factory: Fail Activity”

Azure Data Factory: Filter Activity

In the previous post, we discussed the Switch Activity, which is useful for branching the control flow based on some condition. We will discuss about the Filter Activity in this post. The purpose of Filter Activity is to process array items based on some condition. Consider a scenario where we would like to set theContinue reading “Azure Data Factory: Filter Activity”

Azure Data Factory: Validation Activity

We looked at a specific scenario for branching the control flow based on some business rule or condition in the previous post. In this post, we are going to look at a different scenario. Sometimes, during a scheduled data load, the source data store may not be ready i.e., the files in the source folderContinue reading “Azure Data Factory: Validation Activity”