Author Archives: ashish naik

Web Activity is the easiest way to call any REST API endpoints within a Data Factory Pipeline. In today’s post, we will discuss the basic settings of Web activity. To create a new web activity, search for ‘web’ in the activities pane. Alternatively, it can be located under the General group in the activities pane.Continue reading “Azure Data Factory: How to call REST API?”

It is very easy to visually monitor previous pipeline runs in Data Factory using the Monitor page in the Azure Data Factory, which we have already covered in a previous post. There have been some recent improvements to the monitoring view, we will go through these briefly in this post. Data from the Azure MonitorContinue reading “Azure Data Factory: New Monitoring View Features”

While we have discussed various ways for running custom SQL code in Azure Data Factory in a previous post, recently, a new activity has been added to Azure Data Factory called Script Activity, which provides a more flexible way of running custom SQL scripts. As shown in the screenshot above, this activity supports execution ofContinue reading “Azure Data Factory: Script Activity”

Self-hosted integration runtime in the context of Azure data factory is a gateway that connects the on-prem data sources to datastores in the cloud. To know more about Integration runtimes, please refer to the previous post. We have discussed how to check whether Integration Runtime is online or offline using PowerShell command in a previousContinue reading “Azure Data Factory: Monitor Self Hosted Integration Runtime Metrics”

ARM or Azure Resource Manager templates make it easy to manage deployments for Data Factory. When we connect Data Factory to a source control repository (e.g. GitHub or Azure DevOps Git), the data factory along with all its artefacts (pipelines, datasets, linked services etc.) is saved in the repository in the form of ARM templates.Continue reading “Azure Data Factory: How to edit default parameter definition for ARM templates?”

When it comes to transforming structured data, (e.g., applying business logic, standardization etc.) stored in a database, SQL is the most convenient and fit-to-purpose option.  Stored procedures provide a way to store the transformation logic as a set of SQL statements that can be re-executed as pre-compiled code. The Stored Procedure Activity in Data FactoryContinue reading “Azure Data Factory: Stored Procedure Activity”

We have discussed Linked Service parameterization through the UI, in a previous post. But not all Linked Service Types support parametrization using the UI.  In this post, we will discuss the Linked Services that can’t be parameterized using the UI. (i.e., they don’t have any option to add parameter). If you are familiar with AzureContinue reading “Azure Data Factory Linked Service: Advanced Authoring”

During some scenarios in Azure Data Factory, we may want to intentionally stop the execution of the pipeline. An example could be when we want to check the existence of a file or folder using Get Metadata activity. We may want to fail the pipeline if the file/folder does not exist. To achieve this, weContinue reading “Azure Data Factory: Fail Activity”

In one of the previous posts, we discussed how we can use Validation activity to design the Pipeline to wait for a scheduled time and retry. There is another way to introduce a delay in the Pipeline. Wait activity can be used to pause the execution of the Pipeline for a fixed amount of time.Continue reading “Azure Data Factory: Wait Activity”

In the previous post, we discussed the Switch Activity, which is useful for branching the control flow based on some condition. We will discuss about the Filter Activity in this post. The purpose of Filter Activity is to process array items based on some condition. Consider a scenario where we would like to set theContinue reading “Azure Data Factory: Filter Activity”