Author Archives: ashish naik

We have discussed Data Factory Pipelines and Activities in a previous post. Some of these activities let you run other activities iteratively or based on condition evaluation. Let’s consider a scenario where we would like to run another Pipeline from within the current pipeline. This essentially means that we have a hierarchy of pipelines. TheContinue reading “Azure Data Factory: Execute Pipeline Activity”

Management hub is a central location to access various high-level settings, actions, and configurations within Data Factory. Management hub can be accessed by clicking on the Manage tab on the Azure Portal UI. You can see various settings neatly grouped into categories in the screenshot above. Let’s have a look at these one by one:Continue reading “Azure Data Factory: Management Hub”

We discussed how to load Filename list into SQL table using the Get Metadata activity in the previous post. In today’s post, let’s have a look at the Set Variable activity (and its use case). The Set Variable activity, as the name implies, is used to assign a value to an existing variable in DataContinue reading “Azure Data Factory: Increment Variable using Set Variable Activity”

In the previous post, we learnt about the Get Metadata activity and the various metadata types that can be extracted using the activity. One of the most common use cases of the Get Metadata activity is to extract a list of files from a storage folder. This is a very common scenario while loading dataContinue reading “Azure Data Factory: Load Filename list into an SQL Table”

We have looked at some Azure Data Factory activities such as, For Each Activity and Azure Function Activity in previous posts. In this post, lets have a look at another important activity, the Get Metadata Activity. The purpose of the Get Metadata Activity, as the name suggests, is to get the metadata of any datastoreContinue reading “Azure Data Factory: Get Metadata Activity”

Power Query is usually associated with Microsoft’s premier BI and Data Visualization tool/service, Power BI. But there is an activity available in Azure Data Factory, called Power Query. (Currently in Preview). Power Query activity has been designed to allow users to create data mash-ups using the powerful M language (which is the same language thatContinue reading “Power Query in Azure Data Factory”

We have discussed the For Each activity in a previous post. In today’s post, let’s have a look at other Iteration and conditional Activities in Azure Data Factory. In the Data Factory UI, the list of Activities has been neatly classified into groups. The group of activities that we are going to discuss in thisContinue reading “Iteration and Conditional Activities in Azure Data Factory”

One of the primary objectives of any ETL tool or service that is designed to interact with databases, is to enable users to perform transformations quickly and efficiently on the data. Azure Data Factory has various built in transformation activities to perform specific tasks. But sometimes, it is more convenient to write custom SQL codeContinue reading “Execute custom SQL code in Azure Data Factory”

We have demonstrated how to parameterize the database name in a Data Factory Linked Service, in a previous post. In that example, we created a parameter called ‘DB Name’ which was at the linked service level, i.e., it could be referenced only within that linked service. However, there is a feature in Data Factory thatContinue reading “How to use Global Parameters in Azure Data Factory?”

We have discussed Azure Data Lake Storage (ADLS) in a previous post. The main application of a Data Lake is to store various types of data at a low cost. But with the evolution of the Lakehouse architecture, it is becoming increasingly simple to analyse massive amounts of data and extract information cost effectively. ToContinue reading “Azure Data Lake Analytics and USQL”