How to install Az PowerShell module correctly?

Az PowerShell is a powerful and convenient way to manage an Azure subscription with PowerShell scripts. It is an add-on module to the PowerShell scripting tool pre-installed with Windows. It is very useful for automating tasks and monitoring resources on the Azure cloud. Let’s have a look at the steps to install the Az PowerShellContinue reading “How to install Az PowerShell module correctly?”

When to (and when not to) use Replicated tables in Synapse SQL Pool?

As we know, Azure Synapse Analytics has an MPP architecture, which means there are multiple nodes which can process queries in parallel. Another important concept is table distribution. There are various table distribution options suited to different use cases. We have already discussed Synapse table distributions in a previous post. Replicated tables are replicated acrossContinue reading “When to (and when not to) use Replicated tables in Synapse SQL Pool?”

How to optimize DWUs in Azure Synapse Analytics ?

Optimizing performance is one of the most important design decisions while implementing a Data Warehouse using dedicated SQL Pool in Azure Synapse Analytics. We have discussed how to update Data Warehousing Units in Azure Synapse Analytics in previous post. Data Warehousing Units or DWUs (cDWUs) determine the query performance level of the Synapse Analytics dedicatedContinue reading “How to optimize DWUs in Azure Synapse Analytics ?”

Azure Data Factory : For Each Activity

For Each activity is a Control Flow activity available in Azure Data Factory that lets user iterate through a collection and execute specific activities in a loop. To understand what is control flow, please read my previous post on Azure Data Factory control flows and data flows. If you have worked in the data analyticsContinue reading “Azure Data Factory : For Each Activity”

Parameterize Linked Service in Azure Data Factory

A relatively new feature in Azure Data Factory is support for parameterization of Linked Services. You can find more details about Linked Services in Azure Data Factory in this previous post. Parameterization allows the same linked service to be used for multiple data sources of the same type. e.g. we can now parameterize the connectionContinue reading “Parameterize Linked Service in Azure Data Factory”

Azure Data Factory Templates

Azure Data Factory is the primary task orchestration and data transformation service on the Azure cloud. This means that Azure Data Factory can be used for connecting to various Azure Cloud as well as external services to perform automated tasks. To make it easier for users to create pipelines corresponding to various scenarios, Azure DataContinue reading “Azure Data Factory Templates”

Azure Data Factory: Linked Services and Datasets

We have discussed about source and sink components of Azure Data Factory in the previous post. There are other components that need to be created before we can start creating Data Factory Pipelines. Some of these are Linked Services and Datasets. Linked Service: A linked service contains the connection details (connection string), e.g. Database server,Continue reading “Azure Data Factory: Linked Services and Datasets”

Azure Data Factory: Source and Sink

Azure Data Factory is the primary task orchestration/data transformation and load (ETL) tool on the Azure cloud. The easiest way to move and transform data using Azure Data Factory is to use the Copy Activity within a Pipeline.  To read more about Azure Data Factory Pipelines and Activities, please have a look at this post.Continue reading “Azure Data Factory: Source and Sink”

Azure Data Engineering Interview Questions

The idea behind starting this blog was to help people who are interested in data engineering as a career. The reason this blog is named Azure Data Engineering is because my experience is mostly with Microsoft Technologies. For the 100th post, I have listed the top 50 questions that are most likely to be askedContinue reading “Azure Data Engineering Interview Questions”

Estimating costs for Azure Synapse Analytics

Before starting a project, it is very important to understand and plan for the estimated costs associated with creating and running an Azure Synapse Analytics instance. Azure Pricing Calculator provides an easy-to-use interface where the users can input the numbers and get an overall estimate of the associated costs. To use the Azure Pricing CalculatorContinue reading “Estimating costs for Azure Synapse Analytics”