Monthly Archives: June 2020

Azure Cosmos DB is a premier NoSQL database service. Cosmos DB is available across all Azure regions worldwide. Microsoft provides the diagram below to better understand how Global data distribution is managed in Cosmos DB: Azure Regions can have multiple data centres, where stamps (massive racks) of machines are located. These stamps can be dividedContinue reading “Azure Cosmos DB Global Data Distribution”

Azure Cognitive Search (previously, Azure Search)  is a cloud-based search service, that allows users to provide custom search functionality on  web, mobile and enterprise applications. The cognitive search solution sits in between content ingestion and search queries i.e. the Index and query engines. The data ingestion process creates a search index which is used forContinue reading “Azure Cognitive Search”

We discussed about table entity properties in the previous post. Before we discuss partitions, it will be good to learn some more details about the entity properties. We also discussed about the PartitionKey and RowKey properties in the previous post. These two properties create a clustered index on the table. The data is sorted inContinue reading “Azure Table Storage Partitions”

Azure Table Storage is the storage service optimized for storing structured data. Table Entities are the basic units of data that can be stored in Azure Table Storage, Table. Table entities are equivalent to rows in a relational database table. In other words, Entities are collections of properties and each property is represented as aContinue reading “Azure Table Storage: Table Entities”

Azure Data Share enables organizations to share datasets securely with their customers and external partners. These days data security is extremely high on the priority list for most organizations. Moreover, there are laws in many geographies that make it mandatory for organizations to monitor and audit sharing of sensitive data externally. Azure Data Share providesContinue reading “Azure Data Share”

We have already discussed Azure Data Factory Integration Runtimes in a previous post. In this post, we are going to discuss ways to monitor the performance of the Integration runtimes. The PowerShell command below can be used to get performance stats for Integration runtimes: The Get-AzDataFactoryV2IntegrationRuntime cmdlet returns the current status of the Integration runtimeContinue reading “Monitor Azure Data Factory Integration Runtime using PowerShell”

Partitioning Schemes are a way to optimize the performance of Data Flow. This setting can be accessed on the Optimize tab in the Configuration panel for the Data Flow Activity. Microsoft recommends leaving the setting to default “Use current partitioning” in most cases. This sends an instruction to the data factory to use native partitioning schemes.Continue reading “Azure Data Factory Data Flow Partitioning Schemes”

To run Azure Functions from Azure Data Factory, we can create an Azure Function Activity. We must create a linked service to the Azure Function for this. Linked Services are similar to connection strings. A linked service contains all the information required to connect and authenticate the data factory with the service or data storeContinue reading “Azure Data Factory Azure Function Activity”

What is Network Transfer in the context of Azure Data Transfer Solutions? a. Online data transfer b. Offline data transfer c. Recurring data transfer d. One-time data transfer You are working as Data Engineer at ADE Inc. You have been asked to provide a timeline for some old data which is stored in Azure BlobContinue reading “ADE Q04: Azure Storage and Azure Data Factory Quiz”

If you are familiar with SQL Service Integration Services (SSIS), then you must understand the difference between Control Flow  and Data Flow. The concepts apply to Azure Data Factory as well. Control Flow Activity is an activity that affects the path of execution of the Data Factory pipeline. E.g. for each activity, which creates aContinue reading “Azure Data Factory Control Flow vs Data Flow”