If you are familiar with Azure Data Factory, then you must be familiar with the concept of activity orchestration through Azure Data Factory Pipelines. For a brief overview of Azure Data Factory pipelines, please have a look at this previous post.
Synapse Analytics is positioned as the primary hybrid Data Warehousing and Big Data analytics service on the Azure platform. Data integration from various sources is one of the primary requirements for building a data warehouse/big data analytics store. Therefore, having robust integration features is a must-have for any analytics platform.
Pipelines in Azure Synapse Analytics or Synapse Pipelines for short, make this primary functionality available internally within Azure Synapse Analytics. This is handy as this eliminates the need of going to Azure Data Factory to do simple integrations e.g. for loading data into synapse SQL pool.
Synapse pipelines can be accessed through the Orchestrate tab in Synapse Analytics. The pipeline building interface resembles the interface from Azure Data Factory. Also, under the hood, Synapse pipelines use the same structures as Azure Data Factory pipelines.
As one would expect, not all features of Data Factory pipelines are available in Synapse pipelines e.g. SSIS and SSIS integration runtime are not supported in Synapse Pipelines. We will discuss the feature comparison and differences between the two in a future post.
2 thoughts on “Pipelines in Azure Synapse Analytics”