In the previous post, we discussed about creating an Elapsed Time metric to monitor long-running Pipelines in real time using Azure Monitor. In this post, lets discuss about Azure Monitor and how it can be used to monitor Data Factory Pipelines.
The easiest way to monitor Pipelines visually, is to go the Azure Monitor UI within the Data Factory Studio. To be able to do this, click on the Azure Monitor icon as shown in the screenshot below:
Let’s go through the tabs under Monitor:
- Dashboard: This tab gives you a graphical high-level status of the Pipelines, Activities and Triggers in the Data Factory:
- Pipeline Runs: This tab shows you the status of the individual Pipelines. We can further drill down to check the status of the underlying child Pipelines and Activities.
Please note the Triggered and Debug filters at the top. This means that this screen can also be used to monitor the Debug (Manual test) runs of the Pipelines.
- Trigger Runs: As the name suggests, this tab shows the status of the Triggers that have been created in the Data Factory:
- Integration Runtimes: This tab can be used to monitor the Integration Runtimes that are linked to the Data Factory.
- Data flow debug: If you have created any Dataflows as a part of your Pipelines, they can be monitored here.
- Alters & Metrics: Last but by no means the least, Alerts and Metrics tab lets you create Alert rules based on various Data Factory metrics captured by Azure Monitor.