Handle Interruptions – Design and Implement a Data Stream Processing Solution

As previously mentioned, the platform will handle data stream interruptions when caused by a node failure, an OS upgrade, or product upgrades. However, how can you handle an exception that starts happening unexpectedly? Look back at Figure 7.51 and notice an option named + New Alert Rule, just above the JSON tab. When you select that link, a page will render that walks you through the configuration of actions to be taken when an operation completes with a Failed status. This is covered in more detail in Chapter 10.

Ingest and Transform Data

Chapter 5 covered data ingestion and transformation in detail. In this chapter you learned how a data stream is ingested and how it can then be transformed. Technologies like Azure Event Hubs, Azure IoT Hub, and Azure Data Lake Storage containers are all very common products used for data stream ingestion. Azure Stream Analytics and Azure Databricks receive the data stream and then process the data. The processed data stream is then passed along to a downstream consumer.

Transform Data Using Azure Stream Analytics

Exercise 7.5 is a very good example of transforming data that is ingested from a stream. You created a tumbling window size of 5 seconds, and all the brain wave readings that were received in that time window were transformed. The query that transformed those brain wave readings calculated the median for each frequency. Once calculated, the median values were compared against estimated frequency values for the meditation scenario, and the result was then passed to Power BI for real‐time visualization.

Monitor Data Storage and Data Processing

The monitoring capabilities concerning stream processing are covered in Chapter 9. This section is added to help you navigate through the book while referring to the official online DP‐203 exam objectives/Exam Study Guide, currently accessible from https://learn.microsoft.com/en-us/certifications/exams/dp-203.

Monitor Stream Processing

You will find some initial information about monitoring stream processing in the “Monitor for Performance and Functional Regressions” section in this chapter. You will find a more complete discussion of this topic in Chapter 9, in the section with the same name, “Monitor for Performance and Functional Regressions.”