Managing Azure VMs with Project Flash

Microsoft has a long history of turning its internal tools into products, especially as part of Azure. That’s not surprising: The company is building and running the same cloud-native applications as its customers, with the same requirements and the same problems, only doing it a couple of years ahead of them.

Many of its tools are geared toward running and managing applications at scale, especially monitoring and managing virtual machines. No matter what you’re building on Azure, you’re using virtual machines. Some are yours, running infrastructure as a service (IaaS), with others underpinning the platform features that help power your code.

The scale of public cloud services like Azure allows applications to behave in new ways, scaling rapidly across the globe. You might not know how many VMs are running at any one time, only that your applications are stable and your users have access to the services they need. Keeping on top of the telemetry and observability data to manage a modern application can be hard, especially when collecting data across multiple virtual infrastructure instances and geographies.

Introducing Project Flash

Microsoft has been developing a set of tools to help manage Azure-hosted VMs for some time now under its Project Flash banner. These are designed to help bring the data you need into a central dashboard while delivering categorized events to automate management actions.

Project Flash is still under development; however, the current set of releases is mature enough to start using them in your own deployments, with a mix of general availability and public previews. The current set of tools includes the Azure Resource Graph and Resource Health visibility in the Azure portal, as well as previews of VM metrics in Azure Monitor and a set of labeled events for use with Azure Event Grid.

Using Project Flash with the Azure Resource Graph

Support for the Azure Resource Graph is one of the more interesting parts of the project. The framework investigates failures and downtime using the familiar Kusto Query Language. You can use this to track virtual machine availability; a 14-day history lets you see what changes have been made to your virtual infrastructure to help with debugging. The data stored in the Resource Graph can also help you produce reports to find out if your application is meeting its service-level agreements.

Data is stored in the HealthResources table of the Resource Graph. By using the Azure Portal, you can construct the queries to build dashboard elements. Queries can show past and current health status for VMs, which can help track performance over time. Once you’re sure of the query to use, you can move from the Azure Portal to the Resource Graph’s REST API and build your queries into your own devops tools.

Using Project Flash with Event Grids

As applications scale, you need information from all your VMs in one place so you can quickly process health data. With VMs spinning up and down with services like Virtual Machine Scale Sets, traditional monitoring tools may not be able to register new systems, so alternate methods of getting essential data are necessary.

Microsoft has built a publish-and-subscribe messaging architecture into Azure, initially to support event delivery from edge systems and Internet of Things hardware. This can deliver messages from remote monitoring to a central platform, forward messages as new systems come online, and use filters to direct them to appropriate applications.

Using Azure Resource Notifications, Project Flash can direct health information to your choice of endpoints, using Event Grid to route and queue notifications. Microsoft recommends using Event Hubs as an endpoint when you’re bringing notifications from many VMs into one service; this will allow you to write code that formats and displays messages and handles alerts for outages.

It’s easy enough to add Project Flash messages to Event Grid, using the microsoft resourcenotifications healthresources topic. This uses Cloud Events-formatted messages to deliver events that show changes in availability status, which can then be directed to your choice of endpoint. For example, you can use Azure Monitor to send alerts if a VM becomes unavailable or if it is queued for deactivation when no longer needed. Azure Monitor will manage parsing the Cloud Events schema, something you’ll need to do if you build your own endpoints.

New tools to bridge the physical and virtual

The Project Flash tools provide a basic framework for delivering the information you need to check your virtual machines’ status. It’s a service that’s very much under development, adding new features regularly.

Microsoft recently announced a new set of preview features, with new VM Metrics for Azure Monitor. One important new metric is a set of improvements for the VM availability feature, adding context to the information it delivers. Knowing if an issue was caused by Azure or by your application will help you prioritize debugging and support calls. This new feature adds labels to metrics, such as platform, customer, and unknown. These can filter results in your dashboard views, giving a quick view of where to focus to get a root-cause analysis of a problem, especially if it caused an outage.

Another new option allows you to combine Project Flash’s Event Grid support with Azure Monitor. Events can now be sent through Event Grid to Azure Monitor, bringing distributed systems events into the same metric framework used to watch core systems. You no longer need to write event-handling code for Project Flash events; instead you can rely on Azure Monitor’s existing alert features, including its support for SMS and push alerts. This approach feeds in events from multiple sources and adds a real-time alert option.

Microsoft also wants to have Project Flash give you information about the underlying Azure infrastructure and platform. For example, one planned new feature will give details of issues with rack-level networking hardware, as well as predictive failure alerts so you can move operations to another region in advance of planned or unplanned data center maintenance. The intent is for a mix of Project Flash events and scheduled event notifications to give you enough warning to alert users and migrate workloads, as well as show when services are recovered and ready for use. Scheduled events are intended to give up to 15 minutes’ warning, which should be enough time to stand up a backup instance of an application and begin rerouting traffic.

Why use Project Flash?

Project Flash and other tools like it help us understand the effects of unprecedented levels of automation on our applications. The abstractions that come with IaaS hide the physical systems that support those virtual machines, so we can’t tell when they are being upgraded or when they fail—or the difference between those states. By combining these tools and Azure’s other monitoring and event-handling services, we can start to build our own dashboards and automations.

There’s an interesting possibility here: combining Project Flash’s Cloud Events outputs with the Drasi change detection tool. This would allow us to build a new control plane that combines both virtual infrastructures and cloud-native applications, increasing the maturity of platform operations with new notification and automation features. This would help us run and manage increasingly complex platforms and the applications that depend on them.