With Microsoft adding more services into their offering I am starting to see logging appearing more often and with that the idea of storing and processing those logs. Services like Microsoft Sentinel for example store log information which then needs to be analysed and potentially actioned against, Application Insights collects performance information which can be analysed over time to pick up trends relating to the applications you are running. This month I thought I would discuss Azure Monitor and its components which are used for logging.
Azure Monitor is really a bunch of functionality grouped together under the one branding, at its heart though it is as the name suggests, designed around monitoring which is typically split into two kinds of rough categories, Metrics (Azure Monitor Metrics) and Logs (Azure Monitor Logs). Think of metrics as being more around performance related data points, they are typically relating to numerical data, CPU performance usage for example would be considered a metric. Think of logs as being for text-based data, say you are recording the console of a virtual machine or the output of an Azure Function, that would be more likely to be stored within logs.
With all this data, there needs to be a place to store it, typically with Metrics data will be retained by Azure for between 30 and 90 days (depending on how the data is collected), if you need to retain this information for longer than it can be translated and stored within Logs. Log data is stored within something called a Workspace, which is effectively a high-performance database with tables which represent the log information you are collecting. Storage within a Workspace is free for 31 days, however there is a processing fee for data to get into the Workspace, but you do receive 5GB of free ingress processing per month. If you require your logs to be stored for more than 31 days, then there is a storage fee applicable.
Now you have your data stored, you need a way of querying it which is where the Kusto Query Language (KQL) comes into play. KQL is Microsoft’s data query language which you can see used within the likes of a Workspace, and other services like Azure Data Explorer.
We have the log information stored in Workspaces, and we have the KQL query language to interact with this data, you can now start to pull out useful information from this data and act based on it. As an example, potentially you may want to keep an eye on your virtual machines within an environment and when some start to hit a consistently high CPU usage over say a 15-minute period you can tie an action into it and have a Logic App kick in and notify you or take a course of action based on some predefined set of conditions.
For me Azure Monitor is a good tool to have in your tool chest for monitoring of your Azure environment and to tie multiple metric and log data points together to provide better insights into your workloads. KQL is also a good language to wrap your head around if you are looking to explore more of the data and AI functionality within Azure.
I have included some links below around Azure Monitor and the areas discussed above.
Azure Monitor Metrics https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/data-platform-metrics
Azure Monitor Logs https://docs.microsoft.com/en-us/azure/azure-monitor/logs/data-platform-logs
Workspaces https://docs.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-workspace-overview
Kusto Query Language https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/
