Introduction to Azure Monitor Log Analytics

Azure Monitor is the name of a suite of solutions built within the Azure platform to collect logs and metrics, with that information then being used to create insights, visualizations, and automated responses. Log Analytics is one of the main services created to analyze the logs gathered. The platform supports near real-time scenarios, is automatically scaled, and is available to multiple services across Azure (including Azure Sentinel). Using a version of the Kusto Query Language (KQL), the query language used to obtain information from logs, complex information can be queried quickly and the queries are saved for future use. In this book, we will refer to this service simply as Log Analytics.

In order to create a Log Analytics workspace, you must first have an Azure subscription. Each subscription is based on a specific geographic location that ties the data storage to that region. The region selection is decided based on where you want your data to be stored; consider the distance between the source data and the Azure data center (region), alongside the legal requirements for data sovereignty requirements for your organization. The selection of the region will also impact the costs associated with both Log Analytics and Azure Sentinel.

Each workspace has its own separate data repository, and each can be configured uniquely to meet the business and technical requirements for security, governance, and cost management. Azure Sentinel can only be enabled to use a single Log Analytics workspace; we therefore recommend that you centralize all your security logs to a dedicated central workspace. If your organization requires you to create more than one location for your data for the legal or technical requirements mentioned before, then you will need to run multiple instances of Azure Sentinel (one per Log Analytics workspace), and each instance would need to be monitored, in which case you should consider the deployment of Azure Lighthouse.

Azure Lighthouse

Microsoft has addressed the need to manage multiple Azure subscriptions and resources in a centralized console. This is usually deployed by managed service providers who have multiple customers, although it may also be used by organizations with complex requirements and who have deployed multiple Azure subscriptions. Azure Sentinel is now a supported resource for this portal, and more features are expected to be added in time to ensure strong parity for interacting directly with Azure Sentinel.

The following diagram shows how Log Analytics workspaces relate to the rest of Azure. Each workspace resides in a single resource group, although there can be multiple workspaces in a single resource group and most likely other Azure resources. Each resource group belongs to a single subscription, and each subscription belongs to a single Azure Tenant. There can be, and usually are, multiple resource groups in a subscription, and many companies will have multiple subscriptions in a tenant:

Figure 2.1 – Azure Tenant for Log Analytics

Once created, a workspace can be used to gather information from many different sources, including the following:

• Azure resources in the same subscription
• Azure resources in different subscriptions
• Data from other cloud services (such as Amazon Web Services and Google Cloud Platform)
• Data from your private data center (on-premises or third-party hosted)
• Data from on-premises resources (via secure internet connections)
• Data from IoT and industrial control systems

To protect the data collected, security and compliance standards are built into this solution; the Log Analytics service manages your data in a secure cloud data repository and ensures that the data is secured with multiple layers of protection, including the following:

• Data segregation and isolation, with geographic sovereignty
• Data retention and deletion policies, per data source type
• Internationally certified standards for physical security, inherited from the Azure subscription (commercial and government)
• Microsoft-managed incident management processes
• Certified conformity to multiple compliance and regulatory standards
• Secure channels for sending data to Log Analytics; certificate-based authentication, and SSL via port 443
• Workspace and role-based access permissions, managed by the customer; more details later in this chapter

For more information on the way data is secured, we recommend reading the official Microsoft documentation: https://docs.microsoft.com/en-us/azure/ azure-monitor/platform/data-security.

Planning a workspace

While it is easy to just go and create a workspace, it is better to plan out the workspace configuration beforehand so as to avoid having to perform reworking later. Some of the aspects to take into account include the following:

• The name of the workspace: This must be unique across all of Azure. It should be descriptive enough to show what service this workspace provides just by looking at the name. It is recommended that your company and the word Sentinel are used in the name.

  If this raises concerns that the name will make it a bigger target for bad actors, use whatever naming convention makes sense and meets corporate standards. Just remember that it must be unique across all of Azure. That name must also be between 4 and 64 letters or digits, and should not be the first or last character.

• The subscription the workspace belongs to: It is recommended that a separate subscription is created just for Azure Sentinel use in order to limit access to it. If this is not feasible, choose an appropriate subscription to use.
• The location of the workspace: The workspace should reside in the same location as the resources feeding it in order to prevent egress charges that would occur if you send the data from one location to another. Large companies will most likely have resources in many different locations, in which case it would make sense to use the location that has the most resources and has the lowest price for the workspace. Keep in mind that there may be laws in place that denote where the data must reside.

  For more information on Log Analytics pricing, see https://azure. microsoft.com/en-us/pricing/details/monitor.

• Which resource group will the workspace reside in? It is recommended that all the Azure Sentinel resources reside in the same resource group, although that is not a hard and fast rule. If it makes more sense to have all your workspaces in one resource group, Azure Sentinel in another, and the workbooks in a third resource group, then do that. This will not affect the performance of Azure Sentinel at all.
• Which pricing tier to use: If the subscription being used to house the workspace has had a workspace created in it before April 2, 2018, or if the subscription is part of an Enterprise Agreement that was in place prior to February 1, 2019, it will continue to be able to access the legacy pricing tiers. Otherwise, only Per GB (2018) is allowed and data is charged for ingestion and retention per GB. The legacy options are as follows:

  (a) Free: There is no charge for the data being ingested, although there is a 500 MB daily limit and the data is retained for only 7 days. This can only be used for lab and research purposes.

  (b) Standalone (Per GB): This is the same as Per GB (2018).

  (c) Per Node (OMS): Use this one if OMS E1 Suite, OMS E2 Suite, or OMS Add-On for System Center has been purchased to use the authorization that came from those purchases.

Planning your workspace before you create it is very important. Making sure to select a unique and meaningful name, the proper location to avoid egress charges, the correct resource group, and other decisions prior to deployment will save you frustration or complete reworking later.

Creating a workspace using the portal

This section will describe how to create the Log Analytics workspace using the Azure portal website. This is a graphical representation of the PowerShell and CLI commands discussed later and, as such, may be the easiest way to start working with workspaces:

1. In the Azure portal, enter Log Analytics workspaces in the search bar at the top of the screen. This will show a list of services that are related to the search term entered. Locate and click on the Log Analytics workspaces link.
2. Click the Add button to create a new workspace:
   

   Figure 2.2 – Creating a new Log Analytics workspace

3. Enter the required values in the new blade:

   (a) Enter the name for the workspace.

   (b) Select a Subscription where this will reside.

   (c) Choose the Resource group for this workspace.

   (d) Select the Location where this workspace will reside.

   (e) For Pricing tier, Per GB (2018) will automatically be selected.

   The blade will look something like this:

   

   Figure 2.3 – Log Analytics workspace options

4. When all the values have been filled in, click the OK button at the bottom of the screen to continue. Even though the button will change to say Validating, it is also creating the workspace in the background.
5. Once the workspace has been created, you will be taken back to the listing of all the workspaces. You may need to refresh this listing to see your new workspace.

That is all there is to creating a workspace using the Azure portal. While this is very easy to do, there may be times when you will want to perform these same actions using command-line actions, and that will be described next.

Creating a workspace using PowerShell or the CLI

There are times when you need to be able to consistently recreate an Azure Sentinel environment. Perhaps you are just testing all the various configuration options, creating environments for many different subscriptions for an international company, or creating instances for customers. No matter the reason, if you need to create many Azure Sentinel environments that are all the same, using PowerShell or the Command-Line Interface (CLI) is a better option than doing it in the Azure portal.

Creating an Azure Resource Management template

When creating a new Log Analytics workspace using PowerShell in this lab, you will use an Azure Resource Management (ARM) template to perform the actual configuration. While you can create the workspace directly using either technology, using an ARM template provides additional benefits, including being able to easily recreate the workspace, using the ARM template in a DevOps workflow, or using it in Azure Blueprints.

Note

A complete discussion of ARM templates is beyond the scope of this lab, but briefly, an ARM template is a JSON file that describes what needs to be created in Azure. It contains parameters, which are the values that a user will provide to determine items such as name, location, and pricing tier. It can also have variables, which are internal values that can be used to determine other values. It will also have a list of one or more resources, which are the Azure resources to create.

Go to https://docs.microsoft.com/en-us/azure/azure-monitor/platform/template-workspace-configuration and copy the JSON text. You will be pasting this into a file that you create later.

In this example, you will be prompted for the workspace name and the location, but the pricing tier will default to pergb2018 due to the presence of a defaultValue entry. If you do not wish to have those defaults, you can either change the values shown or remove the entire defaultValue line, including the comma at the end, in which case you will be prompted for the values when executing the command.

While JSON is just text, so you can use a program such as Notepad to view it, it is recommended that you use something such as Visual Studio or Visual Studio Code, which provide options including color coding and showing available commands. We will be using a version of Visual Studio Code in the Azure portal for this lab.

In your Azure portal, click on the Cloud Shell icon in the top right-hand corner of the screen:

Figure 2.4 – Launching the Cloud Shell

If prompted, select the subscription you are currently using for Azure Sentinel and then click Create Storage. This will only have to be done once, so if you have used the Cloud Shell previously, you will not be prompted for this.

At the bottom of the screen will be the Cloud Shell, which should look as in the following screenshot. The text may not match exactly what is shown:

Figure 2.5 – Cloud Shell setup

If, in the top left-hand corner, it says Bash rather than PowerShell, use the dropdown to change it to PowerShell, as that will have the command we need for this lab.

Once the Cloud Shell has finished loading, enter the following:

  code deployworkspacetemplate.json

This will start a version of Visual Studio Code that you can use in the Cloud Shell. Type the code from the preceding example. On the right side of the screen, click on the context menu and click Save. You can then either click the X to close the editor or click on the context menu and click on Close Editor:

Figure 2.6 – PowerShell deployment using a JSON template

Note

If you want to run this on your own computer rather than via the Azure portal, go to https://docs.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-3.2.0 to learn how to install the Azure PowerShell module, or https://docs.microsoft.com/ en-us/cli/azure/install-azure-cli?view=azure-cli-latest to install the Azure CLI module.

That is how you can use the CLI to create a new Log Analytics ARM template. This is the external file that we will be using in the following sections to create the new workspace.

Using PowerShell

PowerShell is a scripting language that can be used across various machines, including Windows and Linux computers, that was built on top of .NET. Because of this, it can accept .NET objects, making it incredibly powerful. PowerShell has many different commands, including some specifically created for working with Azure, which will be used here.

Note

You may not have the Azure PowerShell module loaded on your local computer. To install it, follow the instructions located at https://docs.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-3.5.0.

Follow these steps to create a new workspace using PowerShell:

1. Determine in which resource group the workspace will reside. If you have one already created, skip to step 3.
2. To create a new resource group, run this command:

   New-AzResourceGroup -Name <resource-group-name> -Location

   <location>

   Replace <resource-group-name> with the name of the new resource group and <location> with the location where the resource group will reside, such as EastUS or WestUS. If you do not know what to use for your location, run this command:

     Get-AzLocation

   Find the location you want and use the value listed under Location.

3. Enter the following command:

   New-AzResourceGroupDeployment -Name <deployment-name>

   -ResourceGroupName <resource-group-name> -TemplateFile

   $HOME/deployworkspacetemplate.json

   Replace <deployment-name> with the name of this deployment. You can use something like <labworkspace>. It is not important what you enter, as this is just a placeholder name so that if you look at the resource group, you can see the various deployments. Replace <resource-group-name> with the name of the resource group where the workspace will reside.

4. You will be prompted for the Log Analytics workspace name as well. Enter a valid name and press Enter to continue.
5. Once the command has finished running, it will show a success screen with a summary of the values used as follows:
   

   Figure 2.7 – Running New-AzResourceGroupDeployment in PowerShell

   If you get an error screen, read the message, as the messages are usually quite specific as to what caused the error.

6. Close the Cloud Shell session.

That is how you can create a new Log Analytics workspace using an ARM template and PowerShell. This can be preferable to using the Azure portal as it is repeatable. Next, we will look at using the Azure CLI and see how to create a new workspace without using the ARM template.

Using the CLI

The Azure CLI is also a cross-platform scripting tool developed by Microsoft. Initially, it was the only tool that was cross-platform, so if you were working on a computer that was not running Windows, it was your only option. PowerShell is now cross-platform as well, so the main difference between the two is that the Azure CLI can create Azure resources directly without using an ARM template.

The following steps describe how to run the CLI from the Azure portal. If you want to run this on your local computer, you will need to make sure you have the CLI installed. Go to https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest for instructions on how to perform the installation:

1. Determine which resource group the workspace will reside in. If you have already created one, skip to step 3.
2. To create a new resource group, run this command:

   az group create --name <resource-group-name> --location

   <location>

3. Replace <resource-group-name> with the name of the new resource group and <location> with the location where the resource group will reside, such as EastUS or WestUS. If you do not know what to use for your location, run this command:

     az account list-location

4. Find the location you want and use the value listed under name.
5. Enter the following command:

   az group deployment create --resource-group <my-resource- group> --name <my-deployment-name> --template-file deploylaworkspacetemplate.json

6. Replace <deployment-name> with the name of this deployment. You can use something like <labworkspace>. It is not important what you enter, as this is just a placeholder name so that if you look at the resource group, you can see the various deployments. Replace <resource-group-name> with the name of the resource group where the workspace will reside.

   You will be prompted for the Log Analytics workspace name as well. Enter a valid name and press Enter to continue.

   Once the command has finished running, it will show either the JSON values for this workspace, as shown in the following screenshot, or an error message. Note that not all the JSON is shown for brevity:

   

   Figure 2.8 – Running az group deployment in the CLI

   Note that as stated earlier, you can use the Azure CLI to create the Log Analytics workspace directly using the az monitor log-analytics workspace create command. Go to https://docs.microsoft.com/en-us/cli/azure/monitor/log-analytics/workspace?view=azure-cli-latest#az-monitor-log-analytics-workspace-create for more information on this command.

7. Close the Cloud Shell session.

Exploring the Overview page

No matter how you created your Log Analytics workspace, the rest of the work in this lab will be done using the Azure portal:

1. Open the portal and go to the Log Analytics solution page.
2. Find your new Log Analytics workspace for Azure Sentinel and click on it. This will take you to the Overview screen, as shown in the following screenshot:
   

   Figure 2.9 – Overview page for Log Analytics

   Note that only part of the screen is shown due to the amount of information on this page.

3. Starting with the Essentials listing at the top of the page, we can review the following items:

   a) Resource group: The resource group where the workspace resides. Selecting [change] will allow you to move to another resource group.

   b) Status: The status of the workspace should show Active.

   c) Location: The Azure location where the workspace resides.

   d) Subscription name: The subscription this resource is associated with. Selecting [change] will allow you to move to another subscription.

   e) Subscription ID: The unique GUID for the preceding subscription, which is useful when calling Microsoft for technical support.

   f) Workspace name: The name of the Log Analytics workspace.

   g) Workspace ID: The GUID for the workspace, which is also useful when calling Microsoft for technical support.

   h) Pricing tier: The pricing tier for the workspace.

   i) Management services: View the activity log for the workspace.

   j) Access control mode: How users are granted permission to access the information in this workspace. Refer to the following section for more information.

The previous sections described the various ways that you can create a new Log Analytics workspace to use with Azure Sentinel. This can be done either through the Azure portal or programmatically using either PowerShell or CLI commands. Once the workspace has been created, we next need to ensure that access is restricted to only those users who need to access it.