Azure Log Analytics Workspace

Table-Level Retention

Within the Microsoft Azure cloud platform, a Log Analytics workspace is a repository for collecting data of different types.  It can be used by Azure Sentinel for collecting security log data from a wide variety of sources for detecting and proactively protecting against threats to the environment.  It can also collect performance monitoring and diagnostic data for troubleshooting and trend analysis.

The Log Analytics workspace can be configured to retain data for between 30 and 730 days.  However, this value (set in the Usage and estimated costs/Data Retention blade of the Azure portal) applies to all tables within the workspace and data is charged by the amount stored beyond 31 days (or 90 if enabled for Sentinel). 

It might not be appropriate to store short-term diagnostic data for as long as security logs that may be required longer-term compliance adherence.  The Microsoft documentation (https://docs.microsoft.com/en-us/azure/azure-monitor/platform/manage-cost-storage#retention-by-data-type) shows how the REST API can be used to adjust retention values for each table, although it is not straightforward to implement.  The following is a task list that will hopefully summarise the process (which can be performed from a temporary virtual machine created in Azure for instance to avoid installing software locally).

  1. From the Azure portal, navigate to Subscriptions and record the ID of the subscription containing the Log Analytics workspace
  2. Navigate to Log Analytics, select the workspace, choose Logs and identify tables needing a non-default retention period
  3. Open Windows PowerShell ISE
  4. Run the following commands to install the Chocolatey package manager
    • Set-ExecutionPolicy Bypass -Scope Process -Force;
    • iex ((New-Object System.Net.WebClient).DownloadString(‘https://chocolatey.org/install.ps1’))
  5. Run the following command to install the ARM client
    • choco install armclient
  6. Run the following commands to connect to the Azure subscription
    • armclient login
    • Set-AzContext -Subscription ‘subscriptionname
  7. Run the following command to display the properties of the Log Analytics table with updated subscription id, resource group name, workspace name and table name values
    • armclient get “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rgname/providers/microsoft.operationalinsights/workspaces/lawname/Tables/tablename?api-version=2017-04-26-preview”
  8. Run the following command to update the retention period for the specified table
    • armclient PUT /subscriptions/ xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rgname/providers/microsoft.operationalinsights/workspaces/lawname/Tables/tablename?api-version=2017-04-26-preview “{properties: {retentionInDays: 30}}”

This feature will probably make its way into the portal at some point, but it’s worth some time to configure individual tables to avoid charges for data that is no longer needed.

Contact risual on our website or Twitter for more information.

About the author