Azure Automation - Advanced Auditing

Nathan McNulty

Jan 29, 2023 • 7 min read

In Office 365, applying an E5 license with the Advanced Auditing component to a user does not enable all auditable events. For tenants created more than a few years ago, it's possible MailItemsAccessed and Send are not enabled by default, especially if audit events were previously modified. Another important one, SearchQueryInitiated (covers both Exchange and SharePoint) is never enabled by default. We usually want to ensure all auditable events are being collected for appropriately licensed users, and ideally, have it automated.

💡

Side note: If anyone happens to know of a public API to interact with Advanced Auditing policies, please let me know!

I have been using this script for years to add auditing events, but it has a few problems. First, we have to coordinate running it after license assignment/mailbox creation. Second, I often see this done with a user with legacy authentication or a Service Principal with Exchange Admin permissions. Finally, the script doesn't validate licensing, so it doesn't work well for mixed license environments.

This new solution addresses these concerns by using Azure Automation with a Managed Identity, limited permissions, and takes licensing into consideration! :)

Create an Azure Automation Account

First, we need an Automation Account. If you've followed my previous Azure Automation guide, I would highly recommend creating a separate Automation Account for both permissions reasons and the fact that I need to update that one to use Managed Identities once Graph V2 modules adds MI support (soon!).

Go to https://portal.azure.com and search for Automation Accounts

Next, click Create.

I created a Resource Group dedicated to automation (Automation accounts, Logic Apps, etc.). Resource Groups are intended for IAM and lifecycle management, so design accordingly.

Give it a Name, Region, and review other tabs or simply hit Review + Create.

Once our resource is created, click Go to resource, scroll down, and click Identity. The Object (principal) ID will be needed in the next section.

Install PowerShell modules

To set things up, we need to install modules to interact with Azure AD and Exchange Online. With the deprecation of MSOnline and AzureAD modules, I have been forcing myself to only use the Graph modules (more accessible than API calls for most admins), but it has a long ways to go to being as user friendly...

Open PowerShell (requires Run as Admin to install for all users) and run:

# Install for current user only
Install-Module Microsoft.Graph -Scope CurrentUser
Install-Module ExchangeOnlineManagement -Scope CurrentUser

# Install for all users, helpful with OneDrive Known Folder Move
Install-Module Microsoft.Graph -Scope AllUsers
Install-Module ExchangeOnlineManagement -Scope AllUsers

Next, we need to connect to Azure AD, and we'll need to specify scope for the two tasks we're going to perform: 1) Create a dynamic group based on Advanced Audit licensing, 2) Grant API permissions for the Service Principal associated with our Automation Account.

# Connect to MS Graph
Connect-MgGraph -Scopes AppRoleAssignment.ReadWrite.All,Application.Read.All,Group.ReadWrite.All

Now, let's create a dynamic security group based on users who have an Advanced Auditing license assigned to them:

# Create dynamic security group based on Advanced Auditing license
$groupValues = @{
    DisplayName = 'Azure Automation - Advanced Auditing'
    MailEnabled = $False
    MailNickName = 'AzureAutomationAdvancedAuditing'
    SecurityEnabled = $True
    GroupTypes = 'DynamicMembership'
    MembershipRule = 'user.assignedPlans -any (assignedPlan.servicePlanId -eq "2f442157-a11c-46b9-ae5b-6e39ff4e5849" -and assignedPlan.capabilityStatus -eq "Enabled")'
    MembershipRuleProcessingState = 'On'
}

New-MgGroup @groupValues

Finally, let's gather the necessary information about our Managed Identity, assign it ExchangeManageAsApp, and make it so we can use it with the Graph API.

$SP_ID = 'Object ID copied from Azure Automation'
$AppId = (Get-MgServicePrincipal -ServicePrincipalId $SP_ID).AppId

# To see more about the Managed Identity, you can run:
# Get-MgServicePrincipal -ServicePrincipalId $SP_ID | Format-List *

# Grant API permissions to the Managed Identity to act against Exchange
$ResourceID = (Get-MgServicePrincipal -Filter "AppId eq '00000002-0000-0ff1-ce00-000000000000'").Id
New-MgServicePrincipalAppRoleAssignment -ServicePrincipalId $SP_ID -PrincipalId $SP_ID -AppRoleId "dc50a0fb-09a3-484d-be87-e023b12c6440" -ResourceId $ResourceID

# Assign Graph application permissions to managed identity (outside of Azure Automation)
# Modified from @AlexFilipin: https://gist.github.com/AlexFilipin/daace2f2d7989545e8ab0b969de2aaed
"Group.Read.All","User.Read.All" | ForEach-Object {
	$PermissionName = $_
	$GraphSP = Get-MgServicePrincipal -Filter "startswith(DisplayName,'Microsoft Graph')" | Select-Object -first 1 #Graph App ID: 00000003-0000-0000-c000-000000000000
	$AppRole = $GraphSP.AppRoles | Where-Object {$_.Value -eq $PermissionName -and $_.AllowedMemberTypes -contains "Application"}
	New-MgServicePrincipalAppRoleAssignment -AppRoleId $AppRole.Id -ServicePrincipalId $SP_ID -ResourceId $GraphSP.Id -PrincipalId $SP_ID
}

Now we need to create an Exchange Service Principal linked to the Managed Identity and grant it Exchange permissions. This process involves creating a Management Role based on the Audit Logs role, then remove all unnecessary permissions and limit Set-Mailbox parameters, add the Management Role to a Role Group, and assign the Managed Identity to the Role Group.

# Connect to Exchange Online 
# Requires "Install-Module ExchangeOnlineManagement"
Connect-ExchangeOnline

# Create linked Service Principal ($AppId and $SP_ID from earlier)
New-ServicePrincipal -AppId $AppId -ServiceId $SP_ID -DisplayName "exo-automation"

# Create new Management role
New-ManagementRole -Name "Mailbox Auditing" -Parent "Audit Logs" -Verbose

# Remove unnecessary permissions
Get-ManagementRoleEntry "Mailbox Auditing\*" | Where-Object { $_.Name -notin "Get-Mailbox" } | ForEach-Object { Remove-ManagementRoleEntry -Identity "Mailbox Auditing\$($_.Name)" -Verbose -Confirm:$false }

# Add limited Set-Mailbox permissions
Add-ManagementRoleEntry -Identity "Mailbox Auditing\Set-Mailbox" -Parameters "Identity","AuditAdmin","AuditDelegate","AuditOwner"

# Create a Role Group, add our custom Mailbox Auditing role, and add our Service Principal as a member
New-RoleGroup "Advanced Auditing Management" -Description "Limited scope for Azure Automation to set Advanced Auditing entries" -Roles "Mailbox Auditing" -Members $SP_ID -Confirm:$false -Verbose

Now, that our Automation Account has the necessary permissions, let's head back over to the Azure Portal to our Automation Account, then go Modules. We need to add the ExchangeOnlineManagement and a few Graph modules required to run our script.

Start by clicking Browse Gallery.

Search for ExchangeOnlineManagement, then select the ExchangeOnlineManagement modules. It will show a confirmation page with a list of cmdlets, and simply click Next to continue.

Select the Runtime version (only tested with 5.1), then click Import to finish.

Now repeat this process for the following Graph modules in this order: Microsoft.Graph.Authentication, then Microsoft.Graph.Groups

Once you are finished importing modules, scroll up a bit in the Automation Account and click on Runbooks, then click Create a runbook.

Provide a Name, set Runbook type to PowerShell, specify Runtime version (only testing with 5.1), and optionally add a description, then click Create.

Now copy/paste the following code into the code editor, change <YOURDOMAIN>.onmicrosoft.com to your tenant domain, then click Save. You can optionally add -WhatIf to the Set-Mailbox command to see the output. Use the Test Pane button to run a test of the script, but be aware it actually runs and performs tasks despite the name "Test" ;)

# Set Tenant Name
$tenantName = "<YOURDOMAIN>.onmicrosoft.com"

# Connect to Microsoft Graph within Azure Automation
Connect-AzAccount -Identity
$token = Get-AzAccessToken -ResourceUrl "https://graph.microsoft.com"
Connect-MgGraph -AccessToken $token.Token

# Get members of Advanced Auditing dynamic group
$groupId = (Get-MgGroup -Filter "DisplayName eq 'Azure Automation - Advanced Auditing'").Id
$groupMembers = (Get-MgGroupMember -GroupId $groupId -All | Select-Object -ExpandProperty AdditionalProperties).mail

# Connect to Exchange Online using Managed Identity
Connect-ExchangeOnline -ManagedIdentity -Organization $tenantName -Verbose

# I have absolutely no idea why the script fails without running a Get-Mailbox first...
Get-Mailbox -Identity $groupMembers[0] | Out-Null

# Enable all advanced auditing
$groupMembers | ForEach-Object {
    Write-Output $_
    Set-Mailbox -Identity $_ -AuditAdmin @{add="Create","FolderBind","SendAs","SendOnBehalf","SoftDelete","HardDelete","Update","Move","MoveToDeletedItems","UpdateFolderPermissions","ApplyRecord","RecordDelete","Send","UpdateCalendarDelegation","UpdateComplianceTag","UpdateInboxRules","MailItemsAccessed"} -AuditDelegate @{add="Create","FolderBind","SendAs","SendOnBehalf","SoftDelete","HardDelete","Update","Move","MoveToDeletedItems","UpdateFolderPermissions","ApplyRecord","MailItemsAccessed","RecordDelete","UpdateComplianceTag","UpdateInboxRules"} -AuditOwner @{add="Create","SoftDelete","HardDelete","Update","Move","MoveToDeletedItems","UpdateFolderPermissions","ApplyRecord","RecordDelete","Send","UpdateCalendarDelegation","UpdateComplianceTag","UpdateInboxRules","MailItemsAccessed","MailboxLogin","SearchQueryInitiated"}
}

Once we have it working the way we want, click Publish, and then all we need to do is set up a schedule to run it under. Click Schedules, and then click Add a Schedule.

On the next screen, click Link a schedule to your runbook. If you don't have schedules created that will work for this, click Add a schedule and create one!

And that's it! Your script should now run on whatever schedule you chose, and users will now be properly enabled for all auditable events :)

Final note: It is important to keep track of auditable events, and I will do my best to announce new ones when they are added (very rare). For a full list of auditable events, take a look at the docs: Manage mailbox auditing - Microsoft Purview (compliance)

Feedback and Improvements

I greatly appreciate feedback and had several ideas while creating this, so please let me know if you have any ideas or suggestions :)

First, I'm working on a second script that utilizes Get-MgAuditLogDirectoryAudit to query license changes, and because this will run faster, we can run it more often. Think of the one in this blog like an initial sync and the second one as a delta.

Second, I could have used native Graph API calls to do some things a little more efficiently, but I didn't feel that would be as accessible to most admins. Since I already spent more time than I wanted creating this, I don't have time to work on the Graph API calls, but if you'd like to contribute them, I'll gladly reference them!

Third, I could have used whenChanged to limit running against all users every run, but I couldn't get filters working properly to make it run any faster and preferred ensuring audit events were set back in the event they were disabled by someone outside of automation. This is why I decided on a separate script to handle deltas.