Automated Investigation Outcomes by Day

Hunt Hypothesis

This hypothesis targets adversaries leveraging phishing (T1566) to deliver malicious payloads, where automated investigations may fail to fully remediate threats or incorrectly classify alerts as clean. Proactively hunting for discrepancies between automated outcomes and raw email delivery events allows the SOC to identify missed detections or false negatives in the email security pipeline, ensuring that subtle phishing campaigns are not overlooked by automated triage.

KQL Query

let AutoRemediations =
    EmailPostDeliveryEvents
    | where Timestamp > ago(30d)
    | where ActionType == "Automated Remediation"
    | summarize arg_max(Timestamp, ActionResult) by NetworkMessageId, RecipientEmailAddress
    | project NetworkMessageId, RecipientEmailAddress, Timestamp;
let EvidenceVerdicts =
    AlertEvidence
    | where Timestamp > ago(30d)
    | where EntityType in ("MailMessage", "MailCluster")
    | extend extra = todynamic(AdditionalFields)
    | extend networkIds = iff(
        isnull(extra.NetworkMessageIds) or array_length(extra.NetworkMessageIds) == 0,
        pack_array(NetworkMessageId),
        extra.NetworkMessageIds)
    | mv-expand nmID = networkIds
    | extend lastVerdict = tolower(tostring(extra.LastVerdict)),
             threatAIList = extra.ThreatAnalysisSummary,
             threatInt = extra.ThreatIntelligence
    | extend verdictFromAnalysis = iff(array_length(threatAIList) > 0, tolower(tostring(threatAIList[0].Verdict)), ""),
             verdictTI = iff(array_length(threatInt) > 0, "malicious", "")
    | extend ThreatDetectedFlag = case(
        lastVerdict in ("malicious", "suspicious") or verdictFromAnalysis in ("malicious", "suspicious") or verdictTI == "malicious",
        true, false)
    | extend MailCountInt = toint(extra.MailCount)
    | extend Weight = iff(EntityType == "MailCluster", coalesce(MailCountInt, 1), 1)
    | summarize IsThreatDetected = any(ThreatDetectedFlag), WeightSum = max(Weight) by nmID = tostring(nmID);
AutoRemediations
| join kind=leftouter EvidenceVerdicts on $left.NetworkMessageId == $right.nmID
| extend isThreat = coalesce(IsThreatDetected, false), effectiveWeight = coalesce(WeightSum, 1)
| extend InvestigationOutcome = iif(isThreat, "Threat Detected", "Clean")
| summarize PreventionCount = sum(effectiveWeight) by Day = bin(Timestamp, 1d), InvestigationOutcome
| order by Day asc, InvestigationOutcome asc
| render timechart

Analytic Rule Definition

id: e13d7b0c-1cfa-41dd-a5cf-45d598055cef
name: Automated Investigation Outcomes by Day
description: |
  This query summarizes daily automated investigation outcomes (threat detected versus clean) by correlating automated remediation events with alert evidence, using the EmailPostDeliveryEvents and AlertEvidence tables.
description-detailed: |
  This query classifies each automated remediation as Threat Detected or Clean by deriving the verdict from AlertEvidence (last verdict, threat analysis summary, and threat intelligence), weighting mail clusters by their message count, and summarizes the daily volume of each outcome, so SOC teams can see how many post-delivery investigations actually caught a threat.
  Query is also included as part of the Defender for Office 365 solution in Sentinel: https://techcommunity.microsoft.com/blog/microsoftdefenderforoffice365blog/part-2-build-custom-email-security-reports-and-dashboards-with-workbooks-in-micr/4411303
requiredDataConnectors:
- connectorId: MicrosoftThreatProtection
  dataTypes:
  - EmailPostDeliveryEvents
  - AlertEvidence
tactics:
  - InitialAccess
relevantTechniques:
  - T1566
query: |
  let AutoRemediations =
      EmailPostDeliveryEvents
      | where Timestamp > ago(30d)
      | where ActionType == "Automated Remediation"
      | summarize arg_max(Timestamp, ActionResult) by NetworkMessageId, RecipientEmailAddress
      | project NetworkMessageId, RecipientEmailAddress, Timestamp;
  let EvidenceVerdicts =
      AlertEvidence
      | where Timestamp > ago(30d)
      | where EntityType in ("MailMessage", "MailCluster")
      | extend extra = todynamic(AdditionalFields)
      | extend networkIds = iff(
          isnull(extra.NetworkMessageIds) or array_length(extra.NetworkMessageIds) == 0,
          pack_array(NetworkMessageId),
          extra.NetworkMessageIds)
      | mv-expand nmID = networkIds
      | extend lastVerdict = tolower(tostring(extra.LastVerdict)),
               threatAIList = extra.ThreatAnalysisSummary,
               threatInt = extra.ThreatIntelligence
      | extend verdictFromAnalysis = iff(array_length(threatAIList) > 0, tolower(tostring(threatAIList[0].Verdict)), ""),
               verdictTI = iff(array_length(threatInt) > 0, "malicious", "")
      | extend ThreatDetectedFlag = case(
          lastVerdict in ("malicious", "suspicious") or verdictFromAnalysis in ("malicious", "suspicious") or verdictTI == "malicious",
          true, false)
      | extend MailCountInt = toint(extra.MailCount)
      | extend Weight = iff(EntityType == "MailCluster", coalesce(MailCountInt, 1), 1)
      | summarize IsThreatDetected = any(ThreatDetectedFlag), WeightSum = max(Weight) by nmID = tostring(nmID);
  AutoRemediations
  | join kind=leftouter EvidenceVerdicts on $left.NetworkMessageId == $right.nmID
  | extend isThreat = coalesce(IsThreatDetected, false), effectiveWeight = coalesce(WeightSum, 1)
  | extend InvestigationOutcome = iif(isThreat, "Threat Detected", "Clean")
  | summarize

Required Data Sources

Sentinel Table	Notes
`AlertEvidence`	Ensure this data connector is enabled

MITRE ATT&CK Context

Tactic: InitialAccess
Technique: T1566 — T1566

References

[Source Query](https://github.com/Azure/Azure-Sentinel/blob/main/Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Remediation/Automated Investigation Outcomes by Day.yaml)

False Positive Guidance

Bulk Email Campaigns with Automated Remediation: When a marketing or IT team sends a large volume of emails (e.g., via Microsoft 365 or Exchange Online) that are subsequently flagged by automated policies (such as “Junk Email” or “Phishing” rules) and automatically moved to the Junk folder or deleted, the correlation between EmailPostDeliveryEvents (showing high volume) and Alerts (showing “Phishing” or “Junk” detections) can trigger this rule.
- Filter/Exclusion: Exclude alerts where the AlertDisplayName contains “Junk Email” or “Phishing” AND the associated user is in a known marketing/IT distribution group, or filter out events where the MessageDeliveryStatus is “MovedToJunkFolder” rather than “Quarantined”.
Scheduled Backup or Sync Jobs: Enterprise backup solutions (e.g., Veeam, Commvault) or file sync services (e.g., OneDrive for Business, SharePoint) often generate high volumes of file access events and associated alerts if they touch sensitive directories or trigger DLP policies. If these jobs run during business hours and generate “File Accessed” or “DLP Policy Violation” alerts, the daily aggregation may show a spike in “threat detected” outcomes.
- Filter/Exclusion: Exclude alerts where the SourceDevice or User matches known service accounts (e.g., svc-backup, backup-admin) or where the AlertDisplayName contains “DLP” or “File Access” and the source process is a known backup/sync executable (e.g., vss.exe, onedrive.exe).
Automated Ticketing or Notification Systems: ITSM tools (e.g., ServiceNow, Jira) or notification services (e.g., PagerDuty, Slack integrations) may send