User Submissions by Detection Method - Phish (FP)

Hunt Hypothesis

This hypothesis targets the exploitation of the Phish filter verdict, where adversaries craft emails that bypass automated detection or are intentionally misclassified to evade user suspicion and facilitate credential harvesting. Proactively hunting for these false positive submissions allows the SOC to identify patterns in user-reported messages that consistently evade the phish filter, enabling the team to refine detection logic and mitigate the risk of successful phishing attacks in the Azure Sentinel environment.

KQL Query

let Submissions = CloudAppEvents
    | where Timestamp > ago(30d)
    | where ActionType == "UserSubmission"
    | extend RawData = parse_json(RawEventData)
    | where tostring(RawData.SubmissionContentType) == "Mail" and tostring(RawData.SubmissionType) == "3"
    | extend NetworkMessageId = tostring(RawData.ObjectId), RecipientObjectId = AccountObjectId
    | distinct NetworkMessageId, RecipientObjectId;
EmailEvents
| where Timestamp > ago(30d)
| join kind=inner Submissions on NetworkMessageId, RecipientObjectId
// De-duplicate to the latest EmailEvents record per message and recipient so the verdict is read once
| summarize arg_max(Timestamp, *) by NetworkMessageId, RecipientObjectId
| where EmailDirection == "Inbound" and DetectionMethods has 'Phish'
| mv-expand Phish = parse_json(DetectionMethods).Phish to typeof(string)
| where isnotempty(Phish)
| summarize count() by Phish
| render piechart

Analytic Rule Definition

id: 5990abd3-7e11-4441-947b-17e60c69f28f
name: User Submissions by Detection Method - Phish (FP)
description: |
  This query visualises user false positive submissions by the original phish filter verdict on the reported message.
description-detailed: |
  This query visualises user false positive submissions by the original phish filter verdict on the reported message.
  Query is also included as part of the Defender for Office 365 solution in Sentinel: https://techcommunity.microsoft.com/blog/microsoftdefenderforoffice365blog/part-2-build-custom-email-security-reports-and-dashboards-with-workbooks-in-micr/4411303
requiredDataConnectors:
- connectorId: MicrosoftThreatProtection
  dataTypes:
  - CloudAppEvents
  - EmailEvents
tactics:
  - InitialAccess
relevantTechniques:
  - T1566
query: |
  let Submissions = CloudAppEvents
      | where Timestamp > ago(30d)
      | where ActionType == "UserSubmission"
      | extend RawData = parse_json(RawEventData)
      | where tostring(RawData.SubmissionContentType) == "Mail" and tostring(RawData.SubmissionType) == "3"
      | extend NetworkMessageId = tostring(RawData.ObjectId), RecipientObjectId = AccountObjectId
      | distinct NetworkMessageId, RecipientObjectId;
  EmailEvents
  | where Timestamp > ago(30d)
  | join kind=inner Submissions on NetworkMessageId, RecipientObjectId
  // De-duplicate to the latest EmailEvents record per message and recipient so the verdict is read once
  | summarize arg_max(Timestamp, *) by NetworkMessageId, RecipientObjectId
  | where EmailDirection == "Inbound" and DetectionMethods has 'Phish'
  | mv-expand Phish = parse_json(DetectionMethods).Phish to typeof(string)
  | where isnotempty(Phish)
  | summarize count() by Phish
  | render piechart
version: 1.0.0

Required Data Sources

MITRE ATT&CK Context

• Tactic: InitialAccess
• Technique: T1566 — T1566

References

• [Source Query](https://github.com/Azure/Azure-Sentinel/blob/main/Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Submissions/User Submissions by Detection Method - Phish FP.yaml)

False Positive Guidance

• Scenario: A user reports a legitimate internal security awareness training email (e.g., from GoPhish or KnowBe4) that contains a simulated phishing link or button, which the original phish filter incorrectly flagged as “Phish” due to its external domain or specific URL structure.

  • Filter/Exclusion: Exclude messages where the sender domain matches known internal security training platforms (e.g., *.gophish.io, *.knowbe4.com) or where the subject line contains specific training identifiers (e.g., [Training], [Simulated Phish]).

• Scenario: An automated notification from a CI/CD pipeline (e.g., Jenkins, Azure DevOps, or GitHub Actions) is sent to a distribution list and contains a deep link to a build artifact or log file hosted on an internal or third-party CDN, which the phish filter misclassifies as a malicious external link.

  • Filter/Exclusion: Exclude messages where the sender is a known service account (e.g., ci-bot@company.com, jenkins@company.com) or where the message body contains specific CI/CD keywords (e.g., “Build #”, “Pipeline Run”, “Deployment Status”) and the link domain is in the allowlist of internal infrastructure or approved CDNs.

• Scenario: A user reports a legitimate password reset or MFA enrollment email from an identity provider (e.g., Okta, Azure AD, or OneLogin) that includes a tokenized URL with a long, random path, which the phish filter flags due to the high entropy of the URL or the use of a shortener service.

  • Filter/Exclusion: Exclude messages where the sender domain is in the list of known identity providers (e.g., *.okta.com, login.microsoftonline.com, *.onelogin.com) and the subject line contains