M365 Policy Assistant

Problem

My nonprofit clients often have Conditional Access policies that were set up years ago and nobody fully remembers why. When an access issue comes up, I spend time digging through the Entra ID portal, cross-referencing policy conditions, and reconstructing the intent behind each rule.

I wanted a faster way to answer questions like: “Why would a user on a personal device get blocked when accessing SharePoint from home?”

Approach

1. Export Conditional Access policy JSON from Entra ID using PowerShell (Get-MgIdentityConditionalAccessPolicy)
2. Feed the policies as context to Claude via the API
3. Ask questions in plain English and get back answers that reference the specific policy rules

Current state

I have a working prototype that:

• Reads exported JSON from a local file
• Sends it as part of a system prompt with some structure
• Accepts a question via stdin and returns an explanation

The responses are genuinely useful. Claude correctly identifies policy conflicts and explains exclusions in terms a non-technical stakeholder can follow.

What’s not working yet

• Token limits: large tenants have dozens of policies. I need to either summarize or selectively include policies based on the question
• No live data: currently requires a manual export step instead of querying the Graph API directly
• No UI: it’s a command-line tool right now

Next steps

• Add Graph API integration so I can query policies directly without exporting
• Figure out a retrieval approach for large policy sets (probably embeddings or keyword filtering)
• Wrap it in a minimal web UI or at least a better CLI

What I’ve learned

Structured JSON is actually great context for LLMs. The policy objects have consistent schemas, which makes it easy to write a system prompt that tells Claude exactly how to interpret the data.

The main challenge isn’t the AI part — it’s getting clean, complete data out of Entra ID in the first place.