Straight from Blacksmith: Listen to our discussion about compliant AI on Get NIST-y!

AI and compliance are colliding in a very practical way for MSPs. On one hand, clients want the productivity boost from LLMs; on the other, regulators and insurers are watching closely. At the same time, the security programs you build are often anchored to assessment tools that can change or disappear with little warning.

Using LLMs in regulated environments without leaking data

Banning AI outright is usually a sign that policy has lost touch with reality. People will still experiment — just with unsanctioned tools and zero guardrails. A better approach is to admit that LLMs are here to stay and design controlled, compliant ways to use them.

A few practical principles:

• Separate use cases by data sensitivity.

  • “Green” use: drafting internal comms, summarizing public docs, generating code snippets that do not include client logic or secrets.

  • “Red” use: anything involving PHI, PCI, financial account data, government‑classified information, trade secrets, or unredacted logs and tickets. Those should never go into an external LLM unless you have explicit contractual and technical guarantees.

• Make “what not to paste” painfully clear.

  • Turn data classification into specific examples: “Don’t paste full names plus claim numbers,” “Don’t paste screenshots of EHR screens,” “Don’t paste configs with IPs, keys, or domain details.”

  • Back this with training and short, visual guides for non‑technical staff.

• Prefer enterprise‑grade AI options when possible.

  • Choose tools that support tenant isolation, data residency, logging, and admin controls over prompts and outputs.

  • Ensure vendor terms explicitly limit training on your data or give you a toggle to disable it.

• Log and monitor AI usage.

  • Treat AI tools like any other data egress channel. Centralize access, enforce SSO, and monitor who is using what, for which business functions.

  • Use DLP and CASB where available to detect obvious policy breaks (e.g., bulk uploads of CSVs containing regulated data).

• Document your stance.

  • Write a short AI usage policy that ties back to your existing data classification and acceptable‑use policies.

  • Make it explicit that allowed use is conditional on following those rules, and that the organization — not individuals — is responsible for choosing sanctioned tools.

The goal is to make compliant AI use easier than shadow AI, and to be able to show a regulator you did the work: you classified data, defined boundaries, selected appropriate tools, and monitored behavior.

Picking a replacement framework or assessment when a tool is sunset

When a regulator‑driven portal or checklist is retired, it is tempting to feel like everything has to start over. Usually, you don’t. The key is to separate the tool from the underlying control structure.

Some pragmatic steps:

• Identify the real framework under the hood.

  • Many regulatory tools are just a skin over something like NIST CSF, NIST 800‑53, CIS Controls, ISO 27001, or a custom derivative.

  • Map old questions or controls to a “canonical” framework (e.g., “these items were really about access control,” “these mapped to logging and monitoring,” etc.).

• Preserve your existing work as evidence.

  • Export whatever you can: past assessments, responses, uploaded documents, remediation notes.

  • Store them in your own repository and tag them against the canonical controls you identified, so they remain valid evidence even after the portal is gone.

• Choose a new “anchor” framework, not just a new questionnaire.

  • Pick a broadly recognized framework that fits your client base (e.g., NIST CSF for many, CIS for smaller orgs, ISO when they need international or enterprise‑grade alignment).

  • Evaluate tools and platforms based on how well they implement or align with that framework, rather than chasing the next regulator‑branded portal.

• Build a translation layer, not a reset.

  • Create a simple mapping: “Old Tool Section A → NIST CSF ID.AM-1, ID.AM-3,” etc.

  • Use that mapping to show continuity: your program did not restart, it just shifted views. This makes reviews with auditors, boards, and regulators much smoother.

• Standardize your internal assessment approach.

  • Develop one internal assessment methodology (scoping, evidence collection, scoring, remediation tracking) that you can reuse, regardless of which external tool is “in fashion” this year.

  • Let regulators’ tools be reporting channels or overlays on that core, not the backbone of your program.

This mindset protects you from tool churn. Regulations and portals will evolve, but if your underlying security program is tied to a stable framework and your own evidence, you can adapt without constantly starting from zero.