A vendor questionnaire is not vendor risk management.

It’s a starting point. Sometimes a useful one. But if your process ends with a filled-out form and a SOC report in a folder, you’re not managing risk — you’re documenting optimism.

In this Get NIST-y, Jared and Mike use the Mythos supply chain mess to highlight a hard truth: your vendors’ vendors can absolutely become your problem. And right alongside that is a second issue creeping into a lot of compliance programs — tools that promise automation, but mostly deliver cleaner dashboards and more reassuring green check marks.

Want to hear all the details, use cases, and commentary? Stream the episode above and follow on Spotify!

The Checkbox Illusion

There’s a persistent belief that collecting the “right” documents equals due diligence.

SOC 2, SOC 3, security questionnaires — these are all inputs. None of them, on their own, are conclusions.

A SOC 3 in particular is often treated like a shortcut. It’s easy to share, easy to review, and easy to misunderstand. But on its own, it doesn’t tell you much about how risk actually shows up in your environment.

If your entire vendor review process can be summarized as “we got their report,” then what you have is a checkbox exercise, not a risk management program.

Incidents Matter — But Context Matters More

One of the most common questions is whether recent vulnerabilities or security incidents should change how you score a vendor.

Yes, they should. But not in a vacuum.

Every vendor will have vulnerabilities. Many will have incidents. The differentiator is how those issues are handled:

• How quickly are they disclosed?

• How clearly do they communicate impact?

• Do they tell you whether you were affected?

• Do they explain what changed afterward?

A “critical” vulnerability on paper does not automatically translate to critical risk in your environment. Exposure depends on how the service is used, how it’s configured, and what data is involved.

Blindly escalating risk scores based on headlines isn’t much better than ignoring them altogether. The real work is in the context.

The Vendors Worth Trusting

Silence is a signal.

When something goes wrong, strong vendors don’t disappear behind vague statements and delayed updates. They communicate early and specifically:

• Who was affected

• Where the risk exists

• What actions were taken

• What customers should do next

That level of transparency is part of the risk profile, not just good PR.

If you have to chase a vendor for clarity during an incident, that’s a data point worth factoring into your evaluation.

The Green Check Trap

Automation in compliance is useful. It can reduce manual effort, speed up evidence collection, and improve consistency.

But it has limits — and those limits matter.

Two issues come up repeatedly:

First, automated evidence collection depends on integrations and APIs, which are not always reliable. Data can fail to sync, controls can appear compliant when they’re not, and gaps can go unnoticed without human review.

Second, automation only captures what it’s designed to capture. If a control or system isn’t integrated, it effectively disappears from view.

That creates a subtle but dangerous psychological bias: if it’s not showing up in the platform, it’s easy to assume it doesn’t need attention.

Over time, teams start managing what the tool can see instead of what actually exists.

That’s how you end up with a dashboard full of green checks — and a risk posture that hasn’t meaningfully improved.

Automation Helps. It Doesn’t Own Risk.

Automated evidence collection can absolutely save time. It can make audits smoother and reduce repetitive work.

But it cannot:

• Interpret context

• Evaluate edge cases

• Account for missing data

• Replace human judgment

If your process treats automation as the source of truth instead of a supporting input, you might be obscuring risk rather than reducing it.

The Real Takeaway

Vendor risk management is not about collecting artifacts. It’s about understanding how risk actually flows through your vendors into your environment.

That means:

• Looking beyond reports and questionnaires

• Weighing incidents with context, not just severity labels

• Valuing transparency as part of a vendor’s security posture

• Treating automation as a tool, not a decision-maker

Because a green check mark doesn’t mean you’re secure.

It just means something, somewhere, said you were.

---

Frequently Asked Questions

Is a vendor security questionnaire the same as vendor risk management?

No. A vendor questionnaire is a starting point, not a complete vendor risk program. It gives you inputs, but if your process ends at a filled‑out form and a report in a folder, you are documenting optimism, not actually managing risk.

Why aren’t SOC 2 or SOC 3 reports enough on their own?

SOC reports are useful artifacts, but they don’t automatically tell you how risk shows up in your environment. On their own, they don’t explain how you use the service, how it is configured, or how issues at that vendor could practically affect your data and operations.

How should security incidents and vulnerabilities affect vendor scoring?

Incidents and vulnerabilities should influence vendor risk, but only with context. You need to understand how the service is used, whether your data or configuration was exposed, and how clearly and quickly the vendor communicated what happened and what changed afterward.

What behaviors distinguish vendors that are worth trusting?

Trustworthy vendors communicate early, clearly, and specifically when something goes wrong. They explain who was affected, where the risk exists, what they have done to fix it, and what you should do next, instead of hiding behind vague statements or silence.

What is the “green check” trap in compliance automation?

The green check trap happens when teams start trusting dashboards more than reality. Because automation only sees integrated systems and reliable APIs, it can show a clean, compliant picture while gaps, configuration issues, or non‑integrated controls quietly fall out of view.

What is the real goal of vendor risk management?

The real goal is to understand how risk actually flows from your vendors, and your vendors’ vendors, into your environment. That requires looking past questionnaires and reports, weighing incidents with context, valuing transparent communication, and using automation as a tool to support human judgment—not as the decision‑maker.