Executive Summary
We built a scanner that audits Model Context Protocol configurations for authentication problems, shipped it with 100% module coverage and 843 passing tests, and pointed it at 463 public configs. It was wrong twice, in two different ways, and both times the tests were green while it was wrong. We caught both before publishing and put the corrections in the writeup. The lesson is not “test more.” It is that coverage measures whether your code ran, never whether you were right, and that the discipline separating security research from security marketing is the willingness to check your own headline number before it becomes the thing people quote.
Working through this in production? See how we run a AI security assessment.
There is a version of this piece that opens with the scary number.
Two thirds of remote MCP servers in our corpus declare no authentication at all. 109 out of 163. That number is real, it held up under scrutiny, and it is the finding we published on our site this week.
But the more useful story is the one underneath it: the two times our own tooling told us something false, confidently, with a green test suite behind it.
What we were building
The Model Context Protocol is how an AI agent reaches your tools and data. A JSON file declares which servers the agent may talk to and what credentials it carries. Get that file wrong and you have handed an autonomous system a set of keys with no lock on the door.
We built mcp-warden to gate that file in CI. Static analysis only: it reads the config, flags weak posture, exits non-zero. No server spawned, no network touched. That constraint is deliberate. It makes the tool safe to run anywhere, and it makes it immune to churn in the MCP auth specification, which is still moving.
Then we shipped it, and started using it.
Lie one: the shape we imagined
The day after release, we ran the audit against one of our own real configuration files. It reported three high-severity findings.
All three were wrong.
The value it flagged was Bearer ${GMAIL_TOKEN}.
That is not a leaked credential. That is the single most common correct shape an Authorization header takes. The secret lives in an environment variable; the config carries a reference to it. Doing it that way is the behavior we were trying to encourage.
Our detector only recognised a reference when ${VAR} was the entire value. ${TOKEN} passed. Bearer ${TOKEN} did not.
Here is the part worth sitting with: that module had 100% test coverage. Every line executed. Every branch taken. 843 tests green.
Every fixture we had written used the bare ${VAR} form, because that is the shape the author had in mind when writing the tests. Coverage confirmed the code ran against the examples we imagined. It had nothing to say about the examples the world actually produces.
We fixed it, and then almost introduced a worse bug fixing it. The first permissive version accepted a reference anywhere in the value, and allowed dashes and arbitrary length in whatever surrounded it. That let Bearer sk-abcdefghijklmnopqrst ${T} through as a “scheme word”: a real literal credential sitting right next to a reference, waved past. Tightened to short, purely alphabetic scheme words (Bearer, Token, Basic, ApiKey), with a regression test pinning the mixed case.
A precision fix that becomes a bypass is worse than the false positive it replaced.
Lie two: the number that would have gone viral
With the detector corrected, we assembled a corpus: 463 publicly published MCP configuration files, declaring 998 servers between them.
First pass reported 145 committed credentials.
That is a good number. It is shareable. It would have been the headline, and it would have been the thing quoted back at us for a year.
Before publishing it, we did something that should be routine and is not: we looked at the shapes of the values behind the count, rather than the count.
107 of the 145 were placeholders. YOUR KEY GOES HERE. xxx. <your-api-key>. Template slots in example configs. Not secrets. Nothing to leak.
Several more were reference forms our detector still did not recognise:
${GITHUB_TOKEN:-}, shell expansion with a default valueop://vault/item/field, a 1Password reference URI~/.config/app/keys.json, a path to a credential file, not the credential
Every one of those is a careful operator doing the right thing. We were counting them as leaks.
We fixed the detector again, split placeholders into their own low-severity rule (a template with an unfilled slot is a finding, just not a credential exposure), and re-ran the corpus.
145 became 27.
A five-times overstatement, killed by looking at our own data before we talked about it.
Why this matters more than the finding
Security research and security marketing produce identical-looking artifacts. Both are a number, a chart, and a call to action. The difference is entirely in what happened before publication, and it is invisible in the output.
Nobody would have audited our 145. There is no peer review on a security blog post. If we had shipped it, it would have been cited, aggregated into somebody’s threat report, and quoted back to us as established fact. The correction would never have caught up.
The only thing standing between the wrong number and the public was our own willingness to check it when checking could only make the number smaller.
That is not a moral position. It is an engineering one. A finding you have not tried to falsify is a hypothesis wearing a suit.
The third correction, which happened to us mid-writeup
One more, because it is too apt to leave out.
Our article included Amazon’s documented example AWS key, the AKIA…EXAMPLE placeholder from their public docs, as an illustration of a false positive we had correctly dismissed.
Our CI secret scanner flagged the article. Correctly: a credential-format string is a credential-format string, and a scanner that makes exceptions for “but this one is fake” is a scanner you cannot trust.
So we removed it in a follow-up commit. The scanner still failed.
Because gitleaks reads the diff range, and the value was still sitting in the previous commit. Deleting it in a later commit changed nothing. We had to rewrite the branch history to actually remove it.
Which is, word for word, the advice we had just written three paragraphs earlier in that same article: deleting the file is not enough, because the value stays in git history; the credential itself has to be rotated.
We hit our own advice, from the inside, while publishing it. It is now in the commit message, because that is more useful to the next person than a clean history would have been.
What to actually do
If you publish MCP configs, it is one line:
"Authorization": "Bearer ${TOKEN}"
If a credential already landed in a repository, rotate it at the provider. Removing the file does not help.
If you operate remote MCP servers, assume nothing is authenticating unless you require it. Two thirds of the ones we found are not.
If you run agents in production, put a gate in CI rather than a dashboard in a browser. Plenty of tools report. Very few return a non-zero exit code that a pipeline has to answer for.
pip install mcp-warden-cli
mcp-warden auth audit .mcp.json
And if you publish security findings: look at the shapes behind your number before you publish the number. Ours was wrong by 5x, and the tests were green the entire time.
On the three real ones
The corpus did contain three genuine credentials: a GitHub token, a Slack token, and a JWT.
We never tested any of them. Validating someone else’s key against a live service is unauthorized access, not research, regardless of intent. All three findings are format pattern-matches and nothing more.
Each owner was contacted privately before anything was published. No repository is named in the article, in the thread, or here.
And we destroyed all 463 configurations on every one of our hosts before writing a word. We are a security firm. Holding other people’s credentials while publishing about credential hygiene is not a position worth occupying, and “we needed them for the research” stops being true the moment the aggregate counts are computed.
Data Science & Engineering Experts builds data and AI security for US government and enterprise clients. The scanner in this piece is open source under Apache 2.0. If you are deploying agents and want to know what your configuration surface actually looks like, get in touch.