I buy traffic. Every month I get a report from an anti-fraud vendor telling me how much of it was invalid. The report has a percentage, a seal, and a reassuring tone. It does not have an answer to the only question I actually need answered.
Who checked the checker?
The vendor grades my traffic. Fine. Somebody has to. But the vendor also writes the definition of invalid, labels the examples, runs the test, and marks its own paper. That is not an audit. That is a restaurant printing its own health inspection certificate and laminating it for the table. Nobody has to intend anything dishonest for that to be the outcome. The structure simply never requires an outside opinion.
The hard part of fraud detection is not catching the obvious bot. The hard part is knowing whether the system was right. To know that, you need a set of clicks where you already know the truth. The industry calls it ground truth. It is a lovely phrase. It suggests somebody actually has some.
Where does ground truth come from? In many deployments, the vendor labels it. The vendor decides which clicks are fraudulent and which are human. Then the vendor measures how well its system agrees with those labels. An accuracy number in the report often means the system agreed with the vendor's own answer key. That is not validation. That is an echo.
The buyer often does not get the raw data. The clicks, the labels, the reason codes, the false positives. The buyer gets a summary. The summary says the traffic is clean. The summary does not say how many legitimate users were thrown out with the fraud, because that number would start an argument nobody wants to have.
And who audits the vendor? Sometimes a third party. But the third party is usually paid by the vendor, uses the vendor's definitions, and reports to the vendor. That is not a conspiracy. It is a procurement process. If I hire a firm to inspect my kitchen, and I pay the firm, and I write the inspection checklist, I will probably be very happy with the certificate. The incentive is not to lie. The incentive is to choose a checklist that measures what I already do well.
There is a better question a buyer can ask. Not "what is your fraud rate?" but "what did you block that was human, and how do you know?" The answer is usually silence. Because measuring false positives requires ground truth the vendor does not have. And the vendor is not paid to find it. Very few people receive a bonus for discovering that some of the traffic previously condemned as fraudulent was actually a person on a VPN in a hurry.
The buyer's problem is not that fraud detection is impossible. It is that the person selling certainty is the same person grading the test. The only way out is to demand evidence that can be checked by someone other than the vendor. Raw logs. Labeled data. A definition of invalid that does not change when the vendor's report needs to look good. An audit paid by the buyer, not the audited.
So here is the question I now ask every vendor before I look at the dashboard: "If I pay an independent auditor, will you give them the raw data and your labels, and will you accept their result?" The silence that follows is more informative than the report.