An 8% PDF Redaction Failure Rate That Was Entirely Wrong

Zhenchao Ji · Published August 10, 2026 · Last updated August 10, 2026

We built a PDF privacy scanner to test a specific technical question: can a browser detect text that still exists underneath what appears to be a redaction?

Our first real-world result looked alarming. In a deterministic sample of 100 publicly available US federal court PDFs, the scanner marked eight documents as having a Critical redaction failure. An apparent rate of 8% would have been a striking finding.

It was also wrong.

We rendered every flagged page and reviewed the evidence one document at a time. All eight findings were false positives produced by our detector. The review led to four rounds of engineering corrections involving signature images, OCR text layers, electronic filing stamps, and blank fillable forms.

This is a postmortem of those false positives: how we sampled the documents, why the initial number looked suspicious, how PDF structure defeated seemingly reasonable heuristics, and what the final zero-result does—and does not—tell us.

1. What we wanted to validate

A common failed-redaction pattern is straightforward in principle. A PDF contains extractable text. A filled rectangle or another opaque object is painted later in the page content stream and overlaps the text's bounding box. The page looks covered, but the original text remains selectable or extractable.

Our detector therefore combines two types of evidence:

• text objects, their page-space bounding boxes, and whether they are painted visibly;
• later paint operations such as filled rectangles, image XObjects, and annotation appearance streams.

It normalizes geometry through the page transformation matrix, compares drawing order, and measures overlap. A strong candidate is text followed by a sufficiently opaque covering object over the same region.

Synthetic fixtures made this approach look reliable. Before the real-corpus review, 35 unit tests passed, and a 40-case synthetic benchmark matched every expected finding. That established consistency with our own test assumptions. It did not establish real-world precision.

2. How we selected the corpus

Our first corpus idea was SEC EDGAR filing correspondence. We inspected 50 accession directories and found only one PDF, a 2% availability rate. That path was too sparse for a practical PDF study, so we recorded it as a failed source-selection experiment rather than silently discarding it.

We then used publicly downloadable PDFs indexed by the CourtListener RECAP corpus. Discovery used a fixed 2024 date range, the availability filter, and a fixed seed:

usevelyo-r1-real-world-corpus-v1

Candidates were ordered deterministically from that seed. The sampling process did not search for visual black boxes, redaction terminology, or detector outcomes before selection. We also limited selection to one eligible PDF per docket during candidate construction. Reports retained only non-identifying sample IDs; case names, docket numbers, document names, and source URLs were excluded from published statistics.

The first prevalence run scanned 100 documents from the random tier. A separate set of 30 documents came from a query-targeted tier. The targeted tier was useful for testing the workflow, but it was excluded from prevalence conclusions because its selection method was not neutral.

Later runs split the random corpus using only attachment_number, a structural field unrelated to the detector result:

• main: the main document, where the attachment number was empty or zero;
• attachment: an exhibit or attachment, where the attachment number was at least one.

Each stratum had an independent quota of 100 documents and its own statistics. We did not combine them into one headline rate. Main documents were dominated by orders and motions; attachments were more likely to contain exhibits, forms, and scanned material. Combining them would conceal those differences.

3. The denominator mattered

A redaction detector cannot make the same claim about every PDF.

For a born-digital document with extractable text, the scanner can compare text geometry with later paint operations. For an image-only scan, there may be no text object to test. Reporting both documents in the same denominator would make “no finding” sound like “checked and safe,” even when the relevant evidence was absent.

The initial 100-document random sample contained extractable text in all 100 PDFs. Later stratified reports counted text-layer and image-only documents separately. Critical rates based on text overlap used only text-layer documents as the interpretable denominator, while image-only documents were disclosed separately.

This distinction is not statistical housekeeping. It defines what the measurement means. A zero among text-layer documents says that this detector did not confirm its supported redaction-failure pattern in that slice. It says nothing about pixels that would require OCR, image-only redaction, or unsupported PDF structures.

4. The first result: eight Critical findings

The random run produced eight documents with Critical findings. Seven came from an image_over_text rule and one from an annotation_over_text rule. The original high-confidence black_rect_over_text rule did not trigger once.

Three details argued against accepting the 8% figure immediately:

1. Every confidence score fell within a narrow, relatively low range rather than the detector's strongest range.
2. Six of the eight findings occurred on page one, where letterheads, stamps, and signatures are common.
3. The supposed failures came from newly added image and annotation heuristics, not from the better-constrained filled-rectangle path.

Those were warning signs, not proof. We therefore generated local audit evidence, rendered each flagged page, and reviewed all eight hits.

Every one was wrong.

5. False positive one: signature images

Seven findings were handwritten signature images crossing text near signature lines. The eighth was the same visual pattern embedded through an annotation appearance.

The detector's geometry was accurate: an image was painted after text, and their bounding boxes overlapped. Its interpretation was not. A signature image often has a transparent background and sparse dark strokes. Its rectangular image bounds may cover a large area even though most pixels reveal the content underneath.

Our earlier heuristic treated “later image overlaps text” as weak evidence of concealment without verifying whether the image could actually conceal anything. That shortcut was the defect.

The correction required structural image evidence. Images with a soft mask or alpha channel could not be treated like solid covers. For remaining candidates, the detector required verified opacity, sufficient dark-pixel coverage, and color uniformity before an image could support a failed-redaction finding. The same rule was applied to annotation appearance images. Regression controls covered signatures, seals, logos, watermarks, and annotation-based signatures, while an opaque dark image over text remained a positive case.

Rerunning the same 100-document random sample with the same seed reduced Critical findings from eight to zero.

6. False positive two: OCR text layers

The broader risk audit then exposed another systematic problem: 29 of 34 reviewed signals were hidden_text findings produced by non-painting text.

PDF text rendering modes can place text in the document without painting visible glyphs. That can be suspicious when an isolated hidden phrase has no visible counterpart. It is also how many scanned PDFs become searchable: OCR text is positioned invisibly over a page image that already contains the visible words.

Our detector recognized the rendering mode but initially lacked enough page context. It interpreted normal OCR accessibility/search infrastructure as deliberately hidden content.

The correction preserved the positive case while adding context. A systematic, page-wide invisible layer aligned with a page image or corresponding visible glyph positions is characteristic of OCR or font fallback. An isolated invisible region with no visible counterpart remains a supported warning signal. We did not disable detection for non-painting text; we narrowed what qualifies as suspicious.

7. False positive three: electronic filing stamps

One high-confidence redaction finding came from an electronic court-filing stamp overlapping header text. Again, the geometry was correct. The annotation was opaque, was drawn over extractable text, and met the overlap threshold. But it was a filing mark, not an attempt to conceal information.

The mistake belonged to our detector, not to the document producer.

The correction used structure instead of court-specific words. Annotation subtype and page position matter: a Stamp or FreeText appearance in a narrow header or footer band has a different prior meaning from a dark content-stream rectangle in the body. We explicitly avoided matching a particular court acronym because that would suppress one observed example while leaving the general defect intact.

8. False positive four: blank forms and white rectangles

White overlays were the hardest category because the rendered page could not settle the question. A white rectangle over a white background looks blank whether it is a real white-out redaction or simply the background of an empty form field.

In stratified review, the remaining white-rectangle evidence included a blank fillable court form and a repeated pleading-paper template. A form widget's appearance stream had painted white field backgrounds over template text. Repeated layout elements appeared at the same coordinates across pages. Neither was a redaction.

We could not responsibly solve this by turning off white-overlay detection; white rectangles can be used to hide text. Instead, the detector added non-content structural discriminators:

• whether the overlay came from a content stream, an AcroForm widget, or an annotation appearance;
• whether the same geometry repeated across multiple pages;
• whether the covered text was visibly repeated elsewhere, compared only in memory;
• the covered text length, without retaining the text itself.

AcroForm widget backgrounds and repeated template geometry were excluded as redaction candidates. A one-off white rectangle from the page content stream could still produce a warning, but the white-overlay path no longer produced a Critical result. After this correction, the main and attachment strata both had zero Critical redaction findings in the reproducible runs.

9. What the zero result means

After the four correction rounds, we found no confirmed failed redaction in the reviewed samples. That is the result. We did not find evidence that the problem was common, and we will not turn an absence of findings into a prevalence claim.

The result is limited to RECAP-available federal documents selected under this procedure. The initial sample leaned heavily toward short judicial documents and standardized forms. The stratified attachment sample improved coverage of exhibits and submitted material, but it still does not represent all courts, filing systems, document types, or redaction workflows.

The scanner also does not establish safety for image-only redactions, OCR-only content without supported geometry, clipping masks, arbitrary complex overlays, or every possible PDF construction. A clean report is not a compliance determination or a substitute for manual review.

What the study did establish is narrower and useful: synthetic precision was not enough; the first real-world statistic was a detector-quality signal, not a privacy prevalence result; and PDF structure provided better discriminators than visual overlap alone.

The reproducible ingredients are the fixed seed, neutral candidate ordering, separate random and targeted tiers, independent main and attachment strata, explicit text-layer denominators, and manual review before interpreting any detector hit.

The resulting browser-based scanner is available at /pdf-privacy-scanner. It helps identify supported privacy-risk patterns locally, while presenting the limitations that this study made impossible to ignore.