x-ray is a Python library for finding bad redactions in PDF documents.

Overview

Image of REDACTED STAMP

x-ray is a Python library for finding bad redactions in PDF documents.

Why?

At Free Law Project, we collect millions of PDFs. An ongoing problem is that people fail to properly redact things. Instead of doing it the right way, they just draw a black rectangle or a black highlight on top of black text and call it a day. Well, when that happens you just select the text under the rectangle, and you can read it again. Not great.

After witnessing this problem for years, we decided it would be good to figure out how common it is, so, with some help, we built this simple tool. You give the tool the path to a PDF. It tells you if it has worthless redactions in it.

What next?

Right now, x-ray works pretty well and we are using it to analyze documents in our collections. It could be better though. Bad redactions take many forms. See the issues tab for other examples we don't yet support. We'd love your help solving some of tougher cases.

Installation

With poetry, do:

poetry add x-ray

With pip, that'd be:

pip install x-ray

Usage

You can easily use this on the command line. Once installed, just:

% python -m xray path/to/your/file.pdf
{
  "1": [
    {
      "bbox": [
        58.550079345703125,
        72.19873046875,
        75.65007781982422,
        739.3987426757812
      ],
      "text": "The Ring travels by way of Cirith Ungol"
    }
  ]
}

That'll give you json, so you can use it with tools like jq. The format is as follows:

  • It's a dict.
  • The keys are page numbers.
  • Each page number maps to a list of dicts.
  • Each of those dicts maps to two keys.
  • The first key is bbox. This is a four-tuple that indicates the x,y positions of the upper left corner and then lower right corners of the bad redaction.
  • The second key is text. This is the text under the bad rectangle.

Simple enough.

If you want a bit more, you can use x-ray in Python:

from pprint import pprint
import xray
bad_redactions = xray.inspect("some/path/to/your/file.pdf")
pprint(bad_redactions)
{1: [{'bbox': (58.550079345703125,
               72.19873046875,
               75.65007781982422,
               739.3987426757812),
      'text': 'Aragorn is the one true king.'}]}

The output is the same as above, except it's a Python object, not a JSON object.

If you already have the file contents as a bytes object, that'll work too:

some_bytes = requests.get("https://lotr-secrets.com/some-doc.pdf").content
bad_redactions = xray.inspect(some_bytes)

Note that because the inspect method uses the same signature no matter what, the type of the object you give it is essential. So if you do this, xray will assume your file name (provided as bytes) is file contents and it won't work:

xray.inspect(b"some-file-path.pdf")

That's pretty much it. There are no configuration files or other variables to learn. You give it a file name. If there is a bad redaction in it, you'll soon find out.

How it works

Under the covers, x-ray uses the high-performant PyMuPDF project to parse PDFs.

You can read the source to see how it works, but the general idea is to:

  1. Find rectangles in the PDF.

  2. Find letters that are under those rectangles.

Things get tricky in a couple places:

  • letters without ascenders are taller than they seem and might not be entirely under the rectangle
  • drawings in PDFs can contain multiple rectangles
  • text under redactions can be on purpose (like if it says "XXX" or "privileged", etc)

And so forth. We do our best.

Contributions

Please see the issues list for thinsg we need, or start a conversation if you have questions. Before you do your first contribution, we'll need a signed contributor license agreement. See the template in the repo.

Deployment

Releases happen automatically via Github Actions on any commit that is tagged with something like "v0.0.0".

If you wish to create a new version manually, the process is:

  1. Update version info in pyproject.toml

  2. Configure your Pypi credentials with Poetry

  3. Build and publish the version:

poetry publish --build

License

This repository is available under the permissive BSD license, making it easy and safe to incorporate in your own libraries.

Pull and feature requests welcome. Online editing in GitHub is possible (and easy!).

Owner
Free Law Project
We provide free access to primary legal materials, develop legal research tools, and support academic research on legal corpora.
Free Law Project
CLI tool to generate pdf invoices written in python

invoicepy CLI invoice tool, store and print invoices as pdf. save companies and customers for later use. installation pip install invoicepy config co

Adam Wojtczak 9 Aug 01, 2022
Python PDF Parser (Not actively maintained). Check out pdfminer.six.

PDFMiner PDFMiner is a text extraction tool for PDF documents. Warning: As of 2020, PDFMiner is not actively maintained. The code still works, but thi

Yusuke Shinyama 4.9k Jan 04, 2023
Simple HTML and PDF document generator for Python - with built-in support for popular data analysis and plotting libraries.

Esparto is a simple HTML and PDF document generator for Python. Its primary use is for generating shareable single page reports with content from popular analytics and data science libraries.

Dom 76 Dec 12, 2022
An application which enables the users to perform simple yet intriguing PDF operations

AstutePDF A repository containing the GUI for an application which enables the users to perform simple yet intriguing PDF operations. These include, M

Raghav S 5 Jan 22, 2022
borb is a library for reading, creating and manipulating PDF files in python.

borb is a library for reading, creating and manipulating PDF files in python.

Joris Schellekens 2.9k Jan 01, 2023
Telegram bot that can do a lot of things related to PDF files.

Telegram PDF Bot A Telegram bot that can: Compress, crop, decrypt, encrypt, merge, preview, rename, rotate, scale and split PDF files Compare text dif

130 Dec 26, 2022
Python lib for Simple PDF text extraction

Python lib for Simple PDF text extraction

Jason Alan Palmer 651 Jan 01, 2023
Auto Convert PDFs to png files in python

This python tool, which is an application of PyMuPDF module, could auto convert PDFs to png files

Bo-Yu 4 Dec 05, 2021
Produce pdf in python backend from simple bootstrap vue frontend and download to browser

vollmacht produce pdf in python backend from simple bootstrap vue frontend and download to browser Frontend in one file with bootstrap-vue (allthough

Otto 1 Nov 08, 2020
Scans pdfs for links written in plaintext and checks if they are active or returns an error code.

Scans pdfs for links written in plaintext and checks if they are active or returns an error code. It then generates a report of its findings. Extract references (pdf, url, doi, arxiv) and metadata fr

Marshal Miller 22 Nov 21, 2022
Python bindings for MuPDF's rendering library.

PyMuPDF 1.19.3 Release date: December 15, 2021 On PyPI since August 2016: Author Jorj X. McKie, based on original code by Ruikai Liu. Introduction PyM

Jorj X. McKie 0 Nov 03, 2022
A bot for PDF for doing Many Things....

Telegram PDF Bot A Telegram bot that can: Compress, crop, decrypt, encrypt, merge, preview, rename, rotate, scale and split PDF files Compare text dif

Mr. Developer 60 Dec 27, 2022
Convert Lecture Videos to PDF

Convert Lecture Videos to PDF Description Want to go through lecture videos faster without missing any information? Wish you can read the lecture vide

Emilio Kartono 20 Nov 25, 2022
A tool for certificate PDF generation.

certificate-pdf-generator 获奖证书PDF批量生成工具 | a Tool for certificate PDF generation. ⚠️ 下载前请注意 本项目使用了LFS来存储PDF等大文件。在克隆或下载本仓库前,请先使用apt等包管理器安装git-lfs包。如果已经克

Wanghao Xu 4 Nov 28, 2022
A simple Python script to convert multiple images (well technically also a single image) into a pdf.

PythonImage2PDF A simple Python script to convert multiple images into a single PDF-document. Created basically for only my own needs for converting m

Joona Gynther 1 Jun 28, 2022
JoplinPdf2Images - Converts a PDF to images in Joplin and adds it to the specified note as a printout

joplinPdf2Images Converts a PDF to images in Joplin and adds it to the specified

Morten Haahr Kristensen 2 Apr 20, 2022
A backend for mdbook in Python for generating PDF based on Chrome DevTools Protocol.

mdbook-pdf A backend for mdbook written in Python for generating PDF based on Chrome DevTools Protocol. Python library dependency Usage Put mdbook-pdf

Hollow Man 49 Dec 27, 2022
pikepdf is a Python library for reading and writing PDF files.

A Python library for reading and writing PDF, powered by qpdf

1.6k Jan 03, 2023
PDFSanitizer - Renders possibly unsafe PDF files and outputs harmless PDF files

PDFSanitizer Renders possibly malicious PDF files and outputs harmless PDF files

9 Jan 30, 2022
this is simple program, that converts pdf file to png

author: a5892731 last update:2021-11-01 version: 1.1 resources: -https://pypi.org/project/pdf2image/ -https://github.com/oschwartz10612/poppler-window

1 Nov 01, 2021