Extracting Use-of-Force Records from Vancouver Police PDF

This PDF contains detailed records of Vancouver Police's use-of-force incidents, provided after a public records request by journalists. Challenges include its very small font size and lots of empty whitespace.

from natural_pdf import PDF

pdf = PDF("use-of-force-raw.pdf")
page = pdf.pages[0]
page.show()
Result

Let's find all the headers, they're the text at the top of the pag, which means they have the smallest y0.

headers = page.find_all('text[y0=min()]')
headers.extract_each_text()
['OFFICER', 'DATE OF REPORT', 'DATE OF FORCE', 'TIME OF FORCE', 'PRECINCT', 'CASE #', 'SUBJECT PERSON', 'SEX', 'RACE', 'ETHNICITY', 'SUBJECT CONDITION LEADING TO FORCE', 'REASON FOR USE OF FORCE', 'OFFICER ACTIVITY PRIOR TO FORCE', 'SUBJECT WAS INJURED', 'OFFICER INJURIES', 'SUBJECT INJURIES', 'MENTAL HEALTH INDICATORS', 'FORCE OPTIONS ATTEMPTED', 'EFFECTIVE FORCE OPTIONS', 'TREATMENT RECEIVED OFFICER', 'TREATMENT RECEIVED SUBJECT', 'SUPERVISOR NOTIFIED', 'SUPERVISOR AT SCENE', 'SUPERVISOR NAME', 'CASE COUNT']

We can now use those headers to create guides that fit between each column.

from natural_pdf.analyzers.guides import Guides

guides = Guides(page)
guides.vertical.from_headers(headers)
guides.show()
Result

Once we've established the columns, we're free to extract the table. pdfplumber is smart enough behind the scenes to know what each row is.

guides.extract_table().to_df()
OFFICER DATE OF REPORT DATE OF FORCE TIME OF FORCE PRECINCT CASE # SUBJECT PERSON SEX RACE ETHNICITY SUBJECT CONDITION LEADING TO FORCE REASON FOR USE OF FORCE OFFICER ACTIVITY PRIOR TO FORCE SUBJECT WAS INJURED OFFICER INJURIES SUBJECT INJURIES MENTAL HEALTH INDICATORS FORCE OPTIONS ATTEMPTED EFFECTIVE FORCE OPTIONS TREATMENT RECEIVED OFFICER TREATMENT RECEIVED SUBJECT SUPERVISOR NOTIFIED SUPERVISOR AT SCENE SUPERVISOR NAME CASE COUNT
0 PFEIFER, TIM 12/2/2016 Dec-02-2016 22:30 WEST 23-2016-18335 SOLLERS, CECIL ARTHUR M U H FAILURE TO COMPLY, RESIST ARREST, ATTEMPT TO F... DEFEND SELF, MAKE ARREST, PREVENT ESCAPE FOOT PURSUIT <NA> NONE NONE <NA> TASER, CAROTID RESTRAINT CAROTID RESTRAINT <NA> EMS AT SCENE, HOSPITAL/RELEASED YES YES SGT. GEDDRY / 1315 970
1 NICHOLSON, DUSTIN 12/3/2016 Dec-03-2016 6:45 WEST 23-2016-18348 GUTIERREZ, SAVANNAH D (DOB: Aug-14-1986) F W N FAILURE TO COMPLY, ATTEMPT TO FLEE MAKE ARREST, PREVENT ESCAPE FOOT PURSUIT, FIELD CONTACT <NA> NONE NONE <NA> TAKEDOWNS TAKEDOWNS TREATMENT REFUSED TREATMENT REFUSED YES NO SGT MARTIN / 970
2 JENNINGS, ERIK 12/7/2016 Dec-07-2016 13:05 EAST 23-2016-18571 MCDONALD, ALEMAYOHU J APR-03-1996 M B U FAILURE TO COMPLY MAKE ARREST HANDCUFFING <NA> NONE NONE <NA> TAKEDOWNS TAKEDOWNS <NA> <NA> YES NO BURGARA / 1257 970
3 BATES, JUSTIN 12/13/2016 Dec-02-2016 22:30 WEST 23-2016-18335 SOLLERS, CECIL A (DOB: Sep-25-1989) M U H FAILURE TO COMPLY, RESIST ARREST, ATTEMPT TO FLEE MAKE ARREST, PREVENT ESCAPE FOOT PURSUIT <NA> NONE NONE <NA> POINT TASER POINT TASER <NA> EMS AT SCENE, HOSPITAL/ADMITTED, HOSPITAL/RELE... YES YES CPL PARDUE/1219 970
4 GEDDRY, BLAISE 12/13/2016 Dec-13-2016 2:35 WEST 23-2016-18818 RENFRO, JOHN W M W U RESIST ARREST, DRUGS, SEE NARRATIVE CIVIL HOLD OTHER <NA> NONE NONE MENTAL ILLNESS, ADMISSION TO FACILITY, PRIOR K... TAKEDOWNS, HANDS/FEET TAKEDOWNS, HANDS/FEET <NA> EMS AT SCENE, HOSPITAL/ADMITTED YES YES GEDDRY 1315 970
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
293 PARDUE, BILL 1/19/2018 1/19/2018 1:24 WEST 23-2018-967 RAULS, COURTNEY A F B U FAILURE TO COMPLY, RESIST ARREST, ALCOHOL, DRU... MAKE ARREST FIELD CONTACT <NA> NONE NONE <NA> TAKEDOWNS TAKEDOWNS <NA> <NA> YES YES DUMAS/ 1543 970
294 MARBACH, NICHOLAS 1/20/2018 1/18/2018 23:20 EAST 23-2018-965 ELLIOT, DENNIS RICHARD M U U FAILURE TO COMPLY, RESIST ARREST, ALCOHOL, DRU... CIVIL HOLD ESCORT, HANDCUFFING, FIELD CONTACT PRIOR TO INVOLVMENT <NA> BRUISES MENTAL ILLNESS, ADMISSION TO FACILITY HANDS/FEET HANDS/FEET <NA> EMS AT SCENE, HOSPITAL/ADMITTED YES YES KREBS/DAVIS #1508/1230 970
295 MARBACH, NICHOLAS 1/20/2018 1/18/2018 23:21 EAST 23-2018-965 NAPOLEON, ALICE LANEY F I U FAILURE TO COMPLY, ALCOHOL DEFEND SELF OTHER, FIELD CONTACT <NA> NONE NONE <NA> HANDS/FEET HANDS/FEET <NA> EMS AT SCENE, SELF TREATMENT YES YES KREBS/DAVIS #1508/1230 970
296 HEMSTOCK, STUART 1/23/2018 1/23/2018 8:45 EAST 23-2018-1191 KESSEL, JORDAN P M W N FAILURE TO COMPLY, SEE NARRATIVE <NA> HANDCUFFING, OTHER <NA> NONE NONE <NA> CONTROL HOLDS CAUSING INJURY CONTROL HOLDS CAUSING INJURY <NA> <NA> YES YES HUBERTY/1214 970
297 HAMMOND, SCOTLAND 1/27/2018 1/27/2018 20:20 WEST 23-2018-1420 CONTRERAS, RICKY M W U REPORTED ARMED, FAILURE TO COMPLY, RESIST ARRE... MAKE ARREST HANDCUFFING <NA> NONE NONE <NA> HANDS/FEET HANDS/FEET <NA> EMS AT SCENE YES YES D. KREBS / 1508 970

298 rows × 25 columns

Combining the results for every page

If you provide a list of pages, guides can extract the tables from each of them. It will also do nice things like automatically remove duplicate column headers without you even asking!

df = guides.extract_table(pdf.pages).to_df()
print("You found", len(df), "rows")

df.tail()
Output
You found 970 rows
OFFICER DATE OF REPORT DATE OF FORCE TIME OF FORCE PRECINCT CASE # SUBJECT PERSON SEX RACE ETHNICITY SUBJECT CONDITION LEADING TO FORCE REASON FOR USE OF FORCE OFFICER ACTIVITY PRIOR TO FORCE SUBJECT WAS INJURED OFFICER INJURIES SUBJECT INJURIES MENTAL HEALTH INDICATORS FORCE OPTIONS ATTEMPTED EFFECTIVE FORCE OPTIONS TREATMENT RECEIVED OFFICER TREATMENT RECEIVED SUBJECT SUPERVISOR NOTIFIED SUPERVISOR AT SCENE SUPERVISOR NAME CASE COUNT
965 ALDRIDGE, SANDRA 7/30/2019 7/29/2019 2:50 VPD EAST 23-2019-12373 NICHOLAS CALLAWAY <NA> <NA> <NA> FAILURE TO COMPLY, RESIST ARREST, ALCOHOL MAKE ARREST HANDCUFFING PRIOR TO INVOLVMENT NONE LACERATIONS MENTAL ILLNESS, PRIOR KNOWLEDGE HANDS/FEET HANDS/FEET <NA> EMS AT SCENE, HOSPITAL/ADMITTED, HOSPITAL/RELE... REATMENT REFUSED YES YES BARTON/1551 970
966 RIEDEL, BRANDON 7/30/2019 7/30/2019 3:00 EAST PRECINCT 23-2019-12373 CALLAWAY, NICHOLAS RICHARD M U N FAILURE TO COMPLY, RESIST ARREST, SELF INJURY DEFEND SELF, MAKE ARREST, PREVENT ESCAPE HANDCUFFING DURING ARREST <NA> LACERATIONS <NA> <NA> HANDS/FEET <NA> EMS AT SCENE, HOSPITAL/RELEASED YES YES SERGEANT BARTON/1551 970
967 SCOTT, MICHAEL 7/30/2019 7/30/2019 2:51 EAST 23-2019-12373 CALLAWAY, NICHOLAS M U N FAILURE TO COMPLY, RESIST ARREST, ALCOHOL, SEL... MAKE ARREST HANDCUFFING <NA> NONE NONE <NA> CONTROL HOLDS CAUSING COMPLAINT OF INJURY CONTROL HOLDS CAUSING COMPLAINT OF INJURY <NA> EMS AT SCENE, HOSPITAL/RELEASED YES YES BARTON / 1551 970
968 VIGIL, NICOLE 7/30/2019 7/30/2019 5 MIN EAST 23-2019-12373 CALLAWAY, NICHOLAS M U N FAILURE TO COMPLY, RESIST ARREST, DRUGS MAKE ARREST, PREVENT ESCAPE OTHER PRIOR TO INVOLVMENT NONE ABRASIONS, LACERATIONS <NA> <NA> <NA> <NA> EMS AT SCENE, HOSPITAL/RELEASED YES YES BARTON / 1551 970
969 GIBSON, AARON 8/1/2019 Jun-24-2019 21:55 WEST 23-2019-10327 BLEVINS, ANDREW S (DOB: Apr-20-1990) M W U FAILURE TO COMPLY, RESIST ARREST OTHER OTHER <NA> NONE NONE <NA> HANDS/FEET HANDS/FEET <NA> <NA> YES YES A GIBSON / 1154 970