Skip to content

Commit 4b4e38f

Browse files
rgreinhomergify[bot]
authored andcommitted
Prepare for ScrAPD 3 release. (scrapd#215)
Update the Changelog and the documentation to reflect all the changes brought by ScrAPD 3.
1 parent efdd100 commit 4b4e38f

File tree

2 files changed

+139
-75
lines changed

2 files changed

+139
-75
lines changed

CHANGELOG.md

+22
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
### Added
11+
12+
- Add CLI flag to dump reports with parsing errors on disk. This also adds helper functions and tests to simplify the
13+
debuging of the dumps. [#214]
14+
15+
### Changed
16+
17+
- The internal format was updated to allow automated data validation, prevent invalid values to injected in the data
18+
and handle multiple fatalities in one crash. New fields were added such as `middle name` or `generation`. [#199]
19+
20+
### Fixed
21+
22+
- Fixed parsing issues:
23+
- Fix parsing of `Location` field where its content ws merged with the next field.[#203]
24+
- Handle reports containing an `Arrested` field.[#204]
25+
- Handle `born` as a marker to extract the deceased date of birth. [#208]
26+
- Fix `Date` parsing regex. [#210]
27+
1028
## [[2.1.1]] - 2019-08-22
1129

1230
### Fixed
@@ -191,3 +209,7 @@ This first version allows a user to retrieve traffic fatality repports for a cer
191209
[#171]: https://github.com/scrapd/scrapd/pull/171
192210
[#172]: https://github.com/scrapd/scrapd/pull/172
193211
[#178]: https://github.com/scrapd/scrapd/pull/178
212+
[#203]: https://github.com/scrapd/scrapd/issues/203
213+
[#204]: https://github.com/scrapd/scrapd/issues/204
214+
[#208]: https://github.com/scrapd/scrapd/pull/208
215+
[#210]: https://github.com/scrapd/scrapd/pull/210

README.rst

+117-75
Original file line numberDiff line numberDiff line change
@@ -55,36 +55,50 @@ in `json`::
5555

5656
[
5757
{
58-
"Age": 58,
59-
"Case": "19-0161105",
60-
"DOB": "02/15/1960",
61-
"Date": "01/16/2019",
62-
"Ethnicity": "White",
63-
"Fatal crashes this year": "2",
64-
"First Name": "Ann",
65-
"Gender": "female",
66-
"ID": "19-0161105-0",
67-
"Last Name": "Bottenfield-Seago",
68-
"Link": "http://austintexas.gov/news/traffic-fatality-2-3",
69-
"Location": "West William Cannon Drive and Ridge Oak Road",
70-
"Notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed. APD is investigating this case. Anyone with information regarding this case should call APD\u2019s Vehicular Homicide Unit Detectives at (512) 974-3761. You can also submit tips by downloading APD\u2019s mobile app, Austin PD, for free on iPhone and Android. This is Austin\u2019s second fatal traffic crash of 2018, resulting in two fatalities this year. At this time in 2018, there were two fatal traffic crashes and three traffic fatalities. These statements are based on the initial assessment of the fatal crash and investigation is still pending. Fatality information may change.",
71-
"Time": "3:42 p.m."
58+
"case": "19-0161105",
59+
"crash": 2,
60+
"date": "2019-01-16",
61+
"fatalities": [
62+
{
63+
"age": 58,
64+
"dob": "1960-02-15",
65+
"ethnicity": "White",
66+
"first": "Ann",
67+
"gender": "Female",
68+
"generation": "",
69+
"last": "Bottenfield-Seago",
70+
"middle": ""
71+
}
72+
],
73+
"latitude": 0.0,
74+
"link": "http://austintexas.gov/news/traffic-fatality-2-3",
75+
"location": "West William Cannon Drive and Ridge Oak Road",
76+
"longitude": 0.0,
77+
"notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed.",
78+
"time": "15:42:00"
7279
},
7380
{
74-
"Age": 31,
75-
"Case": "19-0150158",
76-
"DOB": "07/09/1987",
77-
"Date": "01/15/2019",
78-
"Ethnicity": "White",
79-
"Fatal crashes this year": "1",
80-
"First Name": "Hilburn",
81-
"Gender": "male",
82-
"ID": "19-0150158-0",
83-
"Last Name": "Sell",
84-
"Link": "http://austintexas.gov/news/traffic-fatality-1-4",
85-
"Location": "10500 block of N IH 35 SB",
86-
"Notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed. APD is investigating this case. Anyone with information regarding this case should call APD\u2019s Vehicular Homicide Unit Detectives at (512) 974-3761. You can also submit tips by downloading APD\u2019s mobile app, Austin PD, for free on iPhone and Android. This is Austin\u2019s first fatal traffic crash of 2018, resulting in one fatality this year. At this time in 2018, there was one fatal traffic crash and two traffic fatalities. These statements are based on the initial assessment of the fatal crash and investigation is still pending. Fatality information may change.",
87-
"Time": "6:20 a.m."
81+
"case": "19-0150158",
82+
"crash": 1,
83+
"date": "2019-01-15",
84+
"fatalities": [
85+
{
86+
"age": 31,
87+
"dob": "1987-07-09",
88+
"ethnicity": "White",
89+
"first": "David",
90+
"gender": "Male",
91+
"generation": "",
92+
"last": "Sell",
93+
"middle": "Hilburn"
94+
}
95+
],
96+
"latitude": 0.0,
97+
"link": "http://austintexas.gov/news/traffic-fatality-1-4",
98+
"location": "10500 block of N IH 35 SB",
99+
"longitude": 0.0,
100+
"notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.",
101+
"time": "06:20:00"
88102
}
89103
]
90104

@@ -93,9 +107,9 @@ Do the same research but output as CSV::
93107
scrapd --from "Jan 15 2019" --to "Jan 18 2019" --format csv
94108

95109

96-
Fatal crashes this year,Case,Date,Time,Location,First Name,Last Name,Ethnicity,Gender,DOB,Age,Link,Notes
97-
2,19-0161105,01/16/2019,3:42 p.m.,West William Cannon Drive and Ridge Oak Road,Ann,Bottenfield-Seago,White,female,02/15/1960,58,http://austintexas.gov/news/traffic-fatality-2-3,"The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed. APD is investigating this case. Anyone with information regarding this case should call APD’s Vehicular Homicide Unit Detectives at (512) 974-3761. You can also submit tips by downloading APD’s mobile app, Austin PD, for free on iPhone and Android. This is Austin’s second fatal traffic crash of 2018, resulting in two fatalities this year. At this time in 2018, there were two fatal traffic crashes and three traffic fatalities. These statements are based on the initial assessment of the fatal crash and investigation is still pending. Fatality information may change."
98-
1,19-0150158,01/15/2019,6:20 a.m.,10500 block of N IH 35 SB,Hilburn,Sell,White,male,07/09/1987,31,http://austintexas.gov/news/traffic-fatality-1-4,"The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed. APD is investigating this case. Anyone with information regarding this case should call APD’s Vehicular Homicide Unit Detectives at (512) 974-3761. You can also submit tips by downloading APD’s mobile app, Austin PD, for free on iPhone and Android. This is Austin’s first fatal traffic crash of 2018, resulting in one fatality this year. At this time in 2018, there was one fatal traffic crash and two traffic fatalities. These statements are based on the initial assessment of the fatal crash and investigation is still pending. Fatality information may change."
110+
crash,case,date,time,location,first,middle,last,generation,ethnicity,gender,dob,age,link,notes
111+
2,19-0161105,2019-01-16,15:42:00,West William Cannon Drive and Ridge Oak Road,Ann,,Bottenfield-Seago,,White,Female,1960-02-15,58,http://austintexas.gov/news/traffic-fatality-2-3,"The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed."
112+
1,19-0150158,2019-01-15,06:20:00,10500 block of N IH 35 SB,David,Hilburn,Sell,,White,Male,1987-07-09,31,http://austintexas.gov/news/traffic-fatality-1-4,The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.
99113

100114

101115
Retrieve all the traffic fatalities from 2019 (*as of Jan 20th 2019*) in json, and enabling the logging to follow the progress
@@ -106,56 +120,84 @@ of the process::
106120
Fetching page 1...
107121
Fetching page 2...
108122
Fetching page 3...
123+
Fetching page 4...
124+
Fetching page 5...
125+
Fetching page 6...
126+
Fetching page 7...
127+
Fetching page 8...
128+
Fetching page 9...
129+
Fetching page 10...
130+
Fetching page 11...
109131
Total: 3
110-
111132
[
112133
{
113-
"Age": 15,
114-
"Case": "19-0081623",
115-
"DOB": "02/18/2003",
116-
"Date": "01/08/2019",
117-
"Ethnicity": "Hispanic",
118-
"Fatal crashes this year": "3",
119-
"First Name": "Jesus",
120-
"Gender": "male",
121-
"ID": "19-0081623-0",
122-
"Last Name": "Servantez",
123-
"Link": "http://austintexas.gov/news/traffic-fatality-3-4",
124-
"Location": "3600 block of South Capital of Texas Highway SB",
125-
"Notes": "The preliminary investigation shows that the driver of a silver, 2018 KIA was traveling in the center lane of the 3600 block of South Capital of Texas Highway SB when the car collided with Jesus Servantez, a pedestrian in the roadway. The driver remained at the scene and told investigators that he did not see Servantez prior to the crash. Jesus Servantez was transported to Saint David\u2019s South Austin Medical Center where he succumbed to his injuries on January 21, 2019. No charges are expected to be filed. APD is investigating this case. Anyone with information regarding this case should call APD\u2019s Vehicular Homicide Unit Detectives at (512) 974-3761. You can also submit tips by downloading APD\u2019s mobile app, Austin PD, for free on iPhone and Android. This is Austin\u2019s third fatal traffic crash of 2018, resulting in three fatalities this year. At this time in 2018, there were two fatal traffic crashes and three traffic fatalities. These statements are based on the initial assessment of the fatal crash and investigation is still pending. Fatality information may change.",
126-
"Time": "9:37 p.m."
134+
"case": "19-0081623",
135+
"crash": 3,
136+
"date": "2019-01-08",
137+
"fatalities": [
138+
{
139+
"age": 15,
140+
"dob": "2003-02-18",
141+
"ethnicity": "Hispanic",
142+
"first": "Jesus",
143+
"gender": "Male",
144+
"generation": "",
145+
"last": "Servantez",
146+
"middle": ""
147+
}
148+
],
149+
"latitude": 0.0,
150+
"link": "http://austintexas.gov/news/traffic-fatality-3-4",
151+
"location": "3600 block of South Capital of Texas Highway SB",
152+
"longitude": 0.0,
153+
"notes": "The preliminary investigation shows that the driver of a silver, 2018 KIA was traveling in the center lane of the 3600 block of South Capital of Texas Highway SB when the car collided with Jesus Servantez, a pedestrian in the roadway. The driver remained at the scene and told investigators that he did not see Servantez prior to the crash.\n\n\tJesus Servantez was transported to Saint David\u2019s South Austin Medical Center where he succumbed to his injuries on January 21, 2019. No charges are expected to be filed.",
154+
"time": "21:37:00"
127155
},
128156
{
129-
"Age": 58,
130-
"Case": "19-0161105",
131-
"DOB": "02/15/1960",
132-
"Date": "01/16/2019",
133-
"Ethnicity": "White",
134-
"Fatal crashes this year": "2",
135-
"First Name": "Ann",
136-
"Gender": "female",
137-
"ID": "19-0161105-0",
138-
"Last Name": "Bottenfield-Seago",
139-
"Link": "http://austintexas.gov/news/traffic-fatality-2-3",
140-
"Location": "West William Cannon Drive and Ridge Oak Road",
141-
"Notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed. APD is investigating this case. Anyone with information regarding this case should call APD\u2019s Vehicular Homicide Unit Detectives at (512) 974-3761. You can also submit tips by downloading APD\u2019s mobile app, Austin PD, for free on iPhone and Android. This is Austin\u2019s second fatal traffic crash of 2018, resulting in two fatalities this year. At this time in 2018, there were two fatal traffic crashes and three traffic fatalities. These statements are based on the initial assessment of the fatal crash and investigation is still pending. Fatality information may change.",
142-
"Time": "3:42 p.m."
157+
"case": "19-0161105",
158+
"crash": 2,
159+
"date": "2019-01-16",
160+
"fatalities": [
161+
{
162+
"age": 58,
163+
"dob": "1960-02-15",
164+
"ethnicity": "White",
165+
"first": "Ann",
166+
"gender": "Female",
167+
"generation": "",
168+
"last": "Bottenfield-Seago",
169+
"middle": ""
170+
}
171+
],
172+
"latitude": 0.0,
173+
"link": "http://austintexas.gov/news/traffic-fatality-2-3",
174+
"location": "West William Cannon Drive and Ridge Oak Road",
175+
"longitude": 0.0,
176+
"notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed.",
177+
"time": "15:42:00"
143178
},
144179
{
145-
"Age": 31,
146-
"Case": "19-0150158",
147-
"DOB": "07/09/1987",
148-
"Date": "01/15/2019",
149-
"Ethnicity": "White",
150-
"Fatal crashes this year": "1",
151-
"First Name": "Hilburn",
152-
"Gender": "male",
153-
"ID": "19-0150158-0",
154-
"Last Name": "Sell",
155-
"Link": "http://austintexas.gov/news/traffic-fatality-1-4",
156-
"Location": "10500 block of N IH 35 SB",
157-
"Notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed. APD is investigating this case. Anyone with information regarding this case should call APD\u2019s Vehicular Homicide Unit Detectives at (512) 974-3761. You can also submit tips by downloading APD\u2019s mobile app, Austin PD, for free on iPhone and Android. This is Austin\u2019s first fatal traffic crash of 2018, resulting in one fatality this year. At this time in 2018, there was one fatal traffic crash and two traffic fatalities. These statements are based on the initial assessment of the fatal crash and investigation is still pending. Fatality information may change.",
158-
"Time": "6:20 a.m."
180+
"case": "19-0150158",
181+
"crash": 1,
182+
"date": "2019-01-15",
183+
"fatalities": [
184+
{
185+
"age": 31,
186+
"dob": "1987-07-09",
187+
"ethnicity": "White",
188+
"first": "David",
189+
"gender": "Male",
190+
"generation": "",
191+
"last": "Sell",
192+
"middle": "Hilburn"
193+
}
194+
],
195+
"latitude": 0.0,
196+
"link": "http://austintexas.gov/news/traffic-fatality-1-4",
197+
"location": "10500 block of N IH 35 SB",
198+
"longitude": 0.0,
199+
"notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.",
200+
"time": "06:20:00"
159201
}
160202
]
161203

@@ -172,7 +214,7 @@ Combining these two methods provides a high degree of confidence in the parsing
172214

173215
Some statistics:
174216

175-
* **100% of entries detected**
217+
* **99% of entries detected**
176218
* 95% of entries correctly parsed
177219

178220
* 5% of entries failed the parsing only **partially**

0 commit comments

Comments
 (0)