Data pipeline to import events from Ville de Laval to Artsdata.
STOPPED 2025-03-12: Ville de Laval publishes events in a file called calendrier-des-activites on the Donnees Quebec portal.
The script in this repository does the following:
- Download the JSON file calendrier-activites.json from Donnees Québec.
- Convert the JSON to an RDF Graph using the OntoRefine Docker image with config.json that contains the mapping of JSON to RDF.
- Get a list of event places from the calendrier-activites CSV on Donnees Québec and then crawl the URL of each place to extract the JSON-LD and add it to the data to be uploaded.
- Commit the RDF graph in Github in dumps/entities.ttl. This is versioned by Github.
- Call the Artsdata Databus to register a new version of the artifact "calendrier-activites"
- Mint Artsdata URIs for events and places for use in the Signé Laval calendar upcoming-cultural-events.sparql.
The events to be sent to Signé Laval's CMS must meet the following criteria:
- event is in atleast one of 3 categories: Bibliothèques, Expositions et spectacles, Événements et festivals
- event is NOT tagged "Enseignants et étudiants"
- event has a single startDate and location
- location is not a webinar/online
Use this URL to view the data for 5 events that will be sent to CMS.
Description of the directories and the files they contain.
- File from Données Québec in JSON
- File of data converted to RDF turtle
The mapping directory contains the OntoRefine mappings in JSON to convert to RDF and the place mapping in turtle.
- replace-blank-nodes.sparql assigns a place URI to all schema:Place and subtypes including schema:LocalBusiness.
- add-derived-from.sparql adds the webpage to the Place for reference
This data pipeline assumes that event URIs can be consistently generated using the page url + date + time(hour,mintues) https://www.laval.ca/Pages/Fr/Calendrier/rencontrez-votre-elue-louise-lortie.aspx#2023-06-14T16-00
To edit RDF mapping:
- Run ./run_ontorefine.sh (if needed chmod +x run_ontorefine.sh)
- Open localhost:7333 to use Open Refine workbench
- Open the existing project
- Click "Edit RDF Mapping"
- When done making changes save RDF Mapping
- Clean up the history (tab undo/redo) to keep only the steps needed.
- Export > Export project configurations into Github /ontorefine/config.json
- In the summer of 2023 the City of Laval fixed their export of JSON event data which is sent to the Données Quebec Portal.
- 2023-07-18: Culture Creates test loaded the CSV from Données Quebec and sent the event counts per category to Signé Laval with the question of which categories to include.
- 2023-08-16: Signé Laval received the answers from the City of Laval confirming the 3 categories of Events:
- Bibliothèques (http://laval.ca/rdf/52ff792f-a72e-4b2b-859e-2b9faf17491a)
- Événements et festivals (http://laval.ca/rdf/685b3eae-7097-4a44-8e72-1c769ed8f6cf)
- Expositions et spectacles (http://laval.ca/rdf/03e3e7d7-ad9a-4f10-86db-dfc9d49b7abb)
- 2023-09-11: Culture Creates starting loading City of Laval events into Artsdata.
- 2023-09-18: Culture Creates confirms that the feed from Ville de Laval is broken.
- 2023-11-07: Culture Creates confirms that the feed from Ville de Laval is starting to work again.
- 2023-12-04: Culture Creates resumes weekly imports to Artsdata.
- 2024-03-15: Culture Creates automates pull from Donnees Quebec using this repo's workflow
- 2025-03-12: Raynald Girard, Chef de division, Données/IA-GDE, Ville de Laval annonces via email that the new laval.ca site is lanched and they are no longer sending data to Données Quebec Portal.
The events are loaded into Footlight CMS on a schedule located in the Footlight Aggregator workflow.
Future improvement: Check date of event within 23 hrs instead of truncating dateTime.