Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix error when retrieving GeoZones #2570

Closed
wants to merge 4 commits into from
Closed

Fix error when retrieving GeoZones #2570

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9981b81
fix error when retrieving GeoZones
mdamien Nov 8, 2020
c433dec
add changelog entry
mdamien Nov 8, 2020
10e0ef8
retrieve geozones the chunk way
mdamien Nov 10, 2020
6f45fa9
use tempfile to retrieve geozones
mdamien Nov 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
- Escaping XML's forbidden characters [#2562](https://github.com/opendatateam/udata/pull/2562)
- Ignore pattern feature for linkchecker [#2564](https://github.com/opendatateam/udata/pull/2564)
- Fix TypeError when creating a superuser with an incorrect password [#2567](https://github.com/opendatateam/udata/pull/2567)
- Fix error when retrieving GeoZones [#2570](https://github.com/opendatateam/udata/pull/2570)

## 2.4.0 (2020-10-16)

Expand Down
11 changes: 7 additions & 4 deletions udata/core/spatial/commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import shutil
import signal
import sys
import tempfile

from collections import Counter
from contextlib import contextmanager
Expand All @@ -31,7 +32,6 @@


DEFAULT_GEOZONES_FILE = 'https://github.com/etalab/geozones/releases/download/2019.0/geozones-countries-2019-0-msgpack.tar.xz'
GEOZONE_FILENAME = 'geozones.tar.xz'


def level_ref(level):
Expand Down Expand Up @@ -98,8 +98,6 @@ def load_zones(col, path):
def cleanup(prefix):
log.info('Removing temporary files')
tmp.delete(prefix)
if tmp.exists(GEOZONE_FILENAME): # Has been downloaded
tmp.delete(GEOZONE_FILENAME)


@contextmanager
Expand Down Expand Up @@ -139,16 +137,21 @@ def load(filename=DEFAULT_GEOZONES_FILE, drop=False):
'''
ts = datetime.now().isoformat().replace('-', '').replace(':', '').split('.')[0]
prefix = 'geozones-{0}'.format(ts)
file = None
if filename.startswith('http'):
log.info('Downloading GeoZones bundle: %s', filename)
filename, _ = urlretrieve(filename, tmp.path(GEOZONE_FILENAME))
file = tempfile.NamedTemporaryFile()
filename, _ = urlretrieve(filename, file.name)

log.info('Extracting GeoZones bundle')
with handle_error(prefix):
with contextlib.closing(lzma.LZMAFile(filename)) as xz:
with tarfile.open(fileobj=xz) as f:
f.extractall(tmp.path(prefix))

if file:
file.close()

log.info('Loading GeoZones levels')

log.info('Loading levels.msgpack')
Expand Down