From 0453656856310f16ea7749855e0af36c5352d1a1 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 20 Sep 2017 17:39:17 -0400 Subject: [PATCH 001/172] Add API Auth ADR - ADR on how the client app should authenticate its anonymous requests --- doc/arch/adr-005-api-client-app-auth.md | 127 ++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 doc/arch/adr-005-api-client-app-auth.md diff --git a/doc/arch/adr-005-api-client-app-auth.md b/doc/arch/adr-005-api-client-app-auth.md new file mode 100644 index 000000000..837d9613e --- /dev/null +++ b/doc/arch/adr-005-api-client-app-auth.md @@ -0,0 +1,127 @@ +# Geoprocessing API: Only Allow Anonymous Requests From The Client App + +## Context + +We're creating a [proper, publicly documented API](https://github.com/WikiWatershed/model-my-watershed/blob/develop/src/mmw/apps/geoprocessing_api/views.py) around our analyze and +RWD endpoints: the [Geoprocessing API.](https://github.com/WikiWatershed/model-my-watershed/blob/develop/doc/arch/adr-004-geoprocessing-api.md) The analyze and RWD endpoints have +always had allow-any access; we'd now like to secure them via token authentication +so that we can better track individual users, prevent users from sending off too many +requests, revoke the access of problematic users, and some day maybe add a +paid tier of use. + +Enforcing token authentication on these previously open endpoints causes +a problem for our client app; the client app consumes the same API, and +want to continue to allow unauthenticated users access to these endpoints +via the app. We need a way to identify the client app as a special case +from which the app should allow unrestricted requests. + +Because everything we send to and from the client is exposed to the app user, +any mechanism we give the client app to identify itself will be fairly +easy to uncover and then fake by anyone trying to get un-checked access to the +API. With this in mind our solution should prioritize not be overly complicated/difficult +for us when we know there's no truly secure solution (barring doing all our rendering +server-side). + +Some possibilities are: + +1. Give each environment of the client app its own API token + - Keeps a single system for API authentication; easy to reason about; + as a bonus will allow us to cycle the client app's token if someone + decides to start using it + +1. Set a special, is-client-app flag that the API would check to determine if authentication was necessary + - To set the flag we'd have to know the request was from the client app in + the first place. Preliminary checks for doing this via `http_referer` + were unsuccessful, which would leave doing this via custom header (or some + other part of the request). Setting a `X-IsClientApp` custom header might + be overly naive or hacky, and could result in a lot of conditional logic + +1. Do (1) and also enforce the `HTTP_REFERER` header be from an expected domain. +[Google Maps ](https://developers.google.com/maps/documentation/javascript/get-api-key#key-restrictions) +does this. Even though such headers are easily spoofed by a client outside of a browser, +they're fairly effective for browser-based apps and for naive use from a non-browser client + +## Decision + +**Give the client app its own API token** + +Setting up the client app with its own API token will result a cleaner +architecture for the API. All users use a token, no exceptions. In addition to +the simpler mental model the API token method provides, we'll also gain an added +level of token enforcement; if some API user tries to programmatically use the client +app's token instead of their own, we'll be able to easily cycle the client app's token +to de-incentivize them. + +One easy way to cycle the token: +``` +./manage.py drf_create_token -r +``` + +We can always implement (3) as an enhancement later on. + +##### When should the client app send its token + +For all Geoprocessing API requests, whether there's a logged-in user's token available +to use or not, ie., we will continue to use a user's credentials when needed for the +application endpoints. + + +##### How the client app should get the token + +If we hard-code the token on the frontend (or anywhere), we'll have to deploy any time we need +to cycle the token. We should instead pass the token to the client app via `clientSettings`. + +##### How the server should get the token + +To get the token to put in the client's settings, the server can look up the +token from the `authtoken_token` table via the client app's `user_id`. +The `user_id` should remain secret so that there's no programmatic way get +the token outside of swiping it off the client app. + +## Consequences + +#### Backdoor to the API +As discussed in [Context](#Context), allowing the client app to make requests without a user creates a +backdoor into the API. This is acceptable because up until now we've allowed unrestricted API access +(we just haven't advertised the API as something you could use.) There's also no sensitive data available +via the API. While it's not ideal that someone could use the client app token to make unlimited, +resource-intensive requests unchecked the decision will at least allow us to cycle the token whenever +there's an issue. + +#### More Complex Throttling + +We want to throttle users of the API, and if the client app is just another, regular user +of the API with a token it would get throttled the same. All guest MMW users in the world +would share a cumalitive number of requests per minute. + +To fix this we'll need to treat the client app's token as a special case for throttling: + +if token is client app: + +don't throttle or cache number of requests per IP +(like [AnonRateThrottle](http://www.django-rest-framework.org/api-guide/throttling/#anonratethrottle)). +There may be difficulties with this related to which IP DRF uses; +it may use that of our load balancer instead of the client's. This also may +cause issues for our classroom users if set too low. + +else: + +cache number of requests per token (like [UserRateThrottle](http://www.django-rest-framework.org/api-guide/throttling/#userratethrottle)) + +#### Dev Setup + +The `user_id` the app server uses to get the client's token should be fairly +constant for the life of staging and production. Each developer's machine, +however, will have different database instances and, therefore, different +client app `user_id`'s. + +We could: +1. Use some special email or name like `clientapp09212017` for the app server to look up the `user_id` to get the token. This would add a level +of indirection, but would allow production, staging and our local machines +to share a migration that creates the user, and share code to look up the id. +There's precedence for this setup at Azavea in Raster Foundry's airflow user. + +1. Store the `user_id` itself as an environment variable. A migration could +create the user and write its id to an envfile. We would have to include the envfile in our `.gitignore`, and be careful not to tamper with it. + + From 421c1129a03b06884491bc14617c0802ae1b9ecc Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Fri, 22 Sep 2017 13:45:54 -0400 Subject: [PATCH 002/172] Upgrade Celery --- .../ansible/roles/model-my-watershed.celery/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deployment/ansible/roles/model-my-watershed.celery/defaults/main.yml b/deployment/ansible/roles/model-my-watershed.celery/defaults/main.yml index 4e9c8919a..7bd8d06c5 100644 --- a/deployment/ansible/roles/model-my-watershed.celery/defaults/main.yml +++ b/deployment/ansible/roles/model-my-watershed.celery/defaults/main.yml @@ -1,3 +1,3 @@ --- -celery_version: 3.1.18 +celery_version: 4.1.0 djcelery_version: 3.1.16 From bf7e9d0590ff418c68f9ca03361c68916256abf7 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Fri, 22 Sep 2017 17:10:18 -0400 Subject: [PATCH 003/172] Remove unsupported --autoreload command This was experimental and is no longer supported. Furthermore, I doubt it ever worked at all. --- scripts/debugcelery.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/debugcelery.sh b/scripts/debugcelery.sh index 7c9bbab3a..5a32d1c0e 100755 --- a/scripts/debugcelery.sh +++ b/scripts/debugcelery.sh @@ -6,6 +6,6 @@ set -e STOP_SERVICE="(sudo service celeryd stop || /bin/true)" CHANGE_DIR="cd /opt/app/" -RUN_CELERY="envdir /etc/mmw.d/env celery -A 'mmw.celery:app' worker --autoreload -l debug -n debug@%n" +RUN_CELERY="envdir /etc/mmw.d/env celery -A 'mmw.celery:app' worker -l debug -n debug@%n" vagrant ssh worker -c "$STOP_SERVICE && $CHANGE_DIR && $RUN_CELERY" From 7e1b02e98b24a18edfc3f4698410e370f62d236b Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Fri, 22 Sep 2017 18:05:00 -0400 Subject: [PATCH 004/172] Upgrade Celery configuration * Remove old djcelery in favor of new django_celery_results * Remove unsupported celery.task.http * Prefix all Celery settings with CELERY_ and declare that prefix in the config_from_object call * Remove chord_unlock rate limiting hack --- .../defaults/main.yml | 1 - .../model-my-watershed.celery/tasks/main.yml | 1 - src/mmw/apps/modeling/tests.py | 8 +- src/mmw/mmw/celery.py | 78 +------------------ src/mmw/mmw/settings/base.py | 14 ++-- src/mmw/requirements/base.txt | 1 + 6 files changed, 14 insertions(+), 89 deletions(-) diff --git a/deployment/ansible/roles/model-my-watershed.celery/defaults/main.yml b/deployment/ansible/roles/model-my-watershed.celery/defaults/main.yml index 7bd8d06c5..7d8dddc3c 100644 --- a/deployment/ansible/roles/model-my-watershed.celery/defaults/main.yml +++ b/deployment/ansible/roles/model-my-watershed.celery/defaults/main.yml @@ -1,3 +1,2 @@ --- celery_version: 4.1.0 -djcelery_version: 3.1.16 diff --git a/deployment/ansible/roles/model-my-watershed.celery/tasks/main.yml b/deployment/ansible/roles/model-my-watershed.celery/tasks/main.yml index 82a6c4ea6..26213b088 100644 --- a/deployment/ansible/roles/model-my-watershed.celery/tasks/main.yml +++ b/deployment/ansible/roles/model-my-watershed.celery/tasks/main.yml @@ -3,4 +3,3 @@ pip: name="{{ item.name }}" version={{ item.version }} state=present with_items: - { name: "celery[redis]", version: "{{ celery_version }}" } - - { name: "django-celery", version: "{{ djcelery_version }}" } diff --git a/src/mmw/apps/modeling/tests.py b/src/mmw/apps/modeling/tests.py index af0a286a0..f43f468fb 100644 --- a/src/mmw/apps/modeling/tests.py +++ b/src/mmw/apps/modeling/tests.py @@ -202,7 +202,7 @@ def setUp(self): status='started') self.job.save() - @override_settings(CELERY_ALWAYS_EAGER=True) + @override_settings(CELERY_TASK_ALWAYS_EAGER=True) def test_tr55_job_runs_in_chain(self): # For the purposes of this test, there are no modifications self.model_input['modification_pieces'] = [] @@ -228,7 +228,7 @@ def test_tr55_job_runs_in_chain(self): 'complete', 'Job found but incomplete.') - @override_settings(CELERY_ALWAYS_EAGER=True) + @override_settings(CELERY_TASK_ALWAYS_EAGER=True) def test_tr55_job_error_in_chain(self): model_input = { 'inputs': [], @@ -401,7 +401,7 @@ def test_tr55_chain_doesnt_generate_aoi_census_if_it_exists_and_no_mods(self): else False for t in needed_tasks]), 'missing necessary job in chain') - @override_settings(CELERY_ALWAYS_EAGER=True) + @override_settings(CELERY_TASK_ALWAYS_EAGER=True) def test_tr55_chain_generates_modification_censuses_if_they_are_old(self): """If they modification censuses exist in the model input, but the hash stored with the censuses does not match the hash passed in @@ -455,7 +455,7 @@ def test_tr55_chain_generates_modification_censuses_if_they_are_old(self): else False for t in needed_tasks]), 'missing necessary job in chain') - @override_settings(CELERY_ALWAYS_EAGER=True) + @override_settings(CELERY_TASK_ALWAYS_EAGER=True) def test_tr55_chain_generates_both_censuses_if_they_are_missing(self): """If neither the AoI censuses or the modification censuses exist, they are both generated. diff --git a/src/mmw/mmw/celery.py b/src/mmw/mmw/celery.py index 3a94ea79a..8bf22279e 100644 --- a/src/mmw/mmw/celery.py +++ b/src/mmw/mmw/celery.py @@ -2,87 +2,13 @@ import os import rollbar -import logging from celery import Celery -from celery._state import connect_on_app_finalize from celery.signals import task_failure from django.conf import settings -@connect_on_app_finalize -def add_unlock_chord_task_shim(app): - """ - Override native unlock_chord to support configurable max_retries. - Original code taken from https://goo.gl/3mX0ie - - This task is used by result backends without native chord support. - It joins chords by creating a task chain polling the header for completion. - """ - from celery.canvas import maybe_signature - from celery.exceptions import ChordError - from celery.result import allow_join_result, result_from_tuple - - logger = logging.getLogger(__name__) - - MAX_RETRIES = settings.CELERY_CHORD_UNLOCK_MAX_RETRIES - - @app.task(name='celery.chord_unlock', shared=False, default_retry_delay=1, - ignore_result=True, lazy=False, bind=True, - max_retries=MAX_RETRIES) - def unlock_chord(self, group_id, callback, interval=None, - max_retries=MAX_RETRIES, result=None, - Result=app.AsyncResult, GroupResult=app.GroupResult, - result_from_tuple=result_from_tuple, **kwargs): - if interval is None: - interval = self.default_retry_delay - - # check if the task group is ready, and if so apply the callback. - callback = maybe_signature(callback, app) - deps = GroupResult( - group_id, - [result_from_tuple(r, app=app) for r in result], - app=app, - ) - j = deps.join_native if deps.supports_native_join else deps.join - - try: - ready = deps.ready() - except Exception as exc: - raise self.retry( - exc=exc, countdown=interval, max_retries=max_retries) - else: - if not ready: - raise self.retry(countdown=interval, max_retries=max_retries) - - callback = maybe_signature(callback, app=app) - try: - with allow_join_result(): - ret = j(timeout=3.0, propagate=True) - except Exception as exc: - try: - culprit = next(deps._failed_join_report()) - reason = 'Dependency {0.id} raised {1!r}'.format( - culprit, exc, - ) - except StopIteration: - reason = repr(exc) - logger.error('Chord %r raised: %r', group_id, exc, exc_info=1) - app.backend.chord_error_from_stack(callback, - ChordError(reason)) - else: - try: - callback.delay(ret) - except Exception as exc: - logger.error('Chord %r raised: %r', group_id, exc, exc_info=1) - app.backend.chord_error_from_stack( - callback, - exc=ChordError('Callback error: {0!r}'.format(exc)), - ) - return unlock_chord - - # set the default Django settings module for the 'celery' program. os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mmw.settings.production') @@ -91,8 +17,8 @@ def unlock_chord(self, group_id, callback, interval=None, # Using a string here means the worker will not have to # pickle the object when using Windows. -app.config_from_object('django.conf:settings') -app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) +app.config_from_object('django.conf:settings', namespace='CELERY') +app.autodiscover_tasks() rollbar_settings = getattr(settings, 'ROLLBAR', {}) if rollbar_settings: diff --git a/src/mmw/mmw/settings/base.py b/src/mmw/mmw/settings/base.py index 75a962bcf..5ae3fddbe 100644 --- a/src/mmw/mmw/settings/base.py +++ b/src/mmw/mmw/settings/base.py @@ -115,22 +115,21 @@ def get_env_setting(setting): # CELERY CONFIGURATION -BROKER_URL = 'redis://{0}:{1}/2'.format( +CELERY_BROKER_URL = 'redis://{0}:{1}/2'.format( environ.get('MMW_CACHE_HOST', 'localhost'), environ.get('MMW_CACHE_PORT', 6379)) -CELERY_IMPORTS = ('celery.task.http', - # Submodule task is not always autodiscovered - 'apps.modeling.mapshed.tasks', - ) +CELERY_IMPORTS = ( + # Submodule task is not always autodiscovered + 'apps.modeling.mapshed.tasks', +) CELERY_ACCEPT_CONTENT = ['json'] CELERY_TASK_SERIALIZER = 'json' CELERY_RESULT_SERIALIZER = 'json' -CELERY_RESULT_BACKEND = 'djcelery.backends.cache:CacheBackend' +CELERY_RESULT_BACKEND = 'django-cache' STATSD_CELERY_SIGNALS = True CELERY_CREATE_MISSING_QUEUES = True CELERY_CHORD_PROPAGATES = True -CELERY_CHORD_UNLOCK_MAX_RETRIES = 60 CELERY_DEFAULT_QUEUE = STACK_COLOR CELERY_DEFAULT_ROUTING_KEY = "task.%s" % STACK_COLOR # END CELERY CONFIGURATION @@ -294,6 +293,7 @@ def get_env_setting(setting): 'rest_framework_swagger', 'rest_framework.authtoken', 'registration', + 'django_celery_results', ) # THIRD-PARTY CONFIGURATION diff --git a/src/mmw/requirements/base.txt b/src/mmw/requirements/base.txt index 8508c5a92..8839ad100 100644 --- a/src/mmw/requirements/base.txt +++ b/src/mmw/requirements/base.txt @@ -18,3 +18,4 @@ rollbar==0.13.8 retry==0.9.1 python-dateutil==2.6.0 suds==0.4 +django_celery_results==1.0.1 From 24468c9370af504e2d495c454ef2aad51e2b832b Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Fri, 22 Sep 2017 18:22:44 -0400 Subject: [PATCH 005/172] Don't apply error handlers on chains with groups Previously we could attach an error handler to any Celery chain and have it catch an error in any task of the chain. This did not work with groups, which were ignored, so we had to apply the error handler manually to every task in the group. With the latest version of Celery, the groups are no longer ignored. Instead, we get a runtime error saying that chains with groups cannot have error handlers attached to them. Thus, we remove the error handlers on chains that have groups in them. Since all the group tasks have individual error handler tasks already, and the other tasks are simple data transformations in Python, this should not affect the output. --- src/mmw/apps/geoprocessing_api/views.py | 10 +++++++--- src/mmw/apps/modeling/views.py | 2 +- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/src/mmw/apps/geoprocessing_api/views.py b/src/mmw/apps/geoprocessing_api/views.py index cc5fa7be5..31d7b7775 100644 --- a/src/mmw/apps/geoprocessing_api/views.py +++ b/src/mmw/apps/geoprocessing_api/views.py @@ -936,7 +936,7 @@ def start_analyze_climate(request, format=None): group(geotasks), tasks.combine_climate.s(), tasks.collect_climate.s(), - ], area_of_interest, user) + ], area_of_interest, user, link_error=False) def _initiate_rwd_job_chain(location, snapping, data_source, @@ -948,7 +948,7 @@ def _initiate_rwd_job_chain(location, snapping, data_source, .apply_async(link_error=errback) -def start_celery_job(task_list, job_input, user=None): +def start_celery_job(task_list, job_input, user=None, link_error=True): """ Given a list of Celery tasks and it's input, starts a Celery async job with those tasks, adds save_job_result and save_job_error handlers, and returns @@ -957,6 +957,7 @@ def start_celery_job(task_list, job_input, user=None): :param task_list: A list of Celery tasks to execute. Is made into a chain :param job_input: Input to the first task, used in recording started jobs :param user: The user requesting the job. Optional. + :param link_error: Whether or not to apply error handler to entire chain :return: A Response contianing the job id, marked as 'started' """ created = now() @@ -968,7 +969,10 @@ def start_celery_job(task_list, job_input, user=None): error = save_job_error.s(job.id) task_list.append(success) - task_chain = chain(task_list).apply_async(link_error=error) + if link_error: + task_chain = chain(task_list).apply_async(link_error=error) + else: + task_chain = chain(task_list).apply_async() job.uuid = task_chain.id job.save() diff --git a/src/mmw/apps/modeling/views.py b/src/mmw/apps/modeling/views.py index 823a01aad..88d5f3a3c 100644 --- a/src/mmw/apps/modeling/views.py +++ b/src/mmw/apps/modeling/views.py @@ -234,7 +234,7 @@ def _initiate_mapshed_job_chain(mapshed_input, job_id): collect_data.s(area_of_interest).set(link_error=errback) | save_job_result.s(job_id, mapshed_input)) - return chain(job_chain).apply_async(link_error=errback) + return chain(job_chain).apply_async() @decorators.api_view(['POST']) From 7737d61c9afacd8915490b85a04bac6a260836e4 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 26 Sep 2017 11:04:28 -0400 Subject: [PATCH 006/172] Remove extraneous buffer tasks for chained chords The previous version of Celery had a bug which did not allow a chord to be the antepenultimate task in a chain. To get around this, we added extraneous tasks as buffer to get it to work correctly. The latest version of Celery no longer has this bug, so these tasks are collapsed into their successors. This will reduce Celery overhead as we run fewer tasks. See https://github.com/celery/celery/issues/3191 --- src/mmw/apps/geoprocessing_api/tasks.py | 28 ++++++++----------------- src/mmw/apps/geoprocessing_api/views.py | 1 - src/mmw/apps/modeling/mapshed/tasks.py | 17 +++------------ src/mmw/apps/modeling/views.py | 2 -- 4 files changed, 12 insertions(+), 36 deletions(-) diff --git a/src/mmw/apps/geoprocessing_api/tasks.py b/src/mmw/apps/geoprocessing_api/tasks.py index 653984a51..00fc57fad 100644 --- a/src/mmw/apps/geoprocessing_api/tasks.py +++ b/src/mmw/apps/geoprocessing_api/tasks.py @@ -175,38 +175,28 @@ def analyze_climate(result, category, month): @shared_task -def combine_climate(results): +def collect_climate(results): """ Given an array of dictionaries resulting from multiple analyze_climate calls, combines them so that the 'ppt' values are grouped together and 'tmean' together. Each group is a dictionary where the keys are strings of the month '1', '2', ..., '12', and the values the average in the area of interest. + + Then, transforms these dictionaries into a final result of the format used + for all other Analyze operations. The 'categories' contain twelve objects, + one for each month, with a 'month' field containing the name of the month, + and 'ppt' and 'tmean' fields with corresponding values. The 'index' can be + used for sorting purposes on the client side. """ ppt = {k[5:]: v for r in results for k, v in r.items() if 'ppt' in k} tmean = {k[7:]: v for r in results for k, v in r.items() if 'tmean' in k} - return { - 'ppt': ppt, - 'tmean': tmean, - } - - -@shared_task -def collect_climate(results): - """ - Given the two dictionaries from combine_climate, transforms them into a - final result of the format used for all other Analyze operations. The - 'categories' contain twelve objects, one for each month, with a 'month' - field containing the name of the month, and 'ppt' and 'tmean' fields with - corresponding values. The 'index' can be used for sorting purposes on the - client side. - """ categories = [{ 'monthidx': i, 'month': month_name[i], - 'ppt': results['ppt'][str(i)] * CM_PER_MM, - 'tmean': results['tmean'][str(i)], + 'ppt': ppt[str(i)] * CM_PER_MM, + 'tmean': tmean[str(i)], } for i in xrange(1, 13)] return { diff --git a/src/mmw/apps/geoprocessing_api/views.py b/src/mmw/apps/geoprocessing_api/views.py index 31d7b7775..c93ce1b31 100644 --- a/src/mmw/apps/geoprocessing_api/views.py +++ b/src/mmw/apps/geoprocessing_api/views.py @@ -934,7 +934,6 @@ def start_analyze_climate(request, format=None): return start_celery_job([ group(geotasks), - tasks.combine_climate.s(), tasks.collect_climate.s(), ], area_of_interest, user, link_error=False) diff --git a/src/mmw/apps/modeling/mapshed/tasks.py b/src/mmw/apps/modeling/mapshed/tasks.py index 8061e8250..de59a1b1e 100644 --- a/src/mmw/apps/modeling/mapshed/tasks.py +++ b/src/mmw/apps/modeling/mapshed/tasks.py @@ -45,7 +45,9 @@ @shared_task -def collect_data(geop_result, geojson): +def collect_data(geop_results, geojson): + geop_result = {k: v for r in geop_results for k, v in r.items()} + geom = GEOSGeometry(geojson, srid=4326) area = geom.transform(5070, clone=True).area # Square Meters @@ -491,19 +493,6 @@ def geoprocessing_chains(aoi, wkaoi, errback): ] -@shared_task -def combine(geop_results): - """ - Flattens the incoming results dictionaries into one - which has all the keys of the components. - - This could be a part of collect_data, but we need - a buffer in a chord as a workaround to - https://github.com/celery/celery/issues/3191 - """ - return {k: v for r in geop_results for k, v in r.items()} - - def get_lu_index(nlcd): # Convert NLCD code into MapShed Land Use Index if nlcd == 81: diff --git a/src/mmw/apps/modeling/views.py b/src/mmw/apps/modeling/views.py index 88d5f3a3c..7797d997a 100644 --- a/src/mmw/apps/modeling/views.py +++ b/src/mmw/apps/modeling/views.py @@ -31,7 +31,6 @@ from apps.core.decorators import log_request from apps.modeling import tasks, geoprocessing from apps.modeling.mapshed.tasks import (geoprocessing_chains, - combine, collect_data, ) from apps.modeling.models import Project, Scenario @@ -230,7 +229,6 @@ def _initiate_mapshed_job_chain(mapshed_input, job_id): job_chain = ( group(geoprocessing_chains(area_of_interest, wkaoi, errback)) | - combine.s() | collect_data.s(area_of_interest).set(link_error=errback) | save_job_result.s(job_id, mapshed_input)) From 8adcbc3599fe01374698b0771d0d18562b99adf5 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 26 Sep 2017 12:11:06 -0400 Subject: [PATCH 007/172] Switch to new errback format Previously, a Celery error callback was only given the UUID of the failing job. To retrieve additional information about it, we had to query the app with it. To get the app reference, the task had to be bound to the app. With new Celery, there is an issue with using bound tasks for error handling. See https://github.com/celery/celery/issues/3723 Fortunately, in the new error callback format in Celery 4, the handler is given a request, exception, and traceback as arguments. This obviates the need for an app reference, since these were the values we were fetching from it. Now we use them directly. --- src/mmw/apps/core/tasks.py | 15 +++++++-------- src/mmw/apps/modeling/geoprocessing.py | 2 +- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/src/mmw/apps/core/tasks.py b/src/mmw/apps/core/tasks.py index ed3b958ea..09c90e916 100644 --- a/src/mmw/apps/core/tasks.py +++ b/src/mmw/apps/core/tasks.py @@ -14,8 +14,8 @@ logger = logging.getLogger(__name__) -@shared_task(bind=True) -def save_job_error(self, uuid, job_id): +@shared_task +def save_job_error(request, exc, traceback, job_id): """ A handler task attached to the Celery chain. Any exception thrown along the chain will trigger this task, which logs the failure to the Job row so that @@ -24,15 +24,14 @@ def save_job_error(self, uuid, job_id): To see failing task in Flower, follow instructions here: https://github.com/WikiWatershed/model-my-watershed/pull/551#issuecomment-119333146 """ - result = self.app.AsyncResult(uuid) error_message = 'Task {0} run from job {1} raised exception: {2}\n{3}' - logger.error(error_message.format(str(uuid), str(job_id), - str(result.result), - str(result.traceback))) + logger.error(error_message.format(str(request.id), str(job_id), + str(exc), + str(traceback))) try: job = Job.objects.get(id=job_id) - job.error = result.result - job.traceback = result.traceback + job.error = exc + job.traceback = traceback job.delivered_at = now() job.status = 'failed' job.save() diff --git a/src/mmw/apps/modeling/geoprocessing.py b/src/mmw/apps/modeling/geoprocessing.py index 159dde16a..9307e4b98 100644 --- a/src/mmw/apps/modeling/geoprocessing.py +++ b/src/mmw/apps/modeling/geoprocessing.py @@ -77,7 +77,7 @@ def run(self, opname, input_data, wkaoi=None, cache_key=''): raise r except Exception as x: return { - 'error': x.message + 'error': str(x) } From b7f8f2c0fa343f5f778557c4f5252cc26623582c Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 26 Sep 2017 12:19:28 -0400 Subject: [PATCH 008/172] Simplify error message if geoprocessing down In case the geoprocessing service is unreachable, we now show this simplified message, instead of exposing inner details to the user. Without this, the error message would be something like: [analyze_nlcd] HTTPConnectionPool(host='localhost', port=8090): Max retries exceeded with url: /run (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) --- src/mmw/apps/modeling/geoprocessing.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/mmw/apps/modeling/geoprocessing.py b/src/mmw/apps/modeling/geoprocessing.py index 9307e4b98..4bb62969f 100644 --- a/src/mmw/apps/modeling/geoprocessing.py +++ b/src/mmw/apps/modeling/geoprocessing.py @@ -75,6 +75,10 @@ def run(self, opname, input_data, wkaoi=None, cache_key=''): return result except Retry as r: raise r + except ConnectionError: + return { + 'error': 'Could not reach the geoprocessing service' + } except Exception as x: return { 'error': str(x) From 08e748a372ad34c93620013b592411bf6fc33f4d Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 26 Sep 2017 12:34:46 -0400 Subject: [PATCH 009/172] Reduce geoprocessing max retries Previously, since Spark JobServer would take a long time and much resources to accomplish, sometimes to the point of not accepting new requests, we had to retry a large number of times in hopes of making it in eventually. Since the new geoprocessing service is quite fast, and is built on Akka which more readily accepts new requests even when running processing on background threads, we don't need to retry as much. By reducing excessive retries, we make sure to not clog up the Celery worker capacity. --- src/mmw/apps/modeling/geoprocessing.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mmw/apps/modeling/geoprocessing.py b/src/mmw/apps/modeling/geoprocessing.py index 4bb62969f..a82d28b24 100644 --- a/src/mmw/apps/modeling/geoprocessing.py +++ b/src/mmw/apps/modeling/geoprocessing.py @@ -19,7 +19,7 @@ from django.conf import settings -@shared_task(bind=True, default_retry_delay=1, max_retries=42) +@shared_task(bind=True, default_retry_delay=1, max_retries=6) def run(self, opname, input_data, wkaoi=None, cache_key=''): """ Run a geoprocessing operation. From 89277d6e7282d0b2985bf5782b35cf92efcb85b0 Mon Sep 17 00:00:00 2001 From: Kelly Innes Date: Mon, 25 Sep 2017 17:03:35 -0400 Subject: [PATCH 010/172] Add RWD simplify param to geoprocessing API - document RWD simplify param in Swagger docs - parse RWD simplify param in RWD endpoint - adjust sample RWD response to 5 trailing digits - change example Swagger input point to one which generates a smaller unsimplified shape --- src/mmw/apps/geoprocessing_api/tasks.py | 17 ++++++++++++-- src/mmw/apps/geoprocessing_api/views.py | 31 +++++++++++++++---------- 2 files changed, 34 insertions(+), 14 deletions(-) diff --git a/src/mmw/apps/geoprocessing_api/tasks.py b/src/mmw/apps/geoprocessing_api/tasks.py index 653984a51..c9f5e69c7 100644 --- a/src/mmw/apps/geoprocessing_api/tasks.py +++ b/src/mmw/apps/geoprocessing_api/tasks.py @@ -5,6 +5,7 @@ import os import logging +import urllib from ast import literal_eval as make_tuple from calendar import month_name @@ -35,7 +36,7 @@ @shared_task -def start_rwd_job(location, snapping, data_source): +def start_rwd_job(location, snapping, simplify, data_source): """ Calls the Rapid Watershed Delineation endpoint that is running in the Docker container, and returns @@ -47,9 +48,21 @@ def start_rwd_job(location, snapping, data_source): rwd_url = 'http://%s:%s/%s/%f/%f' % (RWD_HOST, RWD_PORT, end_point, lat, lng) + params = {} + # The Webserver defaults to enable snapping, uses 1 (true) 0 (false) if not snapping: - rwd_url += '?snapping=0' + params['snapping'] = 0 + + # RWD also defaults to simplify the shape according to a tolerance. + # Passing it `?simplify=0` returns the unsimplified result. + if simplify is not False: + params['simplify'] = simplify + + query_string = urllib.urlencode(params) + + if query_string: + rwd_url += ('?%s' % query_string) logger.debug('rwd request: %s' % rwd_url) diff --git a/src/mmw/apps/geoprocessing_api/views.py b/src/mmw/apps/geoprocessing_api/views.py index cc5fa7be5..b3661c5d5 100644 --- a/src/mmw/apps/geoprocessing_api/views.py +++ b/src/mmw/apps/geoprocessing_api/views.py @@ -57,11 +57,17 @@ def start_rwd(request, format=None): `snappingOn` (`boolean`): Snap to the nearest stream? Default is false + `simplify` (`number`): Simplify tolerance for delineated watershed shape in + response. Use `0` to receive an unsimplified shape. When this + parameter is not supplied, `simplify` defaults to `0.0001` for + "drb" and is a function of the shape's area for "nhd". + **Example** { - "location": [39.97185812402583,-75.16742706298828], + "location": [39.67185812402583,-75.76742706298828], "snappingOn": true, + "simplify": 0, "dataSource":"nhd" } @@ -86,12 +92,12 @@ def start_rwd(request, format=None): "coordinates": [ [ [ - -75.24776006176894, - 39.98166667527191 + -75.24776, + 39.98166 ], [ - -75.24711191361516, - 39.98166667527191 + -75.24711, + 39.98166 ] ], ... ] @@ -119,8 +125,8 @@ def start_rwd(request, format=None): "geometry": { "type": "Point", "coordinates": [ - -75.24938043215342, - 39.97875000854888 + -75.24938, + 39.97875 ] }, "type": "Feature", @@ -165,12 +171,13 @@ def start_rwd(request, format=None): location = request.data['location'] data_source = request.data.get('dataSource', 'drb') snapping = request.data.get('snappingOn', False) + simplify = request.data.get('simplify', False) job = Job.objects.create(created_at=created, result='', error='', traceback='', user=user, status='started') - task_list = _initiate_rwd_job_chain(location, snapping, data_source, - job.id) + task_list = _initiate_rwd_job_chain(location, snapping, simplify, + data_source, job.id) job.uuid = task_list.id job.save() @@ -939,12 +946,12 @@ def start_analyze_climate(request, format=None): ], area_of_interest, user) -def _initiate_rwd_job_chain(location, snapping, data_source, +def _initiate_rwd_job_chain(location, snapping, simplify, data_source, job_id, testing=False): errback = save_job_error.s(job_id) - return chain(tasks.start_rwd_job.s(location, snapping, data_source), - save_job_result.s(job_id, location)) \ + return chain(tasks.start_rwd_job.s(location, snapping, simplify, + data_source), save_job_result.s(job_id, location)) \ .apply_async(link_error=errback) From d158b61a705332ed7ffcc62e9aaa4796542496c6 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 27 Sep 2017 10:01:18 -0400 Subject: [PATCH 011/172] Add MMW_CLIENT_APP_USER_PASSWORD Env Variable * We're adding a django user for the client app so that the client app can have its own geoprocessing API token * Add the client app's account's password as an environment variable - Add to ansible mmw default role - Add to cloud formation set up - Add to sample deployment configuration file - Add to Django App settings Refs #2270 --- deployment/ansible/group_vars/all | 2 ++ .../roles/model-my-watershed.base/defaults/main.yml | 1 + deployment/cfn/application.py | 12 +++++++++++- deployment/default.yml.example | 1 + src/mmw/mmw/settings/base.py | 4 ++++ 5 files changed, 19 insertions(+), 1 deletion(-) diff --git a/deployment/ansible/group_vars/all b/deployment/ansible/group_vars/all index b53f6a419..9a695f007 100644 --- a/deployment/ansible/group_vars/all +++ b/deployment/ansible/group_vars/all @@ -14,6 +14,8 @@ stack_color: "Black" itsi_client_id: "model-my-watershed" +client_app_user_password: "mmw" + postgresql_username: mmw postgresql_password: mmw postgresql_database: mmw diff --git a/deployment/ansible/roles/model-my-watershed.base/defaults/main.yml b/deployment/ansible/roles/model-my-watershed.base/defaults/main.yml index c6f16c167..673fd063b 100644 --- a/deployment/ansible/roles/model-my-watershed.base/defaults/main.yml +++ b/deployment/ansible/roles/model-my-watershed.base/defaults/main.yml @@ -17,6 +17,7 @@ envdir_config: MMW_ITSI_CLIENT_ID: "{{ itsi_client_id }}" MMW_ITSI_SECRET_KEY: "{{ itsi_secret_key }}" MMW_ITSI_BASE_URL: "{{ itsi_base_url }}" + MMW_CLIENT_APP_USER_PASSWORD: "{{ client_app_user_password }}" MMW_STACK_COLOR: "{{ stack_color }}" MMW_TILECACHE_BUCKET: "{{ tilecache_bucket_name }}" MMW_STACK_TYPE: "{{ stack_type }}" diff --git a/deployment/cfn/application.py b/deployment/cfn/application.py index 2d14258f8..db8b37f68 100644 --- a/deployment/cfn/application.py +++ b/deployment/cfn/application.py @@ -70,6 +70,7 @@ class Application(StackNode): 'ITSISecretKey': ['global:ITSISecretKey'], 'RollbarServerSideAccessToken': ['global:RollbarServerSideAccessToken'], + 'ClientAppUserPassword': ['global:ClientAppUserPassword'], } DEFAULTS = { @@ -240,6 +241,11 @@ def set_up_stack(self): Description='Secret key for ITSI portal integration' ), 'ITSISecretKey') + self.client_app_user_password = self.add_parameter(Parameter( + 'ClientAppUserPassword', Type='String', NoEcho=True, + Description='Password for the client apps django account', + ), 'ClientAppUserPassword') + app_server_lb_security_group, \ app_server_security_group = self.create_security_groups() app_server_lb, \ @@ -582,7 +588,11 @@ def get_cloud_config(self, tile_distribution_endpoint): ' - path: /etc/mmw.d/env/MMW_ITSI_SECRET_KEY\n', ' permissions: 0750\n', ' owner: root:mmw\n', - ' content: ', Ref(self.itsi_secret_key)] + ' content: ', Ref(self.itsi_secret_key), '\n', + ' - path: /etc/mmw.d/env/MMW_CLIENT_APP_USER_PASSWORD\n', + ' permissions: 0750\n', + ' owner: root:mmw\n', + ' content: ', Ref(self.client_app_user_password)] def create_cloud_watch_resources(self, app_server_lb): self.add_resource(cw.Alarm( diff --git a/deployment/default.yml.example b/deployment/default.yml.example index 6afa9fa4e..9e58c873b 100644 --- a/deployment/default.yml.example +++ b/deployment/default.yml.example @@ -64,3 +64,4 @@ WorkerAutoScalingScheduleEndRecurrence: '0 1 * * *' ITSIBaseURL: '' ITSISecretKey: '' RollbarServerSideAccessToken: '' +ClientAppUserPassword: '' diff --git a/src/mmw/mmw/settings/base.py b/src/mmw/mmw/settings/base.py index 5ae3fddbe..e84e2f7fe 100644 --- a/src/mmw/mmw/settings/base.py +++ b/src/mmw/mmw/settings/base.py @@ -585,6 +585,10 @@ def get_env_setting(setting): TILER_HOST = environ.get('MMW_TILER_HOST', 'localhost') # END TILER CONFIGURATION +# UI ("CLIENT APP") USER CONFIGURATION +CLIENT_APP_USER_PASSWORD = environ.get('MMW_CLIENT_APP_USER_PASSWORD', 'mmw') +# END UI ("CLIENT APP") USER CONFIGURATION + # UI CONFIGURATION DRAW_TOOLS = [ From f309bd9e2f3c99fa8e55dca9c89528c229ac4344 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 27 Sep 2017 10:50:42 -0400 Subject: [PATCH 012/172] Add migration to add client app user * Create a user for the client app so that it can get a token to send geoprocessing API requests --- .../user/migrations/0003_client_app_user.py | 30 +++++++++++++++++++ src/mmw/mmw/settings/base.py | 1 + 2 files changed, 31 insertions(+) create mode 100644 src/mmw/apps/user/migrations/0003_client_app_user.py diff --git a/src/mmw/apps/user/migrations/0003_client_app_user.py b/src/mmw/apps/user/migrations/0003_client_app_user.py new file mode 100644 index 000000000..37555888d --- /dev/null +++ b/src/mmw/apps/user/migrations/0003_client_app_user.py @@ -0,0 +1,30 @@ +# -*- coding: utf-8 -*- +from __future__ import unicode_literals + +from django.db import migrations +from django.conf import settings +from django.contrib.auth.models import User + +def add_client_app_user(apps, schema_editor): + web_app_user = User.objects.create_user( + username=settings.CLIENT_APP_USERNAME, + password=settings.CLIENT_APP_USER_PASSWORD) + web_app_user.save() + + +def remove_client_app_user(apps, schema_editor): + User.objects.filter(username=settings.CLIENT_APP_USERNAME).delete() + + +class Migration(migrations.Migration): + + dependencies = [ + ('authtoken', '0001_initial'), + ('user', '0001_initial'), + ('user', '0002_auth_tokens') + ] + + operations = [ + migrations.RunPython(add_client_app_user, + remove_client_app_user) + ] diff --git a/src/mmw/mmw/settings/base.py b/src/mmw/mmw/settings/base.py index e84e2f7fe..000ade3ae 100644 --- a/src/mmw/mmw/settings/base.py +++ b/src/mmw/mmw/settings/base.py @@ -586,6 +586,7 @@ def get_env_setting(setting): # END TILER CONFIGURATION # UI ("CLIENT APP") USER CONFIGURATION +CLIENT_APP_USERNAME = 'mmw|client_app_user' CLIENT_APP_USER_PASSWORD = environ.get('MMW_CLIENT_APP_USER_PASSWORD', 'mmw') # END UI ("CLIENT APP") USER CONFIGURATION From d530911cfaf42fd1cc210613fff4b998ed0b204a Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 27 Sep 2017 11:41:58 -0400 Subject: [PATCH 013/172] Add Client App Geoprocessing API Token To Client Settings * Pass along the client app dummy user's auth token to the clientSettings so the client can use it for its geoprocessing API requests * For now, simply looks up the client app dummy user via username -- there's already an index on the username field, so should be performant enough --- src/mmw/apps/home/views.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/mmw/apps/home/views.py b/src/mmw/apps/home/views.py index 0fb6355fd..dc39e0599 100644 --- a/src/mmw/apps/home/views.py +++ b/src/mmw/apps/home/views.py @@ -12,6 +12,9 @@ from django.template import RequestContext from django.template.context_processors import csrf from django.conf import settings +from django.contrib.auth.models import User + +from rest_framework.authtoken.models import Token from apps.modeling.models import Project, Scenario @@ -115,6 +118,16 @@ def set_url(layer): layer.update({'url': get_layer_url(layer)}) +def get_api_token(): + try: + client_app_user = User.objects.get( + username=settings.CLIENT_APP_USERNAME) + token = Token.objects.get(user=client_app_user) + return token.key + except User.DoesNotExist, Token.DoesNotExist: + return None + + def get_client_settings(request): # BiG-CZ mode applies when either request host contains predefined host, or # ?bigcz query parameter is present. This covers staging sites, etc. @@ -143,6 +156,7 @@ def get_client_settings(request): 'data_catalog_page_size': settings.BIGCZ_CLIENT_PAGE_SIZE, 'itsi_enabled': not bigcz, 'title': title, + 'api_token': get_api_token(), }), 'google_maps_api_key': settings.GOOGLE_MAPS_API_KEY, 'title': title, From 8bb0a421ee6ed3c4afdc9ea0809d8ebfc4f60c81 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 27 Sep 2017 14:58:15 -0400 Subject: [PATCH 014/172] Send Authorization Header With Geoprocessing API Requests * Add token attribute to the TaskModel and use it for Authorization headers where applicable (currently for all geoprocessing api requests) --- src/mmw/js/src/analyze/models.js | 3 ++- src/mmw/js/src/core/models.js | 13 +++++++++++-- src/mmw/js/src/draw/models.js | 4 +++- 3 files changed, 16 insertions(+), 4 deletions(-) diff --git a/src/mmw/js/src/analyze/models.js b/src/mmw/js/src/analyze/models.js index dd47f24fd..58db6c104 100644 --- a/src/mmw/js/src/analyze/models.js +++ b/src/mmw/js/src/analyze/models.js @@ -28,7 +28,8 @@ var AnalyzeTaskModel = coreModels.TaskModel.extend({ area_of_interest: null, wkaoi: null, taskName: 'analyze', - taskType: 'api' + taskType: 'api', + token: settings.get('api_token') }, coreModels.TaskModel.prototype.defaults ), diff --git a/src/mmw/js/src/core/models.js b/src/mmw/js/src/core/models.js index e9291ca5b..7d1be7a5c 100644 --- a/src/mmw/js/src/core/models.js +++ b/src/mmw/js/src/core/models.js @@ -474,6 +474,14 @@ var TaskModel = Backbone.Model.extend({ } }, + headers: function() { + var token = this.get('token'); + if (token) { + return { 'Authorization': 'Token ' + token }; + } + return null; + }, + // Cancels any currently running jobs. The promise returned // by previous calls to pollForResults will be rejected. reset: function() { @@ -505,7 +513,8 @@ var TaskModel = Backbone.Model.extend({ url: self.url(taskHelper.queryParams), method: 'POST', data: taskHelper.postData, - contentType: taskHelper.contentType + contentType: taskHelper.contentType, + headers: self.headers() }), pollingDefer = $.Deferred(); @@ -554,7 +563,7 @@ var TaskModel = Backbone.Model.extend({ return; } - self.fetch() + self.fetch({ headers: self.headers() }) .done(function(response) { console.log('Polling ' + self.url()); if (response.status === 'started') { diff --git a/src/mmw/js/src/draw/models.js b/src/mmw/js/src/draw/models.js index 61007f05d..5b8181111 100644 --- a/src/mmw/js/src/draw/models.js +++ b/src/mmw/js/src/draw/models.js @@ -2,6 +2,7 @@ var Backbone = require('../../shim/backbone'), _ = require('jquery'), + settings = require('../core/settings'), coreModels = require('../core/models'); var ToolbarModel = Backbone.Model.extend({ @@ -53,7 +54,8 @@ var ToolbarModel = Backbone.Model.extend({ var RwdTaskModel = coreModels.TaskModel.extend({ defaults: _.extend( { taskName: 'rwd', - taskType: 'api' + taskType: 'api', + token: settings.get('api_token') }, coreModels.TaskModel.prototype.defaults ) }); From 6b3c523b6949438aa1066a9d9122b3f14218f7c2 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 27 Sep 2017 15:10:19 -0400 Subject: [PATCH 015/172] Require Token Authentication On Geoprocessing API Endpoints Not that the client app sends a proper auth token, we can secure the geoprocessing API as an token-auth-ed only service * Geop API only endpoints (analyze + RWD): require IsAuthenticated permissions and only allow token authentication * The Model + Geop API endpoint (GET /jobs): As before, AllowAny, but only give response if authenticated user matches job's user (if it exists). Get the user either from the Token or Session, when present --- src/mmw/apps/core/permissions.py | 24 ------------------------ src/mmw/apps/geoprocessing_api/views.py | 16 ++++++++-------- src/mmw/apps/modeling/views.py | 3 +-- 3 files changed, 9 insertions(+), 34 deletions(-) delete mode 100644 src/mmw/apps/core/permissions.py diff --git a/src/mmw/apps/core/permissions.py b/src/mmw/apps/core/permissions.py deleted file mode 100644 index f232ae2ae..000000000 --- a/src/mmw/apps/core/permissions.py +++ /dev/null @@ -1,24 +0,0 @@ -# -*- coding: utf-8 -*- -from __future__ import print_function -from __future__ import unicode_literals - -from rest_framework.permissions import BasePermission -from rest_framework.authentication import TokenAuthentication - - -class IsTokenAuthenticatedOrNotSwagger(BasePermission): - """ - TODO This is just to test the token authentication - Only anonymous requests from the client app should - be allowed: - https://github.com/WikiWatershed/model-my-watershed/issues/2270 - Currently all anonymous requests are allowed, unless you're using - swagger - """ - - def has_permission(self, request, view): - from_swagger = 'api/docs' in request.META['HTTP_REFERER'] \ - if 'HTTP_REFERER' in request.META else False - token_authenticated = type(request.successful_authenticator) \ - is TokenAuthentication - return not from_swagger or token_authenticated diff --git a/src/mmw/apps/geoprocessing_api/views.py b/src/mmw/apps/geoprocessing_api/views.py index c93ce1b31..289cd6354 100644 --- a/src/mmw/apps/geoprocessing_api/views.py +++ b/src/mmw/apps/geoprocessing_api/views.py @@ -7,6 +7,7 @@ from rest_framework.response import Response from rest_framework import decorators from rest_framework.authentication import TokenAuthentication +from rest_framework.permissions import IsAuthenticated from django.utils.timezone import now from django.core.urlresolvers import reverse @@ -15,7 +16,6 @@ from apps.core.models import Job from apps.core.tasks import (save_job_error, save_job_result) -from apps.core.permissions import IsTokenAuthenticatedOrNotSwagger from apps.core.decorators import log_request from apps.modeling import geoprocessing from apps.modeling.views import load_area_of_interest @@ -24,7 +24,7 @@ @decorators.api_view(['POST']) @decorators.authentication_classes((TokenAuthentication, )) -@decorators.permission_classes((IsTokenAuthenticatedOrNotSwagger, )) +@decorators.permission_classes((IsAuthenticated, )) @log_request def start_rwd(request, format=None): """ @@ -186,7 +186,7 @@ def start_rwd(request, format=None): @decorators.api_view(['POST']) @decorators.authentication_classes((TokenAuthentication, )) -@decorators.permission_classes((IsTokenAuthenticatedOrNotSwagger, )) +@decorators.permission_classes((IsAuthenticated, )) @log_request def start_analyze_land(request, format=None): """ @@ -381,7 +381,7 @@ def start_analyze_land(request, format=None): @decorators.api_view(['POST']) @decorators.authentication_classes((TokenAuthentication, )) -@decorators.permission_classes((IsTokenAuthenticatedOrNotSwagger, )) +@decorators.permission_classes((IsAuthenticated, )) @log_request def start_analyze_soil(request, format=None): """ @@ -508,7 +508,7 @@ def start_analyze_soil(request, format=None): @decorators.api_view(['POST']) @decorators.authentication_classes((TokenAuthentication, )) -@decorators.permission_classes((IsTokenAuthenticatedOrNotSwagger, )) +@decorators.permission_classes((IsAuthenticated, )) @log_request def start_analyze_animals(request, format=None): """ @@ -619,7 +619,7 @@ def start_analyze_animals(request, format=None): @decorators.api_view(['POST']) @decorators.authentication_classes((TokenAuthentication, )) -@decorators.permission_classes((IsTokenAuthenticatedOrNotSwagger, )) +@decorators.permission_classes((IsAuthenticated, )) @log_request def start_analyze_pointsource(request, format=None): """ @@ -709,7 +709,7 @@ def start_analyze_pointsource(request, format=None): @decorators.api_view(['POST']) @decorators.authentication_classes((TokenAuthentication, )) -@decorators.permission_classes((IsTokenAuthenticatedOrNotSwagger, )) +@decorators.permission_classes((IsAuthenticated, )) @log_request def start_analyze_catchment_water_quality(request, format=None): """ @@ -828,7 +828,7 @@ def start_analyze_catchment_water_quality(request, format=None): @decorators.api_view(['POST']) @decorators.authentication_classes((TokenAuthentication, )) -@decorators.permission_classes((IsTokenAuthenticatedOrNotSwagger, )) +@decorators.permission_classes((IsAuthenticated, )) @log_request def start_analyze_climate(request, format=None): """ diff --git a/src/mmw/apps/modeling/views.py b/src/mmw/apps/modeling/views.py index 7797d997a..5016faf9f 100644 --- a/src/mmw/apps/modeling/views.py +++ b/src/mmw/apps/modeling/views.py @@ -27,7 +27,6 @@ from apps.core.models import Job from apps.core.tasks import save_job_error, save_job_result -from apps.core.permissions import IsTokenAuthenticatedOrNotSwagger from apps.core.decorators import log_request from apps.modeling import tasks, geoprocessing from apps.modeling.mapshed.tasks import (geoprocessing_chains, @@ -486,7 +485,7 @@ def drb_point_sources(request): @decorators.api_view(['GET']) @decorators.authentication_classes((TokenAuthentication, SessionAuthentication, )) -@decorators.permission_classes((IsTokenAuthenticatedOrNotSwagger, )) +@decorators.permission_classes((AllowAny, )) @log_request def get_job(request, job_uuid, format=None): """ From c0cff05bd04ca51077009598477fe5f183b15bee Mon Sep 17 00:00:00 2001 From: Kelly Innes Date: Fri, 6 Oct 2017 10:27:11 -0400 Subject: [PATCH 016/172] Rename api/rwd -> api/watershed - rename api/rwd route -> api/watershed --- src/mmw/apps/geoprocessing_api/urls.py | 2 +- src/mmw/js/src/draw/models.js | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/mmw/apps/geoprocessing_api/urls.py b/src/mmw/apps/geoprocessing_api/urls.py index 72b3dc049..872a67edd 100644 --- a/src/mmw/apps/geoprocessing_api/urls.py +++ b/src/mmw/apps/geoprocessing_api/urls.py @@ -30,5 +30,5 @@ url(r'analyze/climate/$', views.start_analyze_climate, name='start_analyze_climate'), url(r'jobs/' + uuid_regex, get_job, name='get_job'), - url(r'rwd/$', views.start_rwd, name='start_rwd'), + url(r'watershed/$', views.start_rwd, name='start_rwd'), ) diff --git a/src/mmw/js/src/draw/models.js b/src/mmw/js/src/draw/models.js index 5b8181111..5d712b737 100644 --- a/src/mmw/js/src/draw/models.js +++ b/src/mmw/js/src/draw/models.js @@ -53,7 +53,7 @@ var ToolbarModel = Backbone.Model.extend({ // Used for running Rapid Watershed Delineation tasks. var RwdTaskModel = coreModels.TaskModel.extend({ defaults: _.extend( { - taskName: 'rwd', + taskName: 'watershed', taskType: 'api', token: settings.get('api_token') }, coreModels.TaskModel.prototype.defaults From 462d348f2948f6c52d6b227bb3a91da764019039 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Thu, 28 Sep 2017 14:42:20 -0400 Subject: [PATCH 017/172] BiG-CZ: Register Clicks Inside Cinergi Bounding Boxes * Previously we didn't fill the Cinergi boxes, so clicks only registered on their borders * Transparently fill the boxes so we can capture all intersecting clicks --- src/mmw/js/src/core/views.js | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/mmw/js/src/core/views.js b/src/mmw/js/src/core/views.js index a4636944c..25bf32848 100644 --- a/src/mmw/js/src/core/views.js +++ b/src/mmw/js/src/core/views.js @@ -25,7 +25,8 @@ var dataCatalogPolygonStyle = { color: 'steelblue', weight: 2, opacity: 1, - fill: false + fill: true, + fillOpacity: 0, }; var dataCatalogPointStyle = _.assign({}, dataCatalogPolygonStyle, { From 8526b6e320c4b79618f0002bd43f24b03cdc34f9 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Thu, 28 Sep 2017 14:45:41 -0400 Subject: [PATCH 018/172] BiG-CZ: Register Clicks For All Intersected Layers * Previously only the topmost catalog feature would fire a click event * We want to show all intersected layers per click, move the popup binding into an on click handler on the whole FeatureGroup * Clicks that only intersect a single feature show popups the same as before * Clicks that intersect multiple features, just log them to the console for now --- src/mmw/js/src/core/views.js | 81 +++++++++++++++++++++++++++++------- 1 file changed, 66 insertions(+), 15 deletions(-) diff --git a/src/mmw/js/src/core/views.js b/src/mmw/js/src/core/views.js index 25bf32848..9d0d33b99 100644 --- a/src/mmw/js/src/core/views.js +++ b/src/mmw/js/src/core/views.js @@ -737,7 +737,9 @@ var MapView = Marionette.ItemView.extend({ onEachFeature: function(feature, layer) { layer.on('mouseover', function() { // Only highlight the layer if detail mode is not active - if (self._dataCatalogDetailLayer.getLayers().length === 0) { + // and the layer bounds are within the viewport + if (self._dataCatalogDetailLayer.getLayers().length === 0 && + self._leafletMap.getBounds().contains(layer.getBounds())) { layer.setStyle(dataCatalogActiveStyle); result.set('active', true); } @@ -748,7 +750,7 @@ var MapView = Marionette.ItemView.extend({ if (geom.type === 'Point') { // Preserve highlight of marker if popup is open. // It will get restyled when the popup is closed. - if (!layer._popup._isOpen) { + if (!layer._popup || !layer._popup._isOpen) { layer.setStyle(dataCatalogPointStyle); result.set('active', false); } @@ -812,21 +814,70 @@ var MapView = Marionette.ItemView.extend({ }, bindDataCatalogPopovers: function(PopoverView, catalogId, resultModels) { - this._dataCatalogResultsLayer.eachLayer(function(layer) { - var result = resultModels.findWhere({ id: layer.options.id }); - layer.on('popupopen', function() { - layer.setStyle(dataCatalogActiveStyle); - result.set('active', true); - }); - layer.on('popupclose', function() { - layer.setStyle(dataCatalogPointStyle); - result.set('active', false); - }); - layer.bindPopup(new PopoverView({ + var handleClick = function(e) { + var clickPoint = e.layerPoint, + clickLatLng = e.latlng, + + intersectsClickBounds = function(layer) { + var shape = layer.getLayers()[0]; + + if (shape instanceof L.Polygon) { + return shape.getBounds().contains(clickLatLng); + } + + if (shape instanceof L.Circle) { + return shape._point.distanceTo(clickPoint) <= shape._radius; + } + + return false; + }, + + // Get a list of results intersecting the clicked point + intersectingFeatures = _.reduce( + e.target._layers, + function(intersectingFeatures, layer) { + if (intersectsClickBounds(layer)) { + intersectingFeatures.push(layer); + } + return intersectingFeatures; + }, []); + + // If nothing intersected the clicked point, finish + if (intersectingFeatures.length === 0) { + return; + } + + // If only a single feature intersected the clicked point + // show its detail popup, put active styling on the feature + if (intersectingFeatures.length === 1) { + var layer = intersectingFeatures[0], + result = resultModels.findWhere({ id: layer.options.id }); + + layer.bindPopup(new PopoverView({ model: result, catalog: catalogId - }).render().el, { className: 'data-catalog-popover' }); - }); + }).render().el, { className: 'data-catalog-popover'}); + + layer.openPopup(); + layer.setStyle(dataCatalogActiveStyle); + result.set('active', true); + + layer.on('popupclose', function() { + layer.setStyle(dataCatalogPointStyle); + result.set('active', false); + }); + + return; + } + + console.log("Click intersects multiple features", intersectingFeatures); + }; + + // Remove all existing event listeners that might be from the other catalogs + this._dataCatalogResultsLayer.removeEventListener(); + + // Listen for clicks on the currently active layer + this._dataCatalogResultsLayer.on('click', handleClick); }, renderSelectedGeocoderArea: function() { From b398ac38b54b82407c897a6aa0050a5ee8f7d354 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Fri, 29 Sep 2017 11:04:15 -0400 Subject: [PATCH 019/172] BiG-CZ: Render List Popup View On Multiple Features Click * Create a new popup view that first shows a list view of all click-intersected results. If a list item is clicked, shows the selected result's detail view --- src/mmw/js/src/core/views.js | 103 ++++++++++-------- src/mmw/js/src/data_catalog/models.js | 9 +- .../templates/resultMapPopoverController.html | 1 + ...pover.html => resultMapPopoverDetail.html} | 0 .../templates/resultMapPopoverList.html | 1 + .../templates/resultMapPopoverListItem.html | 33 ++++++ src/mmw/js/src/data_catalog/views.js | 78 ++++++++++++- src/mmw/sass/base/_map.scss | 17 ++- src/mmw/sass/pages/_data-catalog.scss | 51 ++++++++- 9 files changed, 240 insertions(+), 53 deletions(-) create mode 100644 src/mmw/js/src/data_catalog/templates/resultMapPopoverController.html rename src/mmw/js/src/data_catalog/templates/{resultMapPopover.html => resultMapPopoverDetail.html} (100%) create mode 100644 src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html create mode 100644 src/mmw/js/src/data_catalog/templates/resultMapPopoverListItem.html diff --git a/src/mmw/js/src/core/views.js b/src/mmw/js/src/core/views.js index 9d0d33b99..e23ee7b4f 100644 --- a/src/mmw/js/src/core/views.js +++ b/src/mmw/js/src/core/views.js @@ -4,6 +4,7 @@ var L = require('leaflet'), $ = require('jquery'), _ = require('underscore'), router = require('../router.js').router, + Backbone = require('../../shim/backbone'), Marionette = require('../../shim/backbone.marionette'), TransitionRegion = require('../../shim/marionette.transition-region'), coreUtils = require('./utils'), @@ -813,64 +814,74 @@ var MapView = Marionette.ItemView.extend({ } }, - bindDataCatalogPopovers: function(PopoverView, catalogId, resultModels) { - var handleClick = function(e) { - var clickPoint = e.layerPoint, - clickLatLng = e.latlng, + bindDataCatalogPopovers: function(SinglePopoverView, ListPopoverView, + catalogId, resultModels) { + var self = this, + handleClick = function(e) { + var clickPoint = e.layerPoint, + clickLatLng = e.latlng, - intersectsClickBounds = function(layer) { - var shape = layer.getLayers()[0]; + intersectsClickBounds = function(layer) { + var shape = layer.getLayers()[0]; - if (shape instanceof L.Polygon) { - return shape.getBounds().contains(clickLatLng); - } + if (shape instanceof L.Polygon) { + return shape.getBounds().contains(clickLatLng); + } - if (shape instanceof L.Circle) { - return shape._point.distanceTo(clickPoint) <= shape._radius; - } + if (shape instanceof L.Circle) { + return shape._point.distanceTo(clickPoint) <= shape._radius; + } - return false; - }, + return false; + }, - // Get a list of results intersecting the clicked point - intersectingFeatures = _.reduce( - e.target._layers, - function(intersectingFeatures, layer) { - if (intersectsClickBounds(layer)) { - intersectingFeatures.push(layer); - } - return intersectingFeatures; - }, []); + // Get a list of results intersecting the clicked point + intersectingFeatures = _.filter(e.target._layers, + intersectsClickBounds); - // If nothing intersected the clicked point, finish - if (intersectingFeatures.length === 0) { - return; - } + // If nothing intersected the clicked point, finish + if (intersectingFeatures.length === 0) { + return; + } - // If only a single feature intersected the clicked point - // show its detail popup, put active styling on the feature - if (intersectingFeatures.length === 1) { - var layer = intersectingFeatures[0], - result = resultModels.findWhere({ id: layer.options.id }); + // If only a single feature intersected the clicked point + // show its detail popup, put active styling on the feature + if (intersectingFeatures.length === 1) { + var layer = intersectingFeatures[0], + result = resultModels.findWhere({ id: layer.options.id }); - layer.bindPopup(new PopoverView({ - model: result, - catalog: catalogId - }).render().el, { className: 'data-catalog-popover'}); + layer.bindPopup(new SinglePopoverView({ + model: result, + catalog: catalogId + }).render().el, { className: 'data-catalog-popover'}); - layer.openPopup(); - layer.setStyle(dataCatalogActiveStyle); - result.set('active', true); + layer.openPopup(); + layer.setStyle(dataCatalogActiveStyle); + result.set('active', true); - layer.on('popupclose', function() { - layer.setStyle(dataCatalogPointStyle); - result.set('active', false); - }); + layer.once('popupclose', function() { + layer.setStyle(dataCatalogPointStyle); + result.set('active', false); + }); - return; - } + return; + } - console.log("Click intersects multiple features", intersectingFeatures); + // For multiple intersecting features, show the list popup + var id = function(layer) { return layer.options.id; }, + intersectingFeatureIds = _.map(intersectingFeatures, id), + resultIntersects = function(result) { + return _.includes(intersectingFeatureIds, result.get('id')); + }, + intersectingResults = resultModels.filter(resultIntersects); + + self._leafletMap.openPopup( + new ListPopoverView({ + collection: new Backbone.Collection(intersectingResults), + catalog: catalogId + }).render().el, + clickLatLng, + { className: 'data-catalog-popover-list' }); }; // Remove all existing event listeners that might be from the other catalogs diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index d940899d9..2b23b36d2 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -390,6 +390,12 @@ var SearchForm = Backbone.Model.extend({ } }); +var PopoverControllerModel = Backbone.Model.extend({ + defaults: { + activeResult: null // Result + } +}); + module.exports = { GriddedServicesFilter: GriddedServicesFilter, DateFilter: DateFilter, @@ -398,5 +404,6 @@ module.exports = { Catalogs: Catalogs, Result: Result, Results: Results, - SearchForm: SearchForm + SearchForm: SearchForm, + PopoverControllerModel: PopoverControllerModel, }; diff --git a/src/mmw/js/src/data_catalog/templates/resultMapPopoverController.html b/src/mmw/js/src/data_catalog/templates/resultMapPopoverController.html new file mode 100644 index 000000000..d66b5568a --- /dev/null +++ b/src/mmw/js/src/data_catalog/templates/resultMapPopoverController.html @@ -0,0 +1 @@ +
diff --git a/src/mmw/js/src/data_catalog/templates/resultMapPopover.html b/src/mmw/js/src/data_catalog/templates/resultMapPopoverDetail.html similarity index 100% rename from src/mmw/js/src/data_catalog/templates/resultMapPopover.html rename to src/mmw/js/src/data_catalog/templates/resultMapPopoverDetail.html diff --git a/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html b/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html new file mode 100644 index 000000000..f75119a6d --- /dev/null +++ b/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html @@ -0,0 +1 @@ +
    diff --git a/src/mmw/js/src/data_catalog/templates/resultMapPopoverListItem.html b/src/mmw/js/src/data_catalog/templates/resultMapPopoverListItem.html new file mode 100644 index 000000000..f7f903735 --- /dev/null +++ b/src/mmw/js/src/data_catalog/templates/resultMapPopoverListItem.html @@ -0,0 +1,33 @@ +
  • + +
  • diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index abf7713d8..6ef7a27fa 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -6,6 +6,7 @@ var $ = require('jquery'), App = require('../app'), analyzeViews = require('../analyze/views.js'), settings = require('../core/settings'), + models = require('./models'), errorTmpl = require('./templates/error.html'), dateFilterTmpl = require('./templates/dateFilter.html'), checkboxFilterTmpl = require('./templates/checkboxFilter.html'), @@ -22,7 +23,10 @@ var $ = require('jquery'), resultDetailsHydroshareTmpl = require('./templates/resultDetailsHydroshare.html'), resultDetailsCuahsiTmpl = require('./templates/resultDetailsCuahsi.html'), resultsWindowTmpl = require('./templates/resultsWindow.html'), - resultMapPopoverTmpl = require('./templates/resultMapPopover.html'); + resultMapPopoverDetailTmpl = require('./templates/resultMapPopoverDetail.html'), + resultMapPopoverListTmpl = require('./templates/resultMapPopoverList.html'), + resultMapPopoverListItemTmpl = require('./templates/resultMapPopoverListItem.html'), + resultMapPopoverControllerTmpl = require('./templates/resultMapPopoverController.html'); var ENTER_KEYCODE = 13, PAGE_SIZE = settings.get('data_catalog_page_size'), @@ -171,7 +175,8 @@ var DataCatalogWindow = Marionette.LayoutView.extend({ if (catalog) { App.map.set('dataCatalogResults', catalog.get('results')); - App.getMapView().bindDataCatalogPopovers(ResultMapPopoverView, + App.getMapView().bindDataCatalogPopovers( + ResultMapPopoverDetailView, ResultMapPopoverControllerView, catalog.id, catalog.get('results')); } } @@ -471,8 +476,8 @@ var ResultDetailsView = Marionette.ItemView.extend({ } }); -var ResultMapPopoverView = Marionette.LayoutView.extend({ - template: resultMapPopoverTmpl, +var ResultMapPopoverDetailView = Marionette.LayoutView.extend({ + template: resultMapPopoverDetailTmpl, regions: { 'resultRegion': '.data-catalog-popover-result-region' @@ -501,6 +506,71 @@ var ResultMapPopoverView = Marionette.LayoutView.extend({ } }); +var ResultMapPopoverListItemView = Marionette.ItemView.extend({ + template: resultMapPopoverListItemTmpl, + + ui: { + selectButton: '.data-catalog-popover-list-item-btn' + }, + + events: { + 'click @ui.selectButton': 'selectItem' + }, + + templateHelpers: function() { + return { + catalog: this.options.catalog + }; + }, + + selectItem: function() { + this.options.popoverModel.set('activeResult', this.model); + } +}); + +var ResultMapPopoverListView = Marionette.CompositeView.extend({ + template: resultMapPopoverListTmpl, + childView: ResultMapPopoverListItemView, + childViewContainer: '.data-catalog-popover-result-list', + childViewOptions: function() { + return { + popoverModel: this.options.popoverModel, + catalog: this.options.catalog + }; + } +}); + +var ResultMapPopoverControllerView = Marionette.LayoutView.extend({ + // model: PopoverControllerModel + template: resultMapPopoverControllerTmpl, + + regions: { + container: '.data-catalog-popover-container' + }, + + initialize: function() { + this.model = new models.PopoverControllerModel(); + this.model.on('change:activeResult', this.render); + }, + + onRender: function() { + var activeResult = this.model.get('activeResult'); + if (activeResult) { + this.container.show(new ResultMapPopoverDetailView({ + model: activeResult, + catalog: this.options.catalog + })); + } else { + this.container.show(new ResultMapPopoverListView({ + collection: this.collection, + popoverModel: this.model, + catalog: this.options.catalog + })); + } + } + +}); + var PagerView = Marionette.ItemView.extend({ template: pagerTmpl, diff --git a/src/mmw/sass/base/_map.scss b/src/mmw/sass/base/_map.scss index 692a641d3..5e0fb9e88 100644 --- a/src/mmw/sass/base/_map.scss +++ b/src/mmw/sass/base/_map.scss @@ -275,10 +275,25 @@ div.leaflet-popup.data-catalog-popover.leaflet-zoom-animated -div.leaflet-popup-content-wrapper { +div.leaflet-popup-content-wrapper, +div.leaflet-popup.data-catalog-popover-list.leaflet-zoom-animated + div.leaflet-popup-content-wrapper { border-radius: 0px; +} + + +div.leaflet-popup.data-catalog-popover.leaflet-zoom-animated +div.leaflet-popup-content-wrapper { div.leaflet-popup-content { margin-left: 10px; margin-bottom: 10px; } } + +div.leaflet-popup.data-catalog-popover-list.leaflet-zoom-animated +div.leaflet-popup-content-wrapper { + width: 250px; + div.leaflet-popup-content { + margin: 0; + } +} diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index 327de1576..202b984b9 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -169,7 +169,16 @@ } .data-catalog-window, -.data-catalog-resource-popover { +.data-catalog-resource-popover, +.data-catalog-popover-list-item { + .btn-secondary { + height: auto; + } + + h3 { + margin-bottom: 7px; + } + .resource-description { color: $ui-grey; font-size: $font-size-h5; @@ -228,6 +237,46 @@ } } +.data-catalog-popover-container +.data-catalog-resource-popover { + padding: 10px; +} + +.data-catalog-popover-result-list { + padding: 0px; +} + +.data-catalog-popover-list-item { + display: block; + .data-catalog-popover-list-item-btn { + border: none; + border-bottom: 1px solid $ui-light; + padding: 6px; + background: none; + text-align: left; + width: 100%; + + &:hover { + background: #f3f3f3; + } + } + + .data-catalog-popover-list-item-btn + .list-item-title { + color: $brand-primary; + height: 17px; + white-space: nowrap; + text-overflow: ellipsis; + overflow: hidden; + } + + .data-catalog-popover-list-item-btn + .data-catalog-popover-list-item-date { + color: $ui-grey; + font-size: $font-size-h5; + } +} + .data-catalog-filter-window { padding: 1rem; padding-top: 2rem; From e141c73511798a8f22e93754f1e04905903c9e43 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Fri, 29 Sep 2017 15:02:29 -0400 Subject: [PATCH 020/172] BiG-CZ: Add number of results to popover list view --- .../data_catalog/templates/resultMapPopoverList.html | 4 ++++ src/mmw/js/src/data_catalog/views.js | 6 ++++++ src/mmw/sass/pages/_data-catalog.scss | 11 +++++++++++ 3 files changed, 21 insertions(+) diff --git a/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html b/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html index f75119a6d..656a8610c 100644 --- a/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html +++ b/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html @@ -1 +1,5 @@ +
    + {{ numResults }} results found at
    + this location +
      diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 6ef7a27fa..7389e0ff9 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -537,6 +537,12 @@ var ResultMapPopoverListView = Marionette.CompositeView.extend({ popoverModel: this.options.popoverModel, catalog: this.options.catalog }; + }, + + templateHelpers: function() { + return { + numResults: this.collection.length + }; } }); diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index 202b984b9..de7c54727 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -246,6 +246,17 @@ padding: 0px; } +.data-catalog-popover-num-results { + color: $ui-secondary; + // Important because leaflet popup title + // tries to add a margin-bottom + margin-bottom: 0 !important; + font-weight: 800; + padding: 6px; + padding-top: 0; + border-bottom: 1px solid $ui-light; +} + .data-catalog-popover-list-item { display: block; .data-catalog-popover-list-item-btn { From 0139cd215960d17af3cc9bf336e5ec30aa4f6177 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Fri, 29 Sep 2017 15:29:41 -0400 Subject: [PATCH 021/172] BiG-CZ: Add back button from details popover to list popover --- .../templates/resultMapPopoverController.html | 6 ++++++ src/mmw/js/src/data_catalog/views.js | 13 ++++++++++++- src/mmw/sass/pages/_data-catalog.scss | 8 ++++++++ 3 files changed, 26 insertions(+), 1 deletion(-) diff --git a/src/mmw/js/src/data_catalog/templates/resultMapPopoverController.html b/src/mmw/js/src/data_catalog/templates/resultMapPopoverController.html index d66b5568a..4a1aa898a 100644 --- a/src/mmw/js/src/data_catalog/templates/resultMapPopoverController.html +++ b/src/mmw/js/src/data_catalog/templates/resultMapPopoverController.html @@ -1 +1,7 @@ +{% if activeResult %} + +{% endif %} +
      diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 7389e0ff9..6e294beef 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -554,6 +554,14 @@ var ResultMapPopoverControllerView = Marionette.LayoutView.extend({ container: '.data-catalog-popover-container' }, + ui: { + back: '.data-catalog-popover-back-btn' + }, + + events: { + 'click @ui.back': 'backToList' + }, + initialize: function() { this.model = new models.PopoverControllerModel(); this.model.on('change:activeResult', this.render); @@ -573,8 +581,11 @@ var ResultMapPopoverControllerView = Marionette.LayoutView.extend({ catalog: this.options.catalog })); } - } + }, + backToList: function() { + this.model.set('activeResult', null); + } }); var PagerView = Marionette.ItemView.extend({ diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index de7c54727..a4fff2efd 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -239,7 +239,15 @@ .data-catalog-popover-container .data-catalog-resource-popover { + padding: 0 10px 10px 10px; +} + +.data-catalog-popover-back-btn { + color: $brand-primary; + border: none; + background: none; padding: 10px; + font-size: $font-size-h5; } .data-catalog-popover-result-list { From 88f8caf6c0fc5fa678674e7912dad941c9a4da52 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Fri, 29 Sep 2017 15:40:07 -0400 Subject: [PATCH 022/172] BiG-CZ: Close popovers when switching catalog/detail view * The list popups are directly on the map, not tied to any particular layer. Close them manually when the catalog is switched or we enter a detail view --- src/mmw/js/src/core/views.js | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/mmw/js/src/core/views.js b/src/mmw/js/src/core/views.js index e23ee7b4f..2ecca263b 100644 --- a/src/mmw/js/src/core/views.js +++ b/src/mmw/js/src/core/views.js @@ -776,6 +776,9 @@ var MapView = Marionette.ItemView.extend({ this._dataCatalogResultsLayer.addLayer(layer); } }, this); + + // Close any popup that might be on the map + this._leafletMap.closePopup(); }, renderDataCatalogActiveResult: function() { @@ -790,6 +793,9 @@ var MapView = Marionette.ItemView.extend({ this._renderDataCatalogResult(result, this._dataCatalogDetailLayer, 'bigcz-detail-map', dataCatalogDetailStyle); + + // Close any popup that might be on the map + this._leafletMap.closePopup(); }, _renderDataCatalogResult: function(result, featureGroup, className, style) { @@ -884,8 +890,9 @@ var MapView = Marionette.ItemView.extend({ { className: 'data-catalog-popover-list' }); }; - // Remove all existing event listeners that might be from the other catalogs + // Remove all existing event listeners/popups that might be from the other catalogs this._dataCatalogResultsLayer.removeEventListener(); + this._leafletMap.closePopup(); // Listen for clicks on the currently active layer this._dataCatalogResultsLayer.on('click', handleClick); From 7e49722c882796f3eb89720cb654db57196b1182 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Fri, 29 Sep 2017 17:25:31 -0400 Subject: [PATCH 023/172] BiG-CZ: Paginate Popover List View * Use Backbone Pageable Collection to paginate popover list view when results > 3 items --- src/mmw/js/src/core/models.js | 6 +++++ src/mmw/js/src/core/views.js | 5 ++-- .../templates/resultMapPopoverList.html | 22 ++++++++++++++++ src/mmw/js/src/data_catalog/views.js | 26 ++++++++++++++++++- src/mmw/sass/pages/_data-catalog.scss | 19 +++++++++++++- 5 files changed, 74 insertions(+), 4 deletions(-) diff --git a/src/mmw/js/src/core/models.js b/src/mmw/js/src/core/models.js index 7d1be7a5c..5102ef833 100644 --- a/src/mmw/js/src/core/models.js +++ b/src/mmw/js/src/core/models.js @@ -635,6 +635,11 @@ var CatchmentWaterQualityCensusCollection = Backbone.PageableCollection.extend({ state: { pageSize: 6, firstPage: 1 } }); +var DataCatalogPopoverResultCollection = Backbone.PageableCollection.extend({ + mode: 'client', + state: { pageSize: 3, firstPage: 1, currentPage: 1 } +}); + var GeoModel = Backbone.Model.extend({ M_IN_KM: 1000000, @@ -705,6 +710,7 @@ module.exports = { ClimateCensusCollection: ClimateCensusCollection, PointSourceCensusCollection: PointSourceCensusCollection, CatchmentWaterQualityCensusCollection: CatchmentWaterQualityCensusCollection, + DataCatalogPopoverResultCollection: DataCatalogPopoverResultCollection, GeoModel: GeoModel, AreaOfInterestModel: AreaOfInterestModel, AppStateModel: AppStateModel diff --git a/src/mmw/js/src/core/views.js b/src/mmw/js/src/core/views.js index 2ecca263b..465f95388 100644 --- a/src/mmw/js/src/core/views.js +++ b/src/mmw/js/src/core/views.js @@ -4,10 +4,10 @@ var L = require('leaflet'), $ = require('jquery'), _ = require('underscore'), router = require('../router.js').router, - Backbone = require('../../shim/backbone'), Marionette = require('../../shim/backbone.marionette'), TransitionRegion = require('../../shim/marionette.transition-region'), coreUtils = require('./utils'), + models = require('./models'), drawUtils = require('../draw/utils'), modificationConfigUtils = require('../modeling/modificationConfigUtils'), headerTmpl = require('./templates/header.html'), @@ -883,7 +883,8 @@ var MapView = Marionette.ItemView.extend({ self._leafletMap.openPopup( new ListPopoverView({ - collection: new Backbone.Collection(intersectingResults), + collection: + new models.DataCatalogPopoverResultCollection(intersectingResults), catalog: catalogId }).render().el, clickLatLng, diff --git a/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html b/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html index 656a8610c..beafcf2e7 100644 --- a/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html +++ b/src/mmw/js/src/data_catalog/templates/resultMapPopoverList.html @@ -3,3 +3,25 @@
      this location
        + +{% if hasPrevPage or hasNextPage %} + +
        + + + Page {{ pageNum }} of {{ numPages }} + + +
        + +{% endif %} diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 6e294beef..d148745d1 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -539,10 +539,34 @@ var ResultMapPopoverListView = Marionette.CompositeView.extend({ }; }, + ui: { + prevPage: '[data-action="prev-page"]', + nextPage: '[data-action="next-page"]' + }, + + events: { + 'click @ui.prevPage': 'prevPage', + 'click @ui.nextPage': 'nextPage' + }, + templateHelpers: function() { return { - numResults: this.collection.length + numResults: this.collection.fullCollection.length, + pageNum: this.collection.state.currentPage, + numPages: this.collection.state.totalPages, + hasNextPage: this.collection.hasNextPage(), + hasPrevPage: this.collection.hasPreviousPage() }; + }, + + prevPage: function() { + this.collection.getPreviousPage(); + this.render(); + }, + + nextPage: function() { + this.collection.getNextPage(); + this.render(); } }); diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index a4fff2efd..16d990ddc 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -254,11 +254,28 @@ padding: 0px; } +.data-catalog-popover-pagination { + text-align: center; +} + +.data-catalog-popover-pagination +.data-catalog-popover-pagination-btn { + border: none; + background: none; + &:disabled { + color: $ui-light; + } +} + +.data-catalog-popover-pagination-btn, .data-catalog-popover-num-results { - color: $ui-secondary; // Important because leaflet popup title // tries to add a margin-bottom margin-bottom: 0 !important; +} + +.data-catalog-popover-num-results { + color: $ui-secondary; font-weight: 800; padding: 6px; padding-top: 0; From 7583d7d2dbb68256a5e47b293df827f7e576f629 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Tue, 3 Oct 2017 14:52:37 -0400 Subject: [PATCH 024/172] BiG-CZ: Link Popup Active+Highlight States With Sidebar And Map * Highlight the sidebar list item and map element when hovering a popup list * This involved changing the way we add the "ring" highlight to the map when the shape is outside the viewport bounds for CINERGI - Before we were using jquery to add a "highlight" class to the map container. This caused additional mouseenter and mouseleave events to fire on the popup list element. The cursor and highlighting would flicker - Switch to adding a whole DOM element to show the "highlight" ring. This does not cause additional mouse events to fire. --- src/mmw/js/src/core/views.js | 8 +++++-- src/mmw/js/src/data_catalog/views.js | 16 +++++++++++++- src/mmw/sass/base/_map.scss | 33 ++++++++++++++-------------- 3 files changed, 38 insertions(+), 19 deletions(-) diff --git a/src/mmw/js/src/core/views.js b/src/mmw/js/src/core/views.js index 465f95388..e311cbbbf 100644 --- a/src/mmw/js/src/core/views.js +++ b/src/mmw/js/src/core/views.js @@ -800,7 +800,7 @@ var MapView = Marionette.ItemView.extend({ _renderDataCatalogResult: function(result, featureGroup, className, style) { featureGroup.clearLayers(); - this.$el.removeClass(className); + $("div.map-highlight").remove(); // If nothing is selected, exit early if (!result) { return; } @@ -811,7 +811,11 @@ var MapView = Marionette.ItemView.extend({ if (geom) { if ((geom.type === 'MultiPolygon' || geom.type === 'Polygon') && drawUtils.shapeBoundingBox(geom).contains(mapBounds)) { - this.$el.addClass(className); + + $(".map-container") + .append('
        '); } else { var layer = this.createDataCatalogShape(result); layer.setStyle(style); diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index d148745d1..28ebba4d9 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -514,7 +514,9 @@ var ResultMapPopoverListItemView = Marionette.ItemView.extend({ }, events: { - 'click @ui.selectButton': 'selectItem' + 'click @ui.selectButton': 'selectItem', + 'mouseover': 'highlightResult', + 'mouseout': 'unHighlightResult' }, templateHelpers: function() { @@ -525,6 +527,16 @@ var ResultMapPopoverListItemView = Marionette.ItemView.extend({ selectItem: function() { this.options.popoverModel.set('activeResult', this.model); + }, + + highlightResult: function() { + App.map.set('dataCatalogActiveResult', this.model); + this.model.set('active', true); + }, + + unHighlightResult: function() { + App.map.set('dataCatalogActiveResult', null); + this.model.set('active', false); } }); @@ -599,6 +611,8 @@ var ResultMapPopoverControllerView = Marionette.LayoutView.extend({ catalog: this.options.catalog })); } else { + App.map.set('dataCatalogActiveResult', null); + this.model.set('active', false); this.container.show(new ResultMapPopoverListView({ collection: this.collection, popoverModel: this.model, diff --git a/src/mmw/sass/base/_map.scss b/src/mmw/sass/base/_map.scss index 5e0fb9e88..57b7e24ce 100644 --- a/src/mmw/sass/base/_map.scss +++ b/src/mmw/sass/base/_map.scss @@ -40,22 +40,6 @@ right: 0; left: 0; height: 100%; - - &.bigcz-highlight-map:after { - content: ""; - border: 3px solid gold; - width: 100%; - height: 100%; - position: absolute; - } - - &.bigcz-detail-map:after { - content: ""; - border: 3px solid steelblue; - width: 100%; - height: 100%; - position: absolute; - } } #overlay-subclass-vector .disabled, #overlay-subclass-raster .disabled { @@ -297,3 +281,20 @@ div.leaflet-popup-content-wrapper { margin: 0; } } + +.map-highlight.bigcz-highlight-map, +.map-highlight.bigcz-detail-map { + pointer-events: none; + width: 100%; + height: 100%; + position: absolute; + z-index: 100; +} + +.map-highlight.bigcz-highlight-map { + border: 3px solid gold; +} + +.map-highlight.bigcz-detail-map { + border: 3px solid steelblue; +} From 9bb0e532f96d2b677e323279b3d3a9af732a472e Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Thu, 5 Oct 2017 13:42:27 -0400 Subject: [PATCH 025/172] BiG-CZ: On list popup close, clear active styling --- src/mmw/js/src/core/views.js | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/mmw/js/src/core/views.js b/src/mmw/js/src/core/views.js index e311cbbbf..f7146d7b1 100644 --- a/src/mmw/js/src/core/views.js +++ b/src/mmw/js/src/core/views.js @@ -893,6 +893,13 @@ var MapView = Marionette.ItemView.extend({ }).render().el, clickLatLng, { className: 'data-catalog-popover-list' }); + + self._leafletMap.once('popupclose', function() { + self.model.set('dataCatalogActiveResult', null); + _.forEach(intersectingResults, function(result) { + result.set('active', false); + }); + }); }; // Remove all existing event listeners/popups that might be from the other catalogs From bb0a409e22247f92afefc10f751725fcef50d918 Mon Sep 17 00:00:00 2001 From: Hector Castro Date: Thu, 5 Oct 2017 14:54:42 -0400 Subject: [PATCH 026/172] Adjust PostgreSQL versions to fix package installs The PostgreSQL 10 release has caused a number of changes to the official APT repositories we depend on. Many of the core packages begin with 10.x. Also, Psycopg2 2.6.x had a bug where it was expecting PostgreSQL version numbers in the 9.x range. --- deployment/ansible/group_vars/all | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deployment/ansible/group_vars/all b/deployment/ansible/group_vars/all index 9a695f007..5f70c5d41 100644 --- a/deployment/ansible/group_vars/all +++ b/deployment/ansible/group_vars/all @@ -23,8 +23,8 @@ postgresql_database: mmw postgresql_version: "9.4" postgresql_package_version: "9.4.*.pgdg14.04+1" postgresql_support_repository_channel: "main" -postgresql_support_libpq_version: "9.6.*.pgdg14.04+1" -postgresql_support_psycopg2_version: "2.6" +postgresql_support_libpq_version: "10.0-*.pgdg14.04+1" +postgresql_support_psycopg2_version: "2.7" postgis_version: "2.1" postgis_package_version: "2.1.*.pgdg14.04+1" From decd019543be277b0396fde52fd9b217e6232fac Mon Sep 17 00:00:00 2001 From: Kelly Innes Date: Fri, 6 Oct 2017 12:23:34 -0400 Subject: [PATCH 027/172] Upgrade RWD to 1.2.3 - upgrade RWD to 1.2.3 to incorporate rounding watershed latlngs to 6 trailing decimal places --- .../ansible/roles/model-my-watershed.rwd/defaults/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deployment/ansible/roles/model-my-watershed.rwd/defaults/main.yml b/deployment/ansible/roles/model-my-watershed.rwd/defaults/main.yml index 90199a7ed..de979dfbe 100644 --- a/deployment/ansible/roles/model-my-watershed.rwd/defaults/main.yml +++ b/deployment/ansible/roles/model-my-watershed.rwd/defaults/main.yml @@ -2,7 +2,7 @@ rwd_data_path: "/opt/rwd-data" rwd_host: "localhost" rwd_port: 5000 -rwd_docker_image: "quay.io/wikiwatershed/rwd:1.2.2" +rwd_docker_image: "quay.io/wikiwatershed/rwd:1.2.3" app_config: RWD_HOST: "{{ rwd_host }}" From b9055f4186175d1c6dee321975197fcc85481ba9 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 10:53:42 -0400 Subject: [PATCH 028/172] Upgrade pip and add Ulmo Ulmo (https://github.com/ulmo-dev/ulmo/) is a data access library designed to fetch information from various hydrology and climatology sources. It works on the WaterML standard, which is a protocol built on top of SOAP. We will use it to fetch detailed values for sensor data from CUAHSI. Also upgrade pip which speeds up our Python dependency installation step significantly. --- deployment/ansible/roles.yml | 2 +- src/mmw/requirements/base.txt | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/deployment/ansible/roles.yml b/deployment/ansible/roles.yml index 4cd4b9f0a..7e061e4e5 100644 --- a/deployment/ansible/roles.yml +++ b/deployment/ansible/roles.yml @@ -1,7 +1,7 @@ - src: azavea.ntp version: 0.1.1 - src: azavea.pip - version: 0.1.1 + version: 1.0.0 - src: azavea.nodejs version: 0.3.0 - src: azavea.git diff --git a/src/mmw/requirements/base.txt b/src/mmw/requirements/base.txt index 8839ad100..4d8a84f6a 100644 --- a/src/mmw/requirements/base.txt +++ b/src/mmw/requirements/base.txt @@ -19,3 +19,4 @@ retry==0.9.1 python-dateutil==2.6.0 suds==0.4 django_celery_results==1.0.1 +ulmo==0.8.4 From b3cb9fb1f4905b90b38b6a20148e39e4a6b05d06 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 11:07:05 -0400 Subject: [PATCH 029/172] Return more details for Variables Previously, a CUAHSI search would have a `concept_keywords` array of strings, corresponding to each variable. Now that we want to fetch more detailed values, we switch to having an array of `variables`, each of which in addition to `id`, `name` (that tends to be longer, more like a description than name), and `concept_keyword`, has a `site` and `wsdl` which will be used to fetch values for that variable in a given timespan. This array is sorted by `concept_keyword`, which will also be used as a display label in the UI. --- src/mmw/apps/bigcz/clients/cuahsi/models.py | 7 ++++--- src/mmw/apps/bigcz/clients/cuahsi/search.py | 11 +++++++++-- src/mmw/apps/bigcz/clients/cuahsi/serializers.py | 11 ++++++++++- 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/src/mmw/apps/bigcz/clients/cuahsi/models.py b/src/mmw/apps/bigcz/clients/cuahsi/models.py index 8a5c757db..7e304c00e 100644 --- a/src/mmw/apps/bigcz/clients/cuahsi/models.py +++ b/src/mmw/apps/bigcz/clients/cuahsi/models.py @@ -9,15 +9,16 @@ class CuahsiResource(Resource): def __init__(self, id, description, author, links, title, created_at, updated_at, geom, details_url, sample_mediums, - concept_keywords, service_org, service_code, service_url, - service_title, service_citation, begin_date, end_date): + variables, service_org, service_code, service_url, + service_title, service_citation, + begin_date, end_date): super(CuahsiResource, self).__init__(id, description, author, links, title, created_at, updated_at, geom) self.details_url = details_url self.sample_mediums = sample_mediums - self.concept_keywords = concept_keywords + self.variables = variables self.service_org = service_org self.service_code = service_code self.service_url = service_url diff --git a/src/mmw/apps/bigcz/clients/cuahsi/search.py b/src/mmw/apps/bigcz/clients/cuahsi/search.py index fbc4fee50..3e0f05b93 100644 --- a/src/mmw/apps/bigcz/clients/cuahsi/search.py +++ b/src/mmw/apps/bigcz/clients/cuahsi/search.py @@ -6,7 +6,7 @@ from datetime import date from urllib2 import URLError from socket import timeout -from operator import attrgetter +from operator import attrgetter, itemgetter from suds.client import Client from suds.sudsobject import asdict @@ -126,7 +126,7 @@ def parse_record(record, service): geom=geom, details_url=details_url, sample_mediums=record['sample_mediums'], - concept_keywords=record['concept_keywords'], + variables=record['variables'], service_org=service['organization'], service_code=record['serv_code'], service_url=service['ServiceDescriptionURL'], @@ -189,6 +189,13 @@ def group_series_by_location(series): for r in group]), 'end_date': max([parse_date(r['endDate']) for r in group]), + 'variables': sorted([{ + 'id': r['VarCode'], + 'name': r['VarName'], + 'concept_keyword': r['conceptKeyword'], + 'site': r['location'], + 'wsdl': r['ServURL'], + } for r in group], key=itemgetter('concept_keyword')) }) return records diff --git a/src/mmw/apps/bigcz/clients/cuahsi/serializers.py b/src/mmw/apps/bigcz/clients/cuahsi/serializers.py index 549a971a0..2d30c1893 100644 --- a/src/mmw/apps/bigcz/clients/cuahsi/serializers.py +++ b/src/mmw/apps/bigcz/clients/cuahsi/serializers.py @@ -6,15 +6,24 @@ from rest_framework.serializers import (CharField, DateTimeField, ListField, + Serializer, ) from apps.bigcz.serializers import ResourceSerializer +class CuahsiVariableSetSerializer(Serializer): + id = CharField() + name = CharField() + concept_keyword = CharField() + site = CharField() + wsdl = CharField() + + class CuahsiResourceSerializer(ResourceSerializer): details_url = CharField() sample_mediums = ListField(child=CharField()) - concept_keywords = ListField(child=CharField()) + variables = ListField(child=CuahsiVariableSetSerializer()) service_org = CharField() service_code = CharField() service_url = CharField() From 2c2d41f700cedc9b650decc804d6e4371ceb760f Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 11:41:40 -0400 Subject: [PATCH 030/172] Add `details` and `values` endpoints For fetching CUAHSI variable values, we add two endpoints: `details` expects a `wsdl` and `site` parameter, and uses it to fetch the WaterML siteinfo object for the given site from the given wsdl url. This siteinfo object contains details for the variables in the site, including units, and the time ranges for which each variable has values. This information is important, because if we query for `values` with a time outside this range, the endpoint crashes. `values` expects a `wsdl`, `site`, `variable`, `from_date` and `to_date`, and uses it to fetch the values of that variable at that site in the given time range, and returns the WaterML value object. The expected use is one call to `details` for a site, to fetch units and time ranges for each variable, and then a call to `values` for each variable in the site, fetching its values for the given time range. --- Notes: 1. The WaterML objects themselves are complex and elaborate, with many nested keys and arrays. Fortunately, we don't need to write our own serializers for them, because that is all taken care of within Ulmo. Furthermore, because of WaterML being an underlying standard, we can reasonably expect the data to follow a certain shape, and exercise this assumption all the way on the client side, without having to verify it on the server. 2. Ulmo uses suds under the hood to make SOAP requests, and by default uses the suds cache, which is distinct from the redis cache used elsewhere in the app (for Geoprocessing, etc). I have not looked into changing this default behavior, leaving it to a future investigation should the need arise. --- src/mmw/apps/bigcz/clients/__init__.py | 2 + src/mmw/apps/bigcz/clients/cuahsi/__init__.py | 1 + src/mmw/apps/bigcz/clients/cuahsi/details.py | 55 +++++++++++++ src/mmw/apps/bigcz/urls.py | 2 + src/mmw/apps/bigcz/views.py | 79 ++++++++++++++++++- 5 files changed, 138 insertions(+), 1 deletion(-) create mode 100644 src/mmw/apps/bigcz/clients/cuahsi/details.py diff --git a/src/mmw/apps/bigcz/clients/__init__.py b/src/mmw/apps/bigcz/clients/__init__.py index 5b97a5cdb..a8db2e15f 100644 --- a/src/mmw/apps/bigcz/clients/__init__.py +++ b/src/mmw/apps/bigcz/clients/__init__.py @@ -22,6 +22,8 @@ 'model': cuahsi.model, 'serializer': cuahsi.serializer, 'search': cuahsi.search, + 'details': cuahsi.details, + 'values': cuahsi.values, 'is_pageable': False, }, } diff --git a/src/mmw/apps/bigcz/clients/cuahsi/__init__.py b/src/mmw/apps/bigcz/clients/cuahsi/__init__.py index 57f81b9fb..2dae317d7 100644 --- a/src/mmw/apps/bigcz/clients/cuahsi/__init__.py +++ b/src/mmw/apps/bigcz/clients/cuahsi/__init__.py @@ -8,6 +8,7 @@ # Import catalog name and search function, so it can be exported from here from apps.bigcz.clients.cuahsi.search import CATALOG_NAME, search # NOQA +from apps.bigcz.clients.cuahsi.details import details, values # NOQA model = CuahsiResource serializer = CuahsiResourceSerializer diff --git a/src/mmw/apps/bigcz/clients/cuahsi/details.py b/src/mmw/apps/bigcz/clients/cuahsi/details.py new file mode 100644 index 000000000..fdcbd7aaf --- /dev/null +++ b/src/mmw/apps/bigcz/clients/cuahsi/details.py @@ -0,0 +1,55 @@ +# -*- coding: utf-8 -*- +from __future__ import print_function +from __future__ import unicode_literals +from __future__ import division + +from datetime import date, timedelta + +from rest_framework.exceptions import ValidationError + +from ulmo.cuahsi import wof + +DATE_FORMAT = '%m/%d/%Y' + + +def details(wsdl, site): + if not wsdl: + raise ValidationError({ + 'error': 'Required argument: wsdl'}) + + if not site: + raise ValidationError({ + 'error': 'Required argument: site'}) + + if not wsdl.upper().endswith('?WSDL'): + wsdl += '?WSDL' + + return wof.get_site_info(wsdl, site) + + +def values(wsdl, site, variable, from_date=None, to_date=None): + if not wsdl: + raise ValidationError({ + 'error': 'Required argument: wsdl'}) + + if not site: + raise ValidationError({ + 'error': 'Required argument: site'}) + + if not variable: + raise ValidationError({ + 'error': 'Required argument: variable'}) + + if not to_date: + # Set to default value of today + to_date = date.today().strftime(DATE_FORMAT) + + if not from_date: + # Set to default value of one week ago + from_date = (date.today() - + timedelta(days=7)).strftime(DATE_FORMAT) + + if not wsdl.upper().endswith('?WSDL'): + wsdl += '?WSDL' + + return wof.get_values(wsdl, site, variable, from_date, to_date) diff --git a/src/mmw/apps/bigcz/urls.py b/src/mmw/apps/bigcz/urls.py index db43e0c10..43becf18a 100644 --- a/src/mmw/apps/bigcz/urls.py +++ b/src/mmw/apps/bigcz/urls.py @@ -10,4 +10,6 @@ urlpatterns = patterns( '', url(r'^search$', views.search, name='bigcz_search'), + url(r'^details$', views.details, name='bigcz_details'), + url(r'^values$', views.values, name='bigcz_values'), ) diff --git a/src/mmw/apps/bigcz/views.py b/src/mmw/apps/bigcz/views.py index 5004f9623..3fa0fbbed 100644 --- a/src/mmw/apps/bigcz/views.py +++ b/src/mmw/apps/bigcz/views.py @@ -8,7 +8,7 @@ from django.contrib.gis.geos import GEOSGeometry from django.conf import settings from rest_framework import decorators -from rest_framework.exceptions import ValidationError, ParseError +from rest_framework.exceptions import ValidationError, ParseError, NotFound from rest_framework.permissions import AllowAny from rest_framework.response import Response @@ -87,7 +87,84 @@ def _do_search(request): raise ParseError(ex.message) +def _get_details(request): + params = request.query_params + catalog = params.get('catalog') + + if not catalog: + raise ValidationError({ + 'error': 'Required argument: catalog'}) + + if catalog not in CATALOGS: + raise ValidationError({ + 'error': 'Catalog must be one of: {}' + .format(', '.join(CATALOGS.keys()))}) + + details = CATALOGS[catalog]['details'] + + if not details: + raise NotFound({ + 'error': 'No details endpoint for {}' + .format(catalog)}) + + details_kwargs = { + 'wsdl': params.get('wsdl'), + 'site': params.get('site'), + } + + try: + return details(**details_kwargs) + except ValueError as ex: + raise ParseError(ex.message) + + +def _get_values(request): + params = request.query_params + catalog = params.get('catalog') + + if not catalog: + raise ValidationError({ + 'error': 'Required argument: catalog'}) + + if catalog not in CATALOGS: + raise ValidationError({ + 'error': 'Catalog must be one of: {}' + .format(', '.join(CATALOGS.keys()))}) + + values = CATALOGS[catalog]['values'] + + if not values: + raise NotFound({ + 'error': 'No values endpoint for {}' + .format(catalog)}) + + values_kwargs = { + 'wsdl': params.get('wsdl'), + 'site': params.get('site'), + 'variable': params.get('variable'), + 'from_date': params.get('from_date'), + 'to_date': params.get('to_date'), + } + + try: + return values(**values_kwargs) + except ValueError as ex: + raise ParseError(ex.message) + + @decorators.api_view(['POST']) @decorators.permission_classes((AllowAny,)) def search(request): return Response(_do_search(request)) + + +@decorators.api_view(['GET']) +@decorators.permission_classes((AllowAny,)) +def details(request): + return Response(_get_details(request)) + + +@decorators.api_view(['GET']) +@decorators.permission_classes((AllowAny,)) +def values(request): + return Response(_get_values(request)) From e049d8debf8e68b8e62eeb65a9e83f6c2d192472 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 16:28:53 -0400 Subject: [PATCH 031/172] Include geometry for HydroShare results Previously the HydroShare API did not send any geometries in its responses. Now, the "coverages" key is an array of (at least) two types of values: period and box, for time and geography respectively. We translate the box coverage into a geometry if it is included. It behaves identically to CINERGI in the UI, with respect to multiple overlapping popups and polygons that exceed the map bounds. --- .../apps/bigcz/clients/hydroshare/search.py | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/src/mmw/apps/bigcz/clients/hydroshare/search.py b/src/mmw/apps/bigcz/clients/hydroshare/search.py index f5952aca3..cf66b584f 100644 --- a/src/mmw/apps/bigcz/clients/hydroshare/search.py +++ b/src/mmw/apps/bigcz/clients/hydroshare/search.py @@ -8,6 +8,7 @@ from rest_framework.exceptions import ValidationError from django.conf import settings +from django.contrib.gis.geos import Polygon from apps.bigcz.models import Resource, ResourceLink, ResourceList from apps.bigcz.utils import RequestTimedOutError @@ -21,6 +22,21 @@ def parse_date(value): return dateutil.parser.parse(value) +def parse_geom(coverages): + if coverages: + box = [c['value'] for c in coverages if c['type'] == 'box'][0] + if box: + return Polygon(( + (float(box['westlimit']), float(box['northlimit'])), + (float(box['eastlimit']), float(box['northlimit'])), + (float(box['eastlimit']), float(box['southlimit'])), + (float(box['westlimit']), float(box['southlimit'])), + (float(box['westlimit']), float(box['northlimit'])), + )) + + return None + + def parse_record(item): return Resource( id=item['resource_id'], @@ -32,7 +48,7 @@ def parse_record(item): ], created_at=parse_date(item['date_created']), updated_at=parse_date(item['date_last_updated']), - geom=None) + geom=parse_geom(item['coverages'])) def prepare_bbox(box): From 2b60667c1ec79f956cd347805024695aac7fefe7 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 10 Oct 2017 15:15:54 -0400 Subject: [PATCH 032/172] Reduce precipitation significant digits to 1 It is unlikely that the underlying data is accurate to 1/10 of a millimeter. --- src/mmw/js/src/analyze/templates/climateTable.html | 2 +- src/mmw/js/src/analyze/templates/climateTableRow.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/mmw/js/src/analyze/templates/climateTable.html b/src/mmw/js/src/analyze/templates/climateTable.html index 3da4d7654..9f8b6da83 100644 --- a/src/mmw/js/src/analyze/templates/climateTable.html +++ b/src/mmw/js/src/analyze/templates/climateTable.html @@ -16,7 +16,7 @@ Annual - {{ totalPpt|round(2)|toLocaleString(2) }} + {{ totalPpt|round(1)|toLocaleString(1) }} {{ avgTmean|round(1)|toLocaleString(1) }} diff --git a/src/mmw/js/src/analyze/templates/climateTableRow.html b/src/mmw/js/src/analyze/templates/climateTableRow.html index e87bd0f10..e4761f299 100644 --- a/src/mmw/js/src/analyze/templates/climateTableRow.html +++ b/src/mmw/js/src/analyze/templates/climateTableRow.html @@ -1,4 +1,4 @@ {{ monthidx }} {{ month }} -{{ ppt|round(2)|toLocaleString(2) }} +{{ ppt|round(1)|toLocaleString(1) }} {{ tmean|round(1)|toLocaleString(1) }} From 131a2c7f5fe66227d90cd21f9423b37384e18c55 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 10 Oct 2017 15:17:20 -0400 Subject: [PATCH 033/172] Reduce significant digits in Climate charts Add a yTickFormat option field, defaulting to previously hard-coded 2 decimal places, but allowing the caller to override with a format. For the climate charts, we set to 1 decimal place. --- src/mmw/js/src/analyze/views.js | 1 + src/mmw/js/src/core/chart.js | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/src/mmw/js/src/analyze/views.js b/src/mmw/js/src/analyze/views.js index 091054f9b..812eb2415 100644 --- a/src/mmw/js/src/analyze/views.js +++ b/src/mmw/js/src/analyze/views.js @@ -1228,6 +1228,7 @@ var ClimateChartView = ChartView.extend({ return monthNames[x]; }, xTickValues: lodash.range(12), + yTickFormat: '0.01f', }; $(chartEl).empty(); diff --git a/src/mmw/js/src/core/chart.js b/src/mmw/js/src/core/chart.js index 93b8adad4..d45205889 100644 --- a/src/mmw/js/src/core/chart.js +++ b/src/mmw/js/src/core/chart.js @@ -283,7 +283,8 @@ function renderLineChart(chartEl, data, options) { options = options || {}; _.defaults(options, { - margin: {top: 20, right: 30, bottom: 40, left: 60} + margin: {top: 20, right: 30, bottom: 40, left: 60}, + yTickFormat: '.02f', }); nv.addGraph(function() { @@ -298,7 +299,7 @@ function renderLineChart(chartEl, data, options) { chart.yAxis .axisLabel(options.yAxisLabel) - .tickFormat(d3.format('.02f')); + .tickFormat(d3.format(options.yTickFormat)); chart.tooltip.valueFormatter(function(d) { return chart.yAxis.tickFormat()(d) + ' ' + options.yAxisUnit; From 2c1946e35f3996f87ab91d0e874e6057d075b246 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Tue, 10 Oct 2017 13:10:45 -0400 Subject: [PATCH 034/172] Pass /account browser url to app * We'll be adding a new page: the Account Page * Allow the /account url to feed through to the web app, so that it can handle it accordingly --- src/mmw/apps/home/urls.py | 1 + 1 file changed, 1 insertion(+) diff --git a/src/mmw/apps/home/urls.py b/src/mmw/apps/home/urls.py index 592b0a0f4..11f1b2f57 100644 --- a/src/mmw/apps/home/urls.py +++ b/src/mmw/apps/home/urls.py @@ -11,6 +11,7 @@ '', url(r'^$', home_page, name='home_page'), url(r'^draw/?$', home_page, name='home_page'), + url(r'^account/?$', home_page, name='account'), url(r'^projects/$', projects, name='projects'), url(r'^project/$', project, name='project'), url(r'^project/new/', project, name='project'), From 260fd3f38a4b091594206ec8e0d8ff750087c2e1 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Tue, 10 Oct 2017 14:36:33 -0400 Subject: [PATCH 035/172] Replace stock DRF obtain_auth_token view with custom get_auth_token * New endpoint: /api/token * Add a view to get a user's auth token either via: - session auth (we'll use this view to display/regenerate the token in the web app) - a json body with the username and password. Created a custom Authentication DRF class to mimic the way the stock `obtain_auth_token` endpoint behaved. This will allow programmatic-only users to get their tokens without having to go to the webapp * Display authtoken view in swagger UI --- src/mmw/apps/geoprocessing_api/permissions.py | 26 ++++++ src/mmw/apps/geoprocessing_api/tests.py | 83 ++++++++++++++++++- src/mmw/apps/geoprocessing_api/urls.py | 3 +- src/mmw/apps/geoprocessing_api/views.py | 39 ++++++++- src/mmw/mmw/settings/base.py | 1 - 5 files changed, 147 insertions(+), 5 deletions(-) create mode 100644 src/mmw/apps/geoprocessing_api/permissions.py diff --git a/src/mmw/apps/geoprocessing_api/permissions.py b/src/mmw/apps/geoprocessing_api/permissions.py new file mode 100644 index 000000000..aafbd768a --- /dev/null +++ b/src/mmw/apps/geoprocessing_api/permissions.py @@ -0,0 +1,26 @@ +# -*- coding: utf-8 -*- +from __future__ import print_function +from __future__ import unicode_literals + +from rest_framework import authentication +from rest_framework.authtoken.serializers import AuthTokenSerializer + + +class AuthTokenSerializerAuthentication(authentication.BaseAuthentication): + def authenticate(self, request): + + # if the user has made no attempt to add + # credentials via the request body, + # pass through so other authentication + # can be tried + username = request.data.get('username', None) + password = request.data.get('password', None) + if not username and not password: + return None + + serializer = AuthTokenSerializer(data=request.data, + context={'request': request}) + serializer.is_valid(raise_exception=True) + user = serializer.validated_data['user'] + + return(user, None) diff --git a/src/mmw/apps/geoprocessing_api/tests.py b/src/mmw/apps/geoprocessing_api/tests.py index 63557c788..8eef67868 100644 --- a/src/mmw/apps/geoprocessing_api/tests.py +++ b/src/mmw/apps/geoprocessing_api/tests.py @@ -3,11 +3,92 @@ from __future__ import unicode_literals from __future__ import division -from django.test import TestCase +import json + +from django.test import (Client, + TestCase, + LiveServerTestCase) +from django.contrib.auth.models import User + +from rest_framework.authtoken.models import Token from apps.geoprocessing_api import tasks +class ExerciseManageApiToken(LiveServerTestCase): + TOKEN_URL = 'http://localhost:8081/api/token/' + + def setUp(self): + User.objects.create_user(username='bob', email='bob@azavea.com', + password='bob') + + User.objects.create_user(username='nono', email='nono@azavea.com', + password='nono') + + def get_logged_in_session(self, username, password): + c = Client() + c.login(username=username, + password=password) + return c + + def get_api_token(self, username='', password='', + session=None): + if not session: + session = Client() + + payload = None + if username or password: + payload = {'username': username, + 'password': password} + + return session.post(self.TOKEN_URL, + data=payload) + + def test_get_api_token_no_credentials_returns_400(self): + response = self.get_api_token() + self.assertEqual(response.status_code, 403, + 'Incorrect server response. Expected 403 found %s %s' + % (response.status_code, response.content)) + + def test_get_api_token_bad_body_credentials_returns_400(self): + response = self.get_api_token('bad', 'bad') + self.assertEqual(response.status_code, 400, + 'Incorrect server response. Expected 400 found %s %s' + % (response.status_code, response.content)) + + def test_get_api_token_good_body_credentials_returns_200(self): + response = self.get_api_token('bob', 'bob') + self.assertEqual(response.status_code, 200, + 'Incorrect server response. Expected 200 found %s %s' + % (response.status_code, response.content)) + + def test_get_api_token_good_session_credentials_returns_200(self): + s = self.get_logged_in_session('bob', 'bob') + response = self.get_api_token(session=s) + self.assertEqual(response.status_code, 200, + 'Incorrect server response. Expected 200 found %s %s' + % (response.status_code, response.content)) + + def test_get_api_token_uses_body_credentials_over_session(self): + bob_user = User.objects.get(username='bob') + bob_token = Token.objects.get(user=bob_user) + + s = self.get_logged_in_session('nono', 'nono') + response = self.get_api_token('bob', 'bob', s) + + self.assertEqual(response.status_code, 200, + 'Incorrect server response. Expected 200 found %s %s' + % (response.status_code, response.content)) + + response_token = json.loads(response.content)['token'] + + self.assertEqual(str(response_token), str(bob_token), + """ Incorrect server response. + Expected to get token for user + given in request body %s, but got %s + """ % (bob_token, response_token)) + + class ExerciseAnalyze(TestCase): def test_survey_land(self): self.maxDiff = None diff --git a/src/mmw/apps/geoprocessing_api/urls.py b/src/mmw/apps/geoprocessing_api/urls.py index 872a67edd..7b322ff7e 100644 --- a/src/mmw/apps/geoprocessing_api/urls.py +++ b/src/mmw/apps/geoprocessing_api/urls.py @@ -4,7 +4,6 @@ from __future__ import division from django.conf.urls import include, patterns, url -import rest_framework.authtoken.views from apps.modeling.views import get_job from apps.modeling.urls import uuid_regex @@ -14,7 +13,7 @@ urlpatterns = patterns( '', url(r'^docs/', include('rest_framework_swagger.urls')), - url(r'^api-token-auth/', rest_framework.authtoken.views.obtain_auth_token, + url(r'^token/', views.get_auth_token, name="authtoken"), url(r'analyze/land/$', views.start_analyze_land, name='start_analyze_land'), diff --git a/src/mmw/apps/geoprocessing_api/views.py b/src/mmw/apps/geoprocessing_api/views.py index 04d74671c..b7c7e557c 100644 --- a/src/mmw/apps/geoprocessing_api/views.py +++ b/src/mmw/apps/geoprocessing_api/views.py @@ -6,8 +6,10 @@ from rest_framework.response import Response from rest_framework import decorators -from rest_framework.authentication import TokenAuthentication +from rest_framework.authentication import (TokenAuthentication, + SessionAuthentication) from rest_framework.permissions import IsAuthenticated +from rest_framework.authtoken.models import Token from django.utils.timezone import now from django.core.urlresolvers import reverse @@ -20,6 +22,41 @@ from apps.modeling import geoprocessing from apps.modeling.views import load_area_of_interest from apps.geoprocessing_api import tasks +from apps.geoprocessing_api.permissions import AuthTokenSerializerAuthentication # noqa + + +@decorators.api_view(['POST']) +@decorators.authentication_classes((AuthTokenSerializerAuthentication, + SessionAuthentication, )) +@decorators.permission_classes((IsAuthenticated, )) +def get_auth_token(request, format=None): + """ + Get your API key + + + ## Request Body + + { + "username": "your_username", + "password": "your_password" + } + + ## Sample Response + + { + "token": "ea467ed7f67c53cfdd313198647a1d187b4d3ab9", + "created_at": "2017-09-11T14:50:54.738Z" + } + --- + omit_serializer: true + consumes: + - application/json + produces: + - application/json + """ + token, created = Token.objects.get_or_create(user=request.user) + return Response({'token': token.key, + 'created_at': token.created}) @decorators.api_view(['POST']) diff --git a/src/mmw/mmw/settings/base.py b/src/mmw/mmw/settings/base.py index 000ade3ae..ef92f04b7 100644 --- a/src/mmw/mmw/settings/base.py +++ b/src/mmw/mmw/settings/base.py @@ -308,7 +308,6 @@ def get_env_setting(setting): } SWAGGER_SETTINGS = { - 'exclude_url_names': ['authtoken'], 'exclude_namespaces': ['bigcz', 'mmw', 'user'], From f2101bf93c899d18a93a4702d43d2330cfd31fd7 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 13:14:54 -0400 Subject: [PATCH 036/172] Adapt search results to API updates Previously concept_keywords was a simple array, which could be joined in the template. Now we get a variables array of objects, each of which have a concept_keyword field. We pluck the values in templateHelpers and use the final string in the template. --- .../src/data_catalog/templates/searchResultCuahsi.html | 2 +- src/mmw/js/src/data_catalog/views.js | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/src/mmw/js/src/data_catalog/templates/searchResultCuahsi.html b/src/mmw/js/src/data_catalog/templates/searchResultCuahsi.html index 7dd66856a..41a088fd1 100644 --- a/src/mmw/js/src/data_catalog/templates/searchResultCuahsi.html +++ b/src/mmw/js/src/data_catalog/templates/searchResultCuahsi.html @@ -9,7 +9,7 @@

        - {{ concept_keywords|join("; ") }} + {{ concept_keywords }}
        diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 28ebba4d9..a65a13cff 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -396,6 +396,16 @@ var StaticResultView = Marionette.ItemView.extend({ return CATALOG_RESULT_TEMPLATE[this.options.catalog]; }, + templateHelpers: function() { + if (this.options.catalog === 'cuahsi') { + return { + 'concept_keywords': this.model.get('variables') + .pluck('concept_keyword') + .join('; '), + }; + } + }, + modelEvents: { 'change:active': 'render' }, From 1b92e1182baf0ab1afea62718897653810747eba Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 14:17:40 -0400 Subject: [PATCH 037/172] Robustify toTimeAgo filter Prefer using standard library over hackneyed solution --- src/mmw/js/src/core/filters.js | 26 ++----------------- .../templates/resultDetailsCuahsi.html | 2 +- 2 files changed, 3 insertions(+), 25 deletions(-) diff --git a/src/mmw/js/src/core/filters.js b/src/mmw/js/src/core/filters.js index 2158a9d5d..780c32fd8 100644 --- a/src/mmw/js/src/core/filters.js +++ b/src/mmw/js/src/core/filters.js @@ -3,6 +3,7 @@ var nunjucks = require('nunjucks'); var utils = require('./utils'); var _ = require('lodash'); +var moment = require('moment'); nunjucks.env = new nunjucks.Environment(); @@ -61,30 +62,7 @@ nunjucks.env.addFilter('toDateWithoutTime', function(date) { }); nunjucks.env.addFilter('toTimeAgo', function(date) { - var diff = Date.now() - (new Date(date)).getTime(), - secs = diff / 1000, - mins = secs / 60, - hrs = mins / 60, - days = hrs / 24, - wks = days / 7, - mths = days / 30, - yrs = days / 365; - - if (yrs > 1) { - return Math.floor(yrs) + " years ago"; - } else if (mths > 1) { - return Math.floor(mths) + " months ago"; - } else if (wks > 1) { - return Math.floor(wks) + " weeks ago"; - } else if (days > 1) { - return Math.floor(days) + " days ago"; - } else if (hrs > 1) { - return Math.floor(hrs) + " hours ago"; - } else if (mins > 1) { - return Math.floor(mins) + " minutes ago"; - } else { - return Math.floor(secs) + " seconds ago"; - } + return moment(date).fromNow(); }); nunjucks.env.addFilter('split', function(str, splitChar, indexToReturn) { diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html index baa09da4f..fff26f9ab 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html @@ -47,7 +47,7 @@


        - Last collected value {{ end_date|toTimeAgo }}: + Last value collected {{ end_date|toTimeAgo }}:

        From 0a1067eb07104751f4de864847329098438c0497 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 11:49:02 -0400 Subject: [PATCH 038/172] Abstract out DATE_FORMAT for reuse --- src/mmw/js/src/data_catalog/models.js | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index 2b23b36d2..2f7b8dada 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -10,6 +10,8 @@ var REQUEST_TIMED_OUT_CODE = 408; var DESCRIPTION_MAX_LENGTH = 100; var PAGE_SIZE = settings.get('data_catalog_page_size'); +var DATE_FORMAT = 'MM/DD/YYYY'; + var FilterModel = Backbone.Model.extend({ defaults: { @@ -76,22 +78,21 @@ var DateFilter = FilterModel.extend({ validate: function() { // Only need to validate if there are two dates. Ensure that // before is earlier than after - var dateFormat = "MM/DD/YYYY", - toDate = this.get('toDate'), + var toDate = this.get('toDate'), fromDate = this.get('fromDate'), isValid = true; - if (toDate && !moment(toDate, dateFormat).isValid()) { + if (toDate && !moment(toDate, DATE_FORMAT).isValid()) { isValid = false; } - if (fromDate && !moment(fromDate, dateFormat).isValid()) { + if (fromDate && !moment(fromDate, DATE_FORMAT).isValid()) { isValid = false; } if (toDate && fromDate){ - isValid = moment(fromDate, dateFormat) - .isBefore(moment(toDate, dateFormat)); + isValid = moment(fromDate, DATE_FORMAT) + .isBefore(moment(toDate, DATE_FORMAT)); } this.set('isValid', isValid); From 107129dc1f07521b3bc453d529a09a9a54d5e0a8 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 12:07:43 -0400 Subject: [PATCH 039/172] Clear lingering popovers The .popover('hide') command only works if the corresponding tag is still in the DOM. If it has been unloaded, but the popup still remains, .popover('hide') does not close it. To close such lingering popovers, we add the explicit command to remove all matching divs. --- src/mmw/js/src/app.js | 1 + 1 file changed, 1 insertion(+) diff --git a/src/mmw/js/src/app.js b/src/mmw/js/src/app.js index d3c20da76..7b0d36a33 100644 --- a/src/mmw/js/src/app.js +++ b/src/mmw/js/src/app.js @@ -68,6 +68,7 @@ var App = new Marionette.Application({ // Enabling hiding popovers from within them window.closePopover = function() { $('[data-toggle="popover"]').popover('hide'); + $('.popover').remove(); }; }, From 6e9131424411ba2d35f59f734c2e08d340b6f0de Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 13:00:48 -0400 Subject: [PATCH 040/172] Add CuahsiVariable model and collection CuahsiVariable model and collection represent each variable for a given result. They are initially populated by the /search/ (id, name, concept_keyword, wsdl, site) and /details/ (units, begin_date, end_date) endpoints, but also have additional fields (most_recent_value, values, error) for fetching values. CuahsiVariable support searching, given a time range. The range is validated and capped to the begin and end dates on the model. If none is specified, we go with the most recent week or the entire range, whichever is shorter. When the results return, if there are any values, we populate the most_recent_value field with the last reported value. This field will be used in the table UI. The error field is set if fetching fails, or if the returned set has 0 values. --- src/mmw/js/src/data_catalog/models.js | 91 +++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index 2f7b8dada..6005f3fe0 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -397,6 +397,95 @@ var PopoverControllerModel = Backbone.Model.extend({ } }); +var CuahsiVariable = Backbone.Model.extend({ + url: '/bigcz/values', + + defaults: { + id: '', + name: '', + units: '', + concept_keyword: '', + speciation: '', + sample_medium: '', + wsdl: '', + site: '', + values: null, // CuahsiValues Collection + most_recent_value: null, + begin_date: '', + end_date: '', + error: null, + }, + + search: function(from, to) { + var self = this, + begin_date = moment(this.get('begin_date')), + end_date = moment(this.get('end_date')), + params = { + catalog: 'cuahsi', + wsdl: this.get('wsdl'), + site: this.get('site'), + variable: this.get('id'), + }; + + // If neither from date nor to date is specified, set time interval + // to be either from begin date to end date, or 1 week up to end date, + // whichever is shorter. + if (!from || moment(from).isBefore(begin_date)) { + if (end_date.diff(begin_date, 'weeks', true) > 1) { + params.from_date = end_date.subtract(7, 'days'); + } else { + params.from_date = begin_date; + } + } else { + params.from_date = moment(from); + } + + if (!to || moment(to).isAfter(end_date)) { + params.to_date = end_date; + } else { + params.to_date = moment(to); + } + + params.from_date = params.from_date.format(DATE_FORMAT); + params.to_date = params.to_date.format(DATE_FORMAT); + + this.set('error', null); + + return this.fetch({ + data: params, + processData: true, + }) + .fail(function(error) { + self.set('error', 'Error ' + error.status + ' during fetch'); + }); + }, + + parse: function(response) { + var mrv = null; + + if (response.values && response.values.length > 0) { + var values = this.get('values'); + + values.reset(response.values); + mrv = response.values[response.values.length - 1].value; + + delete response.values; + } else { + this.set('error', 'No values returned from API'); + } + + return { + name: response.variable.name, + units: response.variable.units.abbreviation, + most_recent_value: mrv, + }; + } +}); + +var CuahsiVariables = Backbone.Collection.extend({ + model: CuahsiVariable, +}); + module.exports = { GriddedServicesFilter: GriddedServicesFilter, DateFilter: DateFilter, @@ -407,4 +496,6 @@ module.exports = { Results: Results, SearchForm: SearchForm, PopoverControllerModel: PopoverControllerModel, + CuahsiVariable: CuahsiVariable, + CuahsiVariables: CuahsiVariables, }; From 870abc69126b64324c395cfb8122050d57a42d63 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 13:02:00 -0400 Subject: [PATCH 041/172] Add CuahsiValue model and collection CuahsiValue model corresponds to a value object in WaterML. It represents a single value in a series for a variable for a site. --- src/mmw/js/src/data_catalog/models.js | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index 6005f3fe0..a265ffc19 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -397,6 +397,22 @@ var PopoverControllerModel = Backbone.Model.extend({ } }); +var CuahsiValue = Backbone.Model.extend({ + defaults: { + source_id: '', + source_code: '', + quality_control_level_code: '', + value: null, + datetime: '', + date_time_utc: '', + time_offset: '', + } +}); + +var CuahsiValues = Backbone.Collection.extend({ + model: CuahsiValue, +}); + var CuahsiVariable = Backbone.Model.extend({ url: '/bigcz/values', @@ -416,6 +432,10 @@ var CuahsiVariable = Backbone.Model.extend({ error: null, }, + initialize: function() { + this.set('values', new CuahsiValues()); + }, + search: function(from, to) { var self = this, begin_date = moment(this.get('begin_date')), @@ -496,6 +516,8 @@ module.exports = { Results: Results, SearchForm: SearchForm, PopoverControllerModel: PopoverControllerModel, + CuahsiValue: CuahsiValue, + CuahsiValues: CuahsiValues, CuahsiVariable: CuahsiVariable, CuahsiVariables: CuahsiVariables, }; From cc888135f354b938565044762035f82fb7e690dc Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 13:11:30 -0400 Subject: [PATCH 042/172] Fetch CUAHSI Values for Result For a given CUAHSI Result, we add a `fetchCuahsiValues` function that first fetches the /details/ for the result, then fetches /values/ for each variable in the result. If either fetch fails, we set an `error` field to true. A `fetching` field is added to track when the fetch completes. When a CUAHSI Result page is loaded, each result has some fields of every variable filled out: id, name, concept_keyword, wsdl, and site. Before we can fetch the variable values, we need begin_date and end_date, thus we call /details/ to fill those values. Once that succeeds, the actual values can be fetched. `units` are populated both by /details/ and /values/. In most cases they agree, but in some cases they differ. The response from /values/ always overrides /details/, and is considered more accurate. --- src/mmw/js/src/data_catalog/models.js | 104 +++++++++++++++++++++++++- 1 file changed, 101 insertions(+), 3 deletions(-) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index a265ffc19..92ad033a3 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -11,6 +11,7 @@ var DESCRIPTION_MAX_LENGTH = 100; var PAGE_SIZE = settings.get('data_catalog_page_size'); var DATE_FORMAT = 'MM/DD/YYYY'; +var WATERML_VARIABLE_TIME_INTERVAL = '{http://www.cuahsi.org/water_ml/1.1/}variable_time_interval'; var FilterModel = Backbone.Model.extend({ @@ -315,7 +316,105 @@ var Result = Backbone.Model.extend({ created_at: '', updated_at: '', active: false, - show_detail: false // Show this result as the detail view? + show_detail: false, // Show this result as the detail view? + variables: null, // CuahsiVariables Collection + fetching: false, + error: false, + }, + + initialize: function(attrs) { + // For CUAHSI + if (attrs.variables) { + this.set('variables', new CuahsiVariables(attrs.variables)); + } + }, + + parse: function(response) { + // For CUAHSI + if (response.variables) { + var variables = this.get('variables'); + if (variables instanceof CuahsiVariables) { + variables.reset(response.variables); + delete response.variables; + } + } + + return response; + }, + + fetchCuahsiValues: function(opts) { + if (this.fetchPromise && !this.get('error')) { + return this.fetchPromise; + } + + opts = _.defaults(opts || {}, { + onEachSearchDone: _.noop, + onEachSearchFail: _.noop, + from_date: null, + to_date: null, + }); + + var self = this, + variables = self.get('variables'), + runSearches = function() { + return variables.map(function(v) { + return v.search(opts.from_date, opts.to_date) + .done(opts.onEachSearchDone) + .fail(opts.onEachSearchFail); + }); + }, + setSuccess = function() { + self.set('error', false); + }, + setError = function() { + self.set('error', true); + }, + startFetch = function() { + self.set('fetching', true); + }, + endFetch = function() { + self.set('fetching', false); + }; + + startFetch(); + this.fetchPromise = $.get('/bigcz/details', { + catalog: 'cuahsi', + wsdl: variables.first().get('wsdl'), + site: self.get('id'), + }) + .then(function(response) { + variables.forEach(function(v) { + var info = response.series[v.get('id')] || null, + interval = info && info[WATERML_VARIABLE_TIME_INTERVAL]; + + if (info) { + v.set({ + 'units': info.variable.units.abbreviation, + 'speciation': info.variable.speciation, + 'sample_medium': info.variable.sample_medium, + }); + + if (interval) { + v.set({ + 'begin_date': new Date(interval.begin_date_time), + 'end_date': new Date(interval.end_date_time), + }); + } + } + }); + }, function() { + // Handle error in /details/ + setError(); + endFetch(); + }) + .then(function() { + return $.when.apply($, runSearches()) + .done(setSuccess) + .fail(setError) // Handle error in /values/ + .always(endFetch); + }); + + return this.fetchPromise; }, getSummary: function() { @@ -488,14 +587,13 @@ var CuahsiVariable = Backbone.Model.extend({ values.reset(response.values); mrv = response.values[response.values.length - 1].value; - - delete response.values; } else { this.set('error', 'No values returned from API'); } return { name: response.variable.name, + sample_medium: response.variable.sample_medium, units: response.variable.units.abbreviation, most_recent_value: mrv, }; From 09fe45ca2118f0e6871b15dc95d8cd618de56c04 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Tue, 10 Oct 2017 14:58:19 -0400 Subject: [PATCH 043/172] Add "regenerate" flag to /api/token endpoint * Allow the passing of the flag "regenerate" in the request body of /api/token to signal that the current token should be deleted, and a new one returned * Flag is part of the body because endpoint was already made a "POST" endpoint to mimic the DRF "obtain_auth_token" view setup --- src/mmw/apps/geoprocessing_api/tests.py | 49 +++++++++++++++++++++++-- src/mmw/apps/geoprocessing_api/views.py | 19 ++++++++++ 2 files changed, 64 insertions(+), 4 deletions(-) diff --git a/src/mmw/apps/geoprocessing_api/tests.py b/src/mmw/apps/geoprocessing_api/tests.py index 8eef67868..588a7fcfd 100644 --- a/src/mmw/apps/geoprocessing_api/tests.py +++ b/src/mmw/apps/geoprocessing_api/tests.py @@ -32,14 +32,16 @@ def get_logged_in_session(self, username, password): return c def get_api_token(self, username='', password='', - session=None): + session=None, regenerate=False): if not session: session = Client() - payload = None + payload = {} if username or password: - payload = {'username': username, - 'password': password} + payload.update({'username': username, + 'password': password}) + if regenerate: + payload.update({'regenerate': True}) return session.post(self.TOKEN_URL, data=payload) @@ -88,6 +90,45 @@ def test_get_api_token_uses_body_credentials_over_session(self): given in request body %s, but got %s """ % (bob_token, response_token)) + def test_get_api_token_doesnt_regenerate_token(self): + bob_user = User.objects.get(username='bob') + bob_token_before = Token.objects.get(user=bob_user) + + response = self.get_api_token('bob', 'bob') + + response_token = json.loads(response.content)['token'] + + self.assertEqual(str(response_token), str(bob_token_before), + """ Expected request token to be the same + as token before the request was made + (%s), but got %s + """ % (bob_token_before, response_token)) + + bob_token_after = Token.objects.get(user=bob_user) + self.assertEqual(bob_token_before, bob_token_after, + """ Expected token to be the same + as it was before the request was made + (%s), but got %s + """ % (bob_token_before, bob_token_after)) + + def test_get_api_token_can_regenerate_token(self): + bob_user = User.objects.get(username='bob') + old_bob_token = Token.objects.get(user=bob_user) + + response = self.get_api_token('bob', 'bob', regenerate=True) + + response_token = json.loads(response.content)['token'] + new_bob_token = Token.objects.get(user=bob_user) + + self.assertEqual(str(response_token), str(new_bob_token), + """ Expected regenerated response token to + be the same as stored token (%s), but got %s + """ % (new_bob_token, response_token)) + + self.assertTrue(old_bob_token is not new_bob_token, + """ Expected new token to be created + but token is the same""") + class ExerciseAnalyze(TestCase): def test_survey_land(self): diff --git a/src/mmw/apps/geoprocessing_api/views.py b/src/mmw/apps/geoprocessing_api/views.py index b7c7e557c..262ae6b74 100644 --- a/src/mmw/apps/geoprocessing_api/views.py +++ b/src/mmw/apps/geoprocessing_api/views.py @@ -36,6 +36,20 @@ def get_auth_token(request, format=None): ## Request Body + **Required** + + `username` (`string`): Your username + + `password` (`string`): Your password + + + **Optional** + + `regenerate` (`boolean`): Regenerate your API token? + Default is `false`. + + **Example** + { "username": "your_username", "password": "your_password" @@ -54,6 +68,11 @@ def get_auth_token(request, format=None): produces: - application/json """ + + should_regenerate = request.data.get('regenerate', False) + if should_regenerate: + Token.objects.filter(user=request.user).delete() + token, created = Token.objects.get_or_create(user=request.user) return Response({'token': token.key, 'created_at': token.created}) From 801b46d8b818c405c0ed6b6655458eded27b7a7f Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 11 Oct 2017 14:24:43 -0400 Subject: [PATCH 044/172] Fix django user no-csrf-token test None of the user login tests were using a csrftoken (we assumed it was coming back from a `requests` request, but it wasn't). It appears the token wasn't ever being enforced, which led to us commenting out a test for the endpoint failing when the token wasn't there. * Remove ineffective fetch for csrf token * Modify attempt_login_without_token to use the Django test helper Client, which exposes an `enforce_csrf_checks` flag --- src/mmw/apps/user/tests.py | 40 +++++++++++--------------------------- 1 file changed, 11 insertions(+), 29 deletions(-) diff --git a/src/mmw/apps/user/tests.py b/src/mmw/apps/user/tests.py index 11d42acc4..0844cb3d7 100644 --- a/src/mmw/apps/user/tests.py +++ b/src/mmw/apps/user/tests.py @@ -6,7 +6,8 @@ import requests from django.test import LiveServerTestCase -from django.test import TestCase +from django.test import (TestCase, + Client) from django.contrib.auth.models import User from apps.user.views import trim_to_valid_length @@ -19,36 +20,18 @@ def setUp(self): User.objects.create_user(username='bob', email='bob@azavea.com', password='bob') - def get_token(self): - try: - init_response = requests.get(self.HOMEPAGE_URL) - except requests.RequestException: - init_response = {} - - try: - csrf = init_response.cookies['csrftoken'] - except KeyError: - csrf = None - - return csrf - def attempt_login(self, username, password): - csrf = self.get_token() try: - headers = {'HTTP_X_CSRFTOKEN': csrf} payload = {'username': username, 'password': password} - response = requests.post(self.LOGIN_URL, params=payload, - headers=headers) + response = requests.post(self.LOGIN_URL, params=payload) except requests.RequestException: response = {} return response def attempt_login_without_token(self, username, password): - try: - payload = {'username': username, 'password': password} - response = requests.post(self.LOGIN_URL, params=payload) - except requests.RequestException: - response = {} + c = Client(enforce_csrf_checks=True) + payload = {'username': username, 'password': password} + response = c.post(self.LOGIN_URL, params=payload) return response def test_no_username_returns_400(self): @@ -87,12 +70,11 @@ def test_good_credentials_returns_200(self): 'Incorrect server response. Expected 200 found %s' % response.status_code) - # TODO: commented out because it fails and we don't know why yet. - # def test_no_token_good_credentials_returns_400(self): - # response = self.attempt_login_without_token('bob', 'bob') - # self.assertEqual(response.status_code, 400, - # 'Incorrect server response. Expected 400 found %s' - # % response.status_code) + def test_no_token_good_credentials_returns_400(self): + response = self.attempt_login_without_token('bob', 'bob') + self.assertEqual(response.status_code, 400, + 'Incorrect server response. Expected 400 found %s' + % response.status_code) def test_no_token_bad_credentials_returns_400(self): response = self.attempt_login_without_token('badbob', 'badpass') From 01872ecb89067b4207df82a3d7192d33b85b43b1 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 9 Oct 2017 13:23:37 -0400 Subject: [PATCH 045/172] Render CUAHSI Variables and Values in table * Extract the variable table into its own set of views * Add spinner and error prompts for when fetching / failing * Add switcher between table and chart views (charts not implemented here, upcoming in #2333) * Initialize BootstrapTable on load, and then handle Backbone events on each model with a custom update function that will update the matching row in the BootstrapTable. Also use a custom formatter for rendering the first column of the table. --- src/mmw/js/src/data_catalog/models.js | 1 + .../templates/resultDetailsCuahsi.html | 29 +-- .../templates/resultDetailsCuahsiChart.html | 0 .../templates/resultDetailsCuahsiTable.html | 9 + ...esultDetailsCuahsiTableRowVariableCol.html | 19 ++ src/mmw/js/src/data_catalog/views.js | 193 ++++++++++++++++-- src/mmw/sass/pages/_data-catalog.scss | 25 +++ src/mmw/sass/utils/_quick-classes.scss | 4 + 8 files changed, 247 insertions(+), 33 deletions(-) create mode 100644 src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiChart.html create mode 100644 src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiTable.html create mode 100644 src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiTableRowVariableCol.html diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index 92ad033a3..366b765e6 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -320,6 +320,7 @@ var Result = Backbone.Model.extend({ variables: null, // CuahsiVariables Collection fetching: false, error: false, + mode: 'table', }, initialize: function(attrs) { diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html index fff26f9ab..35f7c8e05 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html @@ -45,28 +45,19 @@

         Web Services

        +
        +
        + +

        +
        + + +

        - Last value collected {{ end_date|toTimeAgo }}: + Last value collected {{ last_date|toTimeAgo }}:

        -

        - - - - - - - - - {% for ck in concept_keywords %} - - - - - - {% endfor %} - -
        VariableValueUnits
        {{ ck }}
        +

        Citation: {{ service_citation }}

        diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiChart.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiChart.html new file mode 100644 index 000000000..e69de29bb diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiTable.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiTable.html new file mode 100644 index 000000000..8387e54dd --- /dev/null +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiTable.html @@ -0,0 +1,9 @@ + + + Variable + Value + Units + + + + diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiTableRowVariableCol.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiTableRowVariableCol.html new file mode 100644 index 000000000..feefb8425 --- /dev/null +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiTableRowVariableCol.html @@ -0,0 +1,19 @@ +{{ concept_keyword }} +{% if name %} + Speciation: {{ speciation }}

        + {% endif %} +

        Medium: {{ sample_medium }}

        "> + +
        +{% endif %} +{% if error %} + + + +{% endif %} diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index a65a13cff..b3a5ad6d0 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -22,6 +22,9 @@ var $ = require('jquery'), resultDetailsCinergiTmpl = require('./templates/resultDetailsCinergi.html'), resultDetailsHydroshareTmpl = require('./templates/resultDetailsHydroshare.html'), resultDetailsCuahsiTmpl = require('./templates/resultDetailsCuahsi.html'), + resultDetailsCuahsiChartTmpl = require('./templates/resultDetailsCuahsiChart.html'), + resultDetailsCuahsiTableTmpl = require('./templates/resultDetailsCuahsiTable.html'), + resultDetailsCuahsiTableRowVariableColTmpl = require('./templates/resultDetailsCuahsiTableRowVariableCol.html'), resultsWindowTmpl = require('./templates/resultsWindow.html'), resultMapPopoverDetailTmpl = require('./templates/resultMapPopoverDetail.html'), resultMapPopoverListTmpl = require('./templates/resultMapPopoverList.html'), @@ -34,11 +37,6 @@ var ENTER_KEYCODE = 13, cinergi: searchResultTmpl, hydroshare: searchResultTmpl, cuahsi: searchResultCuahsiTmpl, - }, - CATALOG_RESULT_DETAILS_TEMPLATE = { - cinergi: resultDetailsCinergiTmpl, - hydroshare: resultDetailsHydroshareTmpl, - cuahsi: resultDetailsCuahsiTmpl, }; var HeaderView = Marionette.LayoutView.extend({ @@ -134,6 +132,7 @@ var DataCatalogWindow = Marionette.LayoutView.extend({ onDetailResultChange: function() { var activeCatalog = this.collection.getActiveCatalog(), + ResultDetailsView = CATALOG_RESULT_DETAILS_VIEW[activeCatalog.id], detailResult = activeCatalog.get('detail_result'); if (!detailResult) { @@ -447,11 +446,7 @@ var ResultsView = Marionette.CollectionView.extend({ } }); -var ResultDetailsView = Marionette.ItemView.extend({ - getTemplate: function() { - return CATALOG_RESULT_DETAILS_TEMPLATE[this.catalog]; - }, - +var ResultDetailsBaseView = Marionette.LayoutView.extend({ ui: { closeDetails: '.close' }, @@ -469,23 +464,193 @@ var ResultDetailsView = Marionette.ItemView.extend({ placement: 'right', trigger: 'click', }); - this.$('[data-toggle="table"]').bootstrapTable(); }, + closeDetails: function() { + this.model.collection.closeDetail(); + } +}); + +var ResultDetailsCinergiView = ResultDetailsBaseView.extend({ + template: resultDetailsCinergiTmpl, +}); + +var ResultDetailsHydroshareView = ResultDetailsBaseView.extend({ + template: resultDetailsHydroshareTmpl, +}); + +var ResultDetailsCuahsiView = ResultDetailsBaseView.extend({ + template: resultDetailsCuahsiTmpl, + templateHelpers: function() { var id = this.model.get('id'), - location = id.substring(id.indexOf(':') + 1); + location = id.substring(id.indexOf(':') + 1), + fetching = this.model.get('fetching'), + error = this.model.get('error'), + last_date = this.model.get('end_date'); + + if (!fetching && !error) { + var variables = this.model.get('variables'), + last_dates = variables.map(function(v) { + var values = v.get('values'); + + if (values.length > 0) { + return new Date(values.last().get('datetime')); + } else { + return new Date('07/04/1776'); + } + }); + + last_dates.push(new Date(last_date)); + last_date = Math.max.apply(null, last_dates); + } return { location: location, + last_date: last_date, }; }, - closeDetails: function() { - this.model.collection.closeDetail(); + regions: { + valuesRegion: '#cuahsi-values-region', + }, + + ui: _.defaults({ + chartButton: '#cuahsi-button-chart', + tableButton: '#cuahsi-button-table', + }, ResultDetailsBaseView.prototype.ui), + + events: _.defaults({ + 'click @ui.chartButton': 'setChartMode', + 'click @ui.tableButton': 'setTableMode', + }, ResultDetailsBaseView.prototype.events), + + modelEvents: { + 'change:fetching': 'render', + 'change:mode': 'showValuesRegion', + }, + + initialize: function() { + this.model.fetchCuahsiValues(); + }, + + onRender: function() { + this.showValuesRegion(); + }, + + onDomRefresh: function() { + window.closePopover(); + this.$('[data-toggle="popover"]').popover({ + placement: 'right', + trigger: 'click', + }); + }, + + showValuesRegion: function() { + if (!this.valuesRegion) { + // Don't attempt to display values if this view + // has been unloaded. + return; + } + + var mode = this.model.get('mode'), + variables = this.model.get('variables'), + view = mode === 'table' ? + new CuahsiTableView({ collection: variables }) : + new CuahsiChartView({ collection: variables }); + + this.valuesRegion.show(view); + }, + + setChartMode: function() { + this.model.set('mode', 'chart'); + this.ui.chartButton.addClass('active'); + this.ui.tableButton.removeClass('active'); + }, + + setTableMode: function() { + this.model.set('mode', 'table'); + this.ui.tableButton.addClass('active'); + this.ui.chartButton.removeClass('active'); } }); +var CATALOG_RESULT_DETAILS_VIEW = { + cinergi: ResultDetailsCinergiView, + hydroshare: ResultDetailsHydroshareView, + cuahsi: ResultDetailsCuahsiView, +}; + +var CuahsiTableView = Marionette.ItemView.extend({ + tagName: 'table', + className: 'table custom-hover', + attributes: { + 'data-toggle': 'table', + }, + template: resultDetailsCuahsiTableTmpl, + + initialize: function() { + var self = this; + + this.variableColumnTmpl = resultDetailsCuahsiTableRowVariableColTmpl; + + this.collection.forEach(function(v, index) { + self.listenTo(v, 'change', _.partial(self.onVariableUpdate, index)); + }); + }, + + onAttach: function() { + var data = this.collection.toJSON(), + variableColumnFormatter = _.bind(this.variableColumnFormatter, this), + enablePopovers = _.bind(this.enablePopovers, this); + + this.$el.bootstrapTable({ + data: data, + columns: [ + { + field: 'concept_keyword', + formatter: variableColumnFormatter, + }, + { + field: 'most_recent_value', + }, + { + field: 'units', + } + ], + onPostBody: enablePopovers, + }); + + enablePopovers(); + }, + + enablePopovers: function() { + this.$('.variable-popover').popover({ + placement: 'right', + trigger: 'focus', + }); + }, + + variableColumnFormatter: function(value, row, index) { + return this.variableColumnTmpl.render( + this.collection.at(index).toJSON() + ); + }, + + onVariableUpdate: function(index) { + var row = this.collection.at(index).toJSON(); + + this.$el.bootstrapTable('updateRow', { + index: index, + row: row, + }); + } +}); + +var CuahsiChartView = Marionette.ItemView.extend({ + template: resultDetailsCuahsiChartTmpl, +}); + var ResultMapPopoverDetailView = Marionette.LayoutView.extend({ template: resultMapPopoverDetailTmpl, diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index 16d990ddc..568d3b00a 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -30,6 +30,17 @@ margin: 5px 0 0 0; } + .spinner:after { + top: 44px; + right: 7px; + } + + .error { + position: absolute; + top: 33px; + right: 10px; + } + .result-detail-header { h2 { display: inline-block; @@ -41,6 +52,20 @@ display: block; } } + + .cuahsi-buttons { + button { + padding: 1px 4px; + margin-top: -4px; + background-color: transparent; + border: 0; + + &.active { + color: $paper; + background-color: $black-54; + } + } + } } } diff --git a/src/mmw/sass/utils/_quick-classes.scss b/src/mmw/sass/utils/_quick-classes.scss index 4764c7458..abdd232c2 100644 --- a/src/mmw/sass/utils/_quick-classes.scss +++ b/src/mmw/sass/utils/_quick-classes.scss @@ -57,3 +57,7 @@ .pad-lg{ padding: 3rem; } + +.ui-danger { + color: $ui-danger; +} From 5a593c708bf67740fc87c8234447d28bf92f8b10 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 26 Sep 2017 10:22:13 -0400 Subject: [PATCH 046/172] Merge pull request #2269 from WikiWatershed/jdf/detail-style Restyle data detail view --- .../templates/resultDetailsCuahsi.html | 66 +++++++++++-------- src/mmw/sass/pages/_data-catalog.scss | 32 ++++++++- 2 files changed, 70 insertions(+), 28 deletions(-) diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html index 35f7c8e05..3dd7a2bb1 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html @@ -6,33 +6,43 @@

        Site: {{ location }} {{ title }}

        -

        - Source: {{ service_org }} {{ service_title }} - - - -

        + + + + + + {% if author %} + + + + + {% endif %} + + {% if begin_date == end_date %} + + + {% else %} + + + {% endif %} + + + + + +
        Source{{ service_org }}
        {{ service_title }} + + + +
        Author + {{ author }} +
        Data collected on{{ begin_date|toDateWithoutTime }}Data collected between{{ begin_date|toDateWithoutTime }} – {{ end_date|toDateWithoutTime }}
        Medium + {{ sample_mediums|join(", ") }} +

        - {% if author %} -

        - {{ author }} -

        - {% endif %} - {% if begin_date == end_date %} -

        - Data collected on: {{ begin_date|toDateWithoutTime }} -

        - {% else %} -

        - Data collected between: {{ begin_date|toDateWithoutTime }} - {{ end_date|toDateWithoutTime }} -

        - {% endif %} -

        - Medium: {{ sample_mediums|join(", ") }} -

        {% if details_url %}


        +

        Source Data

        @@ -60,5 +71,6 @@


        -

        Citation: {{ service_citation }}

        +

        Citation

        +

        {{ service_citation }}

        diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index 568d3b00a..139346af8 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -44,13 +44,17 @@ .result-detail-header { h2 { display: inline-block; - margin: 4px 0; + margin: 10px 0 4px; max-width: 95%; } a.zoom { display: block; } + + .close { + opacity: 0.6; + } } .cuahsi-buttons { @@ -421,3 +425,29 @@ margin-top: 8px; } } + +.result-details-region { + p, + .btn { + font-size: $font-size-h5; + } +} + +.table-data-detail { + font-size: $font-size-h5; + margin: 14px 0 6px; + + td, + th { + vertical-align: top; + padding-bottom: 10px; + } + + td { + color: #777; + } + + th { + padding-right: 20px; + } +} From 43339345e7be606db4b49fe0ef6502f33f08c8fd Mon Sep 17 00:00:00 2001 From: Jeff Frankl Date: Tue, 26 Sep 2017 16:09:15 -0400 Subject: [PATCH 047/172] Merge pull request #2297 from WikiWatershed/jdf/detail-style Style WDC detail view --- .../src/data_catalog/templates/resultDetailsCinergi.html | 7 +++---- .../src/data_catalog/templates/resultDetailsCuahsi.html | 8 ++++++-- .../data_catalog/templates/resultDetailsHydroshare.html | 7 +++---- src/mmw/sass/pages/_data-catalog.scss | 9 ++++++++- 4 files changed, 20 insertions(+), 11 deletions(-) diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html index ac96714b8..0e7e8a41c 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html @@ -1,12 +1,11 @@

        {% if author %} diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html index 3dd7a2bb1..9d984ec1c 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html @@ -1,7 +1,7 @@

        Site: {{ location }} {{ title }} @@ -41,6 +41,10 @@

        {{ sample_mediums|join(", ") }} + + Catalog + Water Data Center +

        @@ -66,7 +70,7 @@

        Source Data

        - Last value collected {{ last_date|toTimeAgo }}: + Last value collected {{ last_date|toTimeAgo }}

        diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsHydroshare.html b/src/mmw/js/src/data_catalog/templates/resultDetailsHydroshare.html index aacadd961..c9c811f42 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsHydroshare.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsHydroshare.html @@ -1,12 +1,11 @@
        +

        {{ title }}

        - - Zoom to extent

        {% if author %} diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index 139346af8..2a5ec02d1 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -53,7 +53,14 @@ } .close { - opacity: 0.6; + float: none; + display: block; + opacity: 1; + color: #389b9b; + text-shadow: none; + font-size: 13px; + font-weight: 400; + padding: 6px 0 2px; } } From af7305f15000b68d56d8d93c2a3c7193ef1cca60 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Thu, 12 Oct 2017 11:48:58 -0400 Subject: [PATCH 048/172] Better handle empty CUAHSI results --- src/mmw/apps/bigcz/clients/cuahsi/search.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/mmw/apps/bigcz/clients/cuahsi/search.py b/src/mmw/apps/bigcz/clients/cuahsi/search.py index 3e0f05b93..9625ab70a 100644 --- a/src/mmw/apps/bigcz/clients/cuahsi/search.py +++ b/src/mmw/apps/bigcz/clients/cuahsi/search.py @@ -257,6 +257,10 @@ def get_series_catalog_in_box(box, from_date, to_date, networkIDs): try: return result['SeriesRecord'] except KeyError: + # Empty object can mean "No results" + if not result: + return [] + # Missing key may indicate a server-side error raise ValueError(result) except TypeError: From b07d9570c29df166b4e7add03512838e5b1831f1 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Thu, 12 Oct 2017 11:53:09 -0400 Subject: [PATCH 049/172] Style updates To account for new header and back button on the left --- src/mmw/sass/pages/_data-catalog.scss | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index 2a5ec02d1..8c8ad64c7 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -31,13 +31,13 @@ } .spinner:after { - top: 44px; + top: 22px; right: 7px; } .error { position: absolute; - top: 33px; + top: 11px; right: 10px; } From d9f676e5f2dbfce014845b609ca535a904a05521 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Tue, 10 Oct 2017 13:13:56 -0400 Subject: [PATCH 050/172] Add Account Page Skeleton * Add new browser route /account and wire it to the new `account/AccountController` * AccountController will behave similarly to the "Projects" page - Render its view, AccountContainerView, in the "footer" region of the app - It's styled to be on top the other page elements - Hide LayerPicker and Geocoder regions * Add two placeholder sub-pages to AccountContainerView: Profile, Account - There will be a third sub-page -- Preferences, so keep it easy to add new sub-pages - Placeholder templates display sub-page header * Add link to "My Account" in header's user dropdown --- src/mmw/js/src/account/controllers.js | 33 +++++++++ src/mmw/js/src/account/models.js | 18 +++++ src/mmw/js/src/account/templates/account.html | 1 + .../js/src/account/templates/container.html | 20 ++++++ src/mmw/js/src/account/templates/profile.html | 1 + src/mmw/js/src/account/views.js | 71 +++++++++++++++++++ src/mmw/js/src/core/templates/header.html | 1 + src/mmw/js/src/core/utils.js | 2 + src/mmw/js/src/routes.js | 2 + src/mmw/sass/main.scss | 1 + src/mmw/sass/pages/_account.scss | 43 +++++++++++ 11 files changed, 193 insertions(+) create mode 100644 src/mmw/js/src/account/controllers.js create mode 100644 src/mmw/js/src/account/models.js create mode 100644 src/mmw/js/src/account/templates/account.html create mode 100644 src/mmw/js/src/account/templates/container.html create mode 100644 src/mmw/js/src/account/templates/profile.html create mode 100644 src/mmw/js/src/account/views.js create mode 100644 src/mmw/sass/pages/_account.scss diff --git a/src/mmw/js/src/account/controllers.js b/src/mmw/js/src/account/controllers.js new file mode 100644 index 000000000..1973362d2 --- /dev/null +++ b/src/mmw/js/src/account/controllers.js @@ -0,0 +1,33 @@ +"use strict"; + +var App = require('../app'), + views = require('./views'), + models = require('./models'), + coreUtils= require('../core/utils'); + +var AccountController = { + accountPrepare: function() { + App.rootView.geocodeSearchRegion.empty(); + }, + + account: function() { + App.rootView.footerRegion.show( + new views.AccountContainerView({ + model: new models.AccountContainerModel() + }) + ); + + App.rootView.layerPickerRegion.empty(); + + App.state.set('active_page', coreUtils.accountPageTitle); + }, + + accountCleanUp: function() { + App.rootView.footerRegion.empty(); + App.showLayerPicker(); + } +}; + +module.exports = { + AccountController: AccountController +}; diff --git a/src/mmw/js/src/account/models.js b/src/mmw/js/src/account/models.js new file mode 100644 index 000000000..0a88c96f7 --- /dev/null +++ b/src/mmw/js/src/account/models.js @@ -0,0 +1,18 @@ +"use strict"; + +var Backbone = require('../../shim/backbone'); + +var ACCOUNT = 'account'; +var PROFILE = 'profile'; + +var AccountContainerModel = Backbone.Model.extend({ + defaults: { + active_page: PROFILE + } +}); + +module.exports = { + ACCOUNT: ACCOUNT, + PROFILE: PROFILE, + AccountContainerModel: AccountContainerModel +}; diff --git a/src/mmw/js/src/account/templates/account.html b/src/mmw/js/src/account/templates/account.html new file mode 100644 index 000000000..aa341db86 --- /dev/null +++ b/src/mmw/js/src/account/templates/account.html @@ -0,0 +1 @@ +

        Account

        diff --git a/src/mmw/js/src/account/templates/container.html b/src/mmw/js/src/account/templates/container.html new file mode 100644 index 000000000..6fc640316 --- /dev/null +++ b/src/mmw/js/src/account/templates/container.html @@ -0,0 +1,20 @@ +
        +
        + + +
        + +
        + diff --git a/src/mmw/js/src/account/templates/profile.html b/src/mmw/js/src/account/templates/profile.html new file mode 100644 index 000000000..3f4631a3a --- /dev/null +++ b/src/mmw/js/src/account/templates/profile.html @@ -0,0 +1 @@ +

        Profile

        diff --git a/src/mmw/js/src/account/views.js b/src/mmw/js/src/account/views.js new file mode 100644 index 000000000..63b9182aa --- /dev/null +++ b/src/mmw/js/src/account/views.js @@ -0,0 +1,71 @@ +"use strict"; + +var Marionette = require('../../shim/backbone.marionette'), + models = require('./models'), + containerTmpl = require('./templates/container.html'), + profileTmpl = require('./templates/profile.html'), + accountTmpl = require('./templates/account.html'); + +var ProfileView = Marionette.ItemView.extend({ + template: profileTmpl +}); + +var AccountView = Marionette.ItemView.extend({ + template: accountTmpl +}); + +var AccountContainerView = Marionette.LayoutView.extend({ + // model AccountContainerModel + + template: containerTmpl, + + ui: { + profile: '[data-action="viewprofile"]', + account: '[data-action="viewaccount"]' + }, + + events: { + 'click @ui.profile': 'viewProfile', + 'click @ui.account': 'viewAccount' + }, + + modelEvents: { + 'change:active_page': 'render' + }, + + regions: { + infoContainer: '.account-page-container' + }, + + showActivePage: function() { + var activePage = this.model.get('active_page'); + + switch(activePage) { + case models.PROFILE: + this.infoContainer.show(new ProfileView()); + break; + case models.ACCOUNT: + this.infoContainer.show(new AccountView()); + break; + default: + console.error("Account page, ", activePage, + ", is not supported."); + } + }, + + onRender: function() { + this.showActivePage(); + }, + + viewProfile: function() { + this.model.set('active_page', models.PROFILE); + }, + + viewAccount: function() { + this.model.set('active_page', models.ACCOUNT); + } +}); + +module.exports = { + AccountContainerView: AccountContainerView +}; diff --git a/src/mmw/js/src/core/templates/header.html b/src/mmw/js/src/core/templates/header.html index 8a796c4cd..38e77d0c9 100644 --- a/src/mmw/js/src/core/templates/header.html +++ b/src/mmw/js/src/core/templates/header.html @@ -20,6 +20,7 @@ {% if not itsi_embed %}
      • My Projects
      • {% endif %} +
      • My Account
      • Logout
      • diff --git a/src/mmw/js/src/core/utils.js b/src/mmw/js/src/core/utils.js index 3ced9e395..84a56d6e5 100644 --- a/src/mmw/js/src/core/utils.js +++ b/src/mmw/js/src/core/utils.js @@ -33,6 +33,8 @@ var utils = { projectsPageTitle: 'Projects', + accountPageTitle: 'Account', + filterNoData: function(data) { if (data && !isNaN(data) && isFinite(data)) { return data.toFixed(3); diff --git a/src/mmw/js/src/routes.js b/src/mmw/js/src/routes.js index 9ee14a5d8..6ed6ea7a2 100644 --- a/src/mmw/js/src/routes.js +++ b/src/mmw/js/src/routes.js @@ -4,6 +4,7 @@ var router = require('./router').router, settings = require('./core/settings'), DrawController = require('./draw/controllers').DrawController, AnalyzeController = require('./analyze/controllers').AnalyzeController, + AccountController = require('./account/controllers').AccountController, DataCatalogController = require('./data_catalog/controllers').DataCatalogController, ModelingController = require('./modeling/controllers').ModelingController, CompareController = require('./compare/controllers').CompareController, @@ -14,6 +15,7 @@ var router = require('./router').router, router.addRoute(/^/, DrawController, 'splash'); router.addRoute(/^draw/, DrawController, 'draw'); router.addRoute(/^analyze/, AnalyzeController, 'analyze'); +router.addRoute(/^account/, AccountController, 'account'); router.addRoute('project/new/:modelPackage(/)', ModelingController, 'makeNewProject'); router.addRoute('project(/:projectId)(/scenario/:scenarioId)(/)', ModelingController, 'project'); router.addRoute('project/:projectId/clone(/)', ModelingController, 'projectClone'); diff --git a/src/mmw/sass/main.scss b/src/mmw/sass/main.scss index 3d7e9fe15..c975ed9e1 100644 --- a/src/mmw/sass/main.scss +++ b/src/mmw/sass/main.scss @@ -77,6 +77,7 @@ "pages/model", "pages/compare", "pages/projects", + "pages/account", "pages/water-balance", "pages/registration", "pages/data-catalog"; diff --git a/src/mmw/sass/pages/_account.scss b/src/mmw/sass/pages/_account.scss new file mode 100644 index 000000000..f79c94daa --- /dev/null +++ b/src/mmw/sass/pages/_account.scss @@ -0,0 +1,43 @@ +#account-container { + padding: 2em; + display: flex; + flex-direction: row; + position: absolute; + top: 44px; + left: 0; + right: 0; + bottom: 0; + z-index: 100; + background-color: #eceff1; + transition: 0.3s ease left, 0.3s ease right; + + > .page-toggle-column { + width: 100px; + } + + > .account-page-column { + flex-grow: 1; + } +} + +#account-container .page-toggle-button { + background: none; + border: none; + display: block; + font-size: $font-size-h4; + font-weight: lighter; + + &.active { + font-weight: bolder; + color: $brand-primary; + } +} + +#account-container .account-page-container { + background: $paper; + padding: 2rem; + box-sizing: content-box; + box-shadow: 1px 1px 1px 0px rgba(0, 0, 0, 0.5); + margin-left: 8em; + max-width: 700px; +} From cec61103e99f47f83c69558d5af92bf6fce7aa45 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 11 Oct 2017 17:42:32 -0400 Subject: [PATCH 051/172] Show guest user login modal if they try to access account page * If a not-logged in user goes to the account page view the browser route, show them the login modal. Display their account page on success --- src/mmw/js/src/account/controllers.js | 37 ++++++++++++++++++++------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/src/mmw/js/src/account/controllers.js b/src/mmw/js/src/account/controllers.js index 1973362d2..8e3044e27 100644 --- a/src/mmw/js/src/account/controllers.js +++ b/src/mmw/js/src/account/controllers.js @@ -3,6 +3,7 @@ var App = require('../app'), views = require('./views'), models = require('./models'), + router = require('../router').router, coreUtils= require('../core/utils'); var AccountController = { @@ -11,15 +12,7 @@ var AccountController = { }, account: function() { - App.rootView.footerRegion.show( - new views.AccountContainerView({ - model: new models.AccountContainerModel() - }) - ); - - App.rootView.layerPickerRegion.empty(); - - App.state.set('active_page', coreUtils.accountPageTitle); + showLoginOrAccountView(); }, accountCleanUp: function() { @@ -28,6 +21,32 @@ var AccountController = { } }; +function showAccountView() { + App.rootView.footerRegion.show( + new views.AccountContainerView({ + model: new models.AccountContainerModel() + }) + ); + App.rootView.layerPickerRegion.empty(); + + App.state.set('active_page', coreUtils.accountPageTitle); +} + +function showLoginOrAccountView() { + App.user.fetch().always(function() { + if (App.user.get('guest')) { + var loginSuccess = function() { + router.navigate("/account", { trigger: true }); + }; + App.showLoginModal(loginSuccess); + // Until the user has logged in, show the main page + router.navigate("/", { trigger: true }); + } else { + showAccountView(); + } + }); +} + module.exports = { AccountController: AccountController }; From a99045d17da30475770bc4895f96fc07e4cb3324 Mon Sep 17 00:00:00 2001 From: Kelly Innes Date: Mon, 16 Oct 2017 15:24:30 -0400 Subject: [PATCH 052/172] Destroy filterView on clicking back button - destroy the filterView on clicking the back button from the data catalog page to ensure it will re-create properly --- src/mmw/js/src/data_catalog/views.js | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index b3a5ad6d0..871bbe566 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -198,6 +198,10 @@ var FormView = Marionette.ItemView.extend({ 'click @ui.filterToggle': 'onFilterToggle', }, + onBeforeDestroy: function() { + App.rootView.secondarySidebarRegion.empty(); + }, + initialize: function() { var updateFilterSidebar = _.bind(function() { if (App.rootView.secondarySidebarRegion.hasView()) { From 6f9f1487274df6a6b15698c80998a1e8af7ac386 Mon Sep 17 00:00:00 2001 From: jfrankl Date: Tue, 17 Oct 2017 10:33:10 -0400 Subject: [PATCH 053/172] Update observation colors --- src/mmw/js/src/core/vizerLayers.js | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/mmw/js/src/core/vizerLayers.js b/src/mmw/js/src/core/vizerLayers.js index ed4690aeb..974f651a8 100644 --- a/src/mmw/js/src/core/vizerLayers.js +++ b/src/mmw/js/src/core/vizerLayers.js @@ -16,11 +16,11 @@ var $ = require('jquery'), // These are likely temporary until we develop custom icons for each type var platformIcons = { - 'River Guage': '#F44336', - 'Weather Station': '#4CAF50', - 'Fixed Shore Platform': '#2196F3', - 'Soil Pit': '#FFEB3B', - 'Well': '#795548' + 'River Guage': '#10A579', + 'Weather Station': '#F2B703', + 'Fixed Shore Platform': '#7FB95A', + 'Soil Pit': '#CF1D90', + 'Well': '#A5A998' }, error_msg = 'Unable to load Observation data'; @@ -57,7 +57,8 @@ function VizerLayers() { return L.circleMarker([asset.lat, asset.lon], { attributes: asset, fillColor: color, - fillOpacity: 1.0 + fillOpacity: 0.6, + color: color }); }); From 6d25d22e4160f991f595bdd651969accd97e0a7f Mon Sep 17 00:00:00 2001 From: Kelly Innes Date: Mon, 16 Oct 2017 17:57:01 -0400 Subject: [PATCH 054/172] Clear details geom on leaving DataCatalogWindow - reset details geom in dataCatalogCleanup method --- src/mmw/js/src/data_catalog/controllers.js | 1 + 1 file changed, 1 insertion(+) diff --git a/src/mmw/js/src/data_catalog/controllers.js b/src/mmw/js/src/data_catalog/controllers.js index 13f83f234..d26b41a8e 100644 --- a/src/mmw/js/src/data_catalog/controllers.js +++ b/src/mmw/js/src/data_catalog/controllers.js @@ -64,6 +64,7 @@ var DataCatalogController = { App.map.set({ dataCatalogResults: null, dataCatalogActiveResult: null, + dataCatalogDetailResult: null, }); App.rootView.sidebarRegion.currentView.collection.forEach( function(catalogModel) { From 027dfcb3332cced599783e95ce929979800da9c4 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 17 Oct 2017 15:55:48 -0400 Subject: [PATCH 055/172] Fetch one month of data instead of one week It's not terribly slow, we can wait for the results. --- src/mmw/js/src/data_catalog/models.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index 366b765e6..252c0674c 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -551,8 +551,8 @@ var CuahsiVariable = Backbone.Model.extend({ // to be either from begin date to end date, or 1 week up to end date, // whichever is shorter. if (!from || moment(from).isBefore(begin_date)) { - if (end_date.diff(begin_date, 'weeks', true) > 1) { - params.from_date = end_date.subtract(7, 'days'); + if (end_date.diff(begin_date, 'months', true) > 1) { + params.from_date = moment(end_date).subtract(1, 'months'); } else { params.from_date = begin_date; } From e4093c898471726af2d1213bd05670dfce255c14 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Tue, 10 Oct 2017 15:41:30 -0400 Subject: [PATCH 056/172] Display user's api token in account page * Add ApiTokenModel to handle fetching and regenerating the token - two of its attributes are currently undisplay/unused: fetching, error * Add token info to account template * Regenerate token on "regenerate api key" button click * Initialize ApiTokenModel in AccountContainerView's initialize() - The api key will be fairly constant. Instead of having to fetch it every time the user switches the sub-page of the account container view, initialize its model when the container view is created - If the token is eventually needed elsewhere, we will want to move its model to the App object --- src/mmw/js/src/account/models.js | 60 ++++++++++++++++++- src/mmw/js/src/account/templates/account.html | 16 ++++- src/mmw/js/src/account/templates/profile.html | 2 +- src/mmw/js/src/account/views.js | 27 ++++++++- src/mmw/sass/pages/_account.scss | 17 ++++++ 5 files changed, 117 insertions(+), 5 deletions(-) diff --git a/src/mmw/js/src/account/models.js b/src/mmw/js/src/account/models.js index 0a88c96f7..01bfeab9f 100644 --- a/src/mmw/js/src/account/models.js +++ b/src/mmw/js/src/account/models.js @@ -1,6 +1,7 @@ "use strict"; -var Backbone = require('../../shim/backbone'); +var _ = require('lodash'), + Backbone = require('../../shim/backbone'); var ACCOUNT = 'account'; var PROFILE = 'profile'; @@ -11,8 +12,65 @@ var AccountContainerModel = Backbone.Model.extend({ } }); +var ApiTokenModel = Backbone.Model.extend({ + url: '/api/token/', + + defaults: { + api_key: '', + created_at: '', // Datetime of api key generation + fetching: false, // Is fetching api key? + error: '' + }, + + initialize: function() { + this.fetchToken(); + }, + + fetchToken: function(regenerate) { + this.set({fetching: true, + error: ''}); + + var data = {}; + + if (regenerate) { + data['regenerate'] = true; + } + + var request = { + data: JSON.stringify(data), + type: 'POST', + dataType: 'json', + contentType: 'application/json' + }, + + failFetch = _.bind(function(error) { + this.set('error', error); + }, this), + + finishFetch = _.bind(function() { + this.set('fetching', false); + }, this); + + return this.fetch(request) + .fail(failFetch) + .always(finishFetch); + }, + + regenerateToken: function() { + this.fetchToken(true); + }, + + parse: function(response) { + return { + api_key: response.token, + created_at: response.created_at + }; + } +}); + module.exports = { ACCOUNT: ACCOUNT, PROFILE: PROFILE, + ApiTokenModel: ApiTokenModel, AccountContainerModel: AccountContainerModel }; diff --git a/src/mmw/js/src/account/templates/account.html b/src/mmw/js/src/account/templates/account.html index aa341db86..316a6148f 100644 --- a/src/mmw/js/src/account/templates/account.html +++ b/src/mmw/js/src/account/templates/account.html @@ -1 +1,15 @@ -

        Account

        +

        Account

        + + +

        You can use your API Key to make up to 2 requests per minute

        + diff --git a/src/mmw/js/src/account/templates/profile.html b/src/mmw/js/src/account/templates/profile.html index 3f4631a3a..dbfc29ce8 100644 --- a/src/mmw/js/src/account/templates/profile.html +++ b/src/mmw/js/src/account/templates/profile.html @@ -1 +1 @@ -

        Profile

        +

        Profile

        diff --git a/src/mmw/js/src/account/views.js b/src/mmw/js/src/account/views.js index 63b9182aa..9417ea0af 100644 --- a/src/mmw/js/src/account/views.js +++ b/src/mmw/js/src/account/views.js @@ -11,7 +11,24 @@ var ProfileView = Marionette.ItemView.extend({ }); var AccountView = Marionette.ItemView.extend({ - template: accountTmpl + // model ApiTokenModel + template: accountTmpl, + + ui: { + regenerateKey: '[data-action="regeneratekey"]' + }, + + events: { + 'click @ui.regenerateKey': 'regenerateApiKey' + }, + + modelEvents: { + 'change': 'render' + }, + + regenerateApiKey: function() { + this.model.regenerateToken(); + } }); var AccountContainerView = Marionette.LayoutView.extend({ @@ -37,6 +54,10 @@ var AccountContainerView = Marionette.LayoutView.extend({ infoContainer: '.account-page-container' }, + initialize: function() { + this.tokenModel = new models.ApiTokenModel(); + }, + showActivePage: function() { var activePage = this.model.get('active_page'); @@ -45,7 +66,9 @@ var AccountContainerView = Marionette.LayoutView.extend({ this.infoContainer.show(new ProfileView()); break; case models.ACCOUNT: - this.infoContainer.show(new AccountView()); + this.infoContainer.show(new AccountView({ + model: this.tokenModel + })); break; default: console.error("Account page, ", activePage, diff --git a/src/mmw/sass/pages/_account.scss b/src/mmw/sass/pages/_account.scss index f79c94daa..2a105b753 100644 --- a/src/mmw/sass/pages/_account.scss +++ b/src/mmw/sass/pages/_account.scss @@ -40,4 +40,21 @@ box-shadow: 1px 1px 1px 0px rgba(0, 0, 0, 0.5); margin-left: 8em; max-width: 700px; + + h2 { + margin-bottom: 2rem; + } +} + +.account-api-key-section { + border: 1px solid $ui-medium-light; + padding: 1rem; + display: inline-block; + margin: 1rem 0; + + > .detail { + color: $ui-grey; + font-weight: 400; + margin: 1rem 0; + } } From f1b87dd8f596519d6a55842cfa241ec29d035451 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Thu, 12 Oct 2017 11:43:30 -0400 Subject: [PATCH 057/172] Format Account Page API Token Created At * Use `moment.js` to display the token's created at datetime like: "Oct 12, 2017, 3:42 PM" --- src/mmw/js/src/account/templates/account.html | 2 +- src/mmw/js/src/account/views.js | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/src/mmw/js/src/account/templates/account.html b/src/mmw/js/src/account/templates/account.html index 316a6148f..6b4670362 100644 --- a/src/mmw/js/src/account/templates/account.html +++ b/src/mmw/js/src/account/templates/account.html @@ -7,7 +7,7 @@ {{ api_key }}

        - Created at {{ created_at }} + Created at {{ created_at_formatted }}

        Created at {{ created_at_formatted }} diff --git a/src/mmw/js/src/account/views.js b/src/mmw/js/src/account/views.js index f066fbfac..de8a97a36 100644 --- a/src/mmw/js/src/account/views.js +++ b/src/mmw/js/src/account/views.js @@ -1,6 +1,7 @@ "use strict"; -var Marionette = require('../../shim/backbone.marionette'), +var Clipboard = require('clipboard'), + Marionette = require('../../shim/backbone.marionette'), moment = require('moment'), models = require('./models'), containerTmpl = require('./templates/container.html'), @@ -16,7 +17,8 @@ var AccountView = Marionette.ItemView.extend({ template: accountTmpl, ui: { - regenerateKey: '[data-action="regeneratekey"]' + regenerateKey: '[data-action="regeneratekey"]', + copyKey: '[data-action="copykey"]' }, events: { @@ -27,6 +29,10 @@ var AccountView = Marionette.ItemView.extend({ 'change': 'render' }, + onRender: function() { + new Clipboard(this.ui.copyKey[0]); + }, + templateHelpers: function() { var dateFormat = 'MMM D, YYYY, h:mm A', formattedCreatedAt = moment(this.model.get('created_at')) diff --git a/src/mmw/sass/pages/_account.scss b/src/mmw/sass/pages/_account.scss index 2a105b753..b317caa6c 100644 --- a/src/mmw/sass/pages/_account.scss +++ b/src/mmw/sass/pages/_account.scss @@ -58,3 +58,24 @@ margin: 1rem 0; } } + +.account-api-key-section > .api-key-row { + display: flex; + flex: 1 1 auto; + + .copiable-input, .btn-copy { + border: 1px solid $ui-medium-light; + } + + .copiable-input { + padding-left: 0.5rem; + width: 345px; + border-radius: 2px 0 0 2px; + border-right: none; + } + + .btn-copy { + background: $ui-light; + border-radius: 0 2px 2px 0; + } +} From 7b107a188f9e3f25c18c0911a7434395039aa133 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Thu, 12 Oct 2017 13:05:04 -0400 Subject: [PATCH 059/172] Add loading and error states to account page's API key section * show a loading spinner over the copy button if the api key is fetching * show an error message if the api key failed to fetch/regenerate. * disable copy button if state is fetching --- src/mmw/js/src/account/models.js | 6 ++++-- src/mmw/js/src/account/templates/account.html | 10 ++++++++++ src/mmw/sass/pages/_account.scss | 14 ++++++++++++++ 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/src/mmw/js/src/account/models.js b/src/mmw/js/src/account/models.js index 01bfeab9f..966206f13 100644 --- a/src/mmw/js/src/account/models.js +++ b/src/mmw/js/src/account/models.js @@ -43,8 +43,10 @@ var ApiTokenModel = Backbone.Model.extend({ contentType: 'application/json' }, - failFetch = _.bind(function(error) { - this.set('error', error); + failFetch = _.bind(function() { + var messageVerb = regenerate ? 'regenerate' : 'get'; + this.set('error', + 'Unable to ' + messageVerb + ' API Key'); }, this), finishFetch = _.bind(function() { diff --git a/src/mmw/js/src/account/templates/account.html b/src/mmw/js/src/account/templates/account.html index 3ba0de050..c788b6fdb 100644 --- a/src/mmw/js/src/account/templates/account.html +++ b/src/mmw/js/src/account/templates/account.html @@ -2,6 +2,11 @@

        Account

        You can use your API Key to make up to 2 requests per minute

        +{% if error %} +

        + {{error}} +

        +{% endif %} + + +
        + +
        diff --git a/src/mmw/js/src/account/views.js b/src/mmw/js/src/account/views.js index f345fed7e..e9e19b984 100644 --- a/src/mmw/js/src/account/views.js +++ b/src/mmw/js/src/account/views.js @@ -3,6 +3,8 @@ var Clipboard = require('clipboard'), Marionette = require('../../shim/backbone.marionette'), moment = require('moment'), + userViews = require('../user/views'), + userModels = require('../user/models'), modalViews = require('../core/modals/views'), modalModels = require('../core/modals/models'), models = require('./models'), @@ -20,11 +22,13 @@ var AccountView = Marionette.ItemView.extend({ ui: { regenerateKey: '[data-action="regeneratekey"]', - copyKey: '[data-action="copykey"]' + copyKey: '[data-action="copykey"]', + resetPassword: '[data-action="resetpassword"]' }, events: { - 'click @ui.regenerateKey': 'regenerateApiKey' + 'click @ui.regenerateKey': 'regenerateApiKey', + 'click @ui.resetPassword': 'resetPassword' }, modelEvents: { @@ -64,6 +68,15 @@ var AccountView = Marionette.ItemView.extend({ modal.on('confirmation', function() { self.model.regenerateToken(); }); + }, + + resetPassword: function() { + var resetPasswordModal = + new userViews.ChangePasswordModalView({ + model: new userModels.ChangePasswordFormModel() + }); + + resetPasswordModal.render(); } }); diff --git a/src/mmw/js/src/user/models.js b/src/mmw/js/src/user/models.js index 8e0e70b18..13870abc8 100644 --- a/src/mmw/js/src/user/models.js +++ b/src/mmw/js/src/user/models.js @@ -218,6 +218,49 @@ var ResendFormModel = ModalBaseModel.extend({ } }); +var ChangePasswordFormModel = ModalBaseModel.extend({ + defaults: { + old_password: null, + new_password1: null, + new_password2: null + }, + + url: '/user/change-password', + + validate: function(attrs) { + var errors = []; + + if (!attrs.old_password) { + errors.push('Please enter your password'); + } + + if (!attrs.new_password1) { + errors.push('Please enter a password'); + } + + if (!attrs.new_password2) { + errors.push('Please repeat the password'); + } + + if (attrs.new_password1 !== attrs.new_password2) { + errors.push('Passwords do not match'); + } + + if (errors.length) { + this.set({ + 'client_errors': errors, + 'server_errors': null + }); + return errors; + } else { + this.set({ + 'client_errors': null, + 'server_errors': null + }); + } + } +}); + var ItsiSignUpFormModel = ModalBaseModel.extend({ defaults: { username: null, @@ -264,5 +307,6 @@ module.exports = { SignUpFormModel: SignUpFormModel, ResendFormModel: ResendFormModel, ForgotFormModel: ForgotFormModel, + ChangePasswordFormModel: ChangePasswordFormModel, ItsiSignUpFormModel: ItsiSignUpFormModel }; diff --git a/src/mmw/js/src/user/templates/changePasswordModal.html b/src/mmw/js/src/user/templates/changePasswordModal.html new file mode 100644 index 000000000..4f4bd79ec --- /dev/null +++ b/src/mmw/js/src/user/templates/changePasswordModal.html @@ -0,0 +1,21 @@ +{% extends './baseModal.html' %} + +{% block form %} + {% if success %} + Thanks! Your password has been updated. + {% else %} + {{ field(old_password, 'old_password', 'Current Password', 'password') }} + {{ field(new_password1, 'new_password1', 'New Password', 'password') }} + {{ field(new_password2, 'new_password2', 'Repeat New Password', 'password') }} + {% endif %} +{% endblock %} + +{% block footer %} + {% if success %} + + {% else %} + + {% endif %} +{% endblock %} diff --git a/src/mmw/js/src/user/views.js b/src/mmw/js/src/user/views.js index 48cef0ca8..85815b07f 100644 --- a/src/mmw/js/src/user/views.js +++ b/src/mmw/js/src/user/views.js @@ -12,6 +12,8 @@ var _ = require('underscore'), signUpModalTmpl = require('./templates/signUpModal.html'), resendModalTmpl = require('./templates/resendModal.html'), forgotModalTmpl = require('./templates/forgotModal.html'), + changePasswordModalTmpl = + require('./templates/changePasswordModal.html'), itsiSignUpModalTmpl = require('./templates/itsiSignUpModal.html'); var ENTER_KEYCODE = 13; @@ -385,10 +387,37 @@ var ItsiSignUpModalView = ModalBaseView.extend({ } }); +var ChangePasswordModalView = ModalBaseView.extend({ + template: changePasswordModalTmpl, + + ui: _.defaults({ + 'oldPassword': '#old_password', + 'password1': '#new_password1', + 'password2': '#new_password2' + }, ModalBaseView.prototype.ui), + + onModalShown: function() { + this.ui.oldPassword.focus(); + }, + + onValidationError: function() { + this.ui.oldPassword.focus(); + }, + + setFields: function() { + this.model.set({ + old_password: $(this.ui.oldPassword.selector).val(), + new_password1: $(this.ui.password1.selector).val(), + new_password2: $(this.ui.password2.selector).val(), + }, { silent: true }); + } +}); + module.exports = { LoginModalView: LoginModalView, SignUpModalView: SignUpModalView, ResendModalView: ResendModalView, ForgotModalView: ForgotModalView, + ChangePasswordModalView: ChangePasswordModalView, ItsiSignUpModalView: ItsiSignUpModalView }; diff --git a/src/mmw/sass/pages/_account.scss b/src/mmw/sass/pages/_account.scss index ba014522a..c48e1161a 100644 --- a/src/mmw/sass/pages/_account.scss +++ b/src/mmw/sass/pages/_account.scss @@ -57,7 +57,7 @@ border: 1px solid $ui-medium-light; padding: 1rem; display: inline-block; - margin: 1rem 0; + margin: 1rem 0 3rem 0; > .detail { color: $ui-grey; From 9d791f7977c1b01b459c8a6b418c35ea2e4141a4 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Tue, 17 Oct 2017 16:29:59 -0400 Subject: [PATCH 064/172] Add "Copy to clipboard" and "Copied!" Tooltips Copy Key Btn * On hover, show "copy to clipboard" tooltip * on successful click, show "Copied!" --- src/mmw/js/src/account/templates/account.html | 3 +++ src/mmw/js/src/account/views.js | 22 ++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/src/mmw/js/src/account/templates/account.html b/src/mmw/js/src/account/templates/account.html index 3329601d1..23c391724 100644 --- a/src/mmw/js/src/account/templates/account.html +++ b/src/mmw/js/src/account/templates/account.html @@ -19,6 +19,9 @@ +
        + +
        diff --git a/src/mmw/js/src/compare/views.js b/src/mmw/js/src/compare/views.js index 2a2a7be98..ff4e93c07 100644 --- a/src/mmw/js/src/compare/views.js +++ b/src/mmw/js/src/compare/views.js @@ -2,9 +2,11 @@ var _ = require('lodash'), $ = require('jquery'), + moment = require('moment'), Marionette = require('../../shim/backbone.marionette'), App = require('../app'), coreModels = require('../core/models'), + coreUtils = require('../core/utils'), coreViews = require('../core/views'), chart = require('../core/chart.js'), modalViews = require('../core/modals/views'), @@ -217,17 +219,27 @@ var InputsView = Marionette.LayoutView.extend({ ui: { chartButton: '#compare-input-button-chart', tableButton: '#compare-input-button-table', + downloadButton: '#compare-input-button-download' }, events: { 'click @ui.chartButton': 'setChartView', 'click @ui.tableButton': 'setTableView', + 'click @ui.downloadButton': 'downloadCSV' }, regions: { precipitationRegion: '.compare-precipitation', }, + modelEvents: { + 'change:polling': 'toggleDownloadButtonActive' + }, + + toggleDownloadButtonActive: function() { + this.ui.downloadButton.prop('disabled', this.model.get('polling')); + }, + onShow: function() { var addOrReplaceInput = _.bind(this.model.addOrReplaceInput, this.model), controlModel = this.model.get('scenarios') @@ -255,6 +267,92 @@ var InputsView = Marionette.LayoutView.extend({ this.ui.tableButton.addClass('active'); this.model.set({ mode: models.constants.TABLE }); }, + + downloadCSV: function() { + var aoi = App.currentProject.get('area_of_interest'), + aoiVolumeModel = new tr55Models.AoiVolumeModel({ areaOfInterest: aoi }), + csvHeadings = [['scenario_name', 'precipitation_cm', 'runoff_cm', + 'evapotranspiration_cm', 'infiltration_cm', 'tss_load_cm', 'tss_runoff_cm', + 'tss_loading_rate_kgha', 'tn_load_cm', 'tn_runoff_cm', 'tn_loading_rate_kgha', + 'tp_load_cm', 'tp_runoff_cm', 'tp_loading_rate_kgha']], + precipitation = this.model.get('scenarios') + .findWhere({ active: true }) + .get('inputs') + .findWhere({ name: 'precipitation' }) + .get('value'), + csvData = this.model.get('scenarios') + .map(function(scenario) { + var result = scenario + .get('results') + .findWhere({ name: 'runoff' }) + .get('result'), + isPreColumbian = scenario.get('is_pre_columbian') || false, + isCurrentConditions = scenario.get('is_current_conditions'), + runoff, + quality, + tss, + tn, + tp; + + if (isPreColumbian) { + runoff = result.runoff.pc_unmodified; + quality = result.quality.pc_unmodified; + } else if (isCurrentConditions) { + runoff = result.runoff.unmodified; + quality = result.quality.unmodified; + } else { + runoff = result.runoff.modified; + quality = result.quality.modified; + } + + tss = quality[0]; + tn = quality[1]; + tp = quality[2]; + + return [ + scenario.get('name'), + coreUtils.convertToMetric(precipitation, 'in').toFixed(2), + runoff.runoff, + runoff.et, + runoff.inf, + tss.load, + tss.runoff, + aoiVolumeModel.getLoadingRate(tss.load), + tn.load, + tn.runoff, + aoiVolumeModel.getLoadingRate(tn.load), + tp.load, + tp.runoff, + aoiVolumeModel.getLoadingRate(tp.load), + ]; + }), + csv = csvHeadings + .concat(csvData) + .map(function (data) { + return data.join(', '); + }) + .join('\n'), + projectName = this.model.get('projectName'), + timeStamp = moment().format('MMDDYYYYHHmmss'), + fileName = projectName.replace(/[^a-z0-9+]+/gi, '_') + '_' + + timeStamp + '.csv', + blob = new Blob([csv], { type: 'application/octet-stream' }); + + if (window.navigator && window.navigator.msSaveOrOpenBlob) { + window.navigator.msSaveOrOpenBlob(blob, fileName); + } else { + var url = window.URL.createObjectURL(blob), + tmpLink = document.createElement('a'); + + tmpLink.download = fileName; + tmpLink.href = url; + tmpLink.type = 'attachment/csv;charset=utf-8'; + tmpLink.target = '_blank'; + document.body.appendChild(tmpLink); + tmpLink.click(); + document.body.removeChild(tmpLink); + } + } }); var CompareModificationsPopoverView = Marionette.ItemView.extend({ @@ -932,6 +1030,7 @@ function getCompareScenarios(isTr55) { function showCompare() { var model_package = App.currentProject.get('model_package'), + projectName = App.currentProject.get('name'), isTr55 = model_package === modelingModels.TR55_PACKAGE, scenarios = getCompareScenarios(isTr55), tabs = isTr55 ? getTr55Tabs(scenarios) : getGwlfeTabs(scenarios), @@ -941,6 +1040,7 @@ function showCompare() { controls: controls, tabs: tabs, scenarios: scenarios, + projectName: projectName, }); if (isTr55) { From 0aeee9c8b174394927b75fe0cf1c7544df9a58a6 Mon Sep 17 00:00:00 2001 From: Kelly Innes Date: Thu, 12 Oct 2017 17:07:20 -0400 Subject: [PATCH 066/172] Create climate & precipitation legends - create & configure continuous legend for climate tabs using linear gradients - add label ticks & values --- src/mmw/js/src/core/layerPicker.js | 24 +++++- src/mmw/js/src/core/models.js | 8 ++ .../layerPickerColorRampLegendTmpl.html | 27 +++++++ .../src/core/templates/layerPickerLayer.html | 2 +- src/mmw/mmw/settings/layer_settings.py | 10 ++- src/mmw/sass/components/_layerpicker.scss | 77 ++++++++++++++++++- 6 files changed, 140 insertions(+), 8 deletions(-) create mode 100644 src/mmw/js/src/core/templates/layerPickerColorRampLegendTmpl.html diff --git a/src/mmw/js/src/core/layerPicker.js b/src/mmw/js/src/core/layerPicker.js index a22d11a52..cb7038021 100644 --- a/src/mmw/js/src/core/layerPicker.js +++ b/src/mmw/js/src/core/layerPicker.js @@ -10,6 +10,7 @@ var $ = require('jquery'), layerPickerGroupTmpl = require('./templates/layerPickerGroup.html'), layerPickerLayerTmpl = require('./templates/layerPickerLayer.html'), layerPickerLegendTmpl = require('./templates/layerPickerLegend.html'), + layerPickerColorRampLegendTmpl = require('./templates/layerPickerColorRampLegendTmpl.html'), layerPickerNavTmpl = require('./templates/layerPickerNav.html'), opacityControlTmpl = require('./templates/opacityControl.html'), timeSliderTmpl = require('./templates/timeSliderControl.html'); @@ -112,6 +113,18 @@ var LayerPickerLegendView = Marionette.ItemView.extend({ } }); +var LayerPickerColorRampLegendView = Marionette.ItemView.extend({ + template: layerPickerColorRampLegendTmpl, + + templateHelpers: function() { + return { + colorRampId: this.model.get('colorRampId'), + legendUnitsLabel: this.model.get('legendUnitsLabel'), + legendUnitBreaks: this.model.get('legendUnitBreaks'), + }; + } +}); + /* The individual layers in each layer group */ var LayerPickerLayerView = Marionette.ItemView.extend({ template: layerPickerLayerTmpl, @@ -133,21 +146,24 @@ var LayerPickerLayerView = Marionette.ItemView.extend({ layerDisplay: this.model.get('display'), layerClass: this.model.get('active') ? 'layerpicker-title active' : 'layerpicker-title', isDisabled: this.model.get('disabled'), + useColorRamp: this.model.get('useColorRamp') }; }, onRender: function() { + var legendTooltipContent = this.model.get('useColorRamp') ? + new LayerPickerColorRampLegendView({ model: this.model }) : + new LayerPickerLegendView({ model: this.model }); + this.ui.layerHelpIcon.popover({ trigger: 'focus', viewport: { 'selector': '.map-container', 'padding': 10 }, - content: new LayerPickerLegendView({ - model: this.model, - }).render().el + content: legendTooltipContent.render().el }); - }, + } }); /* The list of layers in a layer group */ diff --git a/src/mmw/js/src/core/models.js b/src/mmw/js/src/core/models.js index 5102ef833..6f4f1cc33 100644 --- a/src/mmw/js/src/core/models.js +++ b/src/mmw/js/src/core/models.js @@ -144,6 +144,10 @@ var LayerModel = Backbone.Model.extend({ legendMapping: null, cssClassPrefix: null, active: false, + useColorRamp: false, + colorRampId: null, + legendUnitsLabel: null, + legendUnitBreaks: null, }, buildLayer: function(layerSettings, layerType, initialActive) { @@ -202,6 +206,10 @@ var LayerModel = Backbone.Model.extend({ legendMapping: layerSettings.legend_mapping, cssClassPrefix: layerSettings.css_class_prefix, active: layerSettings.display === initialActive ? true : false, + useColorRamp: layerSettings.use_color_ramp || false, + colorRampId: layerSettings.color_ramp_id || null, + legendUnitsLabel: layerSettings.legend_units_label || null, + legendUnitBreaks: layerSettings.legend_unit_breaks || null, }); } }); diff --git a/src/mmw/js/src/core/templates/layerPickerColorRampLegendTmpl.html b/src/mmw/js/src/core/templates/layerPickerColorRampLegendTmpl.html new file mode 100644 index 000000000..efab0fab4 --- /dev/null +++ b/src/mmw/js/src/core/templates/layerPickerColorRampLegendTmpl.html @@ -0,0 +1,27 @@ +
        +
        +
        +
        +
        + {% for break in legendUnitBreaks %} +
        + {% endfor %} +
        +
        + {% for break in legendUnitBreaks %} + {% if loop.last %} +
        + {{break}} +
        + {% else %} +
        + {{break}} +
        + {% endif %} + {% endfor %} +
        +
        +
        + {{legendUnitsLabel}} +
        +
        diff --git a/src/mmw/js/src/core/templates/layerPickerLayer.html b/src/mmw/js/src/core/templates/layerPickerLayer.html index 5b718746c..cb8090cde 100644 --- a/src/mmw/js/src/core/templates/layerPickerLayer.html +++ b/src/mmw/js/src/core/templates/layerPickerLayer.html @@ -5,7 +5,7 @@ > {{ layerDisplay }} - {% if not isDisabled and legendMapping %} + {% if not isDisabled and (legendMapping or useColorRamp) %} .popover-content { border: #fff 1px solid; } @@ -19,7 +23,7 @@ left: 265px; bottom: 23px; width: 150px; - padding: 10px; + padding: 8px; background-color: #fff; box-shadow: 0 0 6px $black-74; font-size: 13px; @@ -205,3 +209,74 @@ span.time-slider-end { font-size: 13px; padding: 0 0 8px 6px; } + +#temperature-legend { + height: 20px; + width: 300px; + margin-left: 2px; + margin-right: 2px; + background: linear-gradient(90deg, rgba(46, 0, 103, 1) 0%, rgba(141, 20, 255, 1) 16.666667%, rgba(165, 15, 245, 1) 20.0%, rgba(189, 10, 235, 1) 23.333333%, rgba(213, 5, 225, 1) 26.666667%, rgba(238, 0, 215, 1) 30.0%, rgba(178, 8, 225, 1) 33.333333%, rgba(119, 16, 235, 1) 36.666667%, rgba(59, 25, 245, 1) 40.0%, rgba(0, 34, 255, 1) 43.333333%, rgba(0, 77, 199, 1) 46.666667%, rgba(0, 121, 143, 1) 50.0%, rgba(0, 164, 86, 1) 53.333333%, rgba(0, 208, 31, 1) 56.666667%, rgba(59, 211, 23, 1) 60.0%, rgba(117, 214, 15, 1) 63.333333%, rgba(176, 217, 7, 1) 66.666667%, rgba(236, 221, 0, 1) 70.0%, rgba(240, 165, 0, 1) 73.333333%, rgba(245, 110, 2, 1) 76.666667%, rgba(250, 55, 2, 1) 80.0%, rgba(255, 0, 4, 1) 83.333333%, rgba(87, 0, 16, 1) 100%); +} + +#precipitation-legend { + height: 20px; + width: 300px; + margin-left: 3px; + margin-right: 2px; + background: linear-gradient(90deg, rgba(230, 111, 0, 1) 0%, rgba(246, 155, 56, 1) 8%, rgba(62, 178, 189, 1) 16%, rgba(97, 212, 231, 1) 24%, rgba(3, 116, 116, 1) 32%, rgba(0, 49, 96, 1) 100%) +} + +.climate-legend-tooltip { + height: 65px; +} + +.climate-legend-unit-breaks-labels { + font-size: 0.8em; + position: absolute; + left: 0px; +} + +.climate-legend-label-ticks { + display: table; + width: 100%; + table-layout: fixed; +} + +.climate-legend-label-tick { + white-space: nowrap; + overflow: hidden; + text-align: center; + text-overflow: ellipsis; + display: table-cell; +} + +.climate-legend-label-tick::after { + display: block; + text-align: center; + content: '|'; +} + +.climate-legend-label-values { + display: table; + width: 100%; + table-layout: fixed; +} + +.climate-legend-label-value { + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; + margin: 0 auto; + display: table-cell; + text-align: center; +} + +.climate-legend-label-value.max { + padding-right: 10px; +} + +.climate-legend-units-label { + font-weight: bold; + text-align: center; + padding-top: 30px; +} From 50637b8138b9b92c62038a18d6db74ed2d03f3e5 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 17 Oct 2017 16:20:49 -0400 Subject: [PATCH 067/172] Rearchitect WDC Detail View for refreshes The main WDC Detail view is a Marionette.LayoutView. Previously, it featured a number of inline elements that needed to be updated on model changes, such as the fetching spinner, the error icon, and the most recent result date. These were siblings to other Marionette.ItemViews, like the Table View and the Chart View. Unfortunately, when LayoutViews are rendered, all their child views are detached, destroyed, recreated, and reattached. This leads to unnecessary revision of the Table View which is updated separately, and can cause scroll jumps in some cases because the entire view is re-rendering. To get around this, we follow the recommended Marionette pattern of never re-rendering the LayoutView. The static parts of the LayoutView remain there, which are only rendered once when the detail view is loaded. We split the rest into two additional views: a Status which has the error and fetching icons, and a Switcher which has the most recent result date, and the table / chart switcher buttons. All the child ItemViews listen to the same model fields on their own and update accordingly. --- .../templates/resultDetailsCuahsi.html | 17 +-- .../templates/resultDetailsCuahsiStatus.html | 4 + .../resultDetailsCuahsiSwitcher.html | 7 + src/mmw/js/src/data_catalog/views.js | 132 +++++++++++------- 4 files changed, 97 insertions(+), 63 deletions(-) create mode 100644 src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiStatus.html create mode 100644 src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiSwitcher.html diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html index 9d984ec1c..b48bef52a 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsi.html @@ -59,21 +59,12 @@

         Web Services

        -
        -
        - -
        +

        Source Data

        -
        - - -
        -

        - Last value collected {{ last_date|toTimeAgo }} -

        -
        - +
        +
        +

        Citation

        {{ service_citation }}

        diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiStatus.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiStatus.html new file mode 100644 index 000000000..b4111dec7 --- /dev/null +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiStatus.html @@ -0,0 +1,4 @@ +
        +
        + +
        diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiSwitcher.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiSwitcher.html new file mode 100644 index 000000000..9e21b3292 --- /dev/null +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiSwitcher.html @@ -0,0 +1,7 @@ +
        + + +
        +

        + Last value collected {{ last_date|toTimeAgo }} +

        diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index b3a5ad6d0..1cdcf3643 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -22,6 +22,8 @@ var $ = require('jquery'), resultDetailsCinergiTmpl = require('./templates/resultDetailsCinergi.html'), resultDetailsHydroshareTmpl = require('./templates/resultDetailsHydroshare.html'), resultDetailsCuahsiTmpl = require('./templates/resultDetailsCuahsi.html'), + resultDetailsCuahsiStatusTmpl = require('./templates/resultDetailsCuahsiStatus.html'), + resultDetailsCuahsiSwitcherTmpl = require('./templates/resultDetailsCuahsiSwitcher.html'), resultDetailsCuahsiChartTmpl = require('./templates/resultDetailsCuahsiChart.html'), resultDetailsCuahsiTableTmpl = require('./templates/resultDetailsCuahsiTable.html'), resultDetailsCuahsiTableRowVariableColTmpl = require('./templates/resultDetailsCuahsiTableRowVariableCol.html'), @@ -484,58 +486,40 @@ var ResultDetailsCuahsiView = ResultDetailsBaseView.extend({ templateHelpers: function() { var id = this.model.get('id'), - location = id.substring(id.indexOf(':') + 1), - fetching = this.model.get('fetching'), - error = this.model.get('error'), - last_date = this.model.get('end_date'); - - if (!fetching && !error) { - var variables = this.model.get('variables'), - last_dates = variables.map(function(v) { - var values = v.get('values'); - - if (values.length > 0) { - return new Date(values.last().get('datetime')); - } else { - return new Date('07/04/1776'); - } - }); - - last_dates.push(new Date(last_date)); - last_date = Math.max.apply(null, last_dates); - } + location = id.substring(id.indexOf(':') + 1); return { location: location, - last_date: last_date, }; }, - regions: { - valuesRegion: '#cuahsi-values-region', - }, - ui: _.defaults({ - chartButton: '#cuahsi-button-chart', - tableButton: '#cuahsi-button-table', + chartRegion: '#cuahsi-chart-region', + tableRegion: '#cuahsi-table-region', }, ResultDetailsBaseView.prototype.ui), - events: _.defaults({ - 'click @ui.chartButton': 'setChartMode', - 'click @ui.tableButton': 'setTableMode', - }, ResultDetailsBaseView.prototype.events), + regions: { + statusRegion: '#cuahsi-status-region', + switcherRegion: '#cuahsi-switcher-region', + chartRegion: '#cuahsi-chart-region', + tableRegion: '#cuahsi-table-region', + }, modelEvents: { - 'change:fetching': 'render', - 'change:mode': 'showValuesRegion', + 'change:mode': 'showChartOrTable', }, initialize: function() { + this.model.set('mode', 'table'); this.model.fetchCuahsiValues(); }, - onRender: function() { - this.showValuesRegion(); + onShow: function() { + var variables = this.model.get('variables'); + + this.statusRegion.show(new CuahsiStatusView({ model: this.model })); + this.switcherRegion.show(new CuahsiSwitcherView({ model: this.model })); + this.tableRegion.show(new CuahsiTableView({ collection: variables })); }, onDomRefresh: function() { @@ -546,32 +530,80 @@ var ResultDetailsCuahsiView = ResultDetailsBaseView.extend({ }); }, - showValuesRegion: function() { - if (!this.valuesRegion) { - // Don't attempt to display values if this view - // has been unloaded. - return; + showChartOrTable: function() { + if (this.model.get('mode') === 'table') { + this.ui.chartRegion.addClass('hidden'); + this.ui.tableRegion.removeClass('hidden'); + } else { + this.ui.chartRegion.removeClass('hidden'); + this.ui.tableRegion.addClass('hidden'); + + if (!this.chartRegion.hasView()) { + this.chartRegion.show(new CuahsiChartView({ + collection: this.model.get('variables'), + })); + } } + } +}); - var mode = this.model.get('mode'), - variables = this.model.get('variables'), - view = mode === 'table' ? - new CuahsiTableView({ collection: variables }) : - new CuahsiChartView({ collection: variables }); +var CuahsiStatusView = Marionette.ItemView.extend({ + template: resultDetailsCuahsiStatusTmpl, - this.valuesRegion.show(view); + modelEvents: { + 'change:fetching change:error': 'render', + }, +}); + +var CuahsiSwitcherView = Marionette.ItemView.extend({ + template: resultDetailsCuahsiSwitcherTmpl, + + ui: { + chartButton: '#cuahsi-button-chart', + tableButton: '#cuahsi-button-table', + }, + + events: { + 'click @ui.chartButton': 'setChartMode', + 'click @ui.tableButton': 'setTableMode', + }, + + modelEvents: { + 'change:fetching change:mode': 'render', + }, + + templateHelpers: function() { + var fetching = this.model.get('fetching'), + error = this.model.get('error'), + last_date = this.model.get('end_date'); + + if (!fetching && !error) { + var variables = this.model.get('variables'), + last_dates = variables.map(function(v) { + var values = v.get('values'); + + if (values.length > 0) { + return new Date(values.last().get('datetime')); + } else { + return new Date('07/04/1776'); + } + }); + + last_dates.push(new Date(last_date)); + last_date = Math.max.apply(null, last_dates); + } + + return { + last_date: last_date, + }; }, setChartMode: function() { this.model.set('mode', 'chart'); - this.ui.chartButton.addClass('active'); - this.ui.tableButton.removeClass('active'); }, setTableMode: function() { this.model.set('mode', 'table'); - this.ui.tableButton.addClass('active'); - this.ui.chartButton.removeClass('active'); } }); From f08ab89082fb84b5ba51614cdaba954bd9d81c7d Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 17 Oct 2017 16:32:15 -0400 Subject: [PATCH 068/172] Add Chart --- src/mmw/js/src/data_catalog/models.js | 9 ++ .../templates/resultDetailsCuahsiChart.html | 8 ++ src/mmw/js/src/data_catalog/views.js | 117 ++++++++++++++++++ src/mmw/sass/pages/_data-catalog.scss | 8 ++ 4 files changed, 142 insertions(+) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index 252c0674c..f5b4d5df7 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -598,6 +598,15 @@ var CuahsiVariable = Backbone.Model.extend({ units: response.variable.units.abbreviation, most_recent_value: mrv, }; + }, + + getChartData: function() { + return this.get('values').map(function(v) { + return [ + moment(v.get('datetime')).valueOf(), + parseFloat(v.get('value')) + ]; + }); } }); diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiChart.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiChart.html index e69de29bb..31e0a2580 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiChart.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCuahsiChart.html @@ -0,0 +1,8 @@ + +
        diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 1cdcf3643..344c7be6c 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -2,7 +2,9 @@ var $ = require('jquery'), _ = require('lodash'), + Backbone = require('../../shim/backbone'), Marionette = require('../../shim/backbone.marionette'), + HighstockChart = require('../../shim/highstock'), App = require('../app'), analyzeViews = require('../analyze/views.js'), settings = require('../core/settings'), @@ -681,6 +683,121 @@ var CuahsiTableView = Marionette.ItemView.extend({ var CuahsiChartView = Marionette.ItemView.extend({ template: resultDetailsCuahsiChartTmpl, + + ui: { + 'chartDiv': '#cuahsi-variable-chart', + 'select': 'select', + }, + + events: { + 'change @ui.select': 'selectVariable', + }, + + modelEvents: { + 'change:selected': 'renderChart', + }, + + templateHelpers: function() { + var variables = this.collection.map(function(v) { + return { + id: v.get('id'), + concept_keyword: v.get('concept_keyword'), + }; + }); + + return { + variables: variables, + }; + }, + + initialize: function(attrs) { + var selected = this.collection.first().get('id'); + + this.model = new Backbone.Model(); + this.model.set({ + selected: selected, + result: attrs.result, + }); + }, + + selectVariable: function() { + var selected = this.ui.select.val(); + + this.model.set('selected', selected); + }, + + onShow: function() { + this.renderChart(); + }, + + initializeChart: function(variable) { + var chart = new HighstockChart({ + chart: { + renderTo: 'cuahsi-variable-chart', + }, + + rangeSelector: { + selected: 1, + buttons: [ + { type: 'week', count: 1, text: '1w' }, + { type: 'week', count: 2, text: '2w' }, + { type: 'month', count: 1, text: '1m' }, + ], + }, + + xAxis: { + ordinal: false, + }, + + yAxis: { + title: { + text: variable.get('units'), + } + }, + + // TODO Check why this isn't working + lang: { + thousandsSep: ',' + }, + + title : { + text : null + }, + + series : [{ + name : variable.get('concept_keyword'), + data : variable.getChartData(), + color: '#389b9b', + tooltip: { + valueSuffix: ' ' + variable.get('units'), + valueDecimals: 2, + }, + }] + }); + + return chart; + }, + + renderChart: function() { + var id = this.model.get('selected'), + variable = this.collection.findWhere({ id: id }); + + if (!this.chart) { + this.chart = this.initializeChart(variable); + } else { + this.chart.yAxis[0].setTitle({ + text: variable.get('units'), + }); + + this.chart.series[0].update({ + name: variable.get('concept_keyword'), + data: variable.getChartData(), + tooltip: { + valueSuffix: ' ' + variable.get('units'), + }, + }); + } + } }); var ResultMapPopoverDetailView = Marionette.LayoutView.extend({ diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index 8c8ad64c7..a57716817 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -77,6 +77,14 @@ } } } + + #cuahsi-chart-region { + margin-top: 20px; + + .cuahsi-variable-select { + margin-bottom: 10px; + } + } } } From c3a8081c6c55776877f6886120a312ef164d7cd4 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 18 Oct 2017 17:16:21 -0400 Subject: [PATCH 069/172] Fix Predominantly Forested Compare Table Values Identical issue fixed by #2355 -- we were always using the `modified` results. Create two helper functions `getTR55RunoffResult` and `getTR55QualityResult` to handle digging into the proper value within a scenario's tr-55 result. Use them in the compare charts and tables --- src/mmw/js/src/compare/models.js | 40 +++++++------------------------- src/mmw/js/src/core/utils.js | 23 +++++++++++++++++- 2 files changed, 30 insertions(+), 33 deletions(-) diff --git a/src/mmw/js/src/compare/models.js b/src/mmw/js/src/compare/models.js index 8e22188e4..2850f6ddb 100644 --- a/src/mmw/js/src/compare/models.js +++ b/src/mmw/js/src/compare/models.js @@ -43,16 +43,6 @@ var ChartRowsCollection = Backbone.Collection.extend({ this.scenarios.forEach(function(scenario) { scenario.get('results').on('change', update); }); - }, - - getScenarioResults: function(typeKey) { - return this.scenarios.map(function(scenario) { - var resultKey = coreUtils.getTR55ResultKey(scenario), - result = scenario.get('results') - .findWhere({ name: typeKey }) - .get('result'); - return result[typeKey][resultKey]; - }); } }); @@ -62,7 +52,7 @@ var Tr55RunoffCharts = ChartRowsCollection.extend({ .get('inputs') .findWhere({ name: 'precipitation' }), precipitation = coreUtils.convertToMetric(precipitationInput.get('value'), 'in'), - results = this.getScenarioResults('runoff'); + results = this.scenarios.map(coreUtils.getTR55RunoffResult, coreUtils); this.forEach(function(chart) { var key = chart.get('key'), @@ -87,7 +77,7 @@ var Tr55RunoffCharts = ChartRowsCollection.extend({ var Tr55QualityCharts = ChartRowsCollection.extend({ update: function() { var aoivm = this.aoiVolumeModel, - results = this.getScenarioResults('quality'); + results = this.scenarios.map(coreUtils.getTR55WaterQualityResult, coreUtils); this.forEach(function(chart) { var name = chart.get('name'), @@ -135,19 +125,10 @@ var TableRowsCollection = Backbone.Collection.extend({ var Tr55RunoffTable = TableRowsCollection.extend({ update: function() { - var results = this.scenarios.map(function(scenario) { - return scenario.get('results') - .findWhere({ name: 'runoff' }) - .get('result'); - }), - get = function(key) { - return function(result) { - return result.runoff.modified[key]; - }; - }, - runoff = _.map(results, get('runoff')), - et = _.map(results, get('et' )), - inf = _.map(results, get('inf' )), + var results = this.scenarios.map(coreUtils.getTR55RunoffResult, coreUtils), + runoff = _.map(results, 'runoff'), + et = _.map(results, 'et' ), + inf = _.map(results, 'inf' ), rows = [ { name: "Runoff" , unit: "cm", values: runoff }, { name: "Evapotranspiration", unit: "cm", values: et }, @@ -161,15 +142,10 @@ var Tr55RunoffTable = TableRowsCollection.extend({ var Tr55QualityTable = TableRowsCollection.extend({ update: function() { var aoivm = this.aoiVolumeModel, - results = this.scenarios.map(function(scenario) { - return scenario.get('results') - .findWhere({ name: 'quality' }) - .get('result'); - }), + results = this.scenarios.map(coreUtils.getTR55WaterQualityResult, coreUtils), get = function(key) { return function(result) { - var measures = result.quality.modified, - load = _.find(measures, { measure: key }).load; + var load = _.find(result, { measure: key }).load; return aoivm.getLoadingRate(load); }; diff --git a/src/mmw/js/src/core/utils.js b/src/mmw/js/src/core/utils.js index 84a56d6e5..481982e3c 100644 --- a/src/mmw/js/src/core/utils.js +++ b/src/mmw/js/src/core/utils.js @@ -407,7 +407,7 @@ var utils = { // } // Use the scenario attributes to figure out which subresult should be // accessed. - getTR55ResultKey: function(scenario) { + _getTR55ResultKey: function(scenario) { if (scenario.get('is_current_conditions')) { return 'unmodified'; } else if (scenario.get('is_pre_columbian')) { @@ -416,6 +416,27 @@ var utils = { return 'modified'; }, + /** Access a scenario's result for a given type + * @param {ScenarioModel} scenario + * @param {string} typeKey: the type of result you want + * back; 'runoff', 'quality' + **/ + _getTR55ScenarioResult: function(scenario, typeKey) { + var resultKey = this._getTR55ResultKey(scenario), + result = scenario.get('results') + .findWhere({ name: typeKey}) + .get('result'); + return result[typeKey][resultKey]; + }, + + getTR55WaterQualityResult: function(scenario) { + return this._getTR55ScenarioResult(scenario, 'quality'); + }, + + getTR55RunoffResult: function(scenario) { + return this._getTR55ScenarioResult(scenario, 'runoff'); + }, + // Reverse sorting of a Backbone Collection. // Taken from http://stackoverflow.com/a/12220415/2053314 reverseSortBy: function(sortByFunction) { From a1892ad7f865b4b42d55a194fb3076bb14201ceb Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 18 Oct 2017 17:21:19 -0400 Subject: [PATCH 070/172] Clean up TR-55 Views * Remove mention of `compareMode`. We no longer reuse charts from `tr55/quality` or `tr55/runoff` in the compare view. * Use helper functions to access tr-55 results --- .../tr55/quality/templates/result.html | 14 +-- src/mmw/js/src/modeling/tr55/quality/views.js | 109 +++--------------- .../tr55/runoff/templates/result.html | 14 +-- src/mmw/js/src/modeling/tr55/runoff/views.js | 63 +++------- src/mmw/js/src/modeling/views.js | 3 +- 5 files changed, 46 insertions(+), 157 deletions(-) diff --git a/src/mmw/js/src/modeling/tr55/quality/templates/result.html b/src/mmw/js/src/modeling/tr55/quality/templates/result.html index 78a76079e..83c90cd06 100644 --- a/src/mmw/js/src/modeling/tr55/quality/templates/result.html +++ b/src/mmw/js/src/modeling/tr55/quality/templates/result.html @@ -1,11 +1,9 @@ -{% if showModelDescription %} -
        - Total loads delivered in a 24-hour hypothetical storm event -
        -

        - Simulated by EPA's STEP-L model algorithms -

        -{% endif %} +
        + Total loads delivered in a 24-hour hypothetical storm event +
        +

        + Simulated by EPA's STEP-L model algorithms +

        diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 33fa41c8b..54fa62a1f 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -510,9 +510,45 @@ var ResultDetailsCinergiView = ResultDetailsBaseView.extend({ var ResultDetailsHydroshareView = ResultDetailsBaseView.extend({ template: resultDetailsHydroshareTmpl, + modelEvents: { + 'change:fetching': 'render', + }, + + templateHelpers: function() { + var scimeta = this.model.get('scimeta'), + files = this.model.get('files'), + details_url = _.find(this.model.get('links'), {'type': 'details'}), + helpers = { + details_url: details_url ? details_url.href : null, + resource_type: '', + abstract: '', + creators: [], + subjects: '', + files: files ? files.toJSON() : [], + }; + + if (scimeta) { + var type = scimeta.get('type'); + + helpers.resource_type = type.substring(type.lastIndexOf('/') + 1, type.indexOf('Resource')); + helpers.creators = scimeta.get('creators').toJSON(); + helpers.subjects = scimeta.get('subjects').pluck('value').join(', '); + } + + return helpers; + }, + initialize: function() { this.model.fetchHydroshareDetails(); - } + }, + + onDomRefresh: function() { + window.closePopover(); + this.$('[data-toggle="popover"]').popover({ + placement: 'right', + trigger: 'focus', + }); + }, }); var ResultDetailsCuahsiView = ResultDetailsBaseView.extend({ From 12fca097e7a6188f146e7a16178b54dcf22c0da3 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 23 Oct 2017 16:20:14 -0400 Subject: [PATCH 107/172] Add abstract, contents to HydroShare Edit parent tab panel naming so it doesn't interfere with the nested tabs herein. --- src/mmw/js/src/core/filters.js | 24 ++++++++++ .../templates/resultDetailsHydroshare.html | 47 ++++++++++++++++++- src/mmw/js/src/data_catalog/views.js | 6 ++- src/mmw/sass/pages/_data-catalog.scss | 12 ++++- 4 files changed, 84 insertions(+), 5 deletions(-) diff --git a/src/mmw/js/src/core/filters.js b/src/mmw/js/src/core/filters.js index 780c32fd8..be8bfdf21 100644 --- a/src/mmw/js/src/core/filters.js +++ b/src/mmw/js/src/core/filters.js @@ -70,3 +70,27 @@ nunjucks.env.addFilter('split', function(str, splitChar, indexToReturn) { return items[indexToReturn]; }); + +nunjucks.env.addFilter('toFriendlyBytes', function(bytes) { + var roundToOneDecimal = function(x) { return Math.round(x * 10) / 10; }; + + if (bytes < 1024) { + return bytes + ' Bytes'; + } + + bytes /= 1024; + + if (bytes < 1024) { + return roundToOneDecimal(bytes) + ' KB'; + } + + bytes /= 1024; + + if (bytes < 1024) { + return roundToOneDecimal(bytes) + ' MB'; + } + + bytes /= 1024; + + return roundToOneDecimal(bytes) + ' GB'; +}); diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsHydroshare.html b/src/mmw/js/src/data_catalog/templates/resultDetailsHydroshare.html index 6745c7c77..d18ba10a7 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsHydroshare.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsHydroshare.html @@ -74,7 +74,52 @@


        - +
        + +
        +
        +

        + {{ abstract }} +

        +
        +
        +

        + Last Updated {{ updated_at|toDateWithoutTime }} +

        + + + + + + + + + {% for file in files %} + + + + + {% endfor %} + +
        File NameFile Size
        + {{ file.name }} + + {{ file.size|toFriendlyBytes }} +
        +
        +
        +

        Citation

        diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 54fa62a1f..468d9dc58 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -326,7 +326,7 @@ var ErrorView = Marionette.ItemView.extend({ }); var TabContentView = Marionette.LayoutView.extend({ - className: 'tab-pane', + className: 'catalog-tab-pane tab-pane', id: function() { return this.model.id; }, @@ -398,7 +398,7 @@ var TabContentView = Marionette.LayoutView.extend({ }); var TabContentsView = Marionette.CollectionView.extend({ - className: 'tab-content', + className: 'catalog-tab-content tab-content', childView: TabContentView }); @@ -531,6 +531,7 @@ var ResultDetailsHydroshareView = ResultDetailsBaseView.extend({ var type = scimeta.get('type'); helpers.resource_type = type.substring(type.lastIndexOf('/') + 1, type.indexOf('Resource')); + helpers.abstract = scimeta.get('description'); helpers.creators = scimeta.get('creators').toJSON(); helpers.subjects = scimeta.get('subjects').pluck('value').join(', '); } @@ -548,6 +549,7 @@ var ResultDetailsHydroshareView = ResultDetailsBaseView.extend({ placement: 'right', trigger: 'focus', }); + this.$('[data-toggle="table"]').bootstrapTable(); }, }); diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index f1bfa3a36..c84ab08e7 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -85,6 +85,14 @@ margin-bottom: 10px; } } + + .hydroshare-nav-tabs > li > a { + padding: 10px 15px !important; + } + + .hydroshare-tab-content { + padding-top: 15px; + } } } @@ -167,14 +175,14 @@ overflow-y: auto; } - .tab-pane { + .catalog-tab-pane { position: absolute; top: 0; bottom: 0; width: 100%; } - .tab-content { + .catalog-tab-content { padding: 0; background: #E6E6E6; top: 0; From 4b2d220a917475a69f3ff2099f6d83e02579dce8 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 25 Oct 2017 10:49:40 -0400 Subject: [PATCH 108/172] Add More Fields To CINERGI API Results Add contact people, resource type, resource topic categories, web resource links, and web service links to display in the cinergi detail view --- src/mmw/apps/bigcz/clients/cinergi/models.py | 11 +++- src/mmw/apps/bigcz/clients/cinergi/search.py | 52 ++++++++++++++++--- .../apps/bigcz/clients/cinergi/serializers.py | 27 +++++++++- 3 files changed, 80 insertions(+), 10 deletions(-) diff --git a/src/mmw/apps/bigcz/clients/cinergi/models.py b/src/mmw/apps/bigcz/clients/cinergi/models.py index 66992cf7f..28042363f 100644 --- a/src/mmw/apps/bigcz/clients/cinergi/models.py +++ b/src/mmw/apps/bigcz/clients/cinergi/models.py @@ -9,8 +9,10 @@ class CinergiResource(Resource): def __init__(self, id, description, author, links, title, created_at, updated_at, geom, cinergi_url, - source_name, contact_organizations, - categories, begin_date, end_date): + source_name, contact_organizations, contact_people, + categories, begin_date, end_date, + resource_type, resource_topic_categories, + web_resources, web_services): super(CinergiResource, self).__init__(id, description, author, links, title, created_at, updated_at, geom) @@ -18,6 +20,11 @@ def __init__(self, id, description, author, links, title, self.cinergi_url = cinergi_url self.source_name = source_name self.contact_organizations = contact_organizations + self.contact_people = contact_people self.categories = categories self.begin_date = begin_date self.end_date = end_date + self.resource_type = resource_type + self.resource_topic_categories = resource_topic_categories + self.web_resources = web_resources + self.web_services = web_services diff --git a/src/mmw/apps/bigcz/clients/cinergi/search.py b/src/mmw/apps/bigcz/clients/cinergi/search.py index f844b2ab1..25f5f4be5 100644 --- a/src/mmw/apps/bigcz/clients/cinergi/search.py +++ b/src/mmw/apps/bigcz/clients/cinergi/search.py @@ -90,15 +90,15 @@ def parse_cinergi_url(fileid): return '{}/geoportal/?filter=%22{}%22'.format(CINERGI_HOST, fileid) -def parse_contact_organizations(contact_organizations): +def parse_string_or_list(string_or_list): """ - Contact organizations can be either a list of strings, or + Fields like contact_organizations be either a list of strings, or a string. Make it always a list of strings """ - if isinstance(contact_organizations, basestring): - return [contact_organizations] + if isinstance(string_or_list, basestring): + return [string_or_list] - return contact_organizations + return string_or_list def parse_categories(source): @@ -160,6 +160,37 @@ def handle_data(self, data): return parser.contents[0] +def parse_web_resources(raw_resources): + if not raw_resources: + return [] + + resources = raw_resources + + if isinstance(raw_resources, dict): + resources = [raw_resources] + + return [{ + 'url': r.get('url_s'), + 'url_type': r.get('url_type_s') + } for r in resources] + + +def parse_web_services(raw_services): + if not raw_services: + return [] + + services = raw_services + + if isinstance(raw_services, dict): + services = [raw_services] + + return [{ + 'url': s.get('url_s'), + 'url_type': s.get('url_type_s'), + 'url_name': s.get('url_name_s') + } for s in services] + + def parse_record(item): source = item['_source'] geom = parse_geom(source) @@ -176,11 +207,18 @@ def parse_record(item): geom=geom, cinergi_url=parse_cinergi_url(source.get('fileid')), source_name=source.get('src_source_name_s'), - contact_organizations=parse_contact_organizations( + contact_organizations=parse_string_or_list( source.get('contact_organizations_s')), + contact_people=parse_string_or_list( + source.get('contact_people_s')), categories=parse_categories(source), begin_date=parse_date(begin_date), - end_date=parse_date(end_date)) + end_date=parse_date(end_date), + resource_type=source.get('apiso_Type_s'), + resource_topic_categories=parse_string_or_list( + source.get('apiso_TopicCategory_s')), + web_resources=parse_web_resources(source.get('resources_nst')), + web_services=parse_web_services(source.get('services_nst'))) def prepare_bbox(box): diff --git a/src/mmw/apps/bigcz/clients/cinergi/serializers.py b/src/mmw/apps/bigcz/clients/cinergi/serializers.py index 1c2001e95..f0ad22f9a 100644 --- a/src/mmw/apps/bigcz/clients/cinergi/serializers.py +++ b/src/mmw/apps/bigcz/clients/cinergi/serializers.py @@ -5,19 +5,44 @@ from rest_framework.serializers import (CharField, ListField, - DateTimeField) + DateTimeField, + Serializer) from apps.bigcz.serializers import ResourceSerializer +class CinergiWebResourceSerializer(Serializer): + url_type = CharField() + url = CharField() + + +class CinergiWebServiceSerializer(Serializer): + url_type = CharField() + url_name = CharField() + url = CharField() + + class CinergiResourceSerializer(ResourceSerializer): cinergi_url = CharField() source_name = CharField() contact_organizations = ListField( child=CharField() ) + contact_people = ListField( + child=CharField() + ) categories = ListField( child=CharField() ) begin_date = DateTimeField() end_date = DateTimeField() + resource_type = CharField() + resource_topic_categories = ListField( + child=CharField() + ) + web_resources = ListField( + child=CinergiWebResourceSerializer() + ) + web_services = ListField( + child=CinergiWebServiceSerializer() + ) From ffbc66cd584bcf15a2eedf85dcb6df5813c7ca6c Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Wed, 25 Oct 2017 14:22:28 -0400 Subject: [PATCH 109/172] Flesh out CINERGI Detail View --- src/mmw/js/src/data_catalog/models.js | 18 +++ .../templates/resultDetailsCinergi.html | 116 ++++++++++++++++-- src/mmw/js/src/data_catalog/views.js | 30 ++--- 3 files changed, 137 insertions(+), 27 deletions(-) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index c9742d9f4..014993a4a 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -355,6 +355,7 @@ var Result = Backbone.Model.extend({ active: false, show_detail: false, // Show this result as the detail view? variables: null, // CuahsiVariables Collection + categories: null, // Cinergi categories, [String] fetching: false, error: false, mode: 'table', @@ -512,6 +513,23 @@ var Result = Backbone.Model.extend({ return detailsUrl && detailsUrl.href; }, + topCinergiCategories: function(n) { + var categories = _.clone(this.get('categories')); + + if (!categories) { + return null; + } + + // Truncate if longer than n values + if (categories.length > n) { + categories = categories.slice(0, n + 1); + var lastIdx = categories.length - 1; + categories[lastIdx] = categories[lastIdx] + '...'; + } + + return categories.join('; '); + }, + toJSON: function() { return _.assign({}, this.attributes, { summary: this.getSummary(), diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html index 0e7e8a41c..65a949886 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html @@ -4,27 +4,117 @@ Back

        - {{ title }} + Resource: {{ title }}

        + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        Contacts + {% for contact in contacts %} +

        {{ contact }}

        + {% else %} + None provided + {% endfor %} +
        Organizations + {% for org in orgs %} +

        {{ org }}

        + {% else %} + None provided + {% endfor %} +
        Data collected between + {% if begin_date and begin_date|toDateWithoutTime == end_date|toDateWithoutTime %} + {{ begin_date|toDateWithoutTime }} + {% elif begin_date %} + {{ begin_date|toDateWithoutTime }} – {{ end_date|toDateWithoutTime }} + {% else %} + Unknown + {% endif %} +
        Subject + {{ top_categories }} +
        Resource Type + {{ resource_type }} / {{ resource_topic_categories_str }} +
        Source + {{ source_name }} +
        Catalog + CINERGI + + + +
        -
        - {% if author %} -

        - {{ author }} -

        - {% endif %}

        - {{ created_at|toDateFullYear }} + {% if details_url %} + +  Source Data + + {% endif %} + +  Web Services +

        - {% if not (description == "REQUIRED FIELD") %} + +
        + + {% if description and not (description == "REQUIRED FIELD") %} +

        Description

        {{ description }}

        {% endif %}

        - -  Repository - + Last Updated {{ updated_at|toDateWithoutTime }}

        + + {% if web_resources.length > 0 %} +
        +

        Web Resources

        + {% for resource in web_resources %} +
      • + {{resource.url_type}} +
      • + {% endfor %} + {% endif %} + + {% if web_services.length > 0 %} +
        +

        Web Services

        + {% for service in web_services %} +
      • + {{service.url_name}} ({{service.url_type}}) +
      • + {% endfor %} + {% endif %} diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 468d9dc58..8ea6770ff 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -418,21 +418,8 @@ var StaticResultView = Marionette.ItemView.extend({ } if (this.options.catalog === 'cinergi') { - var categories = _.clone(this.model.get('categories')); - - if (!categories) { - return null; - } - - // Truncate if longer than 8 values - if (categories.length > 8) { - categories = categories.slice(0, 9); - var lastIdx = categories.length -1; - categories[lastIdx] = categories[lastIdx] + '...'; - } - return { - 'top_categories': categories.join('; ') + 'top_categories': this.model.topCinergiCategories(8) }; } }, @@ -505,6 +492,21 @@ var ResultDetailsBaseView = Marionette.LayoutView.extend({ var ResultDetailsCinergiView = ResultDetailsBaseView.extend({ template: resultDetailsCinergiTmpl, + + templateHelpers: function() { + var topicCategories = this.model.get('resource_topic_categories'), + contactOrgs = this.model.get('contact_organizations'), + contactPeople = this.model.get('contact_people'); + + return { + 'orgs': contactOrgs ? contactOrgs.filter(utils.distinct) : null, + 'contacts': contactPeople ? contactPeople.filter(utils.distinct) : null, + 'top_categories': this.model.topCinergiCategories(20), + 'resource_topic_categories_str': topicCategories ? + topicCategories.join(", ") : null, + 'details_url': this.model.getDetailsUrl() + }; + } }); var ResultDetailsHydroshareView = ResultDetailsBaseView.extend({ From 21cb18d60dadf8bf168dc79296630ff35a909a90 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Thu, 26 Oct 2017 12:23:52 -0400 Subject: [PATCH 110/172] BiG-CZ: Close any open popups when closing the detail view --- src/mmw/js/src/data_catalog/views.js | 1 + 1 file changed, 1 insertion(+) diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 8ea6770ff..ff9bdc097 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -486,6 +486,7 @@ var ResultDetailsBaseView = Marionette.LayoutView.extend({ }, closeDetails: function() { + window.closePopover(); this.model.collection.closeDetail(); } }); From 7226a5de9ada0c2b9556d4ed6949f02d772943bd Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Thu, 26 Oct 2017 18:40:56 -0400 Subject: [PATCH 111/172] Collapse Cinergi Details Contact and Organizations Lists * When there are more than 2 organizations or contacts, add a "View x more organizations/contacts" button and truncate the remainder --- src/mmw/js/src/data_catalog/models.js | 10 ++++ .../templates/expandableTableRow.html | 12 ++++ .../templates/resultDetailsCinergi.html | 16 +----- src/mmw/js/src/data_catalog/views.js | 55 +++++++++++++++++-- src/mmw/sass/pages/_data-catalog.scss | 4 ++ 5 files changed, 77 insertions(+), 20 deletions(-) create mode 100644 src/mmw/js/src/data_catalog/templates/expandableTableRow.html diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index 014993a4a..261afcdb7 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -775,6 +775,15 @@ var HydroshareFiles = Backbone.Collection.extend({ model: HydroshareFile, }); +var ExpandableListModel = Backbone.Model.extend({ + defaults: { + expanded: false, + list_type: '', + truncate_at: 2, + list: null + } +}); + module.exports = { GriddedServicesFilter: GriddedServicesFilter, PrivateResourcesFilter: PrivateResourcesFilter, @@ -790,4 +799,5 @@ module.exports = { CuahsiValues: CuahsiValues, CuahsiVariable: CuahsiVariable, CuahsiVariables: CuahsiVariables, + ExpandableListModel: ExpandableListModel, }; diff --git a/src/mmw/js/src/data_catalog/templates/expandableTableRow.html b/src/mmw/js/src/data_catalog/templates/expandableTableRow.html new file mode 100644 index 000000000..ff6a9dc4b --- /dev/null +++ b/src/mmw/js/src/data_catalog/templates/expandableTableRow.html @@ -0,0 +1,12 @@ +{% if not list %} +None provided +{% else %} +{% for i in range(0, list.length if expanded else truncate_at) %} +

        {{ list[i] }}

        +{% endfor %} +{% if not expanded and list.length > truncate_at %} + +{% endif %} +{% endif %} diff --git a/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html b/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html index 65a949886..4dadafb44 100644 --- a/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html +++ b/src/mmw/js/src/data_catalog/templates/resultDetailsCinergi.html @@ -9,23 +9,11 @@

        - + - + diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index ff9bdc097..a45593d17 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -35,7 +35,8 @@ var $ = require('jquery'), resultMapPopoverDetailTmpl = require('./templates/resultMapPopoverDetail.html'), resultMapPopoverListTmpl = require('./templates/resultMapPopoverList.html'), resultMapPopoverListItemTmpl = require('./templates/resultMapPopoverListItem.html'), - resultMapPopoverControllerTmpl = require('./templates/resultMapPopoverController.html'); + resultMapPopoverControllerTmpl = require('./templates/resultMapPopoverController.html'), + expandableTableRowTmpl = require('./templates/expandableTableRow.html'); var ENTER_KEYCODE = 13, PAGE_SIZE = settings.get('data_catalog_page_size'), @@ -491,22 +492,64 @@ var ResultDetailsBaseView = Marionette.LayoutView.extend({ } }); +var ExpandableTableRow = Marionette.ItemView.extend({ + // model: ExpandableListModel + template: expandableTableRowTmpl, + ui: { + expandButton: '[data-action="expand"]' + }, + + events: { + 'click @ui.expandButton': 'expand' + }, + + modelEvents: { + 'change:expanded': 'render' + }, + + expand: function() { + this.model.set('expanded', true); + } +}); + var ResultDetailsCinergiView = ResultDetailsBaseView.extend({ template: resultDetailsCinergiTmpl, + regions: { + organizations: '[data-list-region="organizations"]', + contacts: '[data-list-region="contacts"]', + }, + templateHelpers: function() { - var topicCategories = this.model.get('resource_topic_categories'), - contactOrgs = this.model.get('contact_organizations'), - contactPeople = this.model.get('contact_people'); + var topicCategories = this.model.get('resource_topic_categories'); return { - 'orgs': contactOrgs ? contactOrgs.filter(utils.distinct) : null, - 'contacts': contactPeople ? contactPeople.filter(utils.distinct) : null, 'top_categories': this.model.topCinergiCategories(20), 'resource_topic_categories_str': topicCategories ? topicCategories.join(", ") : null, 'details_url': this.model.getDetailsUrl() }; + }, + + onShow: function() { + var contactOrgs = this.model.get('contact_organizations'), + contactPeople = this.model.get('contact_people'), + orgs = contactOrgs ? contactOrgs.filter(utils.distinct) : null, + contacts = contactPeople ? contactPeople.filter(utils.distinct) : null; + + this.organizations.show(new ExpandableTableRow({ + model: new models.ExpandableListModel({ + list_type: 'organizations', + list: orgs + }) + })); + + this.contacts.show(new ExpandableTableRow({ + model: new models.ExpandableListModel({ + list_type: 'contacts', + list: contacts + }) + })); } }); diff --git a/src/mmw/sass/pages/_data-catalog.scss b/src/mmw/sass/pages/_data-catalog.scss index c84ab08e7..a903df1f4 100644 --- a/src/mmw/sass/pages/_data-catalog.scss +++ b/src/mmw/sass/pages/_data-catalog.scss @@ -272,6 +272,10 @@ margin-right: 4px; } } + + .expand-list-btn { + padding: 0; + } } .data-catalog-resource-popover { From 05cb66d9fa4d1eefc7c7e616f3fed036b49b40d6 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Fri, 27 Oct 2017 10:52:23 -0400 Subject: [PATCH 112/172] Update account page to match actual throttle rates * Update the copy on the account page to state the current default throttle rates --- src/mmw/js/src/account/templates/account.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mmw/js/src/account/templates/account.html b/src/mmw/js/src/account/templates/account.html index 23c391724..0d9e31f91 100644 --- a/src/mmw/js/src/account/templates/account.html +++ b/src/mmw/js/src/account/templates/account.html @@ -1,7 +1,7 @@

        Account

        -

        You can use your API Key to make up to 2 requests per minute

        +

        You can use your API Key to make up to 20 requests per minute, and 5000 total per day.

        {% if error %}

        {{error}} From e76168e589345eb7f31484359c70cca89e15f2ca Mon Sep 17 00:00:00 2001 From: Casey Cesari Date: Thu, 26 Oct 2017 16:50:02 -0400 Subject: [PATCH 113/172] Ensure Celery tasks get routed to matching worker Configure Celery to only send tasks to workers that match the specified stack color. This prevents tasks from getting assigned to an old worker in a blue/green deployment. Configuration is largely based on http://docs.celeryproject.org/en/latest/userguide/routing.html#manual-routing and https://github.com/rduplain/celery-simple-example. Refs #2374 Refs #2374 --- src/mmw/mmw/settings/base.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/src/mmw/mmw/settings/base.py b/src/mmw/mmw/settings/base.py index be79a395d..bb9336133 100644 --- a/src/mmw/mmw/settings/base.py +++ b/src/mmw/mmw/settings/base.py @@ -130,8 +130,14 @@ def get_env_setting(setting): STATSD_CELERY_SIGNALS = True CELERY_CREATE_MISSING_QUEUES = True CELERY_CHORD_PROPAGATES = True -CELERY_DEFAULT_QUEUE = STACK_COLOR -CELERY_DEFAULT_ROUTING_KEY = "task.%s" % STACK_COLOR +CELERY_TASK_DEFAULT_QUEUE = STACK_COLOR +CELERY_TASK_QUEUES = { + STACK_COLOR: { + 'binding_key': "task.%s" % STACK_COLOR, + } +} +CELERY_TASK_DEFAULT_EXCHANGE = 'tasks' +CELERY_TASK_DEFAULT_ROUTING_KEY = "task.%s" % STACK_COLOR # END CELERY CONFIGURATION From d8f421bd0a3de366ef4a055e5be42e65a61bb872 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Fri, 27 Oct 2017 16:46:53 -0400 Subject: [PATCH 114/172] Use serverResults field for saving CUAHSI results In preparation for performing client-side free text search on CUAHSI results, where we will get all results from the server once and then filter them on the client side. For the CUAHSI catalog, instead of saving the results to the `results` field directly, we instead use the `serverResults` field. Currently, all the results are copied over to the `results` field, but in coming commits we will add filtering. --- src/mmw/js/src/data_catalog/controllers.js | 1 + src/mmw/js/src/data_catalog/models.js | 33 ++++++++++++++++++---- 2 files changed, 28 insertions(+), 6 deletions(-) diff --git a/src/mmw/js/src/data_catalog/controllers.js b/src/mmw/js/src/data_catalog/controllers.js index f24df368e..e7cb68300 100644 --- a/src/mmw/js/src/data_catalog/controllers.js +++ b/src/mmw/js/src/data_catalog/controllers.js @@ -46,6 +46,7 @@ var DataCatalogController = { name: 'WDC', is_pageable: false, results: new models.Results(null, { catalog: 'cuahsi' }), + serverResults: new models.Results(null, { catalog: 'cuahsi' }), filters: new models.FilterCollection([ dateFilter, new models.GriddedServicesFilter() diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index 261afcdb7..e3f804a6b 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -153,6 +153,7 @@ var Catalog = Backbone.Model.extend({ stale: false, // Should search run when catalog becomes active? active: false, results: null, // Results collection + serverResults: null, // Results collection resultCount: 0, filters: null, // FiltersCollection is_pageable: true, @@ -229,6 +230,9 @@ var Catalog = Backbone.Model.extend({ startSearch: function(page) { var filters = this.get('filters'), + results = this.id === 'cuahsi' ? + this.get('serverResults') : + this.get('results'), dateFilter = filters.findWhere({ id: 'date' }), fromDate = null, toDate = null; @@ -272,7 +276,7 @@ var Catalog = Backbone.Model.extend({ contentType: 'application/json' }; - return this.get('results') + return results .fetch(request) .done(_.bind(this.doneSearch, this)) .fail(_.bind(this.failSearch, this)) @@ -280,12 +284,29 @@ var Catalog = Backbone.Model.extend({ }, doneSearch: function(response) { - var data = _.findWhere(response, { catalog: this.id }); + var data = _.findWhere(response, { catalog: this.id }), + setFields = { + page: data.page || 1, + resultCount: data.count, + }; - this.set({ - page: data.page || 1, - resultCount: data.count, - }); + if (this.id === 'cuahsi') { + var results = this.get('results'), + filtered = this.get('serverResults').toJSON(); + + if (results === null) { + results = new Results(filtered, { catalog: 'cuahsi' }); + } else { + results.reset(filtered); + } + + _.assign(setFields, { + results: results, + resultCount: results.length, + }); + } + + this.set(setFields); }, failSearch: function(response, textStatus) { From 8db76b04272334a7499fb744d3cb63df1dec64a5 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Fri, 27 Oct 2017 16:54:47 -0400 Subject: [PATCH 115/172] Free Text Search A textFilter method is added to Result model that returns truthy if a given query is found in any of the CUAHSI fields. The Results collection also has an identically named method, that filters the collection to those results that satisfy the query. When the first search completes, we filter the serverResults by the given query and include only those in the results field. On subsequent searches, we check to see if the geometry has changed or if only the query has changed. Since we fetch all results for a given geometry in the first search, we don't need to do other server searches until the geometry changes. If the geometry is the same, and only the text has changed, then we re-run the filter on the saved serverResults, and re-populate results with the filtered items. --- src/mmw/js/src/data_catalog/models.js | 54 +++++++++++++++++++++++++-- 1 file changed, 51 insertions(+), 3 deletions(-) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index e3f804a6b..085c76777 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -191,8 +191,25 @@ var Catalog = Backbone.Model.extend({ var self = this, error = this.get('error'), stale = this.get('stale'), - isSameSearch = query === this.get('query') && - geom === this.get('geom'); + isCuahsi = this.id === 'cuahsi', + isSameQuery = query === this.get('query'), + isSameGeom = geom === this.get('geom'), + isSameSearch = isSameQuery && isSameGeom; + + if (isCuahsi && isSameGeom && !isSameQuery) { + this.set({ loading: true }); + + var results = this.get('serverResults').textFilter(query); + this.get('results').reset(results); + + this.set({ + loading: false, + query: query, + resultCount: results.length + }); + + return $.when(); + } if (!isSameSearch || stale || error) { this.cancelSearch(); @@ -292,7 +309,8 @@ var Catalog = Backbone.Model.extend({ if (this.id === 'cuahsi') { var results = this.get('results'), - filtered = this.get('serverResults').toJSON(); + filtered = this.get('serverResults') + .textFilter(this.get('query')); if (results === null) { results = new Results(filtered, { catalog: 'cuahsi' }); @@ -518,6 +536,23 @@ var Result = Backbone.Model.extend({ return this.fetchPromise; }, + textFilter: function(query) { + var fields = [ + this.get('id').toLowerCase(), + this.get('title').toLowerCase(), + this.get('description').toLowerCase(), + (this.get('service_citation') || '').toLowerCase(), + (this.get('service_title') || '').toLowerCase(), + (this.get('service_org') || '').toLowerCase(), + this.get('sample_mediums').join(' ').toLowerCase(), + this.get('variables').pluck('concept_keyword').join(' ').toLowerCase(), + ]; + + return _.some(fields, function(field) { + return field.indexOf(query) >= 0; + }); + }, + getSummary: function() { var text = this.get('description') || ''; if (text.length <= DESCRIPTION_MAX_LENGTH) { @@ -599,6 +634,19 @@ var Results = Backbone.Collection.extend({ } currentDetail.set('show_detail', false); + }, + + textFilter: function(query) { + var lcQueries = query.toLowerCase().split(' ').filter(function(x) { + // Exclude empty, search logic terms + return x !== '' && x !== 'and' && x !== 'or'; + }); + + return this.filter(function(result) { + return _.every(lcQueries, function(lcQuery) { + return result.textFilter(lcQuery); + }); + }); } }); From 3af471f2d4982392b9908f957e28725f056be2d2 Mon Sep 17 00:00:00 2001 From: Taylor Nation Date: Mon, 30 Oct 2017 15:04:07 -0400 Subject: [PATCH 116/172] Pin dependency to numpy 1.13 --- src/mmw/requirements/base.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/src/mmw/requirements/base.txt b/src/mmw/requirements/base.txt index 4d8a84f6a..0a26a4bf7 100644 --- a/src/mmw/requirements/base.txt +++ b/src/mmw/requirements/base.txt @@ -20,3 +20,4 @@ python-dateutil==2.6.0 suds==0.4 django_celery_results==1.0.1 ulmo==0.8.4 +numpy==1.13.0 From aca7ee59c5ae6fbd12c8da70352da762f6fbd9a1 Mon Sep 17 00:00:00 2001 From: Justin Walgran Date: Fri, 27 Oct 2017 11:09:36 -0700 Subject: [PATCH 117/172] Update to latest boostrap-select release from NPM It looks like the issue that was addressed in the Azavea fork of bootstrap-select was fixed in the main repo. Switching to the latest release version does not the behavior. --- src/mmw/npm-shrinkwrap.json | 6 +++--- src/mmw/package.json | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/mmw/npm-shrinkwrap.json b/src/mmw/npm-shrinkwrap.json index 695fdb9d9..7aa131891 100644 --- a/src/mmw/npm-shrinkwrap.json +++ b/src/mmw/npm-shrinkwrap.json @@ -50,9 +50,9 @@ "resolved": "https://registry.npmjs.org/bootstrap-datepicker/-/bootstrap-datepicker-1.7.1.tgz" }, "bootstrap-select": { - "version": "1.6.4", - "from": "../../tmp/npm-25360-e04608df/1472064034158-0.4662157059647143/f1755efa102a41e43fc367ba36ce905b8f2b90e1", - "resolved": "git://github.com/azavea/bootstrap-select#f1755efa102a41e43fc367ba36ce905b8f2b90e1" + "version": "1.12.4", + "from": "https://registry.npmjs.org/bootstrap-select/-/bootstrap-select-1.12.4.tgz", + "resolved": "https://registry.npmjs.org/bootstrap-select/-/bootstrap-select-1.12.4.tgz" }, "bootstrap-table": { "version": "1.11.0", diff --git a/src/mmw/package.json b/src/mmw/package.json index dd3c40544..8de2dfe08 100644 --- a/src/mmw/package.json +++ b/src/mmw/package.json @@ -24,7 +24,7 @@ "blueimp-md5": "1.1.0", "bootstrap": "3.3.4", "bootstrap-datepicker": "1.7.1", - "bootstrap-select": "git://github.com/azavea/bootstrap-select", + "bootstrap-select": "1.12.4", "bootstrap-table": "1.11.0", "browserify": "9.0.3", "chai": "1.10.0", From 68aba6600225e60e2fb6de0c085a619f04ead515 Mon Sep 17 00:00:00 2001 From: Justin Walgran Date: Tue, 24 Oct 2017 17:17:24 -0700 Subject: [PATCH 118/172] Add a profile form to the login process - The form is shown after logging in only if both the `was_skipped` and `is_complete` flags on the `UserProfile` are false. - Making the decision whether or not to show the form required passing the login response as an argument through the chain of success event handlers. - The options for the `country` and `user_type` fields are passed from Python to Javascript via the existing `window.clientSettings` object. - The nunjucks template is built at compile time, not at run time, so there is some Javascript that completes the construction of the option tags on the client. - I edited `bundle.sh` to include the `bootstrap-select` stylesheet that was previously added to the project. --- src/mmw/apps/home/views.py | 8 ++ src/mmw/apps/user/urls.py | 2 + src/mmw/apps/user/views.py | 31 ++++- src/mmw/bundle.sh | 1 + src/mmw/js/src/app.js | 27 +++++ src/mmw/js/src/user/models.js | 27 ++++- src/mmw/js/src/user/templates/baseModal.html | 17 +++ .../src/user/templates/userProfileModal.html | 25 +++++ src/mmw/js/src/user/views.js | 106 +++++++++++++++++- src/mmw/sass/components/_inputs.scss | 17 +++ 10 files changed, 255 insertions(+), 6 deletions(-) create mode 100644 src/mmw/js/src/user/templates/userProfileModal.html diff --git a/src/mmw/apps/home/views.py b/src/mmw/apps/home/views.py index dc39e0599..48448e2c6 100644 --- a/src/mmw/apps/home/views.py +++ b/src/mmw/apps/home/views.py @@ -17,6 +17,8 @@ from rest_framework.authtoken.models import Token from apps.modeling.models import Project, Scenario +from apps.user.models import UserProfile +from apps.user.countries import COUNTRY_CHOICES def home_page(request): @@ -157,6 +159,12 @@ def get_client_settings(request): 'itsi_enabled': not bigcz, 'title': title, 'api_token': get_api_token(), + 'choices': { + 'UserProfile': { + 'user_type': UserProfile.USER_TYPE_CHOICES, + 'country': COUNTRY_CHOICES, + } + }, }), 'google_maps_api_key': settings.GOOGLE_MAPS_API_KEY, 'title': title, diff --git a/src/mmw/apps/user/urls.py b/src/mmw/apps/user/urls.py index 8c89d6855..e45796bd8 100644 --- a/src/mmw/apps/user/urls.py +++ b/src/mmw/apps/user/urls.py @@ -6,6 +6,7 @@ from django.conf.urls import patterns, url from apps.user.views import (login, + profile, sign_up, forgot, resend, @@ -20,6 +21,7 @@ url(r'^itsi/login$', itsi_login, name='itsi_login'), url(r'^itsi/authenticate$', itsi_auth, name='itsi_auth'), url(r'^login$', login, name='login'), + url(r'^profile$', profile, name='profile'), url(r'^sign_up$', sign_up, name='sign_up'), url(r'^resend$', resend, name='resend'), url(r'^forgot$', forgot, name='forgot'), diff --git a/src/mmw/apps/user/views.py b/src/mmw/apps/user/views.py index a667d7eea..75bb5333d 100644 --- a/src/mmw/apps/user/views.py +++ b/src/mmw/apps/user/views.py @@ -48,8 +48,8 @@ def login(request): 'itsi': ItsiUser.objects.filter(user_id=user.id).exists(), 'guest': False, 'id': user.id, - 'profileWasSkipped': profile.was_skipped, - 'profileIsComplete': profile.is_complete + 'profile_was_skipped': profile.was_skipped, + 'profile_is_complete': profile.is_complete, } else: response_data = { @@ -89,6 +89,33 @@ def login(request): return Response(data=response_data, status=status_code) +@decorators.api_view(['POST']) +@decorators.permission_classes((IsAuthenticated, )) +def profile(request): + params = request.POST.dict() + if 'first_name' in params: + request.user.first_name = params['first_name'] + del params['first_name'] + if 'last_name' in request.POST: + request.user.last_name = params['last_name'] + del params['last_name'] + request.user.save() + + if 'was_skipped' in params: + params['was_skipped'] = str(params['was_skipped']).lower() == 'true' + params['is_complete'] = not params['was_skipped'] + else: + params['is_complete'] = True + + profile = UserProfile(**params) + profile.user = request.user + profile.save() + response_data = { + 'result': 'success' + } + return Response(data=response_data, status=status.HTTP_200_OK) + + @decorators.api_view(['GET']) @decorators.permission_classes((AllowAny, )) def logout(request): diff --git a/src/mmw/bundle.sh b/src/mmw/bundle.sh index 1bed2704e..84473523c 100755 --- a/src/mmw/bundle.sh +++ b/src/mmw/bundle.sh @@ -99,6 +99,7 @@ CONCAT_VENDOR_CSS_COMMAND="cat \ ./node_modules/font-awesome/css/font-awesome.min.css \ ./node_modules/bootstrap-table/dist/bootstrap-table.min.css \ ./node_modules/bootstrap-datepicker/dist/css/bootstrap-datepicker3.min.css \ + ./node_modules/bootstrap-select/dist/css/bootstrap-select.min.css \ ./css/shim/nv.d3.min.css \ > $VENDOR_CSS_FILE" diff --git a/src/mmw/js/src/app.js b/src/mmw/js/src/app.js index ec3526cab..3b3dd0e7f 100644 --- a/src/mmw/js/src/app.js +++ b/src/mmw/js/src/app.js @@ -1,6 +1,7 @@ "use strict"; var $ = require('jquery'), + _ = require('lodash'), Marionette = require('../shim/backbone.marionette'), shutterbug = require('../shim/shutterbug'), views = require('./core/views'), @@ -138,10 +139,36 @@ var App = new Marionette.Application({ }); }, + showLoginModal: function(onSuccess) { + var self = this, + promptForProfileIfIncomplete = function(loginResponse) { + if (loginResponse.profile_was_skipped || loginResponse.profile_is_complete) { + if (onSuccess && _.isFunction(onSuccess)) { + onSuccess(loginResponse); + } + } else { + new userViews.UserProfileModalView({ + model: new userModels.UserProfileFormModel({ + successCallback: onSuccess + }), + app: self + }).render(); + } + }; + new userViews.LoginModalView({ model: new userModels.LoginFormModel({ showItsiButton: settings.get('itsi_enabled'), + successCallback: promptForProfileIfIncomplete + }), + app: self + }).render(); + }, + + showProfileModal: function (onSuccess) { + new userViews.UserProfileModalView({ + model: new userModels.UserProfileFormModel({ successCallback: onSuccess }), app: this diff --git a/src/mmw/js/src/user/models.js b/src/mmw/js/src/user/models.js index 13870abc8..aec9c8a15 100644 --- a/src/mmw/js/src/user/models.js +++ b/src/mmw/js/src/user/models.js @@ -55,6 +55,14 @@ var UserModel = Backbone.Model.extend({ } }); +var UserProfileModel = Backbone.Model.extend({ + defaults: { + was_skipped: false, + }, + + url: '/user/profile', +}); + var ModalBaseModel = Backbone.Model.extend({ defaults: { success: false, @@ -97,14 +105,27 @@ var LoginFormModel = ModalBaseModel.extend({ } }, - onSuccess: function() { + onSuccess: function(response) { var callback = this.get('successCallback'); if (callback && _.isFunction(callback)) { - callback(); + callback(response); } } }); +var UserProfileFormModel = ModalBaseModel.extend({ + defaults: { + first_name: null, + last_name: null, + organization: null, + user_type: 'Unspecified', + country: 'US', + was_skipped: false + }, + + url: '/user/profile' +}); + var SignUpFormModel = ModalBaseModel.extend({ defaults: { username: null, @@ -303,7 +324,9 @@ var ItsiSignUpFormModel = ModalBaseModel.extend({ module.exports = { UserModel: UserModel, + UserProfileModel: UserProfileModel, LoginFormModel: LoginFormModel, + UserProfileFormModel: UserProfileFormModel, SignUpFormModel: SignUpFormModel, ResendFormModel: ResendFormModel, ForgotFormModel: ForgotFormModel, diff --git a/src/mmw/js/src/user/templates/baseModal.html b/src/mmw/js/src/user/templates/baseModal.html index 9bbbc0e3d..f50e14ec5 100644 --- a/src/mmw/js/src/user/templates/baseModal.html +++ b/src/mmw/js/src/user/templates/baseModal.html @@ -16,6 +16,23 @@ {% endmacro %} +{% macro select(value, name, model, field, defaultChoice='', label='') %} +

        + + +
        +{% endmacro %} +
        - +
        Contacts - {% for contact in contacts %} -

        {{ contact }}

        - {% else %} - None provided - {% endfor %} -
        Organizations - {% for org in orgs %} -

        {{ org }}

        - {% else %} - None provided - {% endfor %} -
        Data collected between
        CatalogWater Data Center + CUAHSI Water Data Center (WDC) + + + +
        diff --git a/src/mmw/js/src/data_catalog/views.js b/src/mmw/js/src/data_catalog/views.js index 69588386f..e6ff76ae3 100644 --- a/src/mmw/js/src/data_catalog/views.js +++ b/src/mmw/js/src/data_catalog/views.js @@ -914,8 +914,8 @@ var CuahsiChartView = Marionette.ItemView.extend({ selected: 1, buttons: [ { type: 'week', count: 1, text: '1w' }, - { type: 'week', count: 2, text: '2w' }, { type: 'month', count: 1, text: '1m' }, + { type: 'year', count: 1, text: '1y' }, ], }, From 610231c42ed2c5b112cd3c5205a21881bbb04457 Mon Sep 17 00:00:00 2001 From: Matthew McFarland Date: Sat, 4 Nov 2017 17:58:38 -0400 Subject: [PATCH 160/172] Send GA event for AoI creation Send an event for AOI creation and specify the method used for creation. Additionally, for named boundaries, send a boundary name event. --- src/mmw/js/src/draw/views.js | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/src/mmw/js/src/draw/views.js b/src/mmw/js/src/draw/views.js index 23d30828a..9d8e6f535 100644 --- a/src/mmw/js/src/draw/views.js +++ b/src/mmw/js/src/draw/views.js @@ -31,7 +31,8 @@ var selectBoundary = 'selectBoundary', delineateWatershed = 'delineateWatershed', aoiUpload = 'aoiUpload', freeDraw = 'free-draw', - squareKm = 'square-km'; + squareKm = 'square-km', + GA_AOI_CATEGORY = 'AoI Creation'; var codeToLayer = {}; // code to layer mapping @@ -419,6 +420,7 @@ var AoIUploadView = Marionette.ItemView.extend({ var geojson = JSON.parse(jsonString); this.addPolygonToMap(geojson.features[0]); + ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'geojson'); }, handleShpZip: function(zipfile) { @@ -432,6 +434,8 @@ var AoIUploadView = Marionette.ItemView.extend({ self.reprojectAndAddFeature(shp, prj); }) .catch(_.bind(self.handleShapefileError, self)); + + ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'shapefile'); }, reprojectAndAddFeature: function(shp, prj) { @@ -664,6 +668,9 @@ var SelectBoundaryView = DrawToolBaseView.extend({ codeToLayer[layerCode] = ol; grid.on('click', function(e) { + ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'boundary-' + layerCode); + ga('send', 'event', GA_AOI_CATEGORY, 'boundary-aoi-create', e.data.name); + getShapeAndAnalyze(e, self.model, ofg, grid, layerCode, shortDisplay); }); @@ -776,6 +783,7 @@ var DrawAreaView = DrawToolBaseView.extend({ .then(function(shape) { addLayer(shape); navigateToAnalyze(); + ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'freedraw'); }).fail(function(message) { revertLayer(); displayAlert(message, modalModels.AlertTypes.error); @@ -812,6 +820,7 @@ var DrawAreaView = DrawToolBaseView.extend({ return [parseFloat(coord[0]), parseFloat(coord[1])]; }); + ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'squarekm') return box; }).then(validateShape).then(function(polygon) { addLayer(polygon, '1 Square Km'); @@ -902,6 +911,7 @@ var WatershedDelineationView = DrawToolBaseView.extend({ utils.placeMarker(map) .then(function(latlng) { + ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'rwd-' + dataSource); return validatePointWithinDataSourceBounds(latlng, dataSource); }) .then(function(latlng) { From a757d216768f5385c3f11b0f267b484158225093 Mon Sep 17 00:00:00 2001 From: Matthew McFarland Date: Sat, 4 Nov 2017 19:24:03 -0400 Subject: [PATCH 161/172] Send GA event for analysis type When a request for analysis is made, send a GA event that records the analysis type, whether it was in the DRB and the size in km2 of the aoi. --- src/mmw/js/src/analyze/models.js | 8 +++++++- src/mmw/js/src/core/utils.js | 7 +++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/src/mmw/js/src/analyze/models.js b/src/mmw/js/src/analyze/models.js index 58db6c104..cf85ef579 100644 --- a/src/mmw/js/src/analyze/models.js +++ b/src/mmw/js/src/analyze/models.js @@ -5,7 +5,8 @@ var $ = require('jquery'), Backbone = require('../../shim/backbone'), settings = require('../core/settings'), utils = require('../core/utils'), - coreModels = require('../core/models'); + coreModels = require('../core/models'), + turfArea = require('turf-area'); var LayerModel = Backbone.Model.extend({}); @@ -44,6 +45,11 @@ var AnalyzeTaskModel = coreModels.TaskModel.extend({ result = self.get('result'); if (aoi && !result && self.fetchAnalysisPromise === undefined) { + var gaEvent = self.get('name') + '-analyze', + gaLabel = utils.isInDrb(aoi) ? 'drb-aoi' : 'national-aoi', + gaAoiSize = turfArea(aoi) / 1000000; + ga('send', 'event', 'Analyze', gaEvent, gaLabel, parseInt(gaAoiSize)); + var isWkaoi = utils.isWKAoIValid(wkaoi), taskHelper = { contentType: 'application/json', diff --git a/src/mmw/js/src/core/utils.js b/src/mmw/js/src/core/utils.js index fe6309ddb..cb0cbe846 100644 --- a/src/mmw/js/src/core/utils.js +++ b/src/mmw/js/src/core/utils.js @@ -513,6 +513,13 @@ var utils = { // Array.filter(distinct) to get distinct values distinct: function(value, index, self) { return self.indexOf(value) === index; + }, + + isInDrb: function(geom) { + var layers = settings.get('stream_layers'), + drb = _.findWhere(layers, {code: 'drb_streams_v2'}).perimeter; + + return !!intersect(geom, drb); } }; From 8ba260c5d60eb00e981d4ad3c915e517e018d907 Mon Sep 17 00:00:00 2001 From: Kelly Innes Date: Mon, 6 Nov 2017 11:31:20 -0500 Subject: [PATCH 162/172] Add catchment & aoi area guards - add guards to check that catchment & aoi area isn't zero before using those values in division ops - if catchment area is 0, return True (& count the catchment as intersecting), since PostGIS has determined that it intersects prior to reducing the geometry to an empty list in the simplify operation - if aoi area is ever 0, return False since there aren't any possible intersections --- src/mmw/apps/geoprocessing_api/calcs.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/mmw/apps/geoprocessing_api/calcs.py b/src/mmw/apps/geoprocessing_api/calcs.py index a5762bd2e..8be765180 100644 --- a/src/mmw/apps/geoprocessing_api/calcs.py +++ b/src/mmw/apps/geoprocessing_api/calcs.py @@ -182,7 +182,13 @@ def catchment_intersects_aoi(aoi, catchment): catchment_geom = GEOSGeometry(json.dumps(catchment), srid=4326) reprojected_catchment = catchment_geom.transform(5070, clone=True) - if not reprojected_catchment.valid: + if catchment_geom.area == 0: + return True + elif reprojected_catchment.area == 0: + return True + elif aoi.area == 0: + return False + elif not reprojected_catchment.valid: return False aoi_kms = aoi.area / 1000000 From 0c3665edc6e4279d2794512fb2d823932ecd7dde Mon Sep 17 00:00:00 2001 From: Justin Walgran Date: Mon, 6 Nov 2017 12:11:46 -0700 Subject: [PATCH 163/172] Remove bottom margin from modals to prevent unneeded scrolling Removing the bottom margin prevents the page from scrolling when a modal dialog is completely on screen but less than 72px from the bottom of the browser frame. --- src/mmw/sass/components/_modals.scss | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/mmw/sass/components/_modals.scss b/src/mmw/sass/components/_modals.scss index 55c5fa080..013289cad 100644 --- a/src/mmw/sass/components/_modals.scss +++ b/src/mmw/sass/components/_modals.scss @@ -3,7 +3,7 @@ background-color: $black-54; .modal-dialog { - margin: 72px auto; + margin: 72px auto 0 auto; } .modal-content { From 618e4395a82415a3339b2f99849bd8ce71ab4fe9 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Mon, 6 Nov 2017 14:03:50 -0500 Subject: [PATCH 164/172] Add AoI size limit to draw description --- src/mmw/js/src/draw/settings.js | 4 ++-- src/mmw/js/src/draw/templates/window.html | 4 +--- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/src/mmw/js/src/draw/settings.js b/src/mmw/js/src/draw/settings.js index 2eedfb70f..622931dbb 100644 --- a/src/mmw/js/src/draw/settings.js +++ b/src/mmw/js/src/draw/settings.js @@ -29,12 +29,12 @@ var bigCZSplashPageText = { var mmwSelectAreaText = { headerDescription: 'Explore mapped layers, such as streams, land cover, soils, boundaries and observations, using the layer selector in the lower left of the map.', - selectAreaExplanation: 'analyze the factors that impact water in your area and to begin to model different scenarios of human impacts.', + selectAreaExplanation: 'Select an Area of Interest in the continental United States, using the suite of tools below, to analyze the factors that impact water in your area and to begin to model different scenarios of human impacts.', }; var bigCZSelectAreaText = { headerDescription: 'Explore mapped layers, such as streams, land cover, and boundaries, using the layer selector in the lower left of the map.', - selectAreaExplanation: 'analyze the mapped layers within your area and to initiate a search for data.', + selectAreaExplanation: 'Select an Area of Interest (AoI) in the continental United States, using the suite of tools below, to analyze the mapped layers within your area and to initiate a search for datasets and visualize time series data. Currently the AoI must be smaller than 1,500 km2.', }; module.exports = { diff --git a/src/mmw/js/src/draw/templates/window.html b/src/mmw/js/src/draw/templates/window.html index 09e9ddea6..84e1c9e77 100644 --- a/src/mmw/js/src/draw/templates/window.html +++ b/src/mmw/js/src/draw/templates/window.html @@ -4,9 +4,7 @@

        Select Area

        our documentation on layers.

        - Select - an Area of Interest in the continental United States, using the suite - of tools below, to {{ selectAreaText.selectAreaExplanation }} + {{ selectAreaText.selectAreaExplanation }}

        From 328622078701d6eb81b21384c761e82e6d986ba0 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Mon, 6 Nov 2017 15:48:37 -0500 Subject: [PATCH 165/172] Fix Compare View Header Styling * Make input modal label styling more specific, so the label styling doesn't apply to other modals, like the compare view's precipitation label --- src/mmw/js/src/user/templates/baseModal.html | 4 ++-- src/mmw/sass/components/_modals.scss | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/mmw/js/src/user/templates/baseModal.html b/src/mmw/js/src/user/templates/baseModal.html index 752cf55e1..aaccededf 100644 --- a/src/mmw/js/src/user/templates/baseModal.html +++ b/src/mmw/js/src/user/templates/baseModal.html @@ -1,7 +1,7 @@ {% macro field(value, name, label='', type='text') %}
        - - + +
        {% endmacro %} diff --git a/src/mmw/sass/components/_modals.scss b/src/mmw/sass/components/_modals.scss index 55c5fa080..a8391fb62 100644 --- a/src/mmw/sass/components/_modals.scss +++ b/src/mmw/sass/components/_modals.scss @@ -11,11 +11,11 @@ border: 0px solid $black-12; box-shadow: 0 0 6px $black-24; - label { + .modal-text-input-label { display: block; } - input { + .modal-text-input { width: 100%; padding: 0.75rem 0.5rem; margin: 0.5rem 0 2.5rem 0; From e3256f4a9292ed8e6c18edf462c898f76f713a93 Mon Sep 17 00:00:00 2001 From: Matthew McFarland Date: Sun, 5 Nov 2017 11:11:40 -0500 Subject: [PATCH 166/172] Send GA events for Model activities Sends a Google Analytics event for creating a modeling project, creating a scenario and adding modifications. Events or labels include the project model package type. --- src/mmw/js/src/analyze/models.js | 2 +- src/mmw/js/src/draw/views.js | 14 +++++++------- src/mmw/js/src/modeling/constants.js | 8 ++++++++ src/mmw/js/src/modeling/controllers.js | 3 +++ src/mmw/js/src/modeling/models.js | 13 +++++++++++-- 5 files changed, 30 insertions(+), 10 deletions(-) create mode 100644 src/mmw/js/src/modeling/constants.js diff --git a/src/mmw/js/src/analyze/models.js b/src/mmw/js/src/analyze/models.js index cf85ef579..451f86fb0 100644 --- a/src/mmw/js/src/analyze/models.js +++ b/src/mmw/js/src/analyze/models.js @@ -48,7 +48,7 @@ var AnalyzeTaskModel = coreModels.TaskModel.extend({ var gaEvent = self.get('name') + '-analyze', gaLabel = utils.isInDrb(aoi) ? 'drb-aoi' : 'national-aoi', gaAoiSize = turfArea(aoi) / 1000000; - ga('send', 'event', 'Analyze', gaEvent, gaLabel, parseInt(gaAoiSize)); + window.ga('send', 'event', 'Analyze', gaEvent, gaLabel, parseInt(gaAoiSize)); var isWkaoi = utils.isWKAoIValid(wkaoi), taskHelper = { diff --git a/src/mmw/js/src/draw/views.js b/src/mmw/js/src/draw/views.js index 9d8e6f535..8dba561a7 100644 --- a/src/mmw/js/src/draw/views.js +++ b/src/mmw/js/src/draw/views.js @@ -420,7 +420,7 @@ var AoIUploadView = Marionette.ItemView.extend({ var geojson = JSON.parse(jsonString); this.addPolygonToMap(geojson.features[0]); - ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'geojson'); + window.ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'geojson'); }, handleShpZip: function(zipfile) { @@ -435,7 +435,7 @@ var AoIUploadView = Marionette.ItemView.extend({ }) .catch(_.bind(self.handleShapefileError, self)); - ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'shapefile'); + window.ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'shapefile'); }, reprojectAndAddFeature: function(shp, prj) { @@ -668,8 +668,8 @@ var SelectBoundaryView = DrawToolBaseView.extend({ codeToLayer[layerCode] = ol; grid.on('click', function(e) { - ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'boundary-' + layerCode); - ga('send', 'event', GA_AOI_CATEGORY, 'boundary-aoi-create', e.data.name); + window.ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'boundary-' + layerCode); + window.ga('send', 'event', GA_AOI_CATEGORY, 'boundary-aoi-create', e.data.name); getShapeAndAnalyze(e, self.model, ofg, grid, layerCode, shortDisplay); }); @@ -783,7 +783,7 @@ var DrawAreaView = DrawToolBaseView.extend({ .then(function(shape) { addLayer(shape); navigateToAnalyze(); - ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'freedraw'); + window.ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'freedraw'); }).fail(function(message) { revertLayer(); displayAlert(message, modalModels.AlertTypes.error); @@ -820,7 +820,7 @@ var DrawAreaView = DrawToolBaseView.extend({ return [parseFloat(coord[0]), parseFloat(coord[1])]; }); - ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'squarekm') + window.ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'squarekm'); return box; }).then(validateShape).then(function(polygon) { addLayer(polygon, '1 Square Km'); @@ -911,7 +911,7 @@ var WatershedDelineationView = DrawToolBaseView.extend({ utils.placeMarker(map) .then(function(latlng) { - ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'rwd-' + dataSource); + window.ga('send', 'event', GA_AOI_CATEGORY, 'aoi-create', 'rwd-' + dataSource); return validatePointWithinDataSourceBounds(latlng, dataSource); }) .then(function(latlng) { diff --git a/src/mmw/js/src/modeling/constants.js b/src/mmw/js/src/modeling/constants.js new file mode 100644 index 000000000..f19b10eb9 --- /dev/null +++ b/src/mmw/js/src/modeling/constants.js @@ -0,0 +1,8 @@ +module.exports = { + GA: { + MODEL_CATEGORY: "Modeling", + MODEL_CREATE_EVENT: "project-create", + MODEL_SCENARIO_EVENT: "scenario-create", + MODEL_MOD_EVENT: "-modification-create", + } +}; diff --git a/src/mmw/js/src/modeling/controllers.js b/src/mmw/js/src/modeling/controllers.js index 94b324b10..61dbd02ee 100644 --- a/src/mmw/js/src/modeling/controllers.js +++ b/src/mmw/js/src/modeling/controllers.js @@ -4,6 +4,7 @@ var $ = require('jquery'), _ = require('lodash'), Backbone = require('../../shim/backbone'), App = require('../app'), + constants = require('./constants.js'), settings = require('../core/settings'), router = require('../router').router, views = require('./views'), @@ -109,6 +110,8 @@ var ModelingController = { }, makeNewProject: function(modelPackage) { + window.ga('send', 'event', constants.GA.MODEL_CATEGORY, constants.GA.MODEL_CREATE_EVENT, modelPackage); + var project; if (settings.get('itsi_embed')) { project = App.currentProject; diff --git a/src/mmw/js/src/modeling/models.js b/src/mmw/js/src/modeling/models.js index 3557b370d..df783c73f 100644 --- a/src/mmw/js/src/modeling/models.js +++ b/src/mmw/js/src/modeling/models.js @@ -5,6 +5,7 @@ var $ = require('jquery'), _ = require('lodash'), utils = require('../core/utils'), settings = require('../core/settings'), + constants = require('./constants.js'), App = require('../app'), coreModels = require('../core/models'), turfArea = require('turf-area'), @@ -683,11 +684,16 @@ var ScenarioModel = Backbone.Model.extend({ }, addModification: function(modification) { - var modifications = this.get('modifications'); + var modifications = this.get('modifications'), + modelPackage = App.currentProject.get('model_package'), + modKeyName = modelPackage === GWLFE ? 'modKey' : 'value'; + + window.ga('send', 'event', constants.GA.MODEL_CATEGORY, + modelPackage + constants.GA.MODEL_MOD_EVENT, modification.get(modKeyName)); // For GWLFE, first remove existing mod with the same key since it // doesn't make sense to have multiples of the same type of BMP. - if (App.currentProject.get('model_package') === GWLFE) { + if (modelPackage === GWLFE) { var modKey = modification.get('modKey'), matches = modifications.where({'modKey': modKey}); @@ -963,6 +969,7 @@ var ScenariosCollection = Backbone.Collection.extend({ createNewScenario: function() { var currentConditions = this.findWhere({ 'is_current_conditions': true }), + modelPackage = currentConditions.get('taskModel').get('taskName'), scenario = new ScenarioModel({ is_current_conditions: false, name: this.makeNewScenarioName('New Scenario'), @@ -972,6 +979,8 @@ var ScenariosCollection = Backbone.Collection.extend({ inputs: currentConditions.get('inputs').toJSON(), }); + window.ga('send', 'event', constants.GA.MODEL_CATEGORY, constants.GA.MODEL_SCENARIO_EVENT, modelPackage); + this.add(scenario); this.setActiveScenarioByCid(scenario.cid); }, From d57323b6358dd63d236dd9f71a0fb87bc7533ea4 Mon Sep 17 00:00:00 2001 From: Matthew McFarland Date: Mon, 6 Nov 2017 15:01:08 -0500 Subject: [PATCH 167/172] Update tests to define ga function Google Analytics collect function are not available in the test environment and is mocked out to prevent errors. --- src/mmw/js/src/core/testUtils.js | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/mmw/js/src/core/testUtils.js b/src/mmw/js/src/core/testUtils.js index 77225f95d..a11e261ed 100644 --- a/src/mmw/js/src/core/testUtils.js +++ b/src/mmw/js/src/core/testUtils.js @@ -2,6 +2,9 @@ var $ = require('jquery'); +// Mock the Google Analytics function for tests +window.ga = function() {}; + // Should be called after each unit test. function resetApp(app) { app.map.off(); From 1cf390908523f4b081de3824ff6d032627543b14 Mon Sep 17 00:00:00 2001 From: Alice Rottersman Date: Mon, 6 Nov 2017 16:00:31 -0500 Subject: [PATCH 168/172] Style Input Modals * After restyling the login modals, the other input modals (rename scenario, share project url, rename project) lost their styling. * Match their styling to the new login forms --- src/mmw/js/src/core/modals/templates/inputModal.html | 6 ++---- src/mmw/js/src/core/modals/templates/shareModal.html | 6 ++---- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/src/mmw/js/src/core/modals/templates/inputModal.html b/src/mmw/js/src/core/modals/templates/inputModal.html index 5db9ba873..13c5f2471 100644 --- a/src/mmw/js/src/core/modals/templates/inputModal.html +++ b/src/mmw/js/src/core/modals/templates/inputModal.html @@ -5,10 +5,8 @@

        {{ title }}

        diff --git a/src/mmw/js/src/core/modals/templates/shareModal.html b/src/mmw/js/src/core/modals/templates/shareModal.html index c2069a586..5294cdf4b 100644 --- a/src/mmw/js/src/core/modals/templates/shareModal.html +++ b/src/mmw/js/src/core/modals/templates/shareModal.html @@ -8,10 +8,8 @@

        Share {{ text }}

        Sorry, only saved {{ text }}s are able to be shared. Please log in to share your work.

        {% else %}
        - - - - + +
        {% endif %} From 6a7177ed523a47a88b1f4a005215dd82d83cb3f3 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Fri, 3 Nov 2017 14:51:05 -0400 Subject: [PATCH 169/172] Move callback function to options from attributes Previously a success callback function was added to a model's attributes, and was being sent to the server during save, causing server side errors. Now we pass in that callback via options, so it is not added to the list of attributes sent to the server. --- src/mmw/js/src/app.js | 12 +++++++----- src/mmw/js/src/user/models.js | 16 ++++++++-------- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/src/mmw/js/src/app.js b/src/mmw/js/src/app.js index 75488dbd1..b22b7f601 100644 --- a/src/mmw/js/src/app.js +++ b/src/mmw/js/src/app.js @@ -145,7 +145,9 @@ var App = new Marionette.Application({ var loginModalView = new userViews.LoginModalView({ model: new userModels.LoginFormModel({ showItsiButton: settings.get('itsi_enabled'), - successCallback: function(loginResponse) { + }, + { + onSuccess: function(loginResponse) { if (loginResponse.profile_was_skipped || loginResponse.profile_is_complete) { if (onSuccess && _.isFunction(onSuccess)) { onSuccess(loginResponse); @@ -154,8 +156,8 @@ var App = new Marionette.Application({ loginModalView.$el.modal('hide'); loginModalView.$el.on('hidden.bs.modal', function() { new userViews.UserProfileModalView({ - model: new userModels.UserProfileFormModel({ - successCallback: onSuccess + model: new userModels.UserProfileFormModel({}, { + onSuccess: onSuccess }), app: self }).render(); @@ -169,8 +171,8 @@ var App = new Marionette.Application({ showProfileModal: function (onSuccess) { new userViews.UserProfileModalView({ - model: new userModels.UserProfileFormModel({ - successCallback: onSuccess + model: new userModels.UserProfileFormModel({}, { + onSuccess: onSuccess }), app: this }).render(); diff --git a/src/mmw/js/src/user/models.js b/src/mmw/js/src/user/models.js index d4377297e..eb2f879e7 100644 --- a/src/mmw/js/src/user/models.js +++ b/src/mmw/js/src/user/models.js @@ -84,11 +84,18 @@ var LoginFormModel = ModalBaseModel.extend({ defaults: { username: null, password: null, - successCallback: null }, url: '/user/login', + initialize: function(attrs, opts) { + if (opts && opts.onSuccess && _.isFunction(opts.onSuccess)) { + this.onSuccess = opts.onSuccess; + } else { + this.onSuccess = _.noop; + } + }, + validate: function(attrs) { var errors = []; @@ -113,13 +120,6 @@ var LoginFormModel = ModalBaseModel.extend({ }); } }, - - onSuccess: function(response) { - var callback = this.get('successCallback'); - if (callback && _.isFunction(callback)) { - callback(response); - } - } }); var UserProfileFormModel = ModalBaseModel.extend({ From b46de5fabe159edaf0474c4b2f558f9a19002504 Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Tue, 7 Nov 2017 16:49:44 -0500 Subject: [PATCH 170/172] Throttle NWISUV sites to 1 month These sites have a lot of data, and fetching one year worth of it can sometimes overutilize the server's resources. We restrict them to 1 month fetches, keeping the rest at 1 year. --- src/mmw/js/src/data_catalog/models.js | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index 7a3d33338..a065e7d1a 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -13,6 +13,11 @@ var PAGE_SIZE = settings.get('data_catalog_page_size'); var DATE_FORMAT = 'MM/DD/YYYY'; var WATERML_VARIABLE_TIME_INTERVAL = '{http://www.cuahsi.org/water_ml/1.1/}variable_time_interval'; +var SERVICE_PERIODS = { + 'NWISUV': 'months', // For NWISUV sites, fetch 1 month of data + '*' : 'years', // For all else, fetch 1 year of data +}; + var FilterModel = Backbone.Model.extend({ defaults: { @@ -709,14 +714,16 @@ var CuahsiVariable = Backbone.Model.extend({ wsdl: this.get('wsdl'), site: this.get('site'), variable: this.get('id'), - }; + }, + service = params.site.split(':')[0], + duration = SERVICE_PERIODS[service] || 'years'; // If neither from date nor to date is specified, set time interval - // to be either from begin date to end date, or 1 week up to end date, - // whichever is shorter. + // to be either from begin date to end date, or 1 `duration` up to end + // date, whichever is shorter. if (!from || moment(from).isBefore(begin_date)) { - if (end_date.diff(begin_date, 'years', true) > 1) { - params.from_date = moment(end_date).subtract(1, 'years'); + if (end_date.diff(begin_date, duration, true) > 1) { + params.from_date = moment(end_date).subtract(1, duration); } else { params.from_date = begin_date; } From ceaabad12a8ceba8de40a2b53ba664d7264f9dcc Mon Sep 17 00:00:00 2001 From: Terence Tuhinanshu Date: Wed, 8 Nov 2017 10:53:41 -0500 Subject: [PATCH 171/172] Delete fetchPromise on results fetch When a filter changes, all results for a catalog are refetched. It is possible that a filter may include results that were already included. In this case, there isn't a new instance of the result, but the existing instance is refreshed. When this happens, its CuahsiVariables collection is reset, with no detail values. Previously, if a result details page had been viewed, it had fetched the detail values for said CuahsiVariables, and cached them, using a fetchPromise that would only be resolved once. After fetching them for the first time, we simply returned the fetchPromise so they would not be fetched again. When the values are reset, the actual detail values are gone, but the cached fetchPromise remains, thus preventing values from being fetched anew. By deleting the fetchPromise when receiving new values, we ensure that when a detail page is opened for a previously fetched and repeatedly filtered result, the values for its CuahsiVariables are refetched. --- src/mmw/js/src/data_catalog/models.js | 1 + 1 file changed, 1 insertion(+) diff --git a/src/mmw/js/src/data_catalog/models.js b/src/mmw/js/src/data_catalog/models.js index a065e7d1a..3500d6806 100644 --- a/src/mmw/js/src/data_catalog/models.js +++ b/src/mmw/js/src/data_catalog/models.js @@ -421,6 +421,7 @@ var Result = Backbone.Model.extend({ if (variables instanceof CuahsiVariables) { variables.reset(response.variables); delete response.variables; + delete this.fetchPromise; } } From 50481bbc1fd5304065a58bc289e43005871ecaa6 Mon Sep 17 00:00:00 2001 From: Hector Castro Date: Thu, 9 Nov 2017 15:11:59 -0500 Subject: [PATCH 172/172] Update libpq glob to reflect main PostgreSQL repository See: http://apt.postgresql.org/pub/repos/apt/dists/trusty-pgdg/main/binary-amd64/ --- deployment/ansible/group_vars/all | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deployment/ansible/group_vars/all b/deployment/ansible/group_vars/all index 5f70c5d41..4307d8bd7 100644 --- a/deployment/ansible/group_vars/all +++ b/deployment/ansible/group_vars/all @@ -23,7 +23,7 @@ postgresql_database: mmw postgresql_version: "9.4" postgresql_package_version: "9.4.*.pgdg14.04+1" postgresql_support_repository_channel: "main" -postgresql_support_libpq_version: "10.0-*.pgdg14.04+1" +postgresql_support_libpq_version: "10.1-*.pgdg14.04+1" postgresql_support_psycopg2_version: "2.7" postgis_version: "2.1" postgis_package_version: "2.1.*.pgdg14.04+1"