Sensor causing exception - ExternalTargetData not in whitelist

[endar-io]

Hi,

today I started a new docker based project using the latest version 0.13.9.
Currently I have a very simple repository containing one job consisting of one op (only doing log output right now), and one sensor.

As soon as I activate the sensor, I get the following exception (every 4 seconds or so) and the sensor of course does not trigger.
Executing the job itself via launchpad works fine.

ERROR [SensorDaemon] Sensor daemon caught an error for sensor backend_comm_snippets_sensor : dagster.check.CheckError: Invariant failed. Description: Attempted to deserialize class "ExternalTargetData" which is not in the whitelist.

Stack Trace:
  File "/usr/local/lib/python3.8/site-packages/dagster/scheduler/sensor.py", line 184, in execute_sensor_iteration
    repo_location_handle = handle_manager.get_handle(origin)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/host_representation/handle_manager.py", line 33, in get_handle
    existing_handle if existing_handle else repository_location_origin.create_handle()
  File "/usr/local/lib/python3.8/site-packages/dagster/core/host_representation/origin.py", line 209, in create_handle
    return GrpcServerRepositoryLocationHandle(self)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/host_representation/handle.py", line 171, in __init__
    external_repositories_list = sync_get_streaming_external_repositories_grpc(
  File "/usr/local/lib/python3.8/site-packages/dagster/api/snapshot_repository.py", line 64, in sync_get_streaming_external_repositories_grpc
    external_repository_data = deserialize_json_to_dagster_namedtuple(
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 218, in deserialize_json_to_dagster_namedtuple
    dagster_namedtuple = _deserialize_json_to_dagster_namedtuple(
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 229, in _deserialize_json_to_dagster_namedtuple
    return _unpack_value(seven.json.loads(json_str), whitelist_map=whitelist_map)
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 264, in _unpack_value
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 264, in <dictcomp>
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 250, in _unpack_value
    return [_unpack_value(i, whitelist_map) for i in val]
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 250, in <listcomp>
    return [_unpack_value(i, whitelist_map) for i in val]
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 264, in _unpack_value
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 264, in <dictcomp>
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 279, in _unpack_value
    return {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 279, in <dictcomp>
    return {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 253, in _unpack_value
    check.invariant(
  File "/usr/local/lib/python3.8/site-packages/dagster/check/__init__.py", line 168, in invariant
    raise_with_traceback(
  File "/usr/local/lib/python3.8/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
    raise exc.with_traceback(traceback)

The repository looks as following:

from dagster import RunRequest, job, repository, sensor
from .ops import read_files_and_notify_backend

default_config = {"ops": {"read_files_and_notify_backend": {"config": {"queue_directory": "/mnt/queue"}}}}

@job(config=default_config)
def backend_comm_snippets_job():
    read_files_and_notify_backend()

@sensor(job=backend_comm_snippets_job)
def backend_comm_snippets_sensor():
    should_run = True
    if should_run:
        yield RunRequest(run_key=None, run_config=default_config)

@repository
def repository():
    return [
        backend_comm_snippets_sensor,
        backend_comm_snippets_job
    ]

The dagster.yaml and workspace.yaml are defined as following:

telemetry:
  enabled: false

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      hostname: dagster_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      hostname: dagster_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      hostname: dagster_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

load_from:
  - grpc_server:
      host: dagster_jobs
      port: 4001
      location_name: "jobs"

I also attached the content of the docker-compose.yaml below:

version: "3.7"
services:
  dagster_postgresql:
    image: postgres:11
    container_name: dagster_postgresql
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: "postgres_user"
      POSTGRES_PASSWORD: "postgres_password"
      POSTGRES_DB: "postgres_db"
    networks:
      - dagster_network

  dagster_jobs:
    build:
      context: .
      dockerfile: dockers/jobs.dockerfile
    image: dagster-0.13-jobs
    container_name: dagster_jobs
    command: /bin/bash /utils/startup.sh
    ports:
      - "4001:4001"
    environment:
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
    volumes:
      - type: bind
        source: ./tests/queue
        target: /mnt/queue/
    networks:
      - dagster_network

  dagster_dagit:
    build:
      context: .
      dockerfile: dockers/core.dockerfile
    image: dagster-0.13-core
    container_name: dagster_dagit
    entrypoint:
      - dagit
      - -h
      - "0.0.0.0"
      - -p
      - "3000"
      - -w
      - workspace.yaml
    expose:
      - "3000"
    ports:
      - "3000:3000"
    environment:
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
    networks:
      - dagster_network
    depends_on:
      - dagster_postgresql
      - dagster_jobs

  dagster_daemon:
    build:
      context: .
      dockerfile: dockers/core.dockerfile
    image: dagster-core
    container_name: dagster_daemon
    entrypoint:
      - dagster-daemon
      - run
    environment:
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
    networks:
      - dagster_network
    depends_on:
      - dagster_postgresql
      - dagster_jobs

networks:
  dagster_network:
    driver: bridge
    name: dagster_network

Does anyone have an idea what the reason for that error might be, or how I can narrow down the root cause?

Many thanks in advance,
Daniel

[alangenfeld]

The likely cause is different versions of dagster trying to talk to each other.

I suspect

image: dagster-core

vs

image: dagster-0.13-core

could be the cause

[endar-io]

Ugh, thanks!
That was truly a silly mistake. Thanks for finding the root cause.

Thanks, Daniel