Should I use a sensor or an observable source asset?

[sryza]

Sensors and observable source assets are two different approaches to listening for external changes and triggering downstream computation based on them. What's the difference, and when is each one appropriate?

[sryza]

Both sensors and observable source assets listen for external changes.

The main high-level conceptual different between sensors and observable source assets is that observable source assets decouple the "discover what changed" logic from the "decide what downstream assets should be materialized" logic.

• A sensor is responsible for deciding what gets kicked off when it notices an external change.
• An observable source asset just records (observes) the change. The auto-materialize policies on downstream assets determine whether they'll be rematerialized in response to changes.

A few other differences:

• Observable source assets are currently marked experimental, while sensors are marked stable.
• When turned on, sensors are automatically evaluated every 30 seconds. To automatically evaluate an observable source asset, you use the argument auto_observe_interval_minutes=1. Source asset observations occur inside a run, whereas sensor ticks are evaluated outside a run.

[zyd14]

@sryza Are observable source assets still a recommended approach? I thought I had seen some references in communications to SourceAssets being deprecated soon in favor of AssetSpecs and External Assets.

[sryza]

Great question and yes. We are planning to rename them to avoid the deprecated name "source asset", but otherwise they should work the same. I had been hoping to push that out as part of 1.8, but there are some lingering questions about what the API should look like for partitioned observation results that we're still trying to resolve before finalizing the change: #23405.