exp: pygwalker data explorer in streamlit

[zilto]

Description

Add a page to the streamlit dashboard available via dlt pipeline NAME show. It uses the recent Pipeline.dataset interface to pass a dataframe to pygwalker.

2025-01-14.16-15-45.mp4

Goals

When first loading a pipeline, I want to know the properties of the data (null, distribution, outliers). It's reducing the friction of having to create temporary notebooks and install ipykernel/jupyter dependencies

Implementation

• pygwalker is an open-source Python library that produces an interactive widget to explore dataframes (works in Jupyter, VSCode, Streamlit, Marimo) based on Vega.

• Integration surface area is small; only assumes that you can pull the data locally via the Pipeline.dataset feature.

  • Deeper integration could use Pipeline.dataset to query data or pass an sqlalchemy connection to pygwalker to handle data loading.
  • pygwalker can use duckdb for efficient processing of data

TODO

• add useful message for optional/missing dependencies

[netlify]

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	d748340
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/6786d2f89dfa550008c73f7f
😎 Deploy Preview	https://deploy-preview-2216--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.