add initial powerbi assets #127

cnolanminich · 2024-10-09T01:13:09Z

This PR adds Power BI assets to the Hooli project. Features include:

Materializable semantic models
Automatic integration with the lineage graph of an upstream dbt assets
A semantic model with a downstream dashboard (see below for my amazing dashboarding skillz)
Example in the code of using the translator, including adding tags and groups

Open Questions:

Is this something we want to merge or just an example to keep around as a branch deployment?
Is there a "flow" that tells the story better than "analysis of augmented orders"
Should the group be "BI" or tied to some business area?

See below for a screenshot of what it looks like

github-actions · 2024-10-09T01:14:34Z

Your pull request at commit 976f5d4e0aec45bf3a661374368a0c7b39a00837 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_data_ingest	Deploy failed	Oct 09, 2024 at 01:26 AM (UTC)
snowflake_insights	Deploy failed	Oct 09, 2024 at 01:26 AM (UTC)
batch_enrichment	Deploy failed	Oct 09, 2024 at 01:26 AM (UTC)
basics	Deploy failed	Oct 09, 2024 at 01:26 AM (UTC)
data-eng-pipeline	Deploy failed	Oct 09, 2024 at 01:26 AM (UTC)

github-actions · 2024-10-09T14:00:57Z

Your pull request at commit 0a0b1790536b3679b762e90696215d853395a34c is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_data_ingest	View in Cloud	Oct 09, 2024 at 02:13 PM (UTC)
snowflake_insights	View in Cloud	Oct 09, 2024 at 02:13 PM (UTC)
batch_enrichment	View in Cloud	Oct 09, 2024 at 02:13 PM (UTC)
basics	View in Cloud	Oct 09, 2024 at 02:13 PM (UTC)
data-eng-pipeline	View in Cloud	Oct 09, 2024 at 02:13 PM (UTC)

github-actions · 2024-10-31T01:15:48Z

Your pull request at commit c55a2ed3c9d702c4a8abc7a93be2f5e509822d67 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_data_ingest	Building...	Oct 31, 2024 at 01:15 AM (UTC)
snowflake_insights	Building...	Oct 31, 2024 at 01:15 AM (UTC)
batch_enrichment	Building...	Oct 31, 2024 at 01:15 AM (UTC)
basics	Building...	Oct 31, 2024 at 01:15 AM (UTC)
data-eng-pipeline	Building...	Oct 31, 2024 at 01:15 AM (UTC)

github-actions · 2024-10-31T01:29:18Z

Your pull request at commit 3b876cf34e5558449897628df316c4d8264cd7e5 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_data_ingest	View in Cloud	Oct 31, 2024 at 01:41 AM (UTC)
snowflake_insights	View in Cloud	Oct 31, 2024 at 01:41 AM (UTC)
batch_enrichment	View in Cloud	Oct 31, 2024 at 01:41 AM (UTC)
basics	View in Cloud	Oct 31, 2024 at 01:41 AM (UTC)
data-eng-pipeline	View in Cloud	Oct 31, 2024 at 01:41 AM (UTC)

github-actions · 2024-10-31T17:33:07Z

Your pull request at commit f59360f5d3ce07b468721f5b368c28cabaf6b5d6 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_data_ingest	Building...	Oct 31, 2024 at 05:33 PM (UTC)
snowflake_insights	Building...	Oct 31, 2024 at 05:33 PM (UTC)
batch_enrichment	Building...	Oct 31, 2024 at 05:33 PM (UTC)
basics	Building...	Oct 31, 2024 at 05:33 PM (UTC)
data-eng-pipeline	Building...	Oct 31, 2024 at 05:33 PM (UTC)

github-actions · 2024-10-31T17:35:29Z

Your pull request at commit 4c51d6c5fc2062e782cd04b0f7974f1a53ee36c0 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_data_ingest	View in Cloud	Oct 31, 2024 at 05:47 PM (UTC)
snowflake_insights	View in Cloud	Oct 31, 2024 at 05:47 PM (UTC)
batch_enrichment	View in Cloud	Oct 31, 2024 at 05:47 PM (UTC)
basics	View in Cloud	Oct 31, 2024 at 05:47 PM (UTC)
data-eng-pipeline	View in Cloud	Oct 31, 2024 at 05:47 PM (UTC)

slopp

This is awesome! So excited to have it in, thanks for chasing all of this down

Is this something we want to merge or just an example to keep around as a branch deployment?

Deploy

Is there a "flow" that tells the story better than "analysis of augmented orders"
lgtm

Should the group be "BI" or tied to some business area?
BI is my preference

I left a few other comments. My other request, if possible, can we add a link to the dashboard in the asset description? I don't want to have to remember how to get to it, and I think it adds some pizzaz to an otherwise empty asset card/overview page

setup.py

slopp · 2024-10-31T18:11:05Z

hooli_data_eng/resources/__init__.py

@@ -58,6 +59,7 @@

 resource_def = {
    "LOCAL": {
+        "power_bi": power_bi_workspace,


I wonder if there is something we can do here to mock it out so folks who clone and run this locally without the powerBI creds still work?

(Speaking as someone who will want to be able to run the demo locally without worrying if I have sourced the power BI creds correctly)

That 100% makes sense -- I'll look into mocking it, I guess the only question is if we want to mock the power bi assets locally, I'd lean towards leaving it all off from local dev if possible, but I'm not exactly sure how that would work. Maybe @benpankow you have some ideas?

hooli_data_eng/powerbi_workspace.py

hooli_data_eng/definitions.py

cnolanminich · 2024-10-31T18:21:29Z

This is awesome! So excited to have it in, thanks for chasing all of this down

Is this something we want to merge or just an example to keep around as a branch deployment?

Deploy

Just to confirm, by "deploy" you mean merge into master post comment addressing?

Co-authored-by: Sean Lopp <[email protected]>

github-actions · 2024-10-31T18:23:05Z

Your pull request at commit 759ed535d2d5de6d3d3230537296a2891c7cede5 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_data_ingest	View in Cloud	Oct 31, 2024 at 06:36 PM (UTC)
snowflake_insights	View in Cloud	Oct 31, 2024 at 06:36 PM (UTC)
batch_enrichment	View in Cloud	Oct 31, 2024 at 06:36 PM (UTC)
basics	View in Cloud	Oct 31, 2024 at 06:36 PM (UTC)
data-eng-pipeline	View in Cloud	Oct 31, 2024 at 06:36 PM (UTC)

…a-eng-pipelines into add-powerbi

github-actions · 2024-10-31T18:58:33Z

Your pull request at commit cd1b656905f062f8dc6b28bee8803c2d24d7b3a0 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_data_ingest	View in Cloud	Oct 31, 2024 at 07:10 PM (UTC)
snowflake_insights	View in Cloud	Oct 31, 2024 at 07:10 PM (UTC)
batch_enrichment	View in Cloud	Oct 31, 2024 at 07:10 PM (UTC)
basics	View in Cloud	Oct 31, 2024 at 07:10 PM (UTC)
data-eng-pipeline	View in Cloud	Oct 31, 2024 at 07:10 PM (UTC)

github-actions · 2024-10-31T19:27:09Z

Your pull request at commit a9698c0c9625267c2ca9a19ddca8141646958b49 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_data_ingest	View in Cloud	Oct 31, 2024 at 07:39 PM (UTC)
snowflake_insights	View in Cloud	Oct 31, 2024 at 07:39 PM (UTC)
batch_enrichment	View in Cloud	Oct 31, 2024 at 07:39 PM (UTC)
basics	View in Cloud	Oct 31, 2024 at 07:39 PM (UTC)
data-eng-pipeline	View in Cloud	Oct 31, 2024 at 07:39 PM (UTC)

thomasganka

🐐

github-actions · 2024-11-05T19:35:07Z

Your pull request at commit 7b3fc756a297207dc18bbd7fd4c53d380df06214 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_bi	Building...	Nov 05, 2024 at 07:35 PM (UTC)
batch_enrichment	Building...	Nov 05, 2024 at 07:35 PM (UTC)
basics	Building...	Nov 05, 2024 at 07:35 PM (UTC)
data-eng-pipeline	Building...	Nov 05, 2024 at 07:35 PM (UTC)
hooli_data_ingest	Building...	Nov 05, 2024 at 07:35 PM (UTC)
snowflake_insights	Building...	Nov 05, 2024 at 07:35 PM (UTC)

github-actions · 2024-11-06T14:51:36Z

Your pull request at commit 92bdcb36298dffe9460f06c676417aa21f9bb22c is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_bi	Deploy failed	Nov 06, 2024 at 03:09 PM (UTC)
batch_enrichment	Deploy failed	Nov 06, 2024 at 03:09 PM (UTC)
basics	Deploy failed	Nov 06, 2024 at 03:09 PM (UTC)
data-eng-pipeline	Deploy failed	Nov 06, 2024 at 03:09 PM (UTC)
hooli_data_ingest	Deploy failed	Nov 06, 2024 at 03:09 PM (UTC)
snowflake_insights	Deploy failed	Nov 06, 2024 at 03:09 PM (UTC)

github-actions · 2024-11-06T16:21:28Z

Your pull request at commit 1969cdada247a44d09671d01493fe0f3457c2879 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_bi	Building...	Nov 06, 2024 at 04:21 PM (UTC)
batch_enrichment	Building...	Nov 06, 2024 at 04:21 PM (UTC)
basics	Building...	Nov 06, 2024 at 04:21 PM (UTC)
data-eng-pipeline	Building...	Nov 06, 2024 at 04:21 PM (UTC)
hooli_data_ingest	Building...	Nov 06, 2024 at 04:21 PM (UTC)
snowflake_insights	Building...	Nov 06, 2024 at 04:21 PM (UTC)

github-actions · 2024-11-06T16:29:53Z

Your pull request at commit 5d67217a1b30d58297e3187e31af0e2caa29cec1 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_bi	Deploy failed	Nov 06, 2024 at 04:47 PM (UTC)
batch_enrichment	Deploy failed	Nov 06, 2024 at 04:47 PM (UTC)
basics	Deploy failed	Nov 06, 2024 at 04:47 PM (UTC)
data-eng-pipeline	Deploy failed	Nov 06, 2024 at 04:47 PM (UTC)
hooli_data_ingest	Deploy failed	Nov 06, 2024 at 04:47 PM (UTC)
snowflake_insights	Deploy failed	Nov 06, 2024 at 04:47 PM (UTC)

github-actions · 2024-11-06T17:34:08Z

Your pull request at commit 906680b8434c82ef88d26c4d8f4645acdfb8242b is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_bi	Building...	Nov 06, 2024 at 05:34 PM (UTC)
batch_enrichment	Building...	Nov 06, 2024 at 05:34 PM (UTC)
basics	Building...	Nov 06, 2024 at 05:34 PM (UTC)
data-eng-pipeline	Building...	Nov 06, 2024 at 05:34 PM (UTC)
hooli_data_ingest	Building...	Nov 06, 2024 at 05:34 PM (UTC)
snowflake_insights	Building...	Nov 06, 2024 at 05:34 PM (UTC)

github-actions · 2024-11-06T17:48:00Z

Your pull request at commit 6041696783c0abce856f0f3119445e39038b505c is automatically being deployed to Dagster Cloud.

Location	Link	Updated
hooli_bi	View in Cloud	Nov 06, 2024 at 05:59 PM (UTC)
batch_enrichment	View in Cloud	Nov 06, 2024 at 05:59 PM (UTC)
basics	View in Cloud	Nov 06, 2024 at 05:59 PM (UTC)
data-eng-pipeline	View in Cloud	Nov 06, 2024 at 05:59 PM (UTC)
hooli_data_ingest	View in Cloud	Nov 06, 2024 at 05:59 PM (UTC)
snowflake_insights	View in Cloud	Nov 06, 2024 at 05:59 PM (UTC)

add initial powerbi assets

976f5d4

set groups

0a0b179

use newer API + get lineage working

c55a2ed

use refreshable semantic models

3b876cf

move power bi assets to its own file

f59360f

use BI group name

4c51d6c

cnolanminich marked this pull request as ready for review October 31, 2024 17:41

cnolanminich requested review from izzye84, slopp and thomasganka October 31, 2024 17:41

slopp approved these changes Oct 31, 2024

View reviewed changes

Update hooli_data_eng/definitions.py

759ed53

Co-authored-by: Sean Lopp <[email protected]>

cnolanminich added 3 commits October 31, 2024 14:26

remove pin

25a3319

add link to descriptions

a2908f2

Merge branch 'add-powerbi' of https://github.com/dagster-io/hooli-dat…

cd1b656

…a-eng-pipelines into add-powerbi

add tableschema to semantic models use replace and merge

a9698c0

thomasganka approved these changes Oct 31, 2024

View reviewed changes

move to separate code location

7b3fc75

use correct filepath

92bdcb3

remove email sensor for 1.9.0 compat

1969cda

Delete hooli-bi/README.md

5d67217

use system python for docker build

906680b

cnolanminich added 2 commits November 6, 2024 12:45

fix requires_python to satisfy the uv lockfile gods

9790d8e

bump lockfile

6041696

cnolanminich merged commit b6fdbcf into master Nov 6, 2024
1 check passed

cnolanminich mentioned this pull request Nov 12, 2024

[RFC] Only build code locations with changes #130

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add initial powerbi assets #127

add initial powerbi assets #127

cnolanminich commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024

github-actions bot commented Oct 31, 2024 •

edited

Loading

slopp left a comment

slopp Oct 31, 2024

cnolanminich Oct 31, 2024

cnolanminich commented Oct 31, 2024

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

thomasganka left a comment

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 6, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024

github-actions bot commented Nov 6, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024

github-actions bot commented Nov 6, 2024 •

edited

Loading

add initial powerbi assets #127

add initial powerbi assets #127

Conversation

cnolanminich commented Oct 9, 2024 • edited Loading

github-actions bot commented Oct 9, 2024 • edited Loading

github-actions bot commented Oct 9, 2024 • edited Loading

github-actions bot commented Oct 31, 2024

github-actions bot commented Oct 31, 2024 • edited Loading

github-actions bot commented Oct 31, 2024

github-actions bot commented Oct 31, 2024 • edited Loading

slopp left a comment

Choose a reason for hiding this comment

slopp Oct 31, 2024

Choose a reason for hiding this comment

cnolanminich Oct 31, 2024

Choose a reason for hiding this comment

cnolanminich commented Oct 31, 2024

github-actions bot commented Oct 31, 2024 • edited Loading

github-actions bot commented Oct 31, 2024 • edited Loading

github-actions bot commented Oct 31, 2024 • edited Loading

thomasganka left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 5, 2024

github-actions bot commented Nov 6, 2024 • edited Loading

github-actions bot commented Nov 6, 2024

github-actions bot commented Nov 6, 2024 • edited Loading

github-actions bot commented Nov 6, 2024

github-actions bot commented Nov 6, 2024 • edited Loading

cnolanminich commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024 •

edited

Loading