Skip to content

Add Pipeline and Step Comment Attributes #1557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

goodwillpunning
Copy link

@goodwillpunning goodwillpunning commented May 2, 2025

Changes

This PR adds a comment attribute to the PipelineConfig and Step classes.

What does this PR do?

Adds the ability for pipeline authors to document what a pipeline and/or step does for future maintainers (including themselves).

Relevant implementation details

For pipelines containing many steps, it becomes unclear what effect a pipeline or step has. To discern what a step does, for example, a pipeline maintainer must look at the extract_source attribute, navigate to the Python or SQL file, and then read through the file to determine what a step does.

Caveats/things to watch out for when reviewing:

Linked issues

Resolves #1468.

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs remorph ...
  • Augments the PipelineConfig and Step classes with a metadata attribute.

Tests

  • manually tested
  • added unit tests
  • added integration tests

Copy link

github-actions bot commented May 2, 2025

✅ 15/15 passed, 1 skipped, 15s total

Running from acceptance #565

Copy link
Collaborator

@sundarshankar89 sundarshankar89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE]: Generate local Profiler reports
4 participants