Add Pipeline Scheduler #1522

goodwillpunning · 2025-04-23T23:43:35Z

This commit adds system files for scheduling EDW usage collection in Mac and Linux operating systems and a new class, PipelineScheduler, which parses a pipeline definition and executes steps based upon the defined frequency.

github-actions · 2025-04-23T23:47:13Z

❌ 10/16 passed, 6 failed, 1 skipped, 1m11s total

❌ test_days_since_last_run: TypeError: unsupported operand type(s) for //: 'datetime.datetime' and 'int' (2ms)

TypeError: unsupported operand type(s) for //: 'datetime.datetime' and 'int'
[gw4] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python
[gw4] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python

❌ test_hours_since_last_run: TypeError: unsupported operand type(s) for //: 'datetime.datetime' and 'int' (13ms)

TypeError: unsupported operand type(s) for //: 'datetime.datetime' and 'int'
[gw3] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python
[gw3] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python

❌ test_pipeline_scheduler: FileNotFoundError: Credentials file not found at /home/runner/.databricks/labs/remorph/.credentials.yml (22ms)

FileNotFoundError: Credentials file not found at /home/runner/.databricks/labs/remorph/.credentials.yml
[gw5] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python

❌ test_run_python_failure_pipeline: Failed: DID NOT RAISE (254ms)

Failed: DID NOT RAISE <class 'RuntimeError'>
[gw2] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python
14:40 ERROR [root] Python script failed: {"status": "error", "message": "This script is designed to fail"}

14:40 ERROR [root] Script execution failed: {"status": "error", "message": "This script is designed to fail"}
14:40 ERROR [root] Python script failed: {"status": "error", "message": "This script is designed to fail"}

14:40 ERROR [root] Script execution failed: {"status": "error", "message": "This script is designed to fail"}
[gw2] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python

❌ test_run_sql_failure_pipeline: Failed: DID NOT RAISE (1.489s)

Failed: DID NOT RAISE <class 'RuntimeError'>
[gw0] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python
14:40 ERROR [root] SQL execution failed: (pyodbc.ProgrammingError) ('42S02', "[42S02] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Invalid object name 'non_existent_table'. (208) (SQLExecDirectW)")
[SQL: SELECT * FROM non_existent_table;
]
(Background on this error at: https://sqlalche.me/e/20/f405)
14:40 ERROR [root] SQL execution failed: (pyodbc.ProgrammingError) ('42S02', "[42S02] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Invalid object name 'non_existent_table'. (208) (SQLExecDirectW)")
[SQL: SELECT * FROM non_existent_table;
]
(Background on this error at: https://sqlalche.me/e/20/f405)
14:40 ERROR [root] SQL execution failed: (pyodbc.ProgrammingError) ('42S02', "[42S02] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Invalid object name 'non_existent_table'. (208) (SQLExecDirectW)")
[SQL: SELECT * FROM non_existent_table;
]
(Background on this error at: https://sqlalche.me/e/20/f405)
14:40 ERROR [root] SQL execution failed: (pyodbc.ProgrammingError) ('42S02', "[42S02] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Invalid object name 'non_existent_table'. (208) (SQLExecDirectW)")
[SQL: SELECT * FROM non_existent_table;
]
(Background on this error at: https://sqlalche.me/e/20/f405)
[gw0] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python

❌ test_gets_maven_version: AttributeError: 'NoneType' object has no attribute 'split' (1m0.18s)

AttributeError: 'NoneType' object has no attribute 'split'
[gw2] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python
[gw2] linux -- Python 3.10.17 /home/runner/work/remorph/remorph/.venv/bin/python

_{Running from acceptance #488}

sundarshankar89

looks good few nits

sundarshankar89 · 2025-04-24T03:29:05Z

docs/remorph/docs/scheduler/macos/macos_installation.mdx

+3. Next, load the process by executing the following command:
+
+```bash
+$ launchctl load ~/Library/LaunchAgents/com.remorph.usagecollection.plist


does this work, in sequio and above launchctl bootstrap (gui/$id) <> is the way to go?

Seems to work. I've been running this process on my Mac for over a month now. Happy to update to another approach if you feel it's more appropriate.

sundarshankar89 · 2025-04-24T03:31:13Z

src/databricks/labs/remorph/assessments/pipeline.py

@@ -28,16 +35,26 @@ def execute(self):
        for step in self.config.steps:
            if step.flag == "active":
                logging.debug(f"Executing step: {step.name}")
-                self._execute_step(step)
+                self.execute_step(step)


Do you expect to people to run execute and specific step execute_step?

Sorry I meant to sync with you. I had to make this a public function so that the scheduler or another process can parse the pipeline config and execute the step based upon the defined frequency. What do you think? I know you made this private so that the pipeline class can execute entirely. But there needs to be a hook for the scheduler class or the pipeline class needs to parse the scheduled frequency info.

sundarshankar89 · 2025-04-24T03:32:07Z

...atabricks/labs/remorph/assessments/scheduler/install/macos/com.remorph.usagecollection.plist

+    <key>ProgramArguments</key>
+    <array>
+      <string>/Library/Frameworks/Python.framework/Versions/3.11/bin/python3</string>
+      <string>~/Downloads/remorph/scheduler/usage_collector.py</string>


this should be $HOME/.databricks/labs/remorph/....

asnare

This looks like a good start, and I understand its purpose. That said, there are still some rough edges to smooth out.

On the installation side:

We need to be a bit clearer on the pre-requisites for both Linux and macOS.
- Linux: systemd
- macOS: Although we can assume that python is available somewhere, referring to it in the launch agent is an issue. (We can't use /usr/bin/python3: that's Python 3.9 which we don't test against or support.) It should be possible to find it by doing the equivalent of /usr/bin/env python3.11 (an old #! trick for Python scripts).
The install files all assume that remorph has been installed in a specific location due to fixed paths. The user probably has to edit the files.
With all this in mind, I think that rather than having these instructions it might be better to have an installer script. To allow progress I'm happy to accept this as-is (if the instructions are clear for now) and iterate to implement this.
Uninstallation is also important: people won't want this forever.

On the Python side, I think it mostly looks fine although I'd really prefer to see timezone-aware timestamps being used.

asnare · 2025-04-24T08:40:53Z

docs/remorph/docs/scheduler/linux/linux_installation.mdx

These instructions are clear, although I think that we also need to describe how to remove it and clean up afterwards.

Great point. I didn't even think about uninstallation.

asnare · 2025-04-24T08:42:56Z

docs/remorph/docs/scheduler/macos/macos_installation.mdx

+4. Grant the usage collection script with execution permissions:
+
+```bash
+$ chmod +x ./src/databricks/labs/remorph/assessments/scheduler/usage_collector.py
+```


I think this should be earlier… step 2?

Otherwise I think the first run will already be triggered (and fail).

Great catch. Thanks

asnare · 2025-04-24T08:46:45Z

docs/remorph/docs/scheduler/macos/macos_installation.mdx

+
+1. Open a new Terminal window and navigate to the remorph repository
+
+2. Copy the plist file to `~/Library/LaunchAgents/` for a user-specific task or `/Library/LaunchDaemons/` for a system-wide task:


Most users won't know the difference, I don't think?

My advice here is to advise one of these, or help the user choose.

(My preference: user-specific, given the rest of the instructions assume this.)

Yep, totally agree. I'll update the PR to install only a user-specific agent.

asnare · 2025-04-24T08:48:15Z

src/databricks/labs/remorph/assessments/scheduler/install/linux/remorph_usage_collection.timer

+User=root
+
+[Install]
+WantedBy=multi-user.target


EOL at EOF.

asnare · 2025-04-24T08:51:20Z

...atabricks/labs/remorph/assessments/scheduler/install/macos/com.remorph.usagecollection.plist

+
+    <key>ProgramArguments</key>
+    <array>
+      <string>/Library/Frameworks/Python.framework/Versions/3.11/bin/python3</string>


This isn't something you can assume, unfortunately:

% ls -l /Library/Frameworks/Python.framework/Versions/3.11/bin/python3 /Library/Frameworks/Python.framework/Versions/3.11/bin/python3: No such file or directory % sw_vers ProductName: macOS ProductVersion: 15.4.1 BuildVersion: 24E263

Yes, great catch. Will update this to be dynamic.

tests/integration/assessments/test_pipeline_scheduler.py

docs/remorph/docs/scheduler/linux/linux_installation.mdx

Co-authored-by: Andrew Snare <[email protected]>

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

Co-authored-by: Andrew Snare <[email protected]>

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

gueniai · 2025-05-02T15:03:30Z

@goodwillpunning there are some unsigned commits at the beginning, which means you'll have to recreate this one :/

Add PipelineScheduler class and MacOS and Linux system files

8cee97d

goodwillpunning requested a review from a team as a code owner April 23, 2025 23:43

goodwillpunning had a problem deploying to tool April 23, 2025 23:43 — with GitHub Actions Failure

goodwillpunning requested a review from sundarshankar89 April 23, 2025 23:43

goodwillpunning added the feature/profiler label Apr 23, 2025

Merge branch 'main' into feature/add_profiler_scheduler

8e3d6f3

sundarshankar89 had a problem deploying to tool April 24, 2025 03:26 — with GitHub Actions Failure

sundarshankar89 requested review from asnare and gueniai April 24, 2025 03:26

sundarshankar89 requested changes Apr 24, 2025

View reviewed changes

asnare reviewed Apr 24, 2025

View reviewed changes

goodwillpunning mentioned this pull request Apr 25, 2025

Add scheduler components and installation instructions #1503

Closed

Update docs/remorph/docs/scheduler/linux/linux_installation.mdx

5c9aafb

Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning temporarily deployed to tool April 26, 2025 21:27 — with GitHub Actions Inactive

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

ddc4a11

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning had a problem deploying to tool April 26, 2025 21:34 — with GitHub Actions Error

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

cf8725e

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning had a problem deploying to tool April 26, 2025 21:34 — with GitHub Actions Error

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

7774267

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning had a problem deploying to tool April 26, 2025 21:34 — with GitHub Actions Error

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

5893ca9

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning had a problem deploying to tool April 26, 2025 21:34 — with GitHub Actions Error

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

6131a7f

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning had a problem deploying to tool April 26, 2025 21:35 — with GitHub Actions Error

Update tests/integration/assessments/test_pipeline_scheduler.py

cca4b22

Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning had a problem deploying to tool April 26, 2025 21:35 — with GitHub Actions Error

goodwillpunning and others added 3 commits April 26, 2025 17:35

Update tests/integration/assessments/test_pipeline_scheduler.py

a762af5

Co-authored-by: Andrew Snare <[email protected]>

Update tests/integration/assessments/test_pipeline_scheduler.py

490985d

Co-authored-by: Andrew Snare <[email protected]>

Update tests/integration/assessments/test_pipeline_scheduler.py

103d867

Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning had a problem deploying to tool April 26, 2025 21:36 — with GitHub Actions Error

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

e77d7b4

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning had a problem deploying to tool April 26, 2025 21:36 — with GitHub Actions Error

goodwillpunning and others added 3 commits April 26, 2025 17:36

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

17ff6f9

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

c1a8e2b

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

e072ccb

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning had a problem deploying to tool April 26, 2025 21:37 — with GitHub Actions Error

Update src/databricks/labs/remorph/assessments/scheduler/pipeline_sch…

db13f9d

…eduler.py Co-authored-by: Andrew Snare <[email protected]>

goodwillpunning temporarily deployed to tool April 26, 2025 21:37 — with GitHub Actions Inactive

Fix linting errors

f3dc5c3

goodwillpunning had a problem deploying to tool April 27, 2025 03:25 — with GitHub Actions Failure

Fix R0204 pylint error

919307e

goodwillpunning had a problem deploying to tool April 27, 2025 19:34 — with GitHub Actions Failure

Add installer/uninstaller scripts.

a6a6c4f

goodwillpunning had a problem deploying to tool April 28, 2025 05:49 — with GitHub Actions Failure

Add installation/uninstallation scripts for Linux

f0371c7

goodwillpunning had a problem deploying to tool April 28, 2025 15:53 — with GitHub Actions Failure

Add Windows installer/uninstaller

ede9272

goodwillpunning had a problem deploying to tool April 28, 2025 22:27 — with GitHub Actions Failure

Add prerequisite check in installation script.

87b120d

goodwillpunning had a problem deploying to tool April 29, 2025 14:36 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Pipeline Scheduler #1522

Add Pipeline Scheduler #1522

goodwillpunning commented Apr 23, 2025

github-actions bot commented Apr 23, 2025 •

edited

Loading

sundarshankar89 left a comment

sundarshankar89 Apr 24, 2025

goodwillpunning Apr 26, 2025

sundarshankar89 Apr 24, 2025

goodwillpunning Apr 26, 2025

sundarshankar89 Apr 24, 2025

asnare left a comment •

edited

Loading

asnare Apr 24, 2025

goodwillpunning Apr 26, 2025

asnare Apr 24, 2025

goodwillpunning Apr 26, 2025

asnare Apr 24, 2025

goodwillpunning Apr 26, 2025

asnare Apr 24, 2025

asnare Apr 24, 2025

goodwillpunning Apr 26, 2025

gueniai commented May 2, 2025


		1. Open a new Terminal window and navigate to the remorph repository

		2. Copy the plist file to `~/Library/LaunchAgents/` for a user-specific task or `/Library/LaunchDaemons/` for a system-wide task:

Add Pipeline Scheduler #1522

Are you sure you want to change the base?

Add Pipeline Scheduler #1522

Conversation

goodwillpunning commented Apr 23, 2025

github-actions bot commented Apr 23, 2025 • edited Loading

sundarshankar89 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asnare left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gueniai commented May 2, 2025

github-actions bot commented Apr 23, 2025 •

edited

Loading

asnare left a comment •

edited

Loading