[Jobs] Allow logs for finished jobs and add sky jobs logs --refresh for restartin jobs controller
#4380
+177
−52
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Michaelvll
requested review from
cblmemo and
cg505
and removed request for
cblmemo
Michaelvll
changed the title
sky jobs logs --refresh for restartin jobs controller
cblmemo
reviewed
Nov 18, 2024
There was a problem hiding this comment.
Thanks for adding this @Michaelvll ! Left some discussions.
cg505
reviewed
Nov 22, 2024
sky/jobs/state.py
Outdated
|
|
||
| def get_local_log_file(job_id: int, task_id: Optional[int]) -> Optional[str]: | ||
| """Get the local log directory for a job.""" | ||
| task_str = '' if task_id is None else f' AND task_id={task_id}' |
There was a problem hiding this comment.
Comment on lines
+387
to
+394
| # Stream the logs to the console without reading the whole | ||
| # file into memory. | ||
| start_streaming = False | ||
| for line in f: | ||
| if log_lib.LOG_FILE_START_STREAMING_AT in line: | ||
| start_streaming = True | ||
| if start_streaming: | ||
| print(line, end='', flush=True) |
There was a problem hiding this comment.
Should we move this into some utils file?
There was a problem hiding this comment.
Good point! It may require a larger refactoring of the log_lib functions as well. Let's do it in a different PR. Added a TODO.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently,
sky jobs logsdoes not work with finished jobs, although we do have the logs downloaded for failed jobs to the controller. We now enablesky jobs logsfor all finished jobs, including succeeded ones.Tested (run the relevant ones):
bash format.shpytest tests/test_smoke.pypytest tests/test_smoke.py --managed-jobspytest tests/test_smoke.py::test_fill_in_the_namepytest tests/test_smoke.py::test_env_check(with newly added --detach-setup test)conda deactivate; bash -i tests/backward_compatibility_tests.sh