Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Dagger.finish_stream() propagate downstream #579

Closed
wants to merge 56 commits into from

Commits on Nov 15, 2024

  1. Add metadata to EagerThunk

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    90a974f View commit details
    Browse the repository at this point in the history
  2. Sch: Allow occupancy key to be Any

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    cbac605 View commit details
    Browse the repository at this point in the history
  3. Add streaming API

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    e441bd0 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    17096fa View commit details
    Browse the repository at this point in the history
  5. Use Base.promote_op() instead of Base._return_type()

    return_type() is kinda broken in v1.10, see:
    JuliaLang/julia#52385
    
    In any case Base.promote_op() is the official public API for this operation so
    we should use it anyway.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    563f646 View commit details
    Browse the repository at this point in the history
  6. Special-case StreamingFunction in EagerThunkMetadata() constructor

    This always us to handle all the other kinds of task specs.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    2f29be7 View commit details
    Browse the repository at this point in the history
  7. Fix reference to task-queues.md in the docs

    This should get the docs building again.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    d25d6c1 View commit details
    Browse the repository at this point in the history
  8. Delete Dagger.cleanup()

    Because it doesn't actually do anything now.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    aa07cc9 View commit details
    Browse the repository at this point in the history
  9. streaming: Show thunk ID in logs

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    f8d0b8b View commit details
    Browse the repository at this point in the history
  10. streaming: Add tests

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    1be1a41 View commit details
    Browse the repository at this point in the history
  11. Use procs() when initializing EAGER_CONTEXT

    Using `myid()` with `workers()` meant that when the context was initialized with
    a single worker the processor list would be: `[OSProc(1), OSProc(1)]`. `procs()`
    will always include PID 1 and any other workers, which is what we want.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    f58404a View commit details
    Browse the repository at this point in the history
  12. streaming: Fix concurrency issues

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    ee11f3f View commit details
    Browse the repository at this point in the history
  13. Add a --verbose option to runtests.jl

    This is a bit nicer than commenting/uncommenting a line in the code.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    0d70835 View commit details
    Browse the repository at this point in the history
  14. Ensure that stream_fetch_values!() yields in its loop

    Otherwise it may spin (see comments for details). Also refactored it into a
    while-loop instead of using a @goto.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    27392f0 View commit details
    Browse the repository at this point in the history
  15. Add support for limiting the evaluations of a streaming DAG

    This is useful for testing and benchmarking.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    717ceb7 View commit details
    Browse the repository at this point in the history
  16. Dev the migration-helper branch of MemPool.jl

    This is currently necessary for the streaming branch, we'll have to change this
    later but it's good to have CI working for now.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    a0c0805 View commit details
    Browse the repository at this point in the history
  17. Minor style cleanup

    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    f4d709c View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    a7bdfdb View commit details
    Browse the repository at this point in the history
  19. Initial support for robustly migrating streaming tasks

    This works by converting the output buffers into a safely-serializeable
    container and sending that to the new node.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    770a241 View commit details
    Browse the repository at this point in the history
  20. Inherit the top-level testsets in the streaming tests

    This makes them be displayed as if they were running under the original task.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    0b968d6 View commit details
    Browse the repository at this point in the history
  21. Replace rand_finite() with a deterministic Producer functor

    This makes the tests a little easier to control.
    JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    0268b7e View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    0dbdab3 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    1cf99b8 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    71ee854 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    79ee021 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    09e5826 View commit details
    Browse the repository at this point in the history
  27. streaming: Handle cancellation

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    3911a73 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    f71f604 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    b930a42 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    16d73c9 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    2b2da8e View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    5be724f View commit details
    Browse the repository at this point in the history
  33. streaming: Buffers and tasks per input/output

    Instead of taking/putting values sequentially (which may block), runs
    "pull" and "push" tasks for each input and output, respectively. Uses
    buffers to communicate values between pullers/pushers and the streaming
    task, instead of only using one buffer per task-to-task connection.
    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    d545637 View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    61ab9c1 View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    a51cbf9 View commit details
    Browse the repository at this point in the history
  36. Add task_id for DTask

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    31944af View commit details
    Browse the repository at this point in the history
  37. ProcessRingBuffer: Allow closure

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    d5c27ab View commit details
    Browse the repository at this point in the history
  38. Configuration menu
    Copy the full SHA
    fbae73f View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    bf53117 View commit details
    Browse the repository at this point in the history
  40. Configuration menu
    Copy the full SHA
    b9e3c70 View commit details
    Browse the repository at this point in the history
  41. task-tls: Add task_cancel!

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    8908478 View commit details
    Browse the repository at this point in the history
  42. Configuration menu
    Copy the full SHA
    1f21693 View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    c4bc7b2 View commit details
    Browse the repository at this point in the history
  44. Configuration menu
    Copy the full SHA
    51e1606 View commit details
    Browse the repository at this point in the history
  45. tests: Revamp streaming tests

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    4ea09c4 View commit details
    Browse the repository at this point in the history
  46. tests: Add offline mode

    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    8bf5fbf View commit details
    Browse the repository at this point in the history
  47. Configuration menu
    Copy the full SHA
    07ba8b1 View commit details
    Browse the repository at this point in the history
  48. Configuration menu
    Copy the full SHA
    3aba122 View commit details
    Browse the repository at this point in the history
  49. Configuration menu
    Copy the full SHA
    6ac140c View commit details
    Browse the repository at this point in the history
  50. Configuration menu
    Copy the full SHA
    f60cb77 View commit details
    Browse the repository at this point in the history
  51. streaming: Get tests passing

    Switch from RemoteFetcher to RemoteChannelFetcher
    Pass object rather than type to `stream_{push,pull}_values!`
    ProcessRingBuffer: Don't exit on graceful interrupt when non-empty
    jpsamaroo authored and JamesWrigley committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    b3b70e1 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2024

  1. Bump MemPool compat

    JamesWrigley committed Nov 16, 2024
    Configuration menu
    Copy the full SHA
    efc80be View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e6a504d View commit details
    Browse the repository at this point in the history
  3. Streaming tests cleanup and fixes

    - Added some whitespace.
    - Deleted the unused `rand_finite()` methods.
    - Allow passing the `timeout` to `test_finishes()`
    - Fix bug in one of the tests where we weren't waiting for all the tasks to
      finish, which would occasionally cause test failures because of the race
      condition.
    JamesWrigley committed Nov 16, 2024
    Configuration menu
    Copy the full SHA
    149adb5 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2024

  1. Make Dagger.finish_stream() propagate downstream

    Previously a streaming task calling `Dagger.finish_stream()` would only stop the
    caller, but now it will also stop all downstream tasks. This is done by:
    - Getting the output handler tasks to close their `RemoteChannel` when exiting.
    - Making the input handler tasks close their buffers when the `RemoteChannel` is
      closed.
    - Exiting `stream!()` when an input buffer is closed.
    JamesWrigley committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    079e9fa View commit details
    Browse the repository at this point in the history
  2. Fix @test_throws_unwrap tests

    `unwrap_nested_exception()` now supports `DTaskFailedException` so we can match
    against the real exceptions thrown.
    JamesWrigley committed Nov 17, 2024
    Configuration menu
    Copy the full SHA
    999bdd7 View commit details
    Browse the repository at this point in the history