Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: Sid Viz: Error occurred trying to send message over socket on shutdown #2135

Closed
2 tasks done
dagardner-nv opened this issue Jan 27, 2025 · 0 comments
Closed
2 tasks done
Assignees
Labels
bug Something isn't working

Comments

@dagardner-nv
Copy link
Contributor

Version

25.02

Which installation method(s) does this occur on?

Docker

Describe the bug.

Error appears on pipeline shutdown, likely a race condition.

Minimum reproducible example

Follow steps in `examples/sid_visualization/README.md`

Relevant log output

Click here to see error details

====Building Segment Complete!====
Inference rate[ # ] 152160 messages in 00m:26s, Throughput: 5852.31 messages/s
Gen-viz stage completed. Waiting for shutdown
Shutting down queue
Server shut down
Error occurred trying to send message over socket
Traceback (most recent call last):
File "/workspace/python/morpheus/morpheus/stages/postprocess/generate_viz_frames_stage.py", line 206, in client_connected
await websocket.send(next_buffer.to_pybytes())
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 620, in send
await self.ensure_open()
File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 930, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedOK: sent 1001 (going away); then received 1001 (going away)
Disconnected from: 172.80.14.2:34640
Server shut down. Is queue empty: True
Gen-viz shutdown complete
====Pipeline Complete====

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

  • I agree to follow Morpheus' Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@dagardner-nv dagardner-nv added the bug Something isn't working label Jan 27, 2025
@dagardner-nv dagardner-nv self-assigned this Jan 27, 2025
@dagardner-nv dagardner-nv moved this from Todo to In Progress in Morpheus Boards Jan 27, 2025
rapids-bot bot pushed a commit that referenced this issue Jan 28, 2025
… on shutdown (#2137)

* Don't log error messages if `_server_close_event` has already been set.
* Unrelated documentation improvement for the RabbitMQ examples to ensue the `MORPHEUS_ROOT` environment variable is set.

Closes #2135

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Eli Fajardo (https://github.com/efajardo-nv)

URL: #2137
@github-project-automation github-project-automation bot moved this from In Progress to Done in Morpheus Boards Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

No branches or pull requests

1 participant