Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: MonitorStage ensure at least one log line #2148

Closed
2 tasks done
dagardner-nv opened this issue Jan 28, 2025 · 1 comment · Fixed by #2170
Closed
2 tasks done

[BUG]: MonitorStage ensure at least one log line #2148

dagardner-nv opened this issue Jan 28, 2025 · 1 comment · Fixed by #2170
Assignees
Labels
bug Something isn't working

Comments

@dagardner-nv
Copy link
Contributor

dagardner-nv commented Jan 28, 2025

Version

25.02

Which installation method(s) does this occur on?

Source, Docker

Describe the bug.

For short-running pipelines, the monitor stage will potentially not emit any output.
We should always get the final Complete message which contains the total message count along with the message rate.

Minimum reproducible example

Any short-running pipeline will do, in testing this one will reproduce the error if you run it enough time

morpheus --log_level=debug --plugin examples/developer_guide/1_simple_python_stage/pass_thru.py \
  run pipeline-other \
  from-file --filename=examples/data/email_with_addresses.jsonlines \
  pass-thru \
  monitor


Compare with:

morpheus --log_level=debug --plugin examples/developer_guide/1_simple_python_stage/pass_thru.py \
  run --use_cpu_only pipeline-other \
  from-file --filename=examples/data/email_with_addresses.jsonlines \
  pass-thru \
  monitor

Relevant log output

Click here to see error details
Configuring Pipeline via CLI
====Pipeline Pre-build====
====Pre-Building Segment: linear_segment_0====
====Pre-Building Segment Complete!====
====Pipeline Pre-build Complete!====
====Registering Pipeline====
Starting pipeline via CLI... Ctrl+C to Quit
====Building Pipeline====
====Building Pipeline Complete!====
====Registering Pipeline Complete!====
Config: 
{
  "_model_max_batch_size": 8,
  "_pipeline_batch_size": 256,
  "ae": null,
  "class_labels": [],
  "debug": false,
  "edge_buffer_size": 128,
  "execution_mode": "GPU",
  "feature_length": 1,
  "fil": {
    "feature_columns": null
  },
  "frozen": true,
  "log_config_file": null,
  "log_level": 10,
  "mode": "OTHER",
  "num_threads": 64,
  "plugins": [
    "examples/developer_guide/1_simple_python_stage/pass_thru.py"
  ]
}
CPP Enabled: True
====Starting Pipeline====
====Pipeline Started====
====Building Segment: linear_segment_0====
Added source: <from-file-0; FileSourceStage(filename=examples/data/email_with_addresses.jsonlines, iterative=False, file_type=FileTypes.Auto, repeat=1, filter_null=True, filter_null_columns=(), parser_kwargs={})>
  └─> morpheus.MessageMeta
Added stage: <pass-thru-1; PassThruStage()>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
Added stage: <monitor-2; MonitorStage(description=Progress, smoothing=0.05, unit=messages, delayed_start=True, determine_count_fn=None, text_color=IndicatorsTextColor.cyan, font_style=IndicatorsFontStyle.bold, log_level=LogLevels.INFO)>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
====Building Segment Complete!====

====Pipeline Complete====

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

  • I agree to follow Morpheus' Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@dagardner-nv dagardner-nv added the bug Something isn't working label Jan 28, 2025
@dagardner-nv
Copy link
Contributor Author

There is a similar issue with pipelines which run until explicitly stopped with a Ctrl+C like the examples/developer_guide/2_2_rabbitmq/read_simple.py, the monitor produces output, yet the output is overwritten with:

====Building Segment Complete!====
^CStopping pipeline. Please wait... Press Ctrl+C again to kill. messages/s                                                                                                                  
====Stopping Pipeline====

Compared with the output when the --use_cpu_only flag is used:

====Building Segment Complete!====
Stopping pipeline. Please wait... Press Ctrl+C again to kill.
Progress: 100 messages [00:03, 32.97 messages/s]====Stopping Pipeline====
====Pipeline Stopped====
Progress[Complete]: 100 messages [00:00, 515270.76 messages/s]
====Pipeline Complete====

The original monitor output message is overwritten, but then is re-outputed (twice).

@yczhang-nv yczhang-nv self-assigned this Jan 31, 2025
@morpheus-bot-test morpheus-bot-test bot moved this from Todo to Review - Ready for Review in Morpheus Boards Feb 5, 2025
@rapids-bot rapids-bot bot closed this as completed in #2170 Feb 6, 2025
@rapids-bot rapids-bot bot closed this as completed in 2226afe Feb 6, 2025
@github-project-automation github-project-automation bot moved this from Review - Ready for Review to Done in Morpheus Boards Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants