Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Live waveform data in Emap #66

Merged
merged 119 commits into from
Jan 22, 2025
Merged

Live waveform data in Emap #66

merged 119 commits into from
Jan 22, 2025

Conversation

jeremyestein
Copy link
Collaborator

@jeremyestein jeremyestein commented Oct 8, 2024 •
edited
Loading

I think it's time to create this PR now. I would be particularly interested to see @stefpiatek 's feedback.

I appreciate it's quite large, but @skeating has been reviewing it in chunks, going into the sk/waveform-dev branch from which we're now merging. So this is not the first time it's been seen by anyone except me!

waveform_hf_data.md, the main design document, is a good place to start reading.

Project board is at https://github.com/orgs/UCLH-DHCT/projects/3/views/1

@jeremyestein
First pass at adding waveform data to Emap
be8944d
@jeremyestein
Write test which checks some of the data and passes
7911d23
@jeremyestein
checkstyle fixes and make test quicker
50d34f8
@jeremyestein
Merge branch 'develop' into jeremy/hf-data
36f7a84
@jeremyestein
Clarify validation run docs
4aff3a9
@jeremyestein
In new version of checkstyle, the encoding tag is no longer valid, and
5407881 
is not needed as long as project.build.sourceEncoding property is set.
@jeremyestein
Remove duplicated file set_mvn_proxy.sh
a6e11d9
@jeremyestein
Bare bones waveform-reader service. Docker image builds, but is
39a1fa7 
untested.
@jeremyestein
Initialise the fake UDS properly; add fake UDS and waveform reader to
cf27fe5 
emap setup; fix docs
@jeremyestein
Temporarily break the core proc RabbitMQ listener by only listening to
ff7db6f 
the waveform queue. Fixing this a proper way was too awkward.
@jeremyestein
Make data more realistic in terms of volume. Had to add some layers to
d1c1c9c 
get around Spring circular dependency problems.
@jeremyestein
Make message IDs more descriptive to enable better performance tracking
b7f10f2
@jeremyestein
Ten seconds of data per message
a3db3fc
@jeremyestein
call saveAll rather than save many times
a16adbf
@jeremyestein
Batch single-row INSERTs
a209f90
@jeremyestein
Couple of other tweaks, not sure what it'll do, but need a baseline
9615106 
before enabling multi-row INSERTs
@jeremyestein
Add reWriteBatchedInserts option to db connection string so that batched
e9bff1f 
inserts are merged into fewer multi-row inserts
@jeremyestein
Does this help?
3193369
@jeremyestein
Expicitly specify sequence Id generation so we don't have to query the
46ff9a4 
sequence for every single row.
@jeremyestein
Can't use existing sequence as it has a different increment value
5361af2
@jeremyestein
Use SQL arrays (of double precision DB type)
e2635c8
@jeremyestein
Fix copy constructor
7b9b01a
@jeremyestein
Copy constructor checker didn't know about Double[]
182d2c2
@jeremyestein
Put data from multiple patients to test it doesn't get mixed up. Shuffle
5558e48 
order of messages.
@jeremyestein
Make observation time in synthetic data evenly spread out, as real data
49d93d9 
would be
@jeremyestein
Make number of patients, start date, and warp factor configurable to
cd5567f 
allow more data to be put in during validation.
@jeremyestein
Add indexes to Waveform table
55e0c60
@jeremyestein
Cassandra is still running out of memory
73619d4
@jeremyestein
Make configurable the rabbitmq queues that core will listen on,
1c3cc09 
to allow for configurations where not all data sources are present.
@jeremyestein
Make the synthetic data values use a variable amount of digits to test
52dbc3a 
out the numeric SQL data type a bit better.
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 8, 2024

PR checklist

Default guide for a PR (if multiple PRs for the work, only keep one version of it and link to it on the other PRs)

  • From the UCLH data science desktop, a validation run has been set off
  • load times
    in UCL teams has been populated with the run information
  • During the run, glowroot has been checked for any queries which are taking a substantial proportion of the
    total processing time. This can be useful to identify indexes that are required.
  • After the run, look for any unexpected errors in the etl_per_message_logging table, the error_search.sql file
    on the shared drive can be used for this \\sharefs6\UCLH6\EMAP\Shared\EmapSqlScripts\devops\error_search.sql.
    Create an issue if you find an unexpected exception and is not related to the changes you've made, otherwise
    fix them!
  • After the run, populate the end time in
    load times
  • Let Aasiyah know about the completed validation and give her information on the changes and where to start
    with the validation
  • Check validation report and give any feedback to Aasiyah if there are any changes needed on her side,
    iterate on getting the validation to match at least 99% (validation and emap code).

@jeremyestein jeremyestein marked this pull request as ready for review October 9, 2024 15:29
stefpiatek
stefpiatek approved these changes Oct 23, 2024
View reviewed changes
Copy link
Contributor

@stefpiatek stefpiatek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certainly a lot there, hopefully I made sense of most of it. Some comments and questions but nothing blocking

docs/dev/features/waveform_hf_data.md Show resolved Hide resolved
emap-setup/emap_runner/runner.py Outdated Show resolved Hide resolved
core/docker-compose.yml Outdated Show resolved Hide resolved
...in/java/uk/ac/ucl/rits/inform/datasinks/emapstar/controllers/VisitObservationController.java Outdated Show resolved Hide resolved
...in/java/uk/ac/ucl/rits/inform/datasinks/emapstar/controllers/VisitObservationController.java Outdated Show resolved Hide resolved
emap-setup/emap_runner/validation/validation_runner.py Outdated Show resolved Hide resolved
emap-setup/global-configuration-EXAMPLE.yaml Outdated Show resolved Hide resolved
emap-star/emap-star/src/main/java/uk/ac/ucl/rits/inform/informdb/visit_recordings/Waveform.java Show resolved Hide resolved
waveform-reader/src/main/java/uk/ac/ucl/rits/inform/datasources/waveform/LocationMapping.java Show resolved Hide resolved
...orm-reader/src/main/java/uk/ac/ucl/rits/inform/datasources/waveform/hl7parse/Hl7Message.java Show resolved Hide resolved
@jeremyestein jeremyestein mentioned this pull request Jan 21, 2025
Open
@jeremyestein jeremyestein merged commit d7f74bc into develop Jan 22, 2025
7 checks passed
@jeremyestein jeremyestein deleted the sk/waveform-dev branch January 22, 2025 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stefpiatek stefpiatek stefpiatek approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jeremyestein @stefpiatek @skeating