Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The transcriptions do not work sometime #1793

Open
beastoin opened this issue Feb 12, 2025 · 16 comments
Open

The transcriptions do not work sometime #1793

beastoin opened this issue Feb 12, 2025 · 16 comments
Assignees
@beastoin

Comments

@beastoin
Copy link
Collaborator

beastoin commented Feb 12, 2025 •
edited
Loading

Crispy Admin (Discord) reported:

---
@thinh you can see it says connecting, then listening but not actually pulling transcriptions, then connecting again, then it decides to work. I was talking the entire time. its completely random how often or how many times this happens
https://discord.com/channels/1192313062041067520/1324204335474671636/1338901852842102847
---
This one actually lasted for 5 minutes but I cant upload more than 10mb so I cut it down to 40sec, but even after it returns to listening its still not grabbing words
https://discord.com/channels/1192313062041067520/1324204335474671636/1338911389514727556
--

thinh's comments:

case 1: connecting -> listening / without transcriptions in 20s -> connecting -> listening / ok

case 2: connecting -> listening / without transcription in 30s -> listenning / ok
@beastoin
Copy link
Collaborator Author

still need to investigate more, but my thoughts(for the next action) are:

  1. app, ensure only 01 API request is sent to /v3/listen at a time. is this the the socket's service edge case https://github.com/BasedHardware/omi/blob/main/app/lib/services/sockets.dart#L32 ? and, the request must be valid (not malformed), e.g. i noticed some bad requests (without uid) on the omi's system.

Image

  1. backend, double-check the Deepgram 10s in silent (keep-alived) logic and, the first 30s in the speech profile feature (i know it's tricky)

@beastoin beastoin mentioned this issue Feb 12, 2025
Closed
@vonhex
Copy link

vonhex commented Feb 12, 2025

I appreciate the help! (I'm crispy admin)

@beastoin
Copy link
Collaborator Author

beastoin commented Feb 27, 2025 •
edited
Loading

  1. app, ensure only 01 API request is sent to /v3/listen at a time. is this the the socket's service edge case https://github.com/BasedHardware/omi/blob/main/app/lib/services/sockets.dart#L32 ? and, the request must be valid (not malformed), e.g. i noticed some bad requests (without uid) on the omi's system.

I'm pretty sure that the problem is from the app because you can see that from 03:08, in 1 min, there have 3 connections have been sent to the server GET /v3/listen, which were lasting for 20-40 seconds.

I also try on the CLI and the connection still here, connected, for over 5 minutes.

Date Host Service Message
2025-02-27T03:13:51.284Z unknown backend Killing socket2 eLC***L23
2025-02-27T03:13:22.876Z unknown pusher _websocket_util_trigger eLC***L23
2025-02-27T03:13:20.106Z unknown backend Connected to Pusher transcripts trigger WebSocket. eLC***L23
2025-02-27T03:13:20.106Z unknown backend Connecting to Pusher transcripts trigger WebSocket... eLC***L23
2025-02-27T03:12:44.589Z unknown pusher INFO: ('169.254.169.126', 16918) - "WebSocket /v1/trigger/listen?uid=eLC***L23&sample_rate=16000" [accepted]
2025-02-27T03:12:33.477Z unknown backend finalize_processing_memories len(processing): 0 eLC***L23
2025-02-27T03:12:28.740Z unknown backend get_memories eLC***L23 1 0 in_progress
2025-02-27T03:12:27.049Z unknown backend _websocket_util eLC***L23 en 16000 opus True
2025-02-27T03:12:26.876Z unknown backend INFO: ('169.254.169.126', 13312) - "WebSocket /v3/listen?language=en&sample_rate=16000&codec=opus&uid=eLC***L23&include_speech_profile=true&stt_service=soniox" [accepted]
2025-02-27T03:12:18.873Z unknown backend Error closing WebSocket: Cannot call "send" once a close message has been sent. eLC***L23
2025-02-27T03:12:18.873Z unknown backend WebSocket disconnected eLC***L23
2025-02-27T03:12:11.242Z unknown backend Message: type $memory_created eLC***L23
2025-02-27T03:12:11.242Z unknown backend Message: type $memory_processing_started eLC***L23
2025-02-27T03:12:11.242Z unknown backend get_facts eLC***L23 250 0
2025-02-27T03:12:11.242Z unknown backend _create_current_memory eLC***L23
2025-02-27T03:11:51.857Z unknown backend _trigger_create_memory_with_delay not memory or not last session eLC***L23
2025-02-27T03:11:23.094Z unknown backend Killing socket2 eLC***L23
2025-02-27T03:11:23.094Z unknown backend _get_in_progress_memory new id='800b09c1-3e1f-447b-a711-2d6440dc7ec5' eLC***L23
2025-02-27T03:11:23.094Z unknown backend Connected to Pusher transcripts trigger WebSocket. eLC***L23
2025-02-27T03:11:23.094Z unknown backend Connecting to Pusher transcripts trigger WebSocket... eLC***L23
2025-02-27T03:10:28.818Z unknown backend Could not process transcript: error Unexpected ASGI message 'websocket.send', after sending 'websocket.close' or response already completed. eLC***L23
2025-02-27T03:10:28.818Z unknown backend Connected to Pusher transcripts trigger WebSocket. eLC***L23
2025-02-27T03:10:28.818Z unknown backend WebSocket disconnected eLC***L23
2025-02-27T03:10:28.818Z unknown backend Connecting to Pusher transcripts trigger WebSocket... eLC***L23
2025-02-27T03:10:01.547Z unknown pusher Plugins omi-translator-01JCQ2RMAZ3T7B1TPAG000498F is not proactive_notification or data invalid eLC***L23
2025-02-27T03:10:01.547Z unknown pusher Plugins mentor.01 is not proactive_notification or data invalid eLC***L23
2025-02-27T03:10:01.547Z unknown pusher Plugins omi-translator-01JCQ2RMAZ3T7B1TPAG000498F is not proactive_notification or data invalid eLC***L23
2025-02-27T03:10:01.547Z unknown pusher Plugins mentor.01 is not proactive_notification or data invalid eLC***L23
2025-02-27T03:10:01.547Z unknown pusher Plugins omi-translator-01JCQ2RMAZ3T7B1TPAG000498F is not proactive_notification or data invalid eLC***L23
2025-02-27T03:10:01.547Z unknown pusher Plugins mentor.01 is not proactive_notification or data invalid eLC***L23
2025-02-27T03:10:01.547Z unknown pusher _websocket_util_trigger eLC***L23
2025-02-27T03:09:58.794Z unknown plugins INFO: 169.254.169.126:57060 - "POST /mentor?uid=eLC***L23 HTTP/1.1" 200 OK
2025-02-27T03:09:58.794Z unknown plugins eLC***L23 scan_segment 3 0
2025-02-27T03:09:58.782Z plugins POST /mentor 200 114 0.008550651s python-requests/2.32.3 https://plugins-hhibjajaja-uc.a.run.app/mentor?uid=eLC***L23
2025-02-27T03:09:57.771Z unknown plugins INFO: 169.254.169.126:57054 - "POST /mentor?uid=eLC***L23 HTTP/1.1" 200 OK
2025-02-27T03:09:57.771Z unknown plugins eLC***L23 scan_segment 2 0
2025-02-27T03:09:57.759Z plugins POST /mentor 200 114 0.010322245s python-requests/2.32.3 https://plugins-hhibjajaja-uc.a.run.app/mentor?uid=eLC***L23
2025-02-27T03:09:53.883Z unknown plugins INFO: 169.254.169.126:1372 - "POST /mentor?uid=eLC***L23 HTTP/1.1" 200 OK
2025-02-27T03:09:53.883Z unknown plugins eLC***L23 scan_segment 1 0
2025-02-27T03:09:53.863Z plugins POST /mentor 200 114 0.011244516s python-requests/2.32.3 https://plugins-hhibjajaja-uc.a.run.app/mentor?uid=eLC***L23
2025-02-27T03:09:53.042Z unknown backend get_memories eLC***L23 1 0 in_progress
2025-02-27T03:09:50.234Z unknown pusher _websocket_util_trigger eLC***L23
2025-02-27T03:09:48.807Z unknown backend get_memories eLC***L23 50 0
2025-02-27T03:09:46.564Z unknown backend get_memories eLC***L23 50 0
2025-02-27T03:09:44.997Z unknown pusher INFO: ('169.254.169.126', 48016) - "WebSocket /v1/trigger/listen?uid=eLC***L23&sample_rate=16000" [accepted]
2025-02-27T03:09:44.991Z pusher GET /v1/trigger/listen 101 323 152.704518s Python/3.11 websockets/12.0 http://internal-alb.pusher-ep-prod.il7.us-central1.lb.based-hardware.internal/v1/trigger/listen?uid=eLC***L23&sample_rate=16000
2025-02-27T03:09:44.968Z unknown backend finalize_processing_memories len(processing): 0 eLC***L23
2025-02-27T03:09:44.968Z unknown backend _websocket_util eLC***L23 en 16000 opus True
2025-02-27T03:09:44.840Z unknown backend get_memories eLC***L23 1 0 in_progress
2025-02-27T03:09:43.766Z unknown backend INFO: ('169.254.169.126', 29108) - "WebSocket /v3/listen?language=en&sample_rate=16000&codec=opus&uid=eLC***L23&include_speech_profile=true&stt_service=soniox" [accepted]
2025-02-27T03:09:43.748Z backend-listen GET /v3/listen 101 485 145.123966s Dart/3.4 (dart:io) https://api.omi.me/v3/listen?language=en&sample_rate=16000&codec=opus&uid=eLC***L23&include_speech_profile=true&stt_service=soniox
2025-02-27T03:09:30.890Z unknown pusher INFO: ('169.254.169.126', 55028) - "WebSocket /v1/trigger/listen?uid=eLC***L23&sample_rate=16000" [accepted]
2025-02-27T03:09:29.956Z pusher GET /v1/trigger/listen 101 214 34.145147s Python/3.11 websockets/12.0 http://internal-alb.pusher-ep-prod.il7.us-central1.lb.based-hardware.internal/v1/trigger/listen?uid=eLC***L23&sample_rate=16000
2025-02-27T03:09:28.524Z unknown pusher _websocket_util_trigger eLC***L23
2025-02-27T03:09:24.638Z unknown backend Connected to Pusher transcripts trigger WebSocket. eLC***L23
2025-02-27T03:09:24.638Z unknown backend WebSocket disconnected eLC***L23
2025-02-27T03:09:24.638Z unknown backend Connecting to Pusher transcripts trigger WebSocket... eLC***L23
2025-02-27T03:09:24.637Z unknown backend finalize_processing_memories len(processing): 0 eLC***L23
2025-02-27T03:09:24.637Z unknown backend _websocket_util eLC***L23 en 16000 opus True
2025-02-27T03:09:16.540Z unknown pusher INFO: ('169.254.169.126', 25638) - "WebSocket /v1/trigger/listen?uid=eLC***L23&sample_rate=16000" [accepted]
2025-02-27T03:09:16.533Z pusher GET /v1/trigger/listen 101 214 30.083696s Python/3.11 websockets/12.0 http://internal-alb.pusher-ep-prod.il7.us-central1.lb.based-hardware.internal/v1/trigger/listen?uid=eLC***L23&sample_rate=16000
2025-02-27T03:09:08.926Z unknown backend get_memories eLC***L23 1 0 in_progress
2025-02-27T03:09:07.436Z unknown backend INFO: ('169.254.169.126', 59822) - "WebSocket /v3/listen?language=en&sample_rate=16000&codec=opus&uid=eLC***L23&include_speech_profile=true&stt_service=soniox" [accepted]
2025-02-27T03:09:06.200Z backend-listen GET /v3/listen 101 185 22.409050s Dart/3.4 (dart:io) https://api.omi.me/v3/listen?language=en&sample_rate=16000&codec=opus&uid=eLC***L23&include_speech_profile=true&stt_service=soniox
2025-02-27T03:08:29.716Z unknown backend get_memories eLC***L23 50 0
2025-02-27T03:08:19.981Z unknown backend finalize_processing_memories len(processing): 0 eLC***L23
2025-02-27T03:08:19.981Z unknown backend _websocket_util eLC***L23 en 16000 opus True
2025-02-27T03:08:14.695Z unknown backend get_memories eLC***L23 1 0 in_progress
2025-02-27T03:08:13.602Z unknown backend INFO: ('169.254.169.126', 2742) - "WebSocket /v3/listen?language=en&sample_rate=16000&codec=opus&uid=eLC***L23&include_speech_profile=true&stt_service=soniox" [accepted]
2025-02-27T03:08:12.672Z backend-listen GET /v3/listen 101 187 38.359595s Dart/3.4 (dart:io) https://api.omi.me/v3/listen?language=en&sample_rate=16000&codec=opus&uid=eLC***L23&include_speech_profile=true&stt_service=soniox
2025-02-27T03:08:12.273Z unknown backend get_messages eLC***L23 100 0 None True
2025-02-27T03:08:12.088Z unknown backend get_memories eLC***L23 50 0
2025-02-27T03:08:12.086Z unknown backend get_memories eLC***L23 50 0
Image

@beastoin beastoin mentioned this issue Feb 27, 2025
Merged
5 tasks
@beastoin beastoin self-assigned this Feb 28, 2025
@beastoin beastoin moved this to In review in omi TODO / bounties Feb 28, 2025
@beastoin
Copy link
Collaborator Author

beastoin commented Feb 28, 2025 •
edited
Loading

the causes, case by case:

case 1: connecting -> listening / without transcriptions in 20s -> connecting -> listening / ok

1.1 connecting -> listening / without transcriptions in 20s

  • listening means the socket connection was establish, but it took a long time for initiation.
  • the STT is not ready yet. so no transcripts.

1.2 listening / without transcriptions in 20s -> connecting

  • the app reach 20 seconds ping interval without a pong message from the backend
  • the websocket was closed by the app (w/ a normal reason)

1.3 connecting -> listening / ok

  • reconnect, the connection establish fast enough, the STT is ready

case 2: connecting -> listening / without transcription in 30s -> listenning / ok

2.1 connecting -> listening / without transcription in 30s

  • the socket connection was establish, but the STT service took a long time to connect.
  • no transcripts before the STT is ready.

the fixes: #1916

tips: enable diagnostics for further issue detection.

Image

@vonhex
Copy link

vonhex commented Feb 28, 2025

I dont have the diag setting on my app if you wanted my help on this

@beastoin
Copy link
Collaborator Author

beastoin commented Mar 1, 2025

@vonhex join our beta test program https://discord.com/channels/1192313062041067520/1296605387276943360/1296605387276943360 then install the new beta ver - build +251

@vonhex
Copy link

vonhex commented Mar 1, 2025 via email

@beastoin
Copy link
Collaborator Author

beastoin commented Mar 1, 2025

weird! can you share me screenshoots for the Settings + Developer Settings ? @vonhex

@vonhex
Copy link

vonhex commented Mar 1, 2025 via email

@beastoin
Copy link
Collaborator Author

beastoin commented Mar 1, 2025

sry sir? i could not see your screenshots😳

@vonhex
Copy link

vonhex commented Mar 1, 2025

Image

@beastoin
Copy link
Collaborator Author

beastoin commented Mar 1, 2025

the Settings page as well sir ? so that i could see the app version you are using.

the new option is in the 1.0.57 build 251.

Image

@vonhex
Copy link

vonhex commented Mar 1, 2025

Image

@beastoin
Copy link
Collaborator Author

beastoin commented Mar 1, 2025

you are using the old build sir. please update it.

@vonhex
Copy link

vonhex commented Mar 1, 2025 via email

@vonhex
Copy link

vonhex commented Mar 7, 2025

For reference i updated and enabled diag

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@beastoin beastoin

Labels
None yet
Projects
omi TODO / bounties
Status: In review
Milestone
No milestone
Development

No branches or pull requests
2 participants
@beastoin @vonhex