Lumen-ai deployed to k8s stalls with no error message

[MarcSkovMadsen]

lumen==0.8.0rc1

I tried deploying lumen-ai to k8s without. When I upload a small csv it just stalls with no error message in client or on server.

2025-02-02 19:33:46,362 Starting Bokeh server version 3.6.2 (running on Tornado 6.4.2)
2025-02-02 19:33:46,363 User authentication hooks NOT provided (default user enabled)
2025-02-02 19:33:46,367 Bokeh app running at: http://0.0.0.0:80/apps/mt-data-assistant/
2025-02-02 19:33:46,367 Bokeh app running at: http://0.0.0.0:80/apps/mt-data-assistant/data_assistant
2025-02-02 19:33:46,367 Starting Bokeh server with process id: 12
2025-02-02 19:35:45,331 WebSocket connection opened
2025-02-02 19:35:45,331 ServerConnection created
2025-02-02 19:35:45,691 0 (u). Are you there? YES | NO
2025-02-02 19:35:45,692 Length is 1 messages including system

Does 0 (u). Are you there? YES | NO mean its waiting for input from the server terminal?

"""An experimental data assistant based on lumen ai.

Don't yet expect a smooth user experience. This is a work in progress.
"""

import lumen.ai as lmai
import panel as pn
from azure.identity import DefaultAzureCredential
from azure.identity import get_bearer_token_provider


def _create_data_assistant() -> lmai.ui.ExplorerUI:
    token_provider = get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default")

    llm = lmai.llm.AzureOpenAI(
        api_version=...,
        endpoint=...,
        model_kwargs={
            "default": {"model": "gpt4o", "azure_ad_token_provider": token_provider},
            "reasoning": {"model": "gpt4o", "azure_ad_token_provider": token_provider},
        },
    )
    return lmai.ui.ExplorerUI(llm=llm)


if pn.state.served:
    ui = _create_data_assistant()
    ui.servable()

The problem is that the server i blocked too. Thus when I refresh the page it never responds. I'm using

num_procs=1, # panel setting
cpu_request="2.0", # k8s deployment setting below
cpu_limit="4.0",
memory_request="4Gi",
memory_limit="16Gi",

[MarcSkovMadsen]

The apps work when I manually serve the app in my JupyterHub. One difference is that when deployed only /tmp folder is writeable for security reasons. Could that be the issue that it tries to write something to disk and that it fails? But no error is reported.

[ahuang11]

Does 0 (u). Are you there? YES | NO

This means it's checking whether the LLM is available.

[ahuang11]

Maybe the errors are suppressed.

Can you set log_level="DEBUG" in ExplorerUI?

[MarcSkovMadsen]

Done. Don't see any change in server logs:

2025-02-04 18:33:15,826 Starting Bokeh server version 3.6.2 (running on Tornado 6.4.2)
2025-02-04 18:33:15,827 User authentication hooks NOT provided (default user enabled)
2025-02-04 18:33:15,830 Bokeh app running at: http://0.0.0.0:80/apps/mt-data-assistant/
2025-02-04 18:33:15,831 Bokeh app running at: http://0.0.0.0:80/apps/mt-data-assistant/data_assistant
2025-02-04 18:33:15,831 Starting Bokeh server with process id: 12
2025-02-04 18:44:20,424 WebSocket connection opened
2025-02-04 18:44:20,424 ServerConnection created
2025-02-04 18:44:20,886 0 (u). Are you there? YES | NO
2025-02-04 18:44:20,886 Length is 1 messages including system

When uploading a csv file I get timeout.

[ahuang11]

I wonder if the CSV is too large and crashing bokeh's file upload?
https://discourse.holoviz.org/t/file-upload-is-uploading-the-file-but-the-value-is-always-none/7268