Error when using the Android ChatApp with llama3.2-3B model exported with 4096 context length

[mikel-brostrom]

This model works:

python3 -m qai_hub_models.models.llama_v3_2_3b_chat_quantized.export --output-dir genie_bundle --device "Snapdragon 8 Elite QRD" --skip-inferencing --skip-profiling --context-length 2048

This does not:

python3 -m qai_hub_models.models.llama_v3_2_3b_chat_quantized.export --output-dir genie_bundle --device "Snapdragon 8 Elite QRD" --skip-inferencing --skip-profiling --context-length 4096

With the following error:

Failed to create Genie dialog.
2025-02-25 13:57:21.169 26858-26858 libc                    com.quicinc.chatapp                  A  Fatal signal 5 (SIGTRAP), code 1 (TRAP_BRKPT), fault addr 0x7035102c64 in tid 26858 (quicinc.chatapp), pid 26858 (quicinc.chatapp)
---------------------------- PROCESS STARTED (27228) for package com.quicinc.chatapp ----------------------------
2025-02-25 13:57:21.610 27226-27226 DEBUG                   crash_dump64                         A  Cmdline: com.quicinc.chatapp
2025-02-25 13:57:21.610 27226-27226 DEBUG                   crash_dump64                         A  pid: 26858, tid: 26858, name: quicinc.chatapp  >>> com.quicinc.chatapp <<<
2025-02-25 13:57:21.610 27226-27226 DEBUG                   crash_dump64                         A        #00 pc 00000000000bec64  /data/app/~~uQ8L_iMIqAWJciW_0zZOkA==/com.quicinc.chatapp-qFufws9ofCh7xcxTt4ABvw==/lib/arm64/libchatapp.so (Java_com_quicinc_chatapp_GenieWrapper_loadModel+940) (BuildId: 9d0083676a2fa2029d39413b04282ebcccbd4580)
2025-02-25 13:57:21.610 27226-27226 DEBUG                   crash_dump64                         A        #06 pc 0000000000002548  /data/app/~~uQ8L_iMIqAWJciW_0zZOkA==/com.quicinc.chatapp-qFufws9ofCh7xcxTt4ABvw==/base.apk (com.quicinc.chatapp.GenieWrapper.<init>+0)
2025-02-25 13:57:21.611 27226-27226 DEBUG                   crash_dump64                         A        #11 pc 0000000000002318  /data/app/~~uQ8L_iMIqAWJciW_0zZOkA==/com.quicinc.chatapp-qFufws9ofCh7xcxTt4ABvw==/base.apk (com.quicinc.chatapp.Conversation.onCreate+0)
---------------------------- PROCESS ENDED (26858) for package com.quicinc.chatapp ----------------------------

Device specs:

Name: Samsung Galaxy S25 Ultra
Model Name: SM-S938B/DS
Android version: 15

[mestrona-3]

HI @mikel-brostrom, I know we've discussed a bunch about your Llama use, in Slack. Is this still an issue you're hitting?

[mikel-brostrom]

By default only 2048 works in the ChatApp. Don't really understand what is required to solve this 😅

[gustavla]

@mikel-brostrom Are you using your own genie-config.json, or falling back to the default? The default one is set up for 2048. This is the beginning of the config:

$ head src/main/assets/models/llama3_2_3b/genie-config.json
{
    "dialog": {
        "version": 1,
        "type": "basic",
        "context": {
            "version": 1,
            "size": 2048,            # <----
            "n-vocab": 128256,
            "bos-token": -1,
            "eos-token": [128001, 128009, 128008]

Can you try changing this to 4096 and trying again?

We'll make sure this necessary modification is mentioned somewhere. We are also working with Genie SDK team to see if it really is necessary to specify this or if it can be inferred from the context binaries.