libpd to android, NewStringUTF error

[bestsheep1]

It seems that android NDK API crashes sometimes on the JNI part when converting data to a jstring.
This does not happen always, just under intensive message passing and it throws this error:
05-13 11:28:25.387: W/dalvikvm(17566): JNI WARNING: input is not valid Modified UTF-8: illegal start byte 0xf4

i found this workarround that could help:
http://stackoverflow.com/questions/12127817/android-ics-4-0-ndk-newstringutf-is-crashing-down-the-app

[monkeyswarm]

I am getting this as well. (Nexus 7 version 1, android v 4.3) After about 30 minutes of sending constant messages from android to libpd, I get the below stacktrace. Please advise if there is an existing workaround within PdBase

08-23 16:56:56.630: W/dalvikvm(4233): JNI WARNING: input is not valid Modified UTF-8: illegal continuation byte 0x62
08-23 16:56:56.630: W/dalvikvm(4233): string: 'Ȟ�b'
08-23 16:56:56.630: W/dalvikvm(4233): in Lorg/puredata/core/PdBase;.pollPdMessageQueue:()V (NewStringUTF)
08-23 16:56:56.630: I/dalvikvm(4233): "main" prio=5 tid=1 NATIVE
08-23 16:56:56.630: I/dalvikvm(4233): | group="main" sCount=0 dsCount=0 obj=0x4195c578 self=0x417de1c0
08-23 16:56:56.630: I/dalvikvm(4233): | sysTid=4233 nice=0 sched=0/0 cgrp=apps handle=1075486716
08-23 16:56:56.630: I/dalvikvm(4233): | state=R schedstat=( 542269091000 226526141000 743866 ) utm=50805 stm=3421 core=1
08-23 16:56:56.640: I/dalvikvm(4233): #00 pc 000012fe /system/lib/libcorkscrew.so (unwind_backtrace_thread+29)
08-23 16:56:56.640: I/dalvikvm(4233): #1 pc 0005fff6 /system/lib/libdvm.so (dvmDumpNativeStack(DebugOutputTarget const_, int)+33)
08-23 16:56:56.640: I/dalvikvm(4233): #2 pc 00054080 /system/lib/libdvm.so (dvmDumpThreadEx(DebugOutputTarget const_, Thread_, bool)+395)
08-23 16:56:56.640: I/dalvikvm(4233): #3 pc 000540ee /system/lib/libdvm.so (dvmDumpThread(Thread_, bool)+25)
08-23 16:56:56.640: I/dalvikvm(4233): #4 pc 000386a8 /system/lib/libdvm.so
08-23 16:56:56.640: I/dalvikvm(4233): #5 pc 00039a74 /system/lib/libdvm.so
08-23 16:56:56.640: I/dalvikvm(4233): #6 pc 0003be46 /system/lib/libdvm.so
08-23 16:56:56.640: I/dalvikvm(4233): #7 pc 0000362c /data/app-lib/com.iglesiaintermedia.mobmuplat-1/libpdnativeopensl.so (java_sendFloat+47)
08-23 16:56:56.640: I/dalvikvm(4233): #8 pc 000555be /data/app-lib/com.iglesiaintermedia.mobmuplat-1/libpd.so (libpd_queued_receive_pd_messages+209)
08-23 16:56:56.640: I/dalvikvm(4233): #9 pc 0000392c /data/app-lib/com.iglesiaintermedia.mobmuplat-1/libpdnativeopensl.so (Java_org_puredata_core_PdBase_pollPdMessageQueue+7)
08-23 16:56:56.640: I/dalvikvm(4233): #10 pc 0001dc4c /system/lib/libdvm.so (dvmPlatformInvoke+112)
08-23 16:56:56.640: I/dalvikvm(4233): #11 pc 0004dece /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const_, JValue_, Method const_, Thread_)+397)
08-23 16:56:56.640: I/dalvikvm(4233): #12 pc 000270a4 /system/lib/libdvm.so
08-23 16:56:56.640: I/dalvikvm(4233): at org.puredata.core.PdBase.pollPdMessageQueue(Native Method)
08-23 16:56:56.640: I/dalvikvm(4233): at org.puredata.android.io.PdAudio$1.run(PdAudio.java:34)
08-23 16:56:56.640: I/dalvikvm(4233): at android.os.Handler.handleCallback(Handler.java:730)
08-23 16:56:56.640: I/dalvikvm(4233): at android.os.Handler.dispatchMessage(Handler.java:92)
08-23 16:56:56.640: I/dalvikvm(4233): at android.os.Looper.loop(Looper.java:137)
08-23 16:56:56.640: I/dalvikvm(4233): at android.app.ActivityThread.main(ActivityThread.java:5103)
08-23 16:56:56.640: I/dalvikvm(4233): at java.lang.reflect.Method.invokeNative(Native Method)
08-23 16:56:56.640: I/dalvikvm(4233): at java.lang.reflect.Method.invoke(Method.java:525)
08-23 16:56:56.640: I/dalvikvm(4233): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737)
08-23 16:56:56.640: I/dalvikvm(4233): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
08-23 16:56:56.640: I/dalvikvm(4233): at dalvik.system.NativeStart.main(Native Method)
08-23 16:56:56.640: E/dalvikvm(4233): VM aborting
08-23 16:56:56.640: A/libc(4233): Fatal signal 11 (SIGSEGV) at 0xdeadd00d (code=1), thread 4233

[monkeyswarm]

oops, I pulled in the most recent changes, and the problem looks to be fixed. Thanks!

[shauket]

what changes you have pulled you think this is the answer of above questions ? @monkeyswarm

[monkeyswarm]

I actually am still seeing this crash, I spoke too soon before.
It seems, anecdotally, to be more prevalent on 4.4 than on <=4.3 or 5.0, but that could also be ascribable to the devices I am using as much as the OS. In any case, it happens when there is a lot of messages coming in and out, and so I just try to lessen the message bandwidth to avoid it. A fix would be great, though!

[monkeyswarm]

I have identified the issue. It's pretty deep, and I'm not sure how to best solve it.
The issue occurs when sending many lists out of Pd, and there is a race/threading issue between writing and reading to the circular message buffer.

When sending a list out of Pd, we write two chunks of data to the circular buffer.
In z_queued.c's internal_listhook() (line 141), we have:
rb_write_to_buffer(pd_receive_buffer, (const char *)&p, S_PD_PARAMS);
rb_write_to_buffer(pd_receive_buffer, (const char *)argv, n);
The first is the pd_params, and the second is the array of list element atoms. The first is a fixed size (20), and the latter is of size (arc * size of atom).

Normally, the sequence for a 3-element list would be something like this.

1. Write the pd_params to buffer (size 20)
2. Write the argv to buffer (size 3 x 8 = 24)
3. Ask how much is available to read (44 bytes)
4. Read that data, iterate through its pd_params. This assumes that the first bytes in memory are a pd_param, which has a byte for an enum, and then const char* src.

However, the crash occurs when a buffer read occurs between the two buffer writes. This happens

1. Write the pd_params to buffer (size 20)
2. Ask how much is available to read (20 bytes)
3. Read that data as a single pd_params with no argv.
4. Write the argv to buffer (size 3 x 8 = 24)
5. Ask how much is available to read (24 bytes)
6. Read that data, iterate through it assuming it is a pd_param (which it isn't). Assume that it is a pd_param with a (char* src) and try to make a java string out of it (with NewStringUTF). Crash, because of bad encoding.

So how can we make sure that the two buffer writes in listhook() (or message hook or print hook) are treated atomically? I'm rusty on my C-level threading, and I can't find where the read/write threads are set up within PdCore. What would happen if we used the same thread for reading and writing?

Thanks for any help, this would be a huge benefit to the stability of libpd on Android! (And maybe iOS as well, though I haven't really hit this issue there since it wouldn't trigger a crash on malformed data). I may also put this on libpd, since it is within that library that the threading issue occurs...

[monkeyswarm]

P.S. I've made my own workaround by only allowing a single write message at a time:

static void internal_listhook(const char *src, int argc, t_atom *argv) {
  int n = argc * S_ATOM;
  if (rb_available_to_write(pd_receive_buffer) >= S_PD_PARAMS + n) {
    pd_params p = {LIBPD_LIST, src, 0.0f, NULL, argc};
    // Allocate temporary memory to concatenate the pd_params and argv into one block of data.
    char *temp_buffer = (char *)malloc(S_PD_PARAMS + n);
    memcpy(temp_buffer, &p, S_PD_PARAMS);
    memcpy(temp_buffer+S_PD_PARAMS, argv, n);
    rb_write_to_buffer(pd_receive_buffer, (const char *)temp_buffer, S_PD_PARAMS + n);
    free(temp_buffer);
  }
}

It (seems to) solve the issue, but has the downside of allocating and freeing memory on each call. I'm assuming there's a better solution...

[nettoyeurny]

Sorry about the very tardy reply; the past few weeks have been rather insane...

First of all, thanks for investigating and diagnosing the problem, Daniel! My take is that there should be a way to write multiple buffers to the ring buffer at the same time (and update the indices only once). I'm not sure whether this should be done with varargs or whether a second write method for a pair of buffers would do. I'll mull this over and get back to you.

[tkirshboim]

closed by libpd/libpd#95