GraalJS memory leak in Engine

[mdsina]

Hello.
I'm creating service like FaaS for JS scripts with limited API and found memory leak issues on prototyping different approaches while using Graal Polyglot.

Approaches that I've trying to implement:

1. Pool of Context with single Engine. Source evaluated once in Context. Any JS code wrapped to unique namespace, that can be accessible as member of Context and executed.
   Any other invocations just execute existing member and nothing more
2. Single Engine, Sources are cached. On any new invocation we just getting existing Source and evaluate them in new Context that created for every invocation.
   After that getting function member from Context and execute (as in previous approach)
3. New Engine and New Context for every invocation, Sources are built once.

Only third approach have no memory leaks, because GC will grab them out from heap.
In first and second approaches are memory leaks. Looks like Engine have issues on executing some code and generates much data for each invocation, that will be never removed from Engine.

I have a prototype that reproduces the first approach:
https://github.com/mdsina/graaljs-executor-web-service

Also I wrote a load tests that simulate real-world invocations of JS code:
https://github.com/mdsina/graaljs-executor-web-service-load-tests/tree/master
Service was executed with VM options:

-Xmx2g -XX:+UseG1GC

Looks like links to ContextJS never be removed.

[wirthi]

Hi @mdsina,

thanks for your question.

I've not looked at your code yet, but a very generic answer of the best practice when you want code to be cached in principle is:

• Have one Engine, that you share among all your contexts.
• Create a fresh context for ever (independent) execution, AND .close it!
• For (only) those sources you want to share, enable caching (public Source.Builder cached(boolean cached)) Note that you want to reuse the identical source object for sharing to work, otherwise this could lead to a memory leak.

http://www.graalvm.org/docs/graalvm-as-a-platform/embed/#enable-source-caching

As you don't see any memory leak in your third option, I don't think this is a memory leak in our core JS engine code. Before further investigating, please make sure you properly share the engine, close the contexts, and only put the caching flag on sources that you actually want to be cached.

Thanks,
Christian

[mdsina]

Thanks for your answer, @wirthi
Of corse I do what documentation say in second approach:

1. One Engine
2. Contexts are closed after invocation (through try-with-resource statement)
3. Every shared sources are built with cached option

The one thing that I have is that when I build Context and put to JS bindings some members, like require or another one that are present like a singleton Java object (my example have that)

The last one have no problems with memory leaks is obviously. Because Engine exactly keep links to some Context object or what in self. If Engine was destroyed after every invocation so there are no problems
For the second approach I take the example from documentation https://www.graalvm.org/docs/graalvm-as-a-platform/embed/#enable-source-caching
But I want to warm up the contexts and keep them, so that's because I implements the first one approach.

[wirthi]

Hi @mdsina

We have fixed a memory leak problem around Objects with lots of (different) properties. This fix has landed in RC14 and might fix the problem you reported.

Best,
Christian

[wirthi]

Hi @mdsina

can you confirm this problem is solved for you by a newer Release (RC14 or later, or GraalVM 19).

Thanks,
Christian

[blazarus]

I hit what I think was the same problem on rc12 and tried upgrading to rc16 and still saw the memory leak. I haven't tried on v19 yet.

[mdsina]

@wirthi
The problem still presents on warming of graal context with caching

[mdsina]

Checked this out on graal 19.3.0. The same.
Also I forgot to attach heapdump initially. So where it is: https://drive.google.com/file/d/1pGQBM8pEFar1bXK7-3SIbOVXiCMCKgV-/view?usp=sharing

[dariol83]

I believe we are both hitting #268

I am wondering if there is a release plan for GraalVM and if this issue is going to be checked. I can provide some help, eventually.

[wirthi]

Hi,

what would be super-helpful for us would be an example to easily reproduce. Ideally just one Java file without any other dependency. From what I read from your descriptions (especially in #268) stems from DynamicObjects/Shapes, i.e. from Objects on the JavaScript side being created. So, in theory, the problem should be reproducible by something as simple as (pseudocode):

while (true) {
  Context ctx = new Context();
  ctx.eval("js", " //create your JavaScript object here, that you think should be GCed once the context is closed");
  ctx.close();
}

Best,
Christian

[dariol83]

Hi Christian,
I will try to come up with a simple project with a single JUnit test and try to reproduce the problem there. I will put it on Github and share the link if I succeed :)

Thanks for the help,
Dario

[dariol83]

Hi again,

I did my best to try to reproduce the leak I see in my application (#268), but without success. What you see in the attached test application is basically what I exactly do in my software but with this test application there seem to be no leak.

The only difference is that in my application the returned object is saved in memory for some time, while in the test application the returned object is not really used. But I can't believe that the problem is because of this, since typically these objects are primitive objects. Also, in my original application, if I do not cache the Engine object and I put it in a try-with-resources, I see no leak (but performance are bad) while if I cache it, I start seeing the leak... that I do not see in this test application!

In an attempt to work-around the leak, I even cached the Engine object as a SoftReference, so that in case of high memory consuption, the GC could still reclaim it. If I do that, I still see the leak, even though I see via VisualVM that the memory (and the number of related Shape objects) actually goes down a few times due to GC kicking in. I suspect that, since nobody is actually closing the Engine, something remains still laying around somewhere cached in the GraalVM JS implementation...

So I am actually clueless... maybe the problem is somewhere else, but I find it hard to believe, if changing literally two lines of code in a single class (caching the Engine and the Source, or not doing it) makes the problem to appear or disappear.

It is sad, because GraalVM JS is the script engine that provides so far the best performance I could find out there...

TestGraalVmJsLeak.zip