Program Not Responsive on Initial ApiWrapper Call

[apache289]

Environment:

1. AlmaLinux 8, Tomcat 8.5.100, Java 8, Trident 0.9.1
2. Amazaon EC2 t3.xlarge instance, 4vCPUs, 16GB, 120GB gp3

Problem:
My token listener program is supposed to be scheduled to run every 1 minute right after Tomcat starts, but it hangs for 4–7 minutes when it calls new ApiWrapper(endpoint, solidityEndpoint, hexPrivateKey) for the first time. After that, the program returns to normal. No error message is thrown.

Already tried the following, but the problem persists:

1. Set a 10-second timeout when calling new ApiWrapper().
2. Built the latest library on the server using Gradle, overwriting trident JAR files and dependency JAR files.
3. Used ApiWrapper.ofMainnet with a TronGrid API key.
4. Wrote a simple Java class to call ApiWrapper.ofMainnet and new ApiWrapper, ran it with java command but the same problem persists.

This issue only occurs once, right after Tomcat starts.

When I run lsof -p, I see the following two files that have been freezing there until my program returns to normal:

/opt/tomcat/temp/libnetty_transport_native_epoll_x86_643817863052686482907.so
/opt/tomcat/temp/libio_grpc_netty_shaded_netty_transport_native_epoll_x86_641427721799382997163.so

Additionally, I'd like to add that after initializing ApiWrapper, if it becomes unresponsive, Tomcat also becomes unresponsive until it eventually recovers on its own. No issues occur in my Windows development environment.

[angrynurd]

@apache289
As far as I know, Tomcat relies on the SecureRandom class to generate random numbers when using secure connections (HTTPS). On Linux systems, SecureRandom uses /dev/random as the entropy source by default. /dev/random blocks until enough entropy has been collected, which can lead to slow startups in virtual machine environments. Your observation of frozen Netty native library files under /opt/tomcat/temp/ may be caused by SecureRandom blocking waiting for entropy.

try this solution:
Use /dev/urandom as the entropy source. /dev/urandom is non-blocking and will return random numbers even if there is not enough entropy. try setting the JVM parameters:
JAVA_OPTS="$JAVA_OPTS -Djava.security.egd=file:/dev/./urandom"

[apache289]

Thank for replying. As a matter of fact, I did set urandom in my JVM parameters. Here are my parameter settings:

JAVA_OPTS="-server"
JAVA_OPTS="$JAVA_OPTS -Xms4G"
JAVA_OPTS="$JAVA_OPTS -Xmx8G"
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication"
JAVA_OPTS="$JAVA_OPTS -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m"
JAVA_OPTS="$JAVA_OPTS -Djava.library.path=/usr/local/apr/lib"
JAVA_OPTS="$JAVA_OPTS -Djava.security.egd=file:/dev/./urandom"

[angrynurd]

@apache289
could you check if your tomcat startup normal,which means it can serve the requests right after start up?
if tomcat work normally,then you can try to make these checks,as below:

1.Check the network configuration of your Amazon EC2 instance to ensure that it can access the external network properly.
2.Test network connectivity with endpoint and solidityEndpoint using ping or traceroute commands to check for latency or packet loss.
3.Try running the program using another network environment (e.g. a local network) to see if the same problem still exists. This will help determine if the problem is related to the network environment of the Amazon EC2 instance.

[apache289]

1. Verified in AWS console, default is allow all outbound traffic.
2. Ping google.com for 30 minutes without losing any packets. Run the mtr command on one of my public node IPs; over 700 packets were sent, and not a single packet was lost.
3. My local environment is Windows-based, and I do not have this issue. This is not my first time running this program on a production server; it previously ran without issues on dedicated servers. However, we recently encountered this issue when running it on a cloud server, and I experienced the same issue on DigitalOcean as well.
4. If I turn off the token listener and restart the Tomcat service, the web application runs normally.

[apache289]

Updates:
I have migrated my system to a dedicated server, and there are no more unresponsive issues when initializing a new ApiWrapper.