History cache progress is slower when using 64CPUs than when using 16CPUs.

[rochiaw]

When I first indexed AOSP code (customized) using opengrok-indexer, the time it took to complete was several hours longer when using 64CPUs than when using 16CPUs.

I tried changing the disk specifications, but the time it took to create the history cache remained almost the same.

I tried running the indexer with --historyThreads and --historyFileThreads set to 16, and the execution time was almost the same as when using 16vCPUs. I believe the number of CPUs is affecting this.

I'd like to reduce the time it takes to create the history cache.
Could you please give me some advice on tuning methods and the optimal number of CPUs?

Usage environment and time recorded by "Done history cache for all repositories":
opengrok/docker:1.13.28

• AWS r7i.4xlarge 16vCPU 128GiB memory EBS gp3 IOPS: 3000, Throughput: 125MiB/sec → 16.5h
• AWS r7i.16xlarge 64vCPU 512GiB memory EBS gp3 IOPS: 3000, Throughput: 125MiB/sec → 20.0h
• AWS r7i.16xlarge 64vCPU 512GiB memory EBS gp3 IOPS: 16000, Throughput: 1000MiB/sec → 20.5h

Indexer options:
opengrok-indexer
-a /opengrok/lib/opengrok.jar \
-J=-Djava.util.logging.config.file=/opengrok/config/logging.properties.index \
-J=-XX:MaxRAMPercentage=50 \
-- -v \
-s /opengrok/src \
-d /opengrok/data \
-W /opengrok/etc/configuration.xml \
-U "$URI" \
--renamedHistory on \
--depth 50 \
--nestingMaximum 50 \
-c /usr/local/bin/ctags \
-m "256" \
-H\
-P\
-S\
-G

[vladak]

For starters, it would be prudent to see the utilization of the system on various levels when the history is being generated. Also, try running the experiment with renamed file handling off.

Also, I am not sure whether Docker environment could play any role.

For the record, I played a bit with JGit tuning (#4729) however the results were inconclusive.

[rochiaw]

I tested it with --renamedhistory off.

As a result, the history cache creation took about an hour, and the entire indexer process took 2.5 hours.

• AWS r7i.16xlarge 64vCPU 512GiB memory EBS gp3 IOPS: 3000, Throughput: 125MiB/sec → 1.0h

Individual repository history cache creation logs show that repository processing, which took about 17 hours with renamedhistory on, now takes about 40 minutes.
(By the way, individual repository processing with 16 CPUs and renamedhistory on took a maximum of about 8 hours.)

Thank you for letting me know about JGit tuning (#4729).
I've already increased the heap to 400g (← 250g), but it didn't affect the results.

It's wise to monitor system usage at various levels during history generation.

Thank you for this as well.