8277444: Race condition on Instrumentation.retransformClasses() and class linking

[eastig]

There is a race between JvmtiClassFileReconstituter::copy_bytecodes and InstanceKlass::link_class_impl. InstanceKlass::link_class_impl can be rewriting bytecodes. JvmtiClassFileReconstituter::copy_bytecodes will not restore them to the original ones because the flag rewritten is false. This will result in invalid bytecode.

This PR adds a lock (init_lock) to the copy_bytecodes method to prevent reading bytecodes while they are being rewritten during class linking.

Tested fastdebug and release builds: Linux x86_64 and arm64

• The reproducer from JDK-8277444 passed.
• Tier1 - tier3 passed.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8277444: Race condition on Instrumentation.retransformClasses() and class linking (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/26863/head:pull/26863
$ git checkout pull/26863

Update a local copy of the PR:
$ git checkout pull/26863
$ git pull https://git.openjdk.org/jdk.git pull/26863/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 26863

View PR using the GUI difftool:
$ git pr show -t 26863

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/26863.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back eastigeevich! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@eastig The following labels will be automatically applied to this pull request:

• hotspot
• serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[eastig]

Hi @coleenp,
Could you please take a look?

[mlbridge]

Webrevs

• 01: Full - Incremental (d6895181)
• 00: Full (4882b2db)

[dholmes-ora]

@eastig I am not sure about this one. Can you clarify please how you can be transforming a class that has not yet been linked? If this is possible then it seems to me we are missing a call to ensure linkage.