Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ses): Shim compatible with Hermes compiler #2334

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

leotm
Copy link
Contributor

@leotm leotm commented Jun 25, 2024

Description

Add lockdown shim compatible with Hermes
Make current shim compatible with Hermes compiler
for building React Native (RN) prod apps with SES

Generating the release AAB (Android App Bundle)
i.e. npx react-native build-android --mode=release via RN CLI
calls Gradle's bundleRelease task under the hood (bundle build task on release variant)
which calls RNGP (React Native Gradle Plugin) React task createBundleReleaseJsAndAssets and fails
after Metro finishes writing the release bundle (bundle, sourcemaps, assets)

Gradle emits these Hermes errors

(at runtime we can see both are SyntaxErrors)
Resulting in vague java.lang.StackOverflowError (no error message)

The try/catch approach testing language ft support via a new fn works
since RNGP no longer emits the Hermes errors in the task after Metro bundles
and we're conditionally using parts of SES compatible with the Hermes engine

The initial approach involved building a new shim via an env var
which involved a lot of duplicates /src files

Nb: clean before bundling to see changes reflected (avoid cache)
i.e. ./gradlew clean :app:bundleRelease

Nb: async function* a() {}; alone won't emit an error
but using/referencing it and beyond const b = a; will

Nb: eventually we hit RNGP BundleHermesCTask.kt > run > detectedHermesCommand > detectOSAwareHermesCommand from PathUtils.kt, which calls the Hermes compiler default command hermesc - the path of the binary file, to create the optimised bytecode bundle to load/exec at runtime

TODO

  • use standard async functions without arrows (module-load.js)
  • remove async generators from whitelist (permits.js)
  • remove async generators from intrinsics (get-anonymous-intrinsics-hermes.js)
  • remove repairing async generators during taming (tame-function-constructors.js)
  • remove async generator tests (ses, harden, tame-function-unit)
  • ensure 379 ses tests passing
  • test new shim in React Native repro
    • test android bundle via gradle: ./gradlew :app:createBundleReleaseJsAndAssets
    • test android build/runtime via gradle: ./gradlew :app:installRelease -PreactNativeArchitectures=arm64-v8a
    • test android/ios build/runtime via RN CLI: yarn <android/ios> --mode release
  • add new entry points to hook into get-anonymous-intrinsics-hermes.js
    • original: bundle.js > index.js > lockdown-shim.js > lockdown.js
    • hermes: bundle.js > index-hermes.js > lockdown-shim-hermes.js > lockdown-hermes.js
  • support env var to bundle new ses/lockdown shim bundle/terse variants (bundle.js)
  • bundle new shims in prepare npm lifecycle script (package.json)
  • add new shim exports entry points (package.json)
  • conditionally get intrinsics (get-anonymous-intrinsics.js)
  • conditionally repair async generators (tame-function-constructors.js)
  • conditionally add async generators to whitelist (permits.js)
  • revert new publishing changes
  • revert new entry points
  • revert new bundling with env var changes
  • ensure 379 ses tests still passing
  • restore original tests
  • suggested change
  • revert permits.js changes
  • asyncTrampoline non-arrow async function
  • console info msg x2
  • improved catch block
    • do nothing only on the SyntaxError
    • otherwise re-throw
  • add to commons.js and use
    • AsyncGeneratorFunctionInstance
    • AsyncArrowFunctionInstance?
  • CI (new PR, stacked)

Follow-up, CI testing options discussed

  • ubuntu: RN app + SES, cd android && ./gradlew :app:bundleRelease
    • add Java CI docs if included
  • react-native-community/docker-android is available to use
  • CI(macos): RN app test + SES, Xcode release
    • not needed, Xcode does not emit Hermes errors like Gradle
  • test262:hermes script (like test262:xs)
  • Hermes maintains test262 tests (testsuite.py, testsuite_skiplist.py)
  • (eshost) > Hermes CLI v0.12.0 (few years old, newer being discussed)
  • or build and run Hermes (and/or Static Hermes) as a standalone compiler and VM
    • add docs or at least link to Hermes docs

Security Considerations

Does this change introduce new assumptions or dependencies that, if violated, could introduce security vulnerabilities? How does this PR change the boundaries between mutually-suspicious components? What new authorities are introduced by this change, perhaps by new API calls?

Scaling Considerations

Does this change require or encourage significant increase in consumption of CPU cycles, RAM, on-chain storage, message exchanges, or other scarce resources? If so, can that be prevented or mitigated?

Documentation Considerations

Give our docs folks some hints about what needs to be described to downstream users. Backwards compatibility: what happens to existing data or deployments when this code is shipped? Do we need to instruct users to do something to upgrade their saved data? If there is no upgrade path possible, how bad will that be for users?

Testing Considerations

Every PR should of course come with tests of its own functionality. What additional tests are still needed beyond those unit tests? How does this affect CI, other test automation, or the testnet?

Compatibility Considerations

Does this change break any prior usage patterns? Does this change allow usage patterns to evolve?

Upgrade Considerations

What aspects of this PR are relevant to upgrading live production systems, and how should they be addressed?

Include *BREAKING*: in the commit message with migration instructions for any breaking change.

Update NEWS.md for user-facing changes.

Delete guidance from pull request description before merge (including this!)

packages/ses/src/permits.js Outdated Show resolved Hide resolved
packages/ses/index-hermes.js Outdated Show resolved Hide resolved
packages/ses/lockdown-hermes.js Outdated Show resolved Hide resolved
packages/ses/src/module-load.js Outdated Show resolved Hide resolved
packages/ses/src/module-load.js Outdated Show resolved Hide resolved
packages/ses/src/permits.js Outdated Show resolved Hide resolved
packages/ses/src/permits.js Outdated Show resolved Hide resolved
packages/ses/src/permits.js Outdated Show resolved Hide resolved
packages/ses/src/permits.js Outdated Show resolved Hide resolved
@leotm leotm marked this pull request as ready for review July 3, 2024 12:11
@leotm leotm marked this pull request as draft July 3, 2024 12:49
@leotm leotm changed the title feat(ses): Hermes compatible shim variant feat(ses): Shim compatible with Hermes Jul 3, 2024
@leotm leotm marked this pull request as ready for review July 10, 2024 11:56
@leotm
Copy link
Contributor Author

leotm commented Jul 10, 2024

ready for review ^ tackling the testing strategy separately in a follow-up PR, to keep this change small

Copy link
Member

@kriskowal kriskowal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is indeed far more surgical than I expected.

I only request that we capture the async generator function instance in a single try{eval}catch in commons.js so we can just use if blocks to decide whether to include async generators in the various points you’re currently using try/catch.

I would like to make sure @erights looks at these changes as well.

packages/ses/src/get-anonymous-intrinsics.js Outdated Show resolved Hide resolved
packages/ses/src/get-anonymous-intrinsics.js Outdated Show resolved Hide resolved
packages/ses/src/tame-function-constructors.js Outdated Show resolved Hide resolved
@kriskowal
Copy link
Member

ready for review ^ tackling the testing strategy separately in a follow-up PR, to keep this change small

Are you familiar with stacked PRs? You can propose the test changes in a PR based on this branch so they can be reviewed together. I would hesitate to approve code that doesn’t come with tests in the same merge, though I’m fine with reviewing them separately. Much depends on your workflow.

One workflow that I like is individually reviewable commits in a single PR. That does require more commit grooming, and it looks like you intend to squash the 21 commits here when you merge.

Another workflow is to create “stacked” PRs for each review artifact, then merge them top to bottom, so that ultimately the feature and its tests arrive in master with a single merge commit.

@erights
Copy link
Contributor

erights commented Jul 10, 2024

Glad to see this!

I would like to make sure @erights looks at these changes as well.

Hopefully tomorrow.

@erights erights self-requested a review July 14, 2024 08:40
packages/ses/src/module-load.js Outdated Show resolved Hide resolved
@leotm leotm force-pushed the ses-hermes branch 4 times, most recently from 708e9ae to 2d664b7 Compare October 31, 2024 20:38
erights added a commit that referenced this pull request Nov 13, 2024
Closes: #2598 
Refs: #2563
#2334
#1221

## Description

#1221 was supposed to make ses tolerate undeletable `func.prototype`
properties that should be absent, so long as they could be set to
`undefined` instead, making them harmless. This tolerance came with a
warning to flag the remaining non-conformance.

However #2598 explains why #1221 sometimes fails to do this. #1221 did
come with a test, but it fell into the case where #1221 works, which is
a non-toplevel function.

#2563 (and #2334 ?) fell into the trap explained by #2598 and untested
by #1221, which is an undeletable `func.prototype` on a top-level
instrinsic. As a result, #2563 currently contains a workaround for #2598
which this PR would make unnecessary.

This PR fixes the problem by factoring out the `func.prototype`-tolerant
property deletion into a separate `cauterizeProperty` function which it
calls from both places. This PR also adds the test that was missing from
#1221 , having first checked that the test detects #2598 when run
without the rest of this PR.

If this PR gets merged before #2563, then #2563's workaround for #2598
can first be removed before it is merged.

- [ ] TODO should pass a genuine reporter in to all calls to
`cauterizeProperty`. @kriskowal , please advise how intrinsics.js should
arrange to do so.

### Security Considerations

Allowing a `func.prototype` property that really shouldn't be there
seems safe, so long as it is safely set to `undefined` first, which this
PR does, and then checks that it has done so.

### Scaling Considerations

none

### Documentation Considerations

generally, this would be one less thing to worry about, and thus one
less thing that needs to be documented for most users.

### Testing Considerations

Adds the test that was missing from #1221 that let #2598 go unnoticed
until #2563

### Compatibility Considerations

Should be none.

### Upgrade Considerations

Should be none.
@erights
Copy link
Contributor

erights commented Nov 25, 2024

What is the status of this? Is it still needed?

@leotm leotm force-pushed the ses-hermes branch 3 times, most recently from f6fb020 to 7b9eca2 Compare November 27, 2024 17:21
@leotm
Copy link
Contributor Author

leotm commented Nov 27, 2024

What is the status of this? Is it still needed?

yes this is still needed for CI feedback keeping SES compatible with Hermes
after rebasing off master i've added 7b9eca2 for now (tbd)
otherwise 5 failing tests left to resolve and #2334 (comment) (tbd) before ready for final re-review/merge
(unless any further comments on the open threads)

I plan to add draft release notes to NEWS.md separately after the last 2 issues are addressed (removeUnpermittedIntrinsics and assertDirectEvalAvailable) which close #1891 and enable LavaMoat/LavaMoat#1438

@leotm leotm force-pushed the ses-hermes branch 3 times, most recently from 182b0e2 to aa0e8ca Compare December 3, 2024 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Windows in Hermes scripts
7 participants