Splitting rangeStrategy by depedencyType fails yarn immutable

[dbasilio]

Type of discussion.

I'm proposing an idea

Tell us more.

We maintain a few yarn monorepos that are consumed by a ~15ish apps. We previously used "rangeStategy": "update-lockfile" for those monorepos. We found that some apps would lag behind in updates because they were accepting updates from internal packages but not from 3rd parties.

We decided to try this config to help address the problems:

{
  "packageRules": [
    {
      "description": "Cut down on noise to consumers by not updating package.json if it doesn't matter to them",
      "matchDepTypes": ["peerDependencies", "devDependencies"],
      "rangeStrategy": "update-lockfile"
    },
    {
      "description": "Bump dependencies to ensure package testing is accurate to how it will be used in consumers",
      "matchDepTypes": ["dependencies"],
      "rangeStrategy": "bump"
    },
  ]
}

The goal being to maintain update-lockfile for any package bumps that the consumer would not directly action on, but force consumers to update some 3rd party packages by updating the dependencies of our packages.

This accomplishes the goal decently well, but the issue is that now we get broken Renovate PRs for our monorepos. If a 3rd party package (like sentry) is listed as dependency on one package, but is listed as a devDependency on another package (like the root package.json), then Renovate fails to update that dependency.

We require yarn immutable installs in CI and the Renovate PRs are failing that. What I think is happenging is that renovate is updating the package.json for the package that lists sentry as adependency and updating the lockfile based on that, but then not committing the package.json for that package, leading to a failed immutable install but then not updating the lockfile to include the bump in the package.json.

So my question is whether this is even a good idea. Is this a bug or just that we're doing something unexpected so it behaves weirdly. Should we just switch back to the "update-lockfile" to avoid this problem entirely?

For the record, we've had this issue with a few different packages, sentry, babel, webpack, and others.

Yarn Install Failure

10:36:35  yarn install --immutable
10:36:36  ➤ YN0000: ┌ Resolution step
10:36:37  ➤ YN0000: └ Completed in 1s 141ms
10:36:37  
10:36:37  ➤ YN0000: ┌ Post-resolution validation
10:36:37  ➤ YN0000: │ @@ -5788,9 +5788,9 @@
10:36:37  ➤ YN0000: │    checksum: 45dad206a1f600a6a4381c1d6c7809bf915ca6abbb181c9df892897dc163d7a40935fbaf4568c213d91d1da506bf4e11dbc2ee520b4772c7b7e9296554c31614
10:36:37  ➤ YN0000: │    languageName: node
10:36:37  ➤ YN0000: │    linkType: hard
10:36:37  ➤ YN0000: │  
10:36:37  ➤ YN0028: │ -"@sentry/types@npm:7.43.0, @sentry/types@npm:^7.42.0":
10:36:37  ➤ YN0028: │ +"@sentry/types@npm:7.43.0, @sentry/types@npm:^7.42.0, @sentry/types@npm:^7.43.0":
10:36:37  ➤ YN0000: │    version: 7.43.0
10:36:37  ➤ YN0000: │    resolution: "@sentry/types@npm:7.43.0"
10:36:37  ➤ YN0000: │    checksum: fc28b9996f85fde7d9e092f067a5f1e5a9882f9b212014adcf58fa134fc14149f3118240d93c8ae52f33c74cfc889a4f4d1b52bb9f57c2dda295f8c18f242df3
10:36:37  ➤ YN0000: │    languageName: node
10:36:37  ➤ YN0000: │ @@ -13291,9 +13291,9 @@
10:36:37  ➤ YN0000: │    resolution: "@ourNamepace/sentry@workspace:packages/sentry"
10:36:37  ➤ YN0000: │    dependencies:
10:36:37  ➤ YN0000: │      "@sentry/browser": ^7.43.0
10:36:37  ➤ YN0000: │      "@sentry/integrations": ^7.43.0
10:36:37  ➤ YN0028: │ -    "@sentry/types": ^7.42.0
10:36:37  ➤ YN0028: │ +    "@sentry/types": ^7.43.0
10:36:37  ➤ YN0000: │      "@types/jest": ^29.4.1
10:36:37  ➤ YN0000: │      "@types/node": ^18.13.0
10:36:37  ➤ YN0000: │      redux: ^4.2.1
10:36:37  ➤ YN0000: │ 
10:36:37  ➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
10:36:37  ➤ YN0000: └ Completed
10:36:37  ➤ YN0000: Failed with errors in 1s 344ms

[rarkins]

This part sounds like a bug:

renovate is updating the package.json for the package that lists sentry as adependency and updating the lockfile based on that, but then not committing the package.json for that package, leading to a failed immutable install.

(if I understand it right)

You say Renovate updates the package.json, runs Yarn, but then commits only the lock file change and not the package.json change?

If you can reproduce this, we can debug and hopefully fix it, although that may not be the entirety of what you need.

As a general rule, you'll find things a lot easier if you pin dependencies, and for devDependencies there's very few reasons why not to (they are not used downstream..)

[dbasilio]

Either way, the branch still fails yarn install --immutable

[rarkins]

Please paste here the logs for this branch update when the commit was made (instructions below). i.e. capturing the docker run yarn command. The lock file is usually updated by Yarn itself.

The PR (https://github.com/dbasilio/renovate-bug-repro/pull/21/files) shows that the lock file is changed. Can you add a commit on top of that showing what the full change should be?

[dbasilio]

I think these are all the relevant logs

DEBUG: Updating @sentry/types in packages/packageA/package.json(branch="renovate/sentry-javascript-monorepo")
DEBUG: Updated 1 package files(branch="renovate/sentry-javascript-monorepo")
DEBUG: Getting updated lock files(branch="renovate/sentry-javascript-monorepo")
DEBUG: Writing package.json files(branch="renovate/sentry-javascript-monorepo")
{
  "packageFiles": [
    "package.json",
    "packages/packageA/package.json",
    "packages/packageB/package.json"
  ]
}
DEBUG: Writing any updated package files(branch="renovate/sentry-javascript-monorepo")
DEBUG: Writing packages/packageA/package.json(branch="renovate/sentry-javascript-monorepo")
DEBUG: npmrc file found in repository(branch="renovate/sentry-javascript-monorepo")
DEBUG: Writing updated .npmrc file to .npmrc(branch="renovate/sentry-javascript-monorepo")
DEBUG: Generating yarn.lock for .(branch="renovate/sentry-javascript-monorepo")
DEBUG: Spawning yarn install to create yarn.lock(branch="renovate/sentry-javascript-monorepo")
DEBUG: No node constraint found - using latest(branch="renovate/sentry-javascript-monorepo")
DEBUG: Enabling global cache as zero-install is not detected(branch="renovate/sentry-javascript-monorepo")
DEBUG: Performing lockfileUpdate (yarn)(branch="renovate/sentry-javascript-monorepo")
DEBUG: Setting CONTAINERBASE_CACHE_DIR to /tmp/containerbase(branch="renovate/sentry-javascript-monorepo")
DEBUG: Using docker to execute(branch="renovate/sentry-javascript-monorepo")
{
  "image": "sidecar"
}
DEBUG: Resolved stable matching version(branch="renovate/sentry-javascript-monorepo")
{
  "toolName": "node",
  "constraint": null,
  "resolvedVersion": "v18.15.0"
}
DEBUG: Resolved stable matching version(branch="renovate/sentry-javascript-monorepo")
{
  "toolName": "corepack",
  "resolvedVersion": "0.17.1"
}
DEBUG: containerbaseDir is separate from cacheDir(branch="renovate/sentry-javascript-monorepo")
DEBUG: Resolved tag constraint(branch="renovate/sentry-javascript-monorepo")
{
  "image": "ghcr.io/containerbase/sidecar"
}
DEBUG: Docker image is already prefetched: ghcr.io/containerbase/sidecar@sha256:50ebb0ae37d32d33316b54c9c91c537be5730f15c4b3784c1e0a03a2bb012415(branch="renovate/sentry-javascript-monorepo")
DEBUG: Executing command(branch="renovate/sentry-javascript-monorepo")
{
  "command": "docker run --rm --name=renovate_sidecar --label=renovate_child -v \"/mnt/renovate/gh/dbasilio/renovate-bug-repro\":\"/mnt/renovate/gh/dbasilio/renovate-bug-repro\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/containerbase\":\"/tmp/containerbase\" -e NPM_CONFIG_CACHE -e npm_config_store -e CI -e YARN_ENABLE_IMMUTABLE_INSTALLS -e YARN_HTTP_TIMEOUT -e YARN_GLOBAL_FOLDER -e YARN_ENABLE_GLOBAL_CACHE -e BUILDPACK_CACHE_DIR -e CONTAINERBASE_CACHE_DIR -w \"/mnt/renovate/gh/dbasilio/renovate-bug-repro\" ghcr.io/containerbase/sidecar bash -l -c \"install-tool node v18.15.0 && install-tool corepack 0.17.1 && yarn install --mode=update-lockfile && yarn up '@sentry/types@^7.44.2' --mode=update-lockfile\""
}
DEBUG: exec completed(branch="renovate/sentry-javascript-monorepo")
{
  "durationMs": 7120,
  "stdout": "installing v2 tool node v18.15.0\nlinking tool node v18.15.0\nnode: v18.15.0 /usr/local/bin/node\nnpm: 9.5.0  /usr/local/bin/npm\nInstalled v2 /usr/local/buildpack/tools/v2/node.sh in 3 seconds\ninstalling v2 tool corepack v0.17.1\nlinking tool corepack v0.17.1\n0.17.1\nInstalled v2 /usr/local/buildpack/tools/v2/corepack.sh in 1 seconds\n➤ YN0000: ┌ Resolution step\n➤ YN0000: └ Completed in 0s 243ms\n➤ YN0000: ┌ Fetch step\n➤ YN0013: │ No packages were cached - one package had to be fetched (@sentry/types@npm:7.45.0)\n➤ YN0000: └ Completed in 0s 232ms\n➤ YN0000: ┌ Link step\n➤ YN0073: │ Skipped due to mode=update-lockfile\n➤ YN0000: └ Completed\n➤ YN0000: Done with warnings in 0s 486ms\n➤ YN0000: ┌ Resolution step\n➤ YN0000: └ Completed\n➤ YN0000: ┌ Fetch step\n➤ YN0000: └ Completed\n➤ YN0000: ┌ Link step\n➤ YN0073: │ Skipped due to mode=update-lockfile\n➤ YN0000: └ Completed\n➤ YN0000: Done with warnings in 0s 216ms\n",
  "stderr": ""
}
DEBUG: yarn.lock needs updating(branch="renovate/sentry-javascript-monorepo")
DEBUG: updateYarnOffline resolvedPaths(branch="renovate/sentry-javascript-monorepo")
{
  "resolvedPaths": [
    ".yarn/cache",
    ".pnp.cjs",
    ".pnp.js",
    ".pnp.loader.mjs"
  ]
}
DEBUG: Updated 1 lock files(branch="renovate/sentry-javascript-monorepo")
{
  "updatedArtifacts": [
    "yarn.lock"
  ]
}

[dbasilio]

This commit now passes the yarn install --immutable check https://github.com/dbasilio/renovate-bug-repro/pull/21/commits/e38309bc54411234dd431c0908d5154dcb1e64b7

[rarkins]

This is the command which was run yarn install --mode=update-lockfile && yarn up '@sentry/types@^7.44.2' --mode=update-lockfile.

If that command produced an invalid lock file, then it's a Yarn bug and not Renovate.

[github-actions[bot]]

Hi there,

We have found that there's a problem with the logs. Depending on which situation applies follow one, some or all of these instructions.

No logs at all

If you haven't posted any log yet, we need you to find and copy/paste the log into the issue template.

Finding logs on hosted app

Select me to read instructions

If you use the Renovate app (GitHub):

1. Go to the affected PR, and search for "View repository job log here"
2. Select the link to go to the "Mend Renovate Dashboard" and log in
3. You are now in the correct repository log overview screen
4. Copy/paste the correct log
5. Follow the steps in the formatting your logs section

Finding logs when self-hosting

Select me to read instructions

If you're running self-hosted, run with LOG_LEVEL=debug in your environment variables and search for whatever dependency/branch/PR that is causing the problem.

Insufficient logs

Select me to read instructions

If you already gave us a log, and the Renovate team said it's not enough, then follow the instructions from the No logs at all section.

Formatting your logs

Select me to read instructions

Please put your logs in a <details> and <summary> element like this:

<details><summary>Select me to see logs</summary>

```
Copy/paste your logs here, between the starting and ending backticks
```

</details>