Add latest changes from gitlab-org/gitlab@master

Author: GitLab Bot

config/sidekiq_queues.yml

@@ -415,8 +415,6 @@
   - 1
 - - security_findings_delete_by_job_id
   - 1
-- - security_generate_scan_finding_rules
-  - 1
 - - security_orchestration_policy_rule_schedule_namespace
   - 1
 - - security_scans

doc/api/graphql/reference/index.md

@@ -15590,7 +15590,6 @@ Returns [`[SecurityTrainingUrl!]`](#securitytrainingurl).
 | ---- | ---- | ----------- |
 | <a id="projectsecuritytrainingurlsfilename"></a>`filename` | [`String`](#string) | Filename to filter security training URLs by programming language. |
 | <a id="projectsecuritytrainingurlsidentifierexternalids"></a>`identifierExternalIds` | [`[String!]!`](#string) | List of external IDs of vulnerability identifiers. |
-| <a id="projectsecuritytrainingurlslanguage"></a>`language` | [`String`](#string) | Desired language for training urls. |
 
 ##### `Project.sentryDetailedError`
 

doc/development/gemfile.md

@@ -89,6 +89,7 @@ When upgrading the Rails gem and its dependencies, you also should update the fo
 You should also update npm packages that follow the current version of Rails:
 
 - `@rails/ujs`
+  - Run `yarn patch-package @rails/ujs` after updating this to ensure our local patch file version matches.
 - `@rails/actioncable`
 
 ## Upgrading dependencies because of vulnerabilities

doc/development/integrations/secure.md

@@ -338,27 +338,10 @@ To view vulnerabilities, either:
 NOTE:
 This does not apply for the vulnerabilities existing on the default branch.
 
-### Enable report validation
-
-> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/354928) in GitLab 14.9, and planned for removal in GitLab 15.0.
-DISCLAIMER:
-This page contains information related to upcoming products, features, and functionality.
-It is important to note that the information presented is for informational purposes only.
-Please do not rely on this information for purchasing or planning purposes.
-As with all projects, the items mentioned on this page are subject to change or delay.
-The development, release, and timing of any products, features, or functionality remain at the
-sole discretion of GitLab Inc.
-In GitLab 15.0 and later, report validation is enabled and enforced. Reports that fail validation
-are not ingested, and an error message displays on the corresponding pipeline.
-
-In GitLab 14.10 and later, report validation against the schemas is enabled but not enforced.
-Reports that fail validation are ingested but display a warning in the pipeline security tab.
-
-To enforce report validation for GitLab version 14.10 and earlier, set
-[`VALIDATE_SCHEMA`](../../user/application_security/#enable-security-report-validation) to `"true"`.
-
 ### Report validation
 
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/351000) in GitLab 15.0.
+
 You must ensure that reports generated by the scanner pass validation against the schema version
 declared in your reports. Reports that don't pass validation are not ingested by GitLab, and an
 error message displays on the corresponding pipeline.

doc/development/rails_update.md

@@ -24,6 +24,7 @@ We strive to run GitLab using the latest Rails releases to benefit from performa
 1. Run `bundle update --conservative activesupport` in the `qa` folder.
 1. Resolve any Bundler conflicts.
 1. Ensure that `@rails/ujs` and `@rails/actioncable` npm packages match the new rails version in [`package.json`](https://gitlab.com/gitlab-org/gitlab/blob/master/package.json).
+1. Run `yarn patch-package @rails/ujs` after updating this to ensure our local patch file version matches.
 1. Create an MR with the `pipeline:run-all-rspec` label and see if pipeline breaks.
 1. To resolve and debug spec failures use `git bisect` against the rails repository. See the [debugging section](#git-bisect-against-rails) below.
 1. Include links to the Gem diffs between the two versions in the merge request description. For example, this is the gem diff for [`activesupport` 6.1.3.2 to

doc/topics/autodevops/quick_start_guide.md

@@ -236,7 +236,7 @@ you to common environment tasks:
 - **Monitoring** (**{chart}**) - Opens the metrics page where Prometheus collects data
   about the Kubernetes cluster and how the application
   affects it in terms of memory usage, CPU usage, and latency
-- **Deploy to** (**{play}** **{angle-down}**) - Displays a list of environments you can deploy to
+- **Deploy to** (**{play}** **{chevron-down}**) - Displays a list of environments you can deploy to
 - **Terminal** (**{terminal}**) - Opens a [web terminal](../../ci/environments/index.md#web-terminals-deprecated)
   session inside the container where the application is running
 - **Re-deploy to environment** (**{repeat}**) - For more information, see

doc/update/index.md

@@ -367,7 +367,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab
 accordingly, while also consulting the
 [version-specific upgrade instructions](#version-specific-upgrading-instructions):
 
-`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> [`12.10.14`](#12100) -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](#13120) -> [`14.0.12`](#1400) -> [`14.9.5`](#1490) -> [`14.10.Z`](#1410) -> [`15.0.Z`](#1500) -> [latest `15.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases)
+`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> [`12.10.14`](#12100) -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](#13120) -> [`14.0.12`](#1400) -> [`14.3.6`](#1430) -> [`14.9.5`](#1490) -> [`14.10.Z`](#1410) -> [`15.0.Z`](#1500) -> [latest `15.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases)
 
 The following table, while not exhaustive, shows some examples of the supported
 upgrade paths.
@@ -377,7 +377,7 @@ Additional steps between the mentioned versions are possible. We list the minima
 | -------------- | ------------ | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
 | `15.1.0`       | `14.6.2`     | `14.6.2` -> `14.9.5` -> `14.10.4` -> `15.0.2` -> `15.1.0`                                            | Three intermediate versions are required: `14.9` and `14.10`, `15.0`, then `15.1.0`.                                              |
 | `15.0.0`       | `14.6.2`     | `14.6.2` -> `14.9.5` -> `14.10.4` -> `15.0.2`                                                        | Two intermediate versions are required: `14.9` and `14.10`, then `15.0.0`.                                                        |
-| `14.6.2`       | `13.10.2`    | `13.10.2` -> `13.12.15` -> `14.0.12` -> `14.6.2`                                                     | Two intermediate versions are required: `13.12` and `14.0`, then `14.6.2`.                                                        |
+| `14.6.2`       | `13.10.2`    | `13.10.2` -> `13.12.15` -> `14.0.12` -> `14.3.6` => `14.6.2`                                         | Three intermediate versions are required: `13.12` and `14.0`, `14.3`, then `14.6.2`.                                                        |
 | `14.1.8`       | `13.9.2`     | `13.9.2` -> `13.12.15` -> `14.0.12` -> `14.1.8`                                                      | Two intermediate versions are required: `13.12` and `14.0`, then `14.1.8`.                                                        |
 | `13.12.15`     | `12.9.2`     | `12.9.2` -> `12.10.14` -> `13.0.14`  -> `13.1.11` -> `13.8.8` -> `13.12.15`                          | Four intermediate versions are required: `12.10`, `13.0`, `13.1` and `13.8.8`, then `13.12.15`.                                   |
 | `13.2.10`      | `11.5.0`     | `11.5.0` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.2.10` | Six intermediate versions are required: `11.11`, `12.0`, `12.1`, `12.10`, `13.0` and `13.1`, then `13.2.10`.                      |
@@ -706,6 +706,20 @@ for how to proceed.
   sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
   ```
 
+- After upgrading to 14.3, ensure that all the `MigrateMergeRequestDiffCommitUsers` background
+  migration jobs have completed before continuing with upgrading to GitLab 14.5 or later.
+  This is especially important if your GitLab instance has a large
+  `merge_request_diff_commits` table. Any pending
+  `MigrateMergeRequestDiffCommitUsers` background migration jobs are
+  foregrounded in GitLab 14.5, and may take a long time to complete.
+  You can check the count of pending jobs for
+  `MigrateMergeRequestDiffCommitUsers` by using the PostgreSQL console (or `sudo
+  gitlab-psql`):
+
+  ```sql
+  select count(*) from background_migration_jobs where class_name = 'MigrateMergeRequestDiffCommitUsers' and status = 0;
+  ```
+
 - See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
 
 ### 14.2.0

doc/user/project/issues/design_management.md

@@ -92,7 +92,7 @@ The design you selected opens. You can then [zoom in](#zoom-in-on-a-design) on i
 
 When viewing a design, you can move to other designs. To do so, either:
 
-- In the top-right corner, select **Go to previous design** (**{angle-left}**) or **Go to next design** (**{angle-right}**).
+- In the top-right corner, select **Go to previous design** (**{chevron-left}**) or **Go to next design** (**{chevron-right}**).
 - Press <kbd>Left</kbd> or <kbd>Right</kbd> on your keyboard.
 
 To return to the issue view, either:

lib/gitlab/background_migration/nullify_orphan_runner_id_on_ci_builds.rb

@@ -10,9 +10,9 @@ def perform(start_id, end_id, batch_table, batch_column, sub_batch_size, pause_m
         pause_ms = 0 if pause_ms < 0
 
         batch_relation = relation_scoped_to_range(batch_table, batch_column, start_id, end_id)
-        batch_relation.each_batch(column: batch_column, of: sub_batch_size, order_hint: :type) do |sub_batch|
+        batch_relation.each_batch(column: batch_column, of: sub_batch_size) do |sub_batch|
           batch_metrics.time_operation(:update_all) do
-            sub_batch.update_all(runner_id: nil)
+            filtered_sub_batch(sub_batch).update_all(runner_id: nil)
           end
 
           sleep(pause_ms * 0.001)
@@ -31,9 +31,13 @@ def connection
 
       def relation_scoped_to_range(source_table, source_key_column, start_id, stop_id)
         define_batchable_model(source_table, connection: connection)
+          .where(source_key_column => start_id..stop_id)
+      end
+
+      def filtered_sub_batch(sub_batch)
+        sub_batch
           .joins('LEFT OUTER JOIN ci_runners ON ci_runners.id = ci_builds.runner_id')
           .where('ci_builds.runner_id IS NOT NULL AND ci_runners.id IS NULL')
-          .where(source_key_column => start_id..stop_id)
       end
     end
   end

lib/tasks/gitlab/db/validate_config.rake

@@ -141,6 +141,8 @@ namespace :gitlab do
     rescue ActiveRecord::ConnectionNotEstablished, PG::ConnectionBad => err
       warn "WARNING: Could not establish database connection for #{db_config.name}: #{err.message}"
     rescue ActiveRecord::NoDatabaseError
+    rescue PG::ReadOnlySqlTransaction => err
+      warn "WARNING: Could not write to the database #{db_config.name}: #{err.message}"
     end
 
     def get_db_identifier(db_config)

package.json

@@ -150,6 +150,7 @@
     "monaco-yaml": "^2.5.1",
     "mousetrap": "1.6.5",
     "papaparse": "^5.3.1",
+    "patch-package": "^6.4.7",
     "pdfjs-dist": "^2.0.943",
     "pikaday": "^1.8.0",
     "popper.js": "^1.16.1",

patches/@rails+ujs+6.1.4-7.patch

@@ -0,0 +1,16 @@
+diff --git a/node_modules/@rails/ujs/lib/assets/compiled/rails-ujs.js b/node_modules/@rails/ujs/lib/assets/compiled/rails-ujs.js
+index 2176247..1a83d48 100644
+--- a/node_modules/@rails/ujs/lib/assets/compiled/rails-ujs.js
++++ b/node_modules/@rails/ujs/lib/assets/compiled/rails-ujs.js
+@@ -265,11 +265,6 @@ Released under the MIT license
+             try {
+               response = JSON.parse(response);
+             } catch (error) {}
+-          } else if (type.match(/\b(?:java|ecma)script\b/)) {
+-            script = document.createElement('script');
+-            script.setAttribute('nonce', cspNonce());
+-            script.text = response;
+-            document.head.appendChild(script).parentNode.removeChild(script);
+           } else if (type.match(/\b(xml|html|svg)\b/)) {
+             parser = new DOMParser();
+             type = type.replace(/;.+/, '');

qa/Gemfile.lock

@@ -118,7 +118,7 @@ GEM
     gitlab (4.18.0)
       httparty (~> 0.18)
       terminal-table (>= 1.5.1)
-    gitlab-qa (7.29.1)
+    gitlab-qa (7.32.0)
       activesupport (~> 6.1)
       gitlab (~> 4.18.0)
       http (~> 5.0)
@@ -163,7 +163,7 @@ GEM
       http-form_data (~> 2.2)
       llhttp-ffi (~> 0.4.0)
     http-accept (1.7.0)
-    http-cookie (1.0.4)
+    http-cookie (1.0.5)
       domain_name (~> 0.5)
     http-form_data (2.3.0)
     httparty (0.20.0)
@@ -198,7 +198,7 @@ GEM
     multi_xml (0.6.0)
     multipart-post (2.1.1)
     netrc (0.11.0)
-    nokogiri (1.13.3)
+    nokogiri (1.13.6)
       mini_portile2 (~> 2.8.0)
       racc (~> 1.4)
     octokit (4.21.0)
@@ -222,7 +222,7 @@ GEM
     pry-byebug (3.5.1)
       byebug (~> 9.1)
       pry (~> 0.10)
-    public_suffix (4.0.6)
+    public_suffix (4.0.7)
     racc (1.6.0)
     rack (2.2.3.1)
     rack-test (1.1.0)
@@ -295,7 +295,7 @@ GEM
     uber (0.1.0)
     unf (0.1.4)
       unf_ext
-    unf_ext (0.0.8.1)
+    unf_ext (0.0.8.2)
     unicode-display_width (2.1.0)
     unparser (0.4.7)
       abstract_type (~> 0.0.7)

scripts/frontend/postinstall.js

@@ -1,3 +1,4 @@
+const { execSync } = require('child_process');
 const chalk = require('chalk');
 
 // check that fsevents is available if we're on macOS
@@ -20,3 +21,8 @@ if (process.platform === 'darwin') {
 }
 
 console.log(`${chalk.green('success')} Dependency postinstall check passed.`);
+
+// Apply any patches to our packages
+// See https://gitlab.com/gitlab-org/gitlab/-/issues/336138
+execSync('node_modules/.bin/patch-package --error-on-fail');
+console.log(`${chalk.green('success')} Packages successfully patched.`);

spec/frontend/lib/utils/rails_ujs_spec.js

@@ -0,0 +1,78 @@
+import { setHTMLFixture } from 'helpers/fixtures';
+import waitForPromises from 'helpers/wait_for_promises';
+
+beforeAll(async () => {
+  // @rails/ujs expects jQuery.ajaxPrefilter to exist if jQuery exists at
+  // import time. This is only a problem in tests, since we expose jQuery
+  // globally earlier than in production builds. Work around this by pretending
+  // that jQuery isn't available *before* we import @rails/ujs.
+  delete global.jQuery;
+
+  const { initRails } = await import('~/lib/utils/rails_ujs.js');
+  initRails();
+});
+
+function mockXHRResponse({ responseText, responseContentType } = {}) {
+  jest
+    .spyOn(global.XMLHttpRequest.prototype, 'getResponseHeader')
+    .mockReturnValue(responseContentType);
+
+  jest.spyOn(global.XMLHttpRequest.prototype, 'send').mockImplementation(function send() {
+    requestAnimationFrame(() => {
+      Object.defineProperties(this, {
+        readyState: { value: XMLHttpRequest.DONE },
+        status: { value: 200 },
+        response: { value: responseText },
+      });
+      this.onreadystatechange();
+    });
+  });
+}
+
+// This is a test to make sure that the patch-package patch correctly disables
+// script execution for data-remote attributes.
+it('does not perform script execution via data-remote', async () => {
+  global.scriptExecutionSpy = jest.fn();
+
+  mockXHRResponse({
+    responseText: 'scriptExecutionSpy();',
+    responseContentType: 'application/javascript',
+  });
+
+  setHTMLFixture(`
+    <a href="/foo/evil.js"
+       data-remote="true"
+       data-method="get"
+       data-type="script"
+       data-testid="evil-link"
+    >XSS</a>
+  `);
+
+  const link = document.querySelector('[data-testid="evil-link"]');
+  const ajaxSuccessSpy = jest.fn();
+  link.addEventListener('ajax:success', ajaxSuccessSpy);
+
+  link.click();
+
+  await waitForPromises();
+
+  // Make sure Rails ajax machinery finished working as expected to avoid false
+  // positives
+  expect(ajaxSuccessSpy).toHaveBeenCalledTimes(1);
+
+  // If @rails/ujs has been patched correctly, this next assertion should pass.
+  //
+  // Because it's asserting something didn't happen, it is possible for it to
+  // pass for the wrong reason. So, to verify that this test correctly fails
+  // when @rails/ujs has not been patched, run:
+  //
+  //     yarn patch-package --reverse
+  //
+  // And then re-run this test. The spy should now be called, and correctly
+  // fail the test.
+  //
+  // To restore the patch(es), run:
+  //
+  //     yarn install
+  expect(global.scriptExecutionSpy).not.toHaveBeenCalled();
+});

spec/tasks/gitlab/db/validate_config_rake_spec.rb

@@ -205,6 +205,20 @@
         it_behaves_like 'raises an error', /The 'ci' since it is using 'database_tasks: false' should share database with 'main:'/
       end
     end
+
+    context 'one of the databases is in read-only mode' do
+      let(:test_config) do
+        {
+          main: main_database_config
+        }
+      end
+
+      before do
+        expect(ActiveRecord::InternalMetadata).to receive(:upsert).at_least(:once).and_raise(PG::ReadOnlySqlTransaction, "READONLY")
+      end
+
+      it_behaves_like 'validates successfully'
+    end
   end
 
   %w[db:migrate db:schema:load db:schema:dump].each do |task|