Merge #290

290: Fix race condition in queued record & document removal r=brunoocasali a=ellnix

# Pull Request

## Related issue
Fixes #266

## What does this PR do?

The purpose of this PR is to detach the MeiliSearch document deletion process from the ActiveRecord object so that documents corresponding to a record can be deleted even if the record no longer exists in the database.

Tests were also added for new functionality and to hopefully prevent regressions.

## PR checklist
Please check if your PR fulfills the following requirements:
- [X] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [X] Have you read the contributing guidelines?
- [X] Have you made sure that the title is accurate and descriptive of the changes?


## Solution Considerations

Please read #266 for full context:
- The suggested fix in #266 would not work due to the fact that the primary key of a record in an index can be dynamic (by a user-created method) and may not be reconstructed by the record's database ID alone. In addition, a record may have multiple entries in multiple indexes.
- I created a new job so as not to break `MSJob` (although I suspect it would be better if it was replaced with a job dedicated to enqueued indexing, since that's all it does now anyway)
- `ActiveJob` needs to somehow serialize the parameters to a job, default serialization (through `GlobalID`) relies on a database entry to reinitialize the record when the job is run
  - Since the record may no longer exist by the time the job runs, I pass a list of indexes and IDs that _may_ (more on that) belong to the record
- I created a new set of `ms_entries` methods whose purpose is to use the current configuration to list every index the record may have a document in, along with the record's primary key in that index
  - The method does not check `Utilities.indexable?` so it is possible that some of the entries it returns are not actually in MeiliSearch, this is by design since its intention is to be exhaustive and in theory those primary keys _cannot_ belong to another record so deleting them should be ok



Co-authored-by: ellnix <[email protected]>
Co-authored-by: Bruno Casali <[email protected]>

Author: 3 people

.rubocop_todo.yml

@@ -1,6 +1,6 @@
 # This configuration was generated by
 # `rubocop --auto-gen-config`
-# on 2023-11-06 11:54:44 UTC using RuboCop version 1.27.0.
+# on 2024-01-10 10:49:28 UTC using RuboCop version 1.27.0.
 # The point is for the user to remove these configuration records
 # one by one as the offenses are removed from the code base.
 # Note that changes in the inspected code, or installation of new
@@ -30,6 +30,15 @@ Layout/EmptyLinesAroundModuleBody:
   Exclude:
     - 'lib/meilisearch-rails.rb'
 
+# Offense count: 3
+# This cop supports safe auto-correction (--auto-correct).
+# Configuration parameters: EnforcedStyle.
+# SupportedStyles: symmetrical, new_line, same_line
+Layout/MultilineMethodCallBraceLayout:
+  Exclude:
+    - 'spec/integration_spec.rb'
+    - 'spec/ms_clean_up_job_spec.rb'
+
 # Offense count: 1
 # This cop supports safe auto-correction (--auto-correct).
 # Configuration parameters: EnforcedStyle, IndentationWidth.
@@ -44,6 +53,13 @@ Lint/SuppressedException:
   Exclude:
     - 'lib/meilisearch-rails.rb'
 
+# Offense count: 1
+# This cop supports safe auto-correction (--auto-correct).
+# Configuration parameters: IgnoreEmptyBlocks, AllowUnusedKeywordArguments.
+Lint/UnusedBlockArgument:
+  Exclude:
+    - 'lib/meilisearch-rails.rb'
+
 # Offense count: 2
 # This cop supports safe auto-correction (--auto-correct).
 # Configuration parameters: AllowUnusedKeywordArguments, IgnoreEmptyMethods, IgnoreNotImplementedMethods.
@@ -54,7 +70,7 @@ Lint/UnusedMethodArgument:
 # Offense count: 11
 # Configuration parameters: IgnoredMethods, CountRepeatedAttributes.
 Metrics/AbcSize:
-  Max: 102
+  Max: 104
 
 # Offense count: 1
 # Configuration parameters: CountComments, CountAsOne, ExcludedMethods, IgnoredMethods.
@@ -75,17 +91,17 @@ Metrics/CyclomaticComplexity:
 # Offense count: 16
 # Configuration parameters: CountComments, CountAsOne, ExcludedMethods, IgnoredMethods.
 Metrics/MethodLength:
-  Max: 99
+  Max: 103
 
 # Offense count: 1
 # Configuration parameters: CountComments, CountAsOne.
 Metrics/ModuleLength:
-  Max: 435
+  Max: 449
 
 # Offense count: 8
 # Configuration parameters: IgnoredMethods.
 Metrics/PerceivedComplexity:
-  Max: 33
+  Max: 34
 
 # Offense count: 1
 Naming/AccessorMethodName:
@@ -107,7 +123,7 @@ Naming/MethodParameterName:
   Exclude:
     - 'lib/meilisearch-rails.rb'
 
-# Offense count: 15
+# Offense count: 20
 RSpec/BeforeAfterAll:
   Exclude:
     - 'spec/integration_spec.rb'
@@ -118,7 +134,7 @@ RSpec/DescribeClass:
   Exclude:
     - 'spec/integration_spec.rb'
 
-# Offense count: 37
+# Offense count: 46
 # Configuration parameters: CountAsOne.
 RSpec/ExampleLength:
   Max: 19
@@ -132,7 +148,7 @@ RSpec/FilePath:
     - 'spec/settings_spec.rb'
     - 'spec/utilities_spec.rb'
 
-# Offense count: 26
+# Offense count: 25
 # Configuration parameters: AssignmentOnly.
 RSpec/InstanceVariable:
   Exclude:
@@ -150,6 +166,12 @@ RSpec/MultipleDescribes:
   Exclude:
     - 'spec/integration_spec.rb'
 
+# Offense count: 1
+# This cop supports safe auto-correction (--auto-correct).
+RSpec/MultipleSubjects:
+  Exclude:
+    - 'spec/ms_clean_up_job_spec.rb'
+
 # Offense count: 1
 # Configuration parameters: IgnoreNameless, IgnoreSymbolicNames.
 RSpec/VerifiedDoubles:
@@ -188,14 +210,45 @@ Style/InverseMethods:
   Exclude:
     - 'lib/meilisearch-rails.rb'
 
-# Offense count: 8
+# Offense count: 1
+# This cop supports safe auto-correction (--auto-correct).
+Style/MultilineIfModifier:
+  Exclude:
+    - 'lib/meilisearch-rails.rb'
+
+# Offense count: 1
+# This cop supports safe auto-correction (--auto-correct).
+# Configuration parameters: AllowedMethods.
+# AllowedMethods: be, be_a, be_an, be_between, be_falsey, be_kind_of, be_instance_of, be_truthy, be_within, eq, eql, end_with, include, match, raise_error, respond_to, start_with
+Style/NestedParenthesizedCalls:
+  Exclude:
+    - 'spec/ms_clean_up_job_spec.rb'
+
+# Offense count: 9
 # Configuration parameters: AllowedMethods.
 # AllowedMethods: respond_to_missing?
 Style/OptionalBooleanParameter:
   Exclude:
     - 'lib/meilisearch-rails.rb'
 
-# Offense count: 20
+# Offense count: 13
+# This cop supports safe auto-correction (--auto-correct).
+# Configuration parameters: EnforcedStyle, ConsistentQuotesInMultiline.
+# SupportedStyles: single_quotes, double_quotes
+Style/StringLiterals:
+  Exclude:
+    - 'spec/integration_spec.rb'
+    - 'spec/ms_clean_up_job_spec.rb'
+
+# Offense count: 2
+# This cop supports safe auto-correction (--auto-correct).
+# Configuration parameters: EnforcedStyleForMultiline.
+# SupportedStylesForMultiline: comma, consistent_comma, no_comma
+Style/TrailingCommaInArguments:
+  Exclude:
+    - 'spec/integration_spec.rb'
+
+# Offense count: 19
 # This cop supports safe auto-correction (--auto-correct).
 # Configuration parameters: AllowHeredoc, AllowURI, URISchemes, IgnoreCopDirectives, IgnoredPatterns.
 # URISchemes: http, https

lib/meilisearch-rails.rb

@@ -249,6 +249,7 @@ def additional_indexes
       # lazy load the ActiveJob class to ensure the
       # queue is initialized before using it
       autoload :MSJob, 'meilisearch/rails/ms_job'
+      autoload :MSCleanUpJob, 'meilisearch/rails/ms_clean_up_job'
     end
 
     # this class wraps an MeiliSearch::Index document ensuring all raised exceptions
@@ -382,7 +383,11 @@ def meilisearch(options = {}, &block)
 
           proc = if options[:enqueue] == true
                    proc do |record, remove|
-                     MSJob.perform_later(record, remove ? 'ms_remove_from_index!' : 'ms_index!')
+                     if remove
+                       MSCleanUpJob.perform_later(record.ms_entries)
+                     else
+                       MSJob.perform_later(record, 'ms_index!')
+                     end
                    end
                  elsif options[:enqueue].respond_to?(:call)
                    options[:enqueue]
@@ -454,7 +459,7 @@ def meilisearch(options = {}, &block)
               end
             end
           elsif respond_to?(:after_destroy)
-            after_destroy { |searchable| searchable.ms_enqueue_remove_from_index!(ms_synchronous?) }
+            after_destroy_commit { |searchable| searchable.ms_enqueue_remove_from_index!(ms_synchronous?) }
           end
         end
 
@@ -563,6 +568,19 @@ def ms_index!(document, synchronous = false)
         end.compact
       end
 
+      def ms_entries_for(document:, synchronous:)
+        primary_key = ms_primary_key_of(document)
+        raise ArgumentError, 'Cannot index a record without a primary key' if primary_key.blank?
+
+        ms_configurations.filter_map do |options, settings|
+          {
+            synchronous: synchronous || options[:synchronous],
+            index_uid: options[:index_uid],
+            primary_key: primary_key
+          }.with_indifferent_access unless ms_indexing_disabled?(options)
+        end
+      end
+
       def ms_remove_from_index!(document, synchronous = false)
         return if ms_without_auto_index_scope
 
@@ -940,6 +958,10 @@ def ms_synchronous?
         @ms_synchronous
       end
 
+      def ms_entries(synchronous = false)
+        self.class.ms_entries_for(document: self, synchronous: synchronous || ms_synchronous?)
+      end
+
       private
 
       def ms_mark_synchronous

lib/meilisearch/rails/ms_clean_up_job.rb

@@ -0,0 +1,19 @@
+module MeiliSearch
+  module Rails
+    class MSCleanUpJob < ::ActiveJob::Base
+      queue_as :meilisearch
+
+      def perform(documents)
+        documents.each do |document|
+          index = MeiliSearch::Rails.client.index(document[:index_uid])
+
+          if document[:synchronous]
+            index.delete_document!(document[:primary_key])
+          else
+            index.delete_document(document[:primary_key])
+          end
+        end
+      end
+    end
+  end
+end

spec/integration_spec.rb

@@ -628,6 +628,34 @@
     expect(results.size).to eq(1)
   end
 
+  describe '#ms_entries' do
+    it 'returns all 3 indexes for a public book' do
+      book = Book.create!(
+        name: 'Frankenstein', author: 'Mary Shelley',
+        premium: false, released: true
+      )
+
+      expect(book.ms_entries).to contain_exactly(
+        a_hash_including("index_uid" => safe_index_uid('SecuredBook')),
+        a_hash_including("index_uid" => safe_index_uid('BookAuthor')),
+        a_hash_including("index_uid" => safe_index_uid('Book')),
+      )
+    end
+
+    it 'returns all 3 indexes for a non-public book' do
+      book = Book.create!(
+        name: 'Frankenstein', author: 'Mary Shelley',
+        premium: false, released: false
+      )
+
+      expect(book.ms_entries).to contain_exactly(
+        a_hash_including("index_uid" => safe_index_uid('SecuredBook')),
+        a_hash_including("index_uid" => safe_index_uid('BookAuthor')),
+        a_hash_including("index_uid" => safe_index_uid('Book')),
+      )
+    end
+  end
+
   it 'returns facets using max values per facet' do
     10.times do
       Book.create! name: Faker::Book.title, author: Faker::Book.author, genre: Faker::Book.genre
@@ -932,7 +960,10 @@
   end
 
   describe 'ConditionallyEnqueuedDocument' do
-    before { allow(MeiliSearch::Rails::MSJob).to receive(:perform_later).and_return(nil) }
+    before do
+      allow(MeiliSearch::Rails::MSJob).to receive(:perform_later).and_return(nil)
+      allow(MeiliSearch::Rails::MSCleanUpJob).to receive(:perform_later).and_return(nil)
+    end
 
     it 'does not try to enqueue an index job when :if option resolves to false' do
       doc = ConditionallyEnqueuedDocument.create! name: 'test', is_public: false
@@ -952,7 +983,7 @@
 
       doc.destroy!
 
-      expect(MeiliSearch::Rails::MSJob).to have_received(:perform_later).with(doc, 'ms_remove_from_index!')
+      expect(MeiliSearch::Rails::MSCleanUpJob).to have_received(:perform_later).with(doc.ms_entries)
     end
   end
 end
@@ -1049,6 +1080,19 @@
 
     expect(cat_index).to eq(dog_index)
   end
+
+  describe '#ms_entries' do
+    it 'returns the correct entry for each animal' do
+      toby_dog = Dog.create!(name: 'Toby the Dog')
+      taby_cat = Cat.create!(name: 'Taby the Cat')
+
+      expect(toby_dog.ms_entries).to contain_exactly(
+        a_hash_including('primary_key' => /dog_\d+/))
+
+      expect(taby_cat.ms_entries).to contain_exactly(
+        a_hash_including('primary_key' => /cat_\d+/))
+    end
+  end
 end
 
 describe 'Songs' do

spec/ms_clean_up_job_spec.rb

@@ -0,0 +1,72 @@
+require 'spec_helper'
+
+RSpec.describe 'MeiliSearch::Rails::MSCleanUpJob' do
+  include ActiveJob::TestHelper
+
+  def clean_up_indexes
+    indexes.each(&:delete_all_documents)
+  end
+
+  def create_indexed_record
+    record
+
+    indexes.each do |index|
+      index.wait_for_task(index.tasks['results'].last['uid'])
+    end
+  end
+
+  subject(:clean_up) { MeiliSearch::Rails::MSCleanUpJob }
+
+  let(:record) do
+    Book.create name: "Moby Dick", author: "Herman Mellville",
+                premium: false, released: true
+  end
+
+  let(:record_entries) do
+    record.ms_entries(true).each { |h| h[:index_uid] += '_test' }
+  end
+
+  let(:indexes) do
+    %w[SecuredBook BookAuthor Book].map do |uid|
+      Book.index(safe_index_uid uid)
+    end
+  end
+
+  it 'removes record from all indexes' do
+    clean_up_indexes
+
+    create_indexed_record
+
+    clean_up.perform_now(record_entries)
+
+    indexes.each do |index|
+      expect(index.search('*')['hits']).to be_empty
+    end
+  end
+
+  context 'when record is already destroyed' do
+    subject(:record) do
+      Restaurant.create(
+        name: "Los Pollos Hermanos",
+        kind: "Mexican",
+        description: "Mexican chicken restaurant in Albuquerque, New Mexico.")
+    end
+
+    let(:indexes) { [Restaurant.index] }
+
+    it 'successfully deletes its document in the index' do
+      clean_up_indexes
+
+      create_indexed_record
+
+      record.delete # does not run callbacks, unlike #destroy
+
+      clean_up.perform_later(record_entries)
+      expect { perform_enqueued_jobs }.not_to raise_error
+
+      indexes.each do |index|
+        expect(index.search('*')['hits']).to be_empty
+      end
+    end
+  end
+end

spec/ms_job_spec.rb

@@ -1,6 +1,8 @@
 require 'spec_helper'
 
 RSpec.describe 'MeiliSearch::Rails::MSJob' do
+  include ActiveJob::TestHelper
+
   subject(:job) { MeiliSearch::Rails::MSJob }
 
   let(:record) { double }