Introduce have_reported_error matcher #2849

skatkov · 2025-06-04T22:53:33Z

rspec-rails is missing support for Rails ErrorReporter, this was introduced to rails in v7 and has been evolving ever since. With my client, we have moved to using ErrorReporter as a unified error reporting interface, so we can easily move from one error tracking software to another with minimal code changes. And we had a need to test this interface with rspec, so we implemented our own matcher to handle this.

I'm suggesting our internal implementation as is. This is probably not suitable as is for this gem, but I'd like to open discussion with this starting point.

Example usage

@example Checking for any error
  expect { Rails.error.report(StandardError.new) }.to have_reported_error

@example Checking for specific error class
  expect { Rails.error.report(MyError.new) }.to have_reported_error(MyError)

@example Checking for specific error instance with message
  expect { Rails.error.report(MyError.new("message")) }.to have_reported_error(MyError.new("message"))

@example Checking error attributes
  expect { Rails.error.report(StandardError.new, context: "test") }.to have_reported_error.with(context: "test")

@example Checking error message patterns
  expect { Rails.error.report(StandardError.new("test message")) }.to have_reported_error(/test/)

@example Negation
  expect { "safe code" }.not_to have_reported_error

pirj

Thank you!

spec/rspec/rails/matchers/have_reported_error_spec.rb

features/matchers/README.md

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-06-21T06:59:54Z

lib/rspec/rails/matchers/have_reported_error.rb

+          return true if @attributes.empty?
+          return false if @error_subscriber.events.empty?
+
+          event_context = @error_subscriber.events.last[1].with_indifferent_access["context"]


Does it lock and match only on the last one reported?

@pirj yeah, it does. I've been debating this, but this works in our case just fine.

But questions keep popping in my head. Should we look through array of errors? what to do if there will be more than 1? Should we match amount of errors too?

I don't have good answers for these. So this is my dumbest approach -- checking last error only.

We should match like we do with scheduled jobs, if there is a matching one among reported, it matches.

I don’t see a reason to add count matches now. This can be added later.

lib/rspec/rails/matchers/have_reported_error.rb

spec/rspec/rails/matchers/have_reported_error_spec.rb

features/matchers/README.md

features/matchers/have_reported_error_matcher.feature

JonRowe · 2025-06-25T17:14:51Z

features/matchers/have_reported_error_matcher.feature

+  Scenario: Using in controller specs
+    Given a file named "spec/controllers/users_controller_spec.rb" with:
+      """ruby
+      require "rails_helper"
+
+      RSpec.describe UsersController, type: :controller do
+        describe "POST #create" do
+          it "reports validation errors" do
+            expect {
+              post :create, params: { user: { email: "invalid" } }
+            }.to have_reported_error(ValidationError)
+          end
+        end
+      end
+      """
+    When I run `rspec spec/controllers/users_controller_spec.rb`
+    Then the examples should all pass
+
+  Scenario: Using in request specs
+    Given a file named "spec/requests/users_spec.rb" with:
+      """ruby
+      require "rails_helper"
+
+      RSpec.describe "Users", type: :request do
+        describe "POST /users" do
+          it "reports processing errors" do
+            expect {
+              post "/users", params: { user: { name: "Test" } }
+            }.to have_reported_error.with(context: "user_creation")
+          end
+        end
+      end
+      """
+    When I run `rspec spec/requests/users_spec.rb`
+    Then the examples should all pass
+


This isn't a controller / request specific matcher so these feel out of place, why these, why not every aspect of Rails... I'd just cut them.

spec/rspec/rails/matchers/have_reported_error_spec.rb

JonRowe · 2025-06-25T17:25:27Z

👋 I think this would be a great addition, sorry for the size of the review its been on my todo list for a while, its mostly just grammar / wording tweaks plus a few "fit" things.

The one change I do want to see though is dropping instance matching, I don't think it makes sense over just providing class / message / using with.

Commit grammar improvements Co-authored-by: Jon Rowe <[email protected]>

skatkov · 2025-06-27T11:45:47Z

@JonRowe thanks for review.

The one change I do want to see though is dropping instance matching, I don't think it makes sense over just providing class / message / using with.

Which of these you would rather go with?
.have_reported_error(StandardError, /wrong attributes/).with(param1: 'test')

or

.have_reported_error(StandardError).with(message:/wrong attributes/, params1: 'test1')

??

I assume, that first option is prefered to keep similarity with raised_error matcher.
https://rspec.info/features/3-13/rspec-expectations/built-in-matchers/raise-error/

JonRowe · 2025-06-27T11:59:45Z

Given that you already have with, I think:

.have_reported_error(StandardError).with(/wrong attributes/, params1: 'test1')

But if you insisted on message: in with I could be persuaded

pirj · 2025-06-27T14:31:21Z

It makes sense to separate matching of args passed to report and exception’s own args, including the message. Both can receive args, right? It would open a way for ambiguity and confusion.
raise_error accepts have_attributes as an argument matcher. It supports with_message. Wdyt if we reserve with for additional args passed to report, but keep it separate from with_message?

skatkov · 2025-06-29T18:44:41Z

It makes sense to separate matching of args passed to report and exception’s own args, including the message. Both can receive args, right? It would open a way for ambiguity and confusion.

The only argument that the exception class accepts is "message". We can of course subclass it and support way more than that, but I'm not sure if we should support this in a matcher?

I have a feeling, that having a matcher interface similar to raise_error would make it simpler for people to memorize it. I don't mind changing .with to .have_attributes as well.

JonRowe · 2025-06-30T08:02:50Z

I hadn't clocked that the extra attributes were in addition to the error, I think matching the raise_error interface in terms of report_error(ErrorClass, message_or_regexp) makes sense with the existing with becoming something clearer, with_metadata? What does Rails call the extra data?

skatkov · 2025-06-30T09:58:19Z

Here is a source code for Rails.error.report.

https://github.com/rails/rails/blob/36601bbb02bc3570f5609db2127c77afca575d6c/activesupport/lib/active_support/error_reporter.rb#L233C23-L233C112

These attributes in rails case have a name "context". so with_context might be more appropriate.

pirj · 2025-07-02T05:37:44Z

lib/rspec/rails/matchers/have_reported_error.rb

+      # @api private
+      # Sentinel value to distinguish between no argument passed vs explicitly passed nil.
+      # This follows the same pattern as RSpec's raise_error matcher.
+      UndefinedValue = Object.new.freeze


Such a constant is already defined in the basematcher, do we need another one?

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-07-11T18:55:38Z

lib/rspec/rails/matchers/have_reported_error.rb

+        end
+
+        def failure_message
+          if !@error_subscriber.events.empty? && !@attributes.empty?


Cosmetic: any? or present??

pirj · 2025-07-11T18:57:19Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        def failure_message
+          if !@error_subscriber.events.empty? && !@attributes.empty?
+            event_context = @error_subscriber.events.last.attributes[:context]


Won’t it be confusing to use last when several errors were reported?

pirj · 2025-07-11T19:07:30Z

lib/rspec/rails/matchers/have_reported_error.rb

+              return "Expected error message to be '#{@expected_message}', but got: #{reported_errors}"
+            end
+          else
+            if @expected_error && !actual_error.is_a?(@expected_error)


actual_error should be is_a? expected_class, no? Just by looking at the implementation. Or what is the case when they won’t match?

pirj · 2025-07-11T19:22:25Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+          case @expected_message
+          when Regexp
+            error.message&.match(@expected_message)


Cosmetic: is safe nav necessary?

pirj · 2025-07-11T19:27:10Z

spec/rspec/rails/matchers/have_reported_error_spec.rb

+    it "provides details about mismatched attributes" do
+      expect {
+        expect {
+          Rails.error.report(StandardError.new("test"), context: { user_id: 123, context: "actual" })


Nested context key is confusing here. Do you mind to change the inner one to something else?

pirj · 2025-07-11T19:29:24Z

spec/rspec/rails/matchers/have_reported_error_spec.rb

+        expect {
+          Rails.error.report(StandardError.new("test"), context: { user_id: 123, context: "actual" })
+        }.to have_reported_error.with_context(user_id: 456, context: "expected")
+      }.to fail_with(/Expected error attributes to match {user_id: 456, context: "expected"}, but got these mismatches: {user_id: 456, context: "expected"} and actual values are {"user_id" => 123, "context" => "actual"}/)


Can you please fix specs?

Expected error attributes to match {user_id: 456, context: "expected"}, but got these mismatches: {user_id: 456, context: "expected"} and actual values are {"user_id" => 123, "context" => "actual"}/, got #<RSpec::Expectations::ExpectationNotMetError: Expected error attributes to match {:user_id=>456, :context=>"expected"}, but got these mismatches: {:user_id=>456, :context=>"expected"} and actual values are {"user_id"=>123, "context"=>"actual"}

pirj · 2025-07-12T09:56:44Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        private
+
+        def error_matches_expectation?


Is the last condition necessary? We already check if it’s empty? before calling this method.

pirj

A fee minor things and simplifications and it looks good to go.

Let’s leave out matching multiple with chains, severity qualifiers, and all extras for later.

pirj · 2025-07-12T13:03:36Z

lib/rspec/rails/matchers/have_reported_error.rb

+      UndefinedValue = Object.new.freeze
+
+      # @api private
+      class ErrorSubscriber


They have https://github.com/rails/rails/blob/57093484e7926fb33ff266aa25b3c6241b235a68/activesupport/lib/active_support/testing/error_reporter_assertions.rb#L6, can we reuse it?

pirj · 2025-07-12T13:06:35Z

lib/rspec/rails/matchers/have_reported_error.rb

+        end
+
+        def with_context(expected_attributes)
+          @attributes.merge!(expected_attributes)


Cosmetic: Do we need to merge? Maybe just assign?

pirj · 2025-07-12T13:08:34Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        def matches?(block)
+          if block.nil?
+            raise ArgumentError, "this matcher doesn't work with value expectations"


Does this check have the same effect as defining a method supports_value_expectations? that returns false? It should be.

pirj · 2025-07-12T13:10:36Z

lib/rspec/rails/matchers/have_reported_error.rb

+          @error_subscriber = ErrorSubscriber.new
+          ::Rails.error.subscribe(@error_subscriber)
+
+          block.call


This will become something like reports = ErrorCollector.record(&block)

pirj · 2025-07-12T13:16:43Z

lib/rspec/rails/matchers/have_reported_error.rb

+        def error_message_matches?(error)
+          return true if @expected_message.nil?
+
+          case @expected_message


There should be existing examples amongst our matchers of something like value_matches? that would also allow the use of argument matchers like a_string_including("network").

On a second thought, this may make matching more complicated, as if just one argument is passed, and it’s an argument matcher, what should we match it against, the exception itself, or the exception’s message?

This makes it more compelling to build the with_message qualifier. But it then limits the argument to the matcher to only accept the class. But this is what the Rails’ assertion does anyway.

pirj · 2025-07-12T13:17:18Z

lib/rspec/rails/matchers/have_reported_error.rb

+        end
+
+        def actual_error
+          @actual_error ||= matching_event&.error


Is it worth memoizing?

pirj · 2025-07-12T13:18:59Z

lib/rspec/rails/matchers/have_reported_error.rb

+      #
+      # @param expected_error_or_message [Class, String, Regexp, nil] the expected error class, message string, or message pattern
+      # @param expected_message [String, Regexp, nil] the expected error message to match
+      def have_reported_error(expected_error_or_message = UndefinedValue, expected_message = nil)


We have default attributes both here and in the initializer. Worth leaving just here?

pirj · 2025-07-12T13:27:05Z

features/matchers/have_reported_error_matcher.feature

+        end
+
+        def self.process_with_context
+          Rails.error.report(ArgumentError.new("Invalid input"), context: { context: "user_processing", severity: :error })


Nested context here, too. Rename inner to topic: or :section?

pirj · 2025-07-12T13:28:03Z

features/matchers/have_reported_error_matcher.feature

+        end
+
+        def self.process_with_context
+          Rails.error.report(ArgumentError.new("Invalid input"), context: { context: "user_processing", severity: :error })


The severity: here is inside the context:, is this correct?

pirj · 2025-07-12T13:35:26Z

features/matchers/have_reported_error_matcher.feature

+          expect {
+            # Safe operation that doesn't report errors
+            "safe code"
+          }.not_to have_reported_error


I wonder if the negated form should warn if qualifiers were passed to avoid mistakenly letting unmatched reports through:

expect { User.process_custom_error }.not_to have_reported_error("Email is TYPO")

Meaning that if the matcher is used in the negated form, no class, message, or attributes qualifiers are acceptable.

See https://github.com/rspec/rspec/blob/95fbf88255de04e4ba8288172292bda494c4f187/rspec-expectations/lib/rspec/matchers/built_in/raise_error.rb#L162

skatkov force-pushed the have-reported-error branch from 8b837a8 to d2d0e51 Compare June 7, 2025 20:44

pirj reviewed Jun 7, 2025

View reviewed changes

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

skatkov force-pushed the have-reported-error branch 2 times, most recently from b38ff1c to 87ebd2a Compare June 22, 2025 20:50

skatkov added 12 commits June 22, 2025 22:59

first pass at have_reported_error matcher

5e6efcc

Adding a feature

5dd2c5c

simplify matcher/readmer.md

ce1a162

Remove mocks out of tests

550c25e

test failure messages better

e074991

rewrite specs

1fe7e91

should require a block matcher

6b41d2b

Create ErrorEvent struct to store errors

3859b04

simplify matcher

8352668

disable matcher chaining

030c27a

improved argument error message in case block is not passed to matcher

2b9bd02

further clean-up

6135d5d

skatkov force-pushed the have-reported-error branch from 71685ad to 6135d5d Compare June 22, 2025 20:59

pirj reviewed Jun 23, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

JonRowe requested changes Jun 25, 2025

View reviewed changes

skatkov and others added 2 commits June 25, 2025 23:21

Apply suggestions from code review

8335664

Commit grammar improvements Co-authored-by: Jon Rowe <[email protected]>

clean-up features

a14f5c7

skatkov added 6 commits July 1, 2025 23:59

remove a require from test

af58d83

rename .with to .with_context

7259a25

have_reported_error to accept class name and a message

1b6deb6

switch to error_report matcher type of api

c77fca5

refactor

f5e427a

Using UndefinedValue as a default attribute

7b3abc5

skatkov force-pushed the have-reported-error branch 3 times, most recently from c1134da to f90fc3b Compare July 1, 2025 23:20

Look through all errors, don't just go with last error

2488a71

skatkov force-pushed the have-reported-error branch from f90fc3b to 2488a71 Compare July 1, 2025 23:28

pirj reviewed Jul 2, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

warn about nil being passed

c1816e2

pirj reviewed Jul 11, 2025

View reviewed changes

pirj reviewed Jul 12, 2025

View reviewed changes

Uh oh!

Introduce have_reported_error matcher #2849

Are you sure you want to change the base?

Introduce have_reported_error matcher #2849

Conversation

skatkov commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example usage

Uh oh!

pirj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skatkov Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JonRowe commented Jun 25, 2025

Uh oh!

skatkov commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonRowe commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pirj commented Jun 27, 2025

Uh oh!

skatkov commented Jun 29, 2025

Uh oh!

JonRowe commented Jun 30, 2025

Uh oh!

skatkov commented Jun 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pirj left a comment

Choose a reason for hiding this comment

skatkov commented Jun 4, 2025 •

edited

Loading

skatkov Jun 22, 2025 •

edited

Loading

skatkov commented Jun 27, 2025 •

edited

Loading

JonRowe commented Jun 27, 2025 •

edited

Loading

pirj Jul 12, 2025 •

edited

Loading