Rspec-core: Feature request to append values of variables as more context for debugging failures

Created on 9 Apr 2019 · 5Comments · Source: rspec/rspec-core

Subject of the issue

Depending upon how the test is set up, some tests fail only in certain contexts like randomized setup data, time at with it runs, or other variables that make up the context of the test. It can be hard to reverse engineer the specific context under which the test fails.

For such a situation, it would be helpful for debugging purposes. if there was a clean way to append the failure message with a string displaying values of the context variables.

I imagine that one way of handling this is creating a custom matcher with a customized failure message, but it could be a lot niftier to be able to just append to the failure message. Something similar to customized message, but allows for reusing the existing failure message.

This seems like a similar attempt: https://til.hashrocket.com/posts/a50b3d9313-append-an-rspec-failure-message

Your environment

Ruby version:2.6.2
rspec-core version: 3.8.0

Current behavior

1) Colors::awesome_color starts with g
     Failure/Error: expect(subject.awesome_color).to start_with 'g'
       expected "red" to start with "g"
     # ./spec/lib/colors.rb:22:in `block (5 levels) in <top (required)>'

Future behavior

1) Colors::awesome_color starts with g
     Failure/Error: expect(subject.awesome_color).to start_with 'g'
       expected "red" to start with "g"
     Context: 
       Variable @colors was 'red'. Date was '2019-04-10'
     # ./spec/lib/colors.rb:22:in `block (5 levels) in <top (required)>'

Source

gaganawhad

Most helpful comment

Yes, the places you need to look are our formatters (to handle the extra output), you will need to consider how to track the "context" one suggestion from #2616 is to use lets which have been used (checking if they've been memoized and using that value)

JonRowe on 11 Apr 2019

👍2

All 5 comments

:wave: The ability to add a custom failure message already exists:

e.g.

expect(subject.awesome_color).to(
  start_with('g'),
  "Expected colour to start with 'g' was #{subject.awesome_color}\n" \
  "Context: @colours #{@colours}, Date #{@date}"
)

I think I'd support a PR where you could set into the example what certain data was, after all we have example metadata which could have a context included... And then it would be a matter of modifying our formatters to support it...

JonRowe on 10 Apr 2019

Thank you for getting back @JonRowe and for your work on open source!

Yes, I can use that style for now! I would have to do that for several tests. It would nice to support this on context / describe as you are suggesting, to DRY things up and it would be nice to reuse the existing failure message. :-)

I'll keep in mind to consider submitting a PR. I am not super acquainted with the rspec codebase, but if I were to attempt submitting a PR would you or someone else be available to roughly point me in the right direction and discuss style? In the past, I have gotten lost in some other code bases, but I do imagine that the rspec is going to be a lot cleaner and easier to reason with.

Thanks again!

gaganawhad on 10 Apr 2019

JonRowe on 11 Apr 2019

👍2

Would something like this be sufficient for you @gaganawhad ?

  after do |example|
    instance_variables.reject { |v| v =~ /@__/ }.each { |v| puts "#{v}: #{instance_variable_get(v)}" }
  end

It would be possible to include using metadata, i.e.:

  after(:dump_ivars) do |example|
    instance_variables.reject { |v| v =~ /@__/ }.each { |v| puts "#{v}: #{instance_variable_get(v)}" }
  end

  describe 'something', :dump_ivars do
  ...
  end

Unfortunately, it's not yet possible to understand if an example has failed at that stage yet, since an expectation can be set in the after hook, and may change it.

pirj on 10 Jan 2020

I just want aggregate_failures, but with it being active even if there's only a single failure.

it "parses all entries correctly" do
  result = Parser.call(bar_owner)
  result.each do |entry|
    aggregate_failures("Entry #{entry.id}") do
      expect(entry.name).to match(/\A[A-z][a-z]+\z/)
      expect(entry.owner).to eq(bar_owner)
      expect(entry.tags).to be_present
    end
  end
end

If you get two errors in there you get a nice context (Entry 76: …), but when you fix one of them the context is gone again and you are back on an error that is very hard to understand without adding extra print statements in the loop and re-running again.

How about a with_failure_context(extra_message) { … } method that acts and works similar to aggregate_failures?