I'm trying to write tests for before hooks and have run into trouble. When a before(:all)
hook is skipped in an isolated example group it sets the outer example group as pending, but it should pass.
describe do
it 'should pass' do
RSpec.describe do
before(:all) { skip }
it('should be skipped') { }
end.run
end
end
.*
Pending: (Failures listed here are expected and do not affect your suite's status)
1) should be skipped
# No reason given
# ./spec.rb:7
Finished in 0.0013 seconds (files took 0.07034 seconds to load)
2 examples, 0 failures, 1 pending
F*
Pending: (Failures listed here are expected and do not affect your suite's status)
1) should be skipped
# No reason given
# ./spec.rb:7
Failures:
1) should pass FIXED
Expected pending 'No reason given' to fail. No error was raised.
# ./spec.rb:4
Finished in 0.00124 seconds (files took 0.06776 seconds to load)
2 examples, 1 failure, 1 pending
Failed examples:
rspec ./spec.rb:4 # should pass
Strangely, it's fine if the test has multiple isolated example groups:
describe do
it 'should pass' do
RSpec.describe do
it('passes') { }
end.run
RSpec.describe do
before(:all) { skip }
it('should be skipped') { }
end.run
end
end
..*
Pending: (Failures listed here are expected and do not affect your suite's status)
1) should be skipped
# No reason given
# ./spec.rb:11
Finished in 0.00145 seconds (files took 0.0682 seconds to load)
3 examples, 0 failures, 1 pending
:wave: If you have a look at the rspec-core specs you will see all sorts of protection and isolation (sand boxing) to allow us to run specs within rspec, its not a "public api" intended to be used in the way you are using it. You will probably need to replicate what we've done to ensure the "inner rspec" works from the outer rspec, in both your examples you should see only 1 spec output for example...
May I ask why you are trying to test before hooks? Is this report psuedo coded from something else?
@JonRowe We're trying to add some new features on top of RSpec, and would like to write tests for them. You can find more context at: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24942#note_140637060
Yeah, this is a minimal reproduction of something more complex, as @godfat noted.
Essentially, we're trying to allow tests to be skipped unless focused (e.g., as if running rspec with --tag skip
would actually run the skipped tests). We did that via configuration hooks that would skip examples if they had user-defined metadata :quarantine
, unless :quarantine
was focused, in which case it wouldn't skip the tests.
That worked, but when I tried to write unit tests of the configuration I ran into this issue.
Ok so as a minimum you need to setup a reporter of your own, and pass it to run
so that the status of the internal groups don't expand into the global. You then need to check the groups pass / fail as desired.
Then you're going to want RSpec::Core::Sandbox.sandboxed
and wrap that around every one of the outer examples.
You can see examples of this in our spec/spec_helper.rb and spec/support/sandboxing.rb.
Ahh, thanks @JonRowe! That explains a few things about the RSpec unit tests that I wasn't understanding, like how tests of configuration didn't affect other tests.
With that I managed to get a simple test passing. It should be enough to fix the rest of out tests, so I'll close this.
Most helpful comment
Ahh, thanks @JonRowe! That explains a few things about the RSpec unit tests that I wasn't understanding, like how tests of configuration didn't affect other tests.
With that I managed to get a simple test passing. It should be enough to fix the rest of out tests, so I'll close this.