Rspec-core: Add config.around(:all) // current behaviour is confusing, and the same as around(:each)

The big thing against the :all hooks is that shared state across multiple examples can have unintended consequences (we've seen this with usage of let and other RSpec internals), we try to tear down and setup completely clean environments for examples to run in and usage of before(:all) et al complicates this.

Your suggested use case does make some sense, but you would be able to manipulate state in your DB between examples, this not usually desired behaviour, and could open you up to performance problems in the database driver too, (holding open a connection and holding other variables in memory over a large amount of examples could have side effects and performance degradations too.)

Personally I'm not in favour of adding this hook as a matter of "good practice" but I'm not against it if the other core team like the idea... /cc @samphippen @myronmarston @soulcutter

I'd like to see this added (understanding it will need the same kids of caveats documented as before-all does already).
We actually already use this patch: http://myronmars.to/n/dev-blog/2012/03/building-an-around-hook-using-fibers
to add an around(:all) hook, and then use it (with specs for a Rails app) to wrap all (nested) examples in (nested) database transactions. It does require using a slightly different setup when you want to use before-all state (something like a before_all_let!() which refetches the clean object for each example from the transactionally-cleaned database), but it doesn't break regular before/let behaviors when you don't need that.

Great article! Thanks for sharing, @seanwalbran :)

@seanwalbran -- thanks for pointing that out...I was going to point it out as well :).

@rosenfeld -- does the microgem I published work for your use case? I'm open to considering adding this to core but it's such a rare need (given that you're the first to ever ask for it, and I've never wanted or needed an around(:all) hook) I have a preference for keeping it in external gem if we can do so w/o hooking into rspec's internals...and currently that microgem doesn't :). It does require that you're on a ruby interpreter that supports fibers, though.

After reading your microgem for the first time it wasn't clear to me if it would support config.around(:all). Does it simply work?

with regards to fiber support this is a non-issue for all Ruby applications I maintain...

Also if such a gem can be made in a future-proof safe way, I don't actually need it to be integrated in rspec-core...

After reading your microgem for the first time it wasn't clear to me if it would support config.around(:all). Does it simply work?

Hmm, it doesn't directly support config.around(:all), but I think it would work if you included RSpecAroundAll in RSpec::Core::Configuration.

Also if such a gem can be made in a future-proof safe way, I don't actually need it to be integrated in rspec-core...

I consider it already future proof: it only relies on public RSpec APIs (before(:all), after(:all) and RSpec::Configuration#extend).

I have to leave to take my daughter to the doctor for a regular exam and will try your solution once I'm back. Thanks, @myronmarston.

Actually, I just realized that including RSpecAroundAll in RSpec::Core::Configuration won't work because it won't override around like it needs to. If you're on ruby 2.0, you can use prepend, though, or you could do something like this:

extend_method = Object.instance_method(:extend)
extend_method.bind(RSpec.configuration).call(RSpecAroundAll)

(We can't simply use RSpec.configuration.extend(RSpecAroundAll) because RSpec::Core::Configuration overrides extend so that when you use it, the module will be extended onto example groups).

I'm back and gave it a try:

require 'rspec/core'
require 'rspec_around_all'
RSpec::Core::Configuration.send :prepend, RSpecAroundAll

RSpec.configure do |config|
  config.around(:each) {|e| DB.transaction(savepoint: true, rollback: :always) { e.run } }
  config.around(:all) {|g| DB.transaction(savepoint: true, rollback: :always){ g.run_examples } }
end

I made the run_all_in_transaction a no-op and run the suite again twice. Everything seems to be working. I just had to change the snippet in your gem description to tell it not to require RSpecAroundAll when loading Bundler:

group :test, :development do
  gem 'rspec-rails', '~> 2.14'
  gem 'rspec_around_all', git: 'git://gist.github.com/2005175.git', require: false # the require: false was required
end

Would you consider this safe/future proof? If so, would you mind to release this gem as a regular repository so that we can create issues on it and navigate on the project as usual? After that I'll happily create a PR to fix the instructions for usage in Gemfile if you want, as well as another PR to include documentation on how to add a config.before(:all) under Ruby 2.

Thank you for pointing me out how to achieve that :)

Would you consider this safe/future proof?

Yep. It relies only on public RSpec APIs (and ones that have been public for many years and are unlikely to ever change).

If so, would you mind to release this gem as a regular repository so that we can create issues on it and navigate on the project as usual?

As it happens, @seanwalbran kindly offered to take over maintenance and now has it on github and rubygems:

http://rubygems.org/gems/rspec_around_all
https://github.com/seanwalbran/rspec_around_all

After that I'll happily create a PR to fix the instructions for usage in Gemfile if you want, as well as another PR to include documentation on how to add a config.before(:all) under Ruby 2.

It would probably be good for the gem to take care of making config.before(:all) work out of the box so users don't have to mess with that stuff. I'l leave that to you and @seanwalbran to work out, though :).

Thank you very much, @myronmarston!

While exploring this technique, I noticed the group.getobj is an instance of RSpec::Core::Configuration, which means we can't use group.metadata for instance inside config.around(:each). Is there an easy work-around?

@rosenfeld -- you can look into assigning the group from the before(:all) hook so that it can work in both cases.

That makes sense, but I'm thinking more about this and this is not exactly what I want.

Currently, this config.before(:all) will wrap only the top describe blocks before/after(:all), right? But actually I want to wrap all before/after(:all) (specially before(:all)) from all inner contexts as well.

Any ideas on how to get this to work?

Currently, this config.before(:all) will wrap only the top describe blocks before/after(:all), right? But actually I want to wrap all before/after(:all) (specially before(:all)) from all inner contexts as well.

I thought this would add it to all example groups, not just top-level ones. Can you paste an example showing what you are seeing and how you want it to work different?

I was trying to write a spec for it in that gem. I wanted to enable some behavior depending on the describe block metadata in the config.around(:all) block since I was not sure how to test this...

Then I found another bug in this gem. Try to run the test suite with "rspec -e hook" and the examples will fail.

I'll publish my attempt to write some tests in a few minutes to a branch of my fork so that you can take a look at what I'm talking about.

Just by adding this to the start of the specs for that gem will also show an error even though all specs still pass:

RSpec.configuration.around(:all){|g| g.run_examples }

I added support by using this block in the end of the implementation:

RSpec.configure do |c|
  c.extend RSpecAroundAll

  # Add config.around(:all):
  # c.extend overrides the original Object#extend method.
  # See discussion in https://github.com/rspec/rspec-core/issues/1031#issuecomment-22264638
  Object.instance_method(:extend).bind(c).call RSpecAroundAll
end

> rspec
.....
An error occurred in an after(:all) hook.
  FiberError: dead fiber called
  occurred at /home/rodrigo/src-externo/rspec_around_all/lib/rspec_around_all.rb:26:in `resume'



Finished in 0.00513 seconds
5 examples, 0 failures

Also, I think I didn't understand your comment

you can look into assigning the group from the before(:all) hook so that it can work in both cases.

I tried to use the before(:all){|group|...} approach and using it instead of the local group = self and the examples failed. What did you mean exactly?

The problem is that the after(:all) is trying to resume a different fiber (the last created one) instead of that created in the config.around(:all) block after the last example.

This worked for my application because DB.transaction(rollback: :always){} will rescue from any errors and rollback the transactions so I don't really need to run the last after(:all) successfully...

Hmm...that's what I had in mind, but I admittedly haven't looked at that code in over a year.

Anyhow, do you want to report issues against the repo @seanwalbran set up and work with him to address these issues? I'm happy to help advise if there's things you two can't figure out, but it would be good to take the discussion there now that there's a github repo and a maintainer for it :).

ok, thanks, I'll try that tomorrow since my wife is already asking for my help with the baby :)

I just realized after a good sleep that config.around(:all) is not exactly what I want. While before(:each) blocks will be nested, before(:all) won't, which makes sense actually. So, what I need is actually something like a around(:all_nested) which will be triggered once for each context, nested ones included.

But I think I can get it working in that gem. Will start working on it now.

before(:all) are nestable, but you're correct they're not invoked again.

@myronmarston, I believe I got it working, although it's not thread-safe (in case someone is running the examples in parallel, since I'm using Array#<< and #pop for creating a fibers' stack):

https://github.com/seanwalbran/rspec_around_all/pull/2

I'm using fiber stacks instead of the outer local fiber variable because I don't understand why the last created one is being resumed after the last example when it should be the fiber created in the config.around(:all) block...

Please comment there if you think it can be improved.

Would love to see @myronmarston's gem incorporated into core in the future.

Very helpful for drying this:

  config.before(:all, dirty_inside: true) do
    DatabaseCleaner.start
  end

  config.after(:all, dirty_inside: true) do
    DatabaseCleaner.clean
  end

into this:

config.around(:all, dirty_inside: true) do |example|
  DatabaseCleaner.cleaning do
    example.run
  end
end

I think it's fine to let a 3rd party gem handle this, it's an edge case and sorry but I don't want to increase our maintenance burden to save you 2 lines of code :P

Guys, I'm sorry to revive an old discussion, and if there's a new one, point me to it please.
Now I'm puzzled by the apparently biggest obstacle to implementation of this feature: possible misuse. I love ruby community, but sometimes saving the dummies' asses goes a bit too far.

Now I'm working on a video processing gem and I need to generate sample media files — which are quite large for repository storage (and not so fast to produce). The straightforward idea — generate them in an around(:all) block with Tempfiles — met this discussion. before(:each) - check, after(:each) - check, around(:each) - check, before(:all) - check, after(:all) - check, around(:all) - ..not even an error?? Why?

I have an alternative idea, to avoid misuse of anything: let's leave the rspec-core functional and spin off rspec-for-dummies which is safe to use for dummies. Oh, snap, the dummies might suspect something. Another idea then: let's leave rspec-core for dummies and spin off a normal gem: rspec-core-advanced, for people that kinda know what they're doing.

@costa TLDR, your desired effect can be achieved with before(:context) and after(:context).

Longer response, we routinely choose not to add or expand features we think are a bad idea, or simply that aren't in popular demand, not because we're "saving the dummies' asses" but because adding new features creates a maintenance burden upon ourselves which cannot easily be undone. Once a version of RSpec supports something its there until a next major version which could be a long time away, we have several features already that we don't recommend extensive use of (expect_any_instance_of for example, we'd recommend not using it but we know there is popular demand for it so we maintain despite the extra burden it causes) so we're understandably not keen to increase that number.

Additionally I encourage you to read https://github.com/rspec/rspec-core/blob/master/code_of_conduct.md before commenting again, as personally I found your tone to be more than a little aggressive, please remember everyone on the RSpec team has the best interests of the community at heart and are all volunteers. Thanks.

@JonRowe First, I sincerely apologise if anyone might have found my tone inappropriate, it meant to be ironic at best, but apparently I must stick to the formal language — even in the Ruby community nowadays.

That aside, I'm not convinced by your reply, unfortunately. Can you please explain any extra technical difficulties having (and maintaining) around(:all) hook in the presence of before(:all) and after(:all) hooks. It really looks like a few lines of code — https://github.com/seanwalbran/rspec_around_all/blob/master/lib/rspec_around_all.rb — which complete the DSL and make up for those 0.1% of the cases like mine.
Plus, I also think that micro-gems are not always the answer — they might simply be not that efficient in terms of the total community's effort/benefit.

Cheers.

but apparently I must stick to the formal language — even in the Ruby community nowadays.

Not at all, just remember MINSWAN.

That aside, I'm not convinced by your reply, unfortunately. Can you please explain any extra technical difficulties having (and maintaining) around(:all) hook in the presence of before(:all) and after(:all) hooks. It really looks like a few lines of code — https://github.com/seanwalbran/rspec_around_all/blob/master/lib/rspec_around_all.rb — which complete the DSL and make up for those 0.1% of the cases like mine.

If you were to create a PR implementing around(:context) successfully and we were ok with the additional complexity we would look at merging it. Currently we're not interested in doing it ourselves and would rather focus our efforts elsewhere.

Plus, I also think that micro-gems are not always the answer — they might simply be not that efficient in terms of the total community's effort/benefit.

Maybe not, but the total community has more combined effort than the few RSpec maintainers, the more we offload to other gems the more we can focus on core features. This is one of the reasons various parts of RSpec were spun out in RSpec 3 (e.g. rspec-its), similarly we merged in a gem (rspec-fire) we thought would be useful but we also brought it's maintainer onto the core team to help us with this process.

FWIW, I've been thinking of looking into implementing this in rspec-core, primarily to make the API more consistent (e.g. so that you can combine any scope -- example/context/suite -- with any hook type before/after/around). The hook implementation was fairy messy when this issue was last discussed but I completely refactored/rewrote their implementation for 3.2 to support some other features and I think it should be considerably easier now. So I'm planning to look into this in the near future.

FWIW I looked at it at the time of this original issue and it wasn't as trivial as I expected, hopefully you'll have a better time given the refactoring.

@myronmarston Are you still thinking of adding around(:all) to rspec-core in order to be able to create invariant test data inside a database transaction and speed up specs?

If not, I could work on https://github.com/seanwalbran/rspec_around_all next year

Thanks.

It's still on my radar but havent had the time lately.

Hi @JonRowe and @myronmarston I am going to work on this around(:all) feature. I'll setup a project on github and post my work in progress here. Shall I ping the mailing list to ask for feedback / support?

Sure, or you can just mention us on your github

In the meanwhile, my :baby: was born so I am not going to get back to this issue before a while :)

There is a use-case where this would also be helpful: running benchmarks with Benchmark#ips which requires a block. It's a nice option because the benchmarks build on top of Rack::Test and other layers which helps to make things very simple.

@ioquatix you can run around(:example) for that, anything else wouldn't be particularly useful as you wouldn't be able to exclude RSpecs own run time, clean up etc

@JonRowe Actually, I want to run a limited set of specs and use benchmark.compare!. If you just run it around individual examples you loose the ability to compare them.

@ioquatix that wouldn't be a very reliable comparison as it would take into account our internals and potentially even booting things like rails

@JonRowe well, given that I'm not using Rails, and that all the setup happens outside the benchmark code, I'm not really sure your assertions are correct/logical.

RSpec.describe "Utopia Performance" do
    include_context "rack app", "benchmark_spec/config.ru"

    before(:all) do
        @fiber = Fiber.new do
            Benchmark.ips do |benchmark|
                @benchmark = benchmark

                Fiber.yield

                benchmark.compare!
            end
        end

        @fiber.resume
    end

    after(:all) do
        @fiber.resume
    end

    it "should be fast to access basic page" do
        @benchmark.report("/welcome/index") do |i|
            i.times { get "/welcome/index" }
        end
    end

    it "should be fast to invoke a controller" do
        header "Accept", "application/json"

        @benchmark.report("/api/fetch") do |i|
            i.times { get "/api/fetch" }
        end
    end
end

This is what I ended up with, it works quite okay for now. I may actually add some expectations to validate the benchmark is working correctly before calling report. The other option is to have a single spec that does all the benchmarks but then I can't selectively disable them..

Perhaps it would be nice to have a higher level integrated benchmark, I believe minitest has something like that which can validate constant, linear and quadratic performance.

Your version above doesn't have the issues our implementation would as you have more control over where you're running things and the context :)

We measure group and example run time when --profile is used but we don't really have access to anything else (as we don't use external dependencies), comparison between these things would be possible in a custom formatter

What is wrong with:

    around(:all) do
        Benchmark.ips do |benchmark|
            @benchmark = benchmark
             yield
        end
    end

Nothing particularly with your use case, (as you wouldn't be measuring our internals in this instance) but it doesn't exist currently and none of the core team have been particularly interested in implementing it. It's also not a particularly easy change from memory (I briefly attempted to do it but I didn't and still don't have the time to dedicate to do it properly).

@JonRowe That's cool, I get it, it's unpaid open source work :) I just wanted to mention there was, IMHO, a valid use case for this. It helps add to the validity of the ticket and the design of the feature.

At least, the Benchmark::IPS library could be fixed to avoid the need of a block, which might actually not be a bad idea.

FWIW, I've changed my thinking on this a bit. I think RSpec should provide around(:context)/around(:all). Not because of any particular use case, but simply for API consistency. It's much simpler to tell users "there are 3 kinds of hooks (before, after and around) and each can be used with any of 3 scopes (example, context and suite)". Having some kinds of hooks work with only some kinds of scopes makes the API inconsistent and forces us to add special case code to emit warnings and also write extra documentation for this fact.

So at some point I hope to see RSpec gain this feature. At the moment my open source time is much more limited than it used to be so I haven't gotten around to it yet.

@ioquatix for benchmarking I don't understand the point of putting it in a spec. What does that gain you over using benchmark-ips the normal way?

@myronmarston

1/ It fits into existing spec based testing infrastructure nicely, including running on travis, code coverage using SimpleCov, switching between generating a profile (RubyProf), a benchmark (Benchmark::IPS) or normal test run.

2/ Some of my benchmarks do have expect clauses to validate that things are working before invoking the benchmark.

rspec core 3.8.0. There is still no around(:context) / around(:all) in rspec, true?

It's a bit confusing, because you can write around(:all) and get no error (perhaps guessing it existed parallel with around(:each) -- but I _believe_ it doesn't do anything, it does the same thing as around or around(:each).

Confirm?

I would say that the current behaviour is a bug:

an example file:

require 'spec_helper'

RSpec.describe '' do
  it 'works' do
    p 'example'
  end
  it "bees" do
    p 'exampe2'
  end
end

with spec helper:

# This file was generated by the `rspec --init` command. Conventionally, all
# specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`.
# The generated `.rspec` file contains `--require spec_helper` which will cause
# this file to always be loaded, without a need to explicitly require it in any
# files.
#
# Given that it is always loaded, you are encouraged to keep this file as
# light-weight as possible. Requiring heavyweight dependencies from this file
# will add to the boot time of your test suite on EVERY test run, even for an
# individual file that may not need all of that loaded. Instead, consider making
# a separate helper file that requires the additional dependencies and performs
# the additional setup, and require it from the spec files that actually need
# it.
#
# See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration
RSpec.configure do |config|
  config.around(:all) { |ex|
    p 'hi'
    ex.run
    p 'done'
  }
end

prints hi and done as if it was an around(:each) hook, which is about as confusing a behaviour as possible:

"hi"
"example"
"done"
  works
"hi"
"exampe2"
"done"
  bees

Finished in 0.00123 seconds (files took 0.07965 seconds to load)
2 examples, 0 failures

@JonRowe I think we should either make around(:all) show the user an error, or behave more like before(:all). WDYT?

Am I correct to think this applies to to around in a describe block inside a spec file too? I'm not sure if config.around and that kind of around are the same code path -- but the behavior is the same, around(:all) in a describe block in a spec file also seems to do the same thing as around(:each).

@samphippen I think it should how the user an error, as it has never been supported and IMHO it is confusing to make it behave like a before(:all) given thats not the intent.