Rspec-rails: Cannot kill Rspec when running feature tests.

I tried with:

  %w[rspec-core rspec-expectations rspec-mocks rspec-rails rspec-support].each do |lib|
    gem lib, :git => "https://github.com/rspec/#{lib}.git", :branch => 'master'
  end

But same behaviour.

Hmm I'm not sure theres much we can do if Puma is intercepting Ctrl-C, did you try Ctrl-C twice?

Yes, I tried Ctrl-C multiple times but it has no effect. I will raise the issue in the Puma Repo too.

Can you try configuring puma in non cluster mode, you need to set workers to 0 and threads to 0,1 see conversation on #1813

@JonRowe I configured puma.rb to be like this:

threads 0, 1
port        ENV.fetch("PORT") { 3000 }
environment ENV.fetch("RAILS_ENV") { "development" }
workers 0
plugin :tmp_restart

Then I run the tests:

rspec spec/features
Run options:
  include {:focus=>true}
  exclude {:slow=>true}

All examples were filtered out; ignoring {:focus=>true}
Puma starting in single mode...                                                                                                                           |  ETA: ??:??:??
* Version 3.10.0 (ruby 2.4.2-p198), codename: Russell's Teapot
* Min threads: 0, max threads: 4
* Environment: test
* Listening on tcp://127.0.0.1:62430
Use Ctrl-C to stop
^C- Gracefully stopping, waiting for requests to finish
=== puma shutdown: 2017-10-25 08:42:29 +0200 ===
- Goodbye!

Puma was shutdown but rspec was not. I had to close the tab.

Same here. Reverting to 3.6.0 fixes it. We're not using system tests although puma is a dependency in our gem.

rspec-rails 3.7 which introduced system tests seems to cause this because it requires actiondispatch/system_test_case early which starts puma when required through ActionDispatch::SystemTesting::Server using Rack::Handler::Puma which uses Puma::Launcher which hijacks signals, unavoidably. You don't even have to be using system tests, you might only be running a unit test and it still boots puma and the app.

It seems like action_dispatch/system_test_case should not be required by rspec-rails until a SystemExampleGroup is actually started, or at least defined.

Upstream, they should probably be using Puma::Server directly or something.

TBH this seems like a problem that should be handled in Rails upstream, if people are experiencing it here chances are it exists as a problem in Rails too.

I agree, although it's not as much of a problem upstream with minitest unless you require action_dispatch/system_test_case, usually by defining a system test case (lol autoloading). The problem is that rspec-rails has unavoidably required action_dispatch/system_test_case.

Would you accept a patch which delays loading the system test case, avoiding the boot?

The problem is certain, but not all configurations of puma don't shut down correctly, and Rails system tests are configuring it to boot.

Hmm. Yeah, I think puma config should be done better upstream. For instance, puma sets "RACK_ENV" to "development" automatically, which also broke our tests when we simply bumped rspec-rails. But that's a separate issue.

I don't think running a unit test should boot puma or my app, or stomp on my capybara settings. Simply bumping rspec-rails has done all of these things.

I would accept a PR improving the system test setup, but it doesn't "close" this issue.

It would fix this issue when folks are not using SystemExampleGroup (say just using feature specs), but not if you want to use the new system specs — that's going to need an upstream fix. :-(

It won't fix this issue for those people who have the "bad"* configuration who accidentally boot their server in process.

System tests do work correctly in "normal"* configurations.

*by the definitions of whatever Rails is doing :)

Not loading action_dispatch/system_test_case would fix folks in a bad/old/legacy configuration. It won't fix folks in the normal/I'm-using-system-tests-with-rspec configuration — they still won't have Ctrl+C.

Some will, some won't, it depends on the puma config.

Not loading action_dispatch/system_test_case means the puma config won't be touched/loaded/anything.

Agreed. My point is that loading action_dispatch/system_test_case surfaces the issue with puma config, hence normal configuration doesn't have this issue, it's only bad configuration and requiring this that does it. Hence this issue wouldn't be fixed, as its the puma config when loaded that causes this behaviour, and delaying loading action_dispatch/system_test_case merely hides it from people with bad config.

I am totally ok with delaying the loading, just documenting the actual issue for prosperity in this.

Ah yes, I see we're aggressively agreeing 😆

Just a note: You can do a Capybara.server = :webrick and the puma problem goes away. That is what I am doing in the mean time until this is fixed.

Could someone suffering this issue paste their puma config here? I was unable to replicate with a plain rails app, also give #1903 a spin as it delays our system test integration until its used which should help.

@JonRowe Full demo: https://github.com/Nerian/sample-capybara-rspec-issue

Yep that replicates this, and leads me to believe the issue is puma is installing its own signal handler atop of ours, this isn't really a resolvable problem on our end, puma needs to be loaded by rails in a way that doesn't stomp on our interrupt handler. Can you try the new version on master which delays loading system tests thus hopefully not triggering puma at all? The issue will still remain for system tests tho

@JonRowe

I updated my sample and now it uses master:

https://github.com/Nerian/sample-capybara-rspec-issue

This is the output that I get:
https://gist.github.com/Nerian/93716d084bb4a9cfcce0a75f98263277

I let a couple of tests pass, then I hit ctrl-c. The Puma server is shutdown but the rspec process continue to run the tests – and fail because the server is no moro.

Yes your sample demonstrates the issue, its just theres nothing we can do on rspec's side.

@JonRowe, is there anything we can do to resolve this issue? Like opening an issue with Rails? However I feel like I know too little about the problem to write a meaningful issue.

I read a lot about the "bad" and "normal" configurations here, but couldn't figure out what exactly is meant by that, nor examples of a good or bad config. Could you maybe provide one or point me into the right direction?

Would be great if we could resolve this.

Ah, I just found https://github.com/rails/rails/pull/30638 https://github.com/rails/rails/pull/30712, maybe it is resolved in Rails 5.1.5 and 5.2. I'll try 5-1-stable and report back this week or next.

Any news @aried3r? I tried picking up those PR's but I don't see any difference, impossible to interrupt system tests. :thinking:

Same problem here with Rails 5.1.4 and Rspec 3.7.0

@deivid-rodriguez I'm still having this issue in Rails 5.1.5.

I get this issue intermitently, although I haven't seen it in a while (since my last comment). I'm hopefull it'll be fixed in Rails 5.2.

This is due to Rails 5.1 and Capybara launching puma via Rack::Handler::Puma.run which installs signal traps - The fix for Capybara is https://github.com/teamcapybara/capybara/pull/1992

I experienced this issue with thin. Here's the PR I opened with the Capybara team: https://github.com/teamcapybara/capybara/pull/2022

I just took the example code from Nerian's comment and this problem no longer happens.

Based on ignisf's comment this issue in capybara's repo fixed it.

A bundle install on the test case code installed Puma 3.10.0. The issue referenced by deivid-rodriguez indicates it was fixed in Puma 3.6.0.

Magnificent