Sinon: fakeServer is broken with .respond(500) on 1.17.4

I can confirm that this happens on 1.17.4 but not 1.17.3. I have a similar setup with karma-sinon.

My guess is the problem is in this commit https://github.com/sinonjs/sinon/commit/2cfbacd5cea5b63c014076d3a65b6642b2200793

The tag 1.17.4 was pushed to npm registry but I didn't found any trace of this tag on this repository. What happened?

My guess is the tag just hasn't be created yet. It was only released 3 hours ago

@mbarlock Yeah, probably — nonetheless I think would be better if the tag on GitHub was released first. At least we would take a look and help on a PR or something to fix.

My bad. Forgot git push --tags. Thanks for the info on the bug.

I've just confirmed commit 2cfbacd definitely caused the tests to fail for us in mozilla/loop#400.

I applyed the patch from #1031 locally, and it fixes our tests.

If 1.17.4 changes existing behavior, shouldn't it be part of a new major release? Currently, it's considered compatible with 1.17.3, so a package.json specifying a sinon dependency as "^1.17.3" will get 1.17.4 and potentially fail tests that used to work.

The way 1.17.3 works is a regression though, right? Feel free to correct me. If that is the case, it should be fixed and not maintained in its broken state.

UPDATE: This looks like an actual bug. See below.

Oh, I hadn't read your original discussion on https://github.com/sinonjs/sinon/pull/861 @fearphage . It does seem like this is the correct behavior, though I think it'll have more impact than intended, given how long the bug has been in the Sinon codebase for.

Given that, it looks like the right thing to do on my end is to change my tests to rely on xhr.onabort instead of xhr.onerror. I suspect this change will cause confusion for a while though, since locally run unit tests don't automatically re-download all dependencies if they're present in a node_modules directory, so people will find out about this when they add new dependencies to their package.json (I realized quickly because Travis CI does an npm install from scratch for my tests).

I'm not sure what the right course of action is. Maybe add a note to the changelog for 1.17.4 specifying that some tests that rely on "onerror" should use "onabort"? (btw, as of this comment, http://sinonjs.org/Changelog.txt does not include 1.17.4 yet).

I take that back. I looked into fixing my tests and realized that a new bug was introduced in https://github.com/sinonjs/sinon/commit/2cfbacd5cea5b63c014076d3a65b6642b2200793 . The onReadyStateChange function now triggers an "error" ProgressEvent instead of relying on the abort function to call onerror directly if defined. The problem is that FakeXMLHttpRequest currently does not have a listener for the "error" event.

I've created PR https://github.com/sinonjs/sinon/pull/1042 to add the missing error eventListener key. Without this change, any unit tests that check for an onerror handler function being called on a legitimate server error response (like 500) will fail. Let me know your thoughts, @fearphage @fatso83

On my initial read of this thread I didn't see https://github.com/sinonjs/sinon/pull/1041 , which adds the missing eventListener key and extra code to assert the order of events. Feel free to disregard my PR if you go with that instead.

OK, #1041 (same as #1042) has now been merged in.

@overcaffeinated Regarding the missing changelog that is due to a failure in the build script for the docs that prevented me from updating them (see https://github.com/sinonjs/sinon/issues/991#issuecomment-216651347). Sorry about that. Hopefully we can get it working again and add a note on the change there.

Stuff like this makes me really want to get 2.0 out the door so that we don't invest too much energy in the 1.x branch.

1.7.4 caused several of my tests to fail. Do we expect 1.7.5 to resolve this?

Unfortunately it won't fix it for everyone, as #1031 is not fixed yet. I'll try and help that along this evening.

Thanks for the info, @Standard8 !

I fixed (broke?) this based on my interpretation of the XHR specification. It would be nice if there was a browser implementation to compare this to to make sure it's right this time. We need a testbed to compare sinon's fake xhr implementation to reality to make sure the events fire in the correct order and the correct events are fired.

Any takers?

@fearphage any idea how such a testbed would look? a suite of manual tests? kind of hard to simulate "network down" in normal browsers. so I am not really sure how this should be done.

I imagine it would be like browserscope.org or at least it could use that as a backend to store the results.

Imagine http://www.acidtests.org/ for XHR though

Here are some request-based utilities to help:

https://httpbin.org/
http://requestb.in/

That seems like excellent resources! Browserscope is down, btw.

@fearphage I didn't have time to work out how to get something into www.browserscope.org, so I whipped up a web page instead:

http://people.mozilla.org/~mbanner2/sinonXHRBrowserTest.html

Loads in latest versions of Firefox, Chrome, IE and Safari.

@Standard8 I appreciate it. I've been using this to compare sinon's fake server implementation. Thank you.

@fearphage @fatso83, could you please provide an overview of the current status of this issue?

From a quick skim through it sounds like we should consider reverting the v1.17.3 functionality and providing a v1.17.5 release - even tho the old functionality was broken, this represents a breaking API change and should therefore be rolled into the 2.0 release?

I think you summed it up nicely, although @fearphage is more into the details on this one. I keep getting overwhelmed by trying to hold in my head what is in the v1.17 branch and what is in master. I _think_ most problems have been fixed in master, but I don't think that holds for the v1.17 branch (which does not have fixes such as #1031 and #1041 AFAIK). I might be (probably am) mistaken.

I think the most pragmatic solution might be to do as you said:

• revert #1017 (and related?) for the 1.17 branch to lower the noise and keep the network API the same, ship a 1.17.5 release, and just accept that the implementation is less correct than in version 2
• keep the changes in master and just document what has changed in the migration guide (see #1090).

I'm just catching up to what's going on here. Would it be better to fix than revert?

The biggest change is when error/onerror is fired or not, right?

@fearphage Thanks for chiming in! Since I am rather slow, could you please make a five line summary of what API changes would be apparent from the view of an existing end user when all fixes have been applied to the v1.17 branch?

While big changes would be ok for a new major version, anything that starts breaking a lot of tests in 1.x should probably be on hold, but I just wasn't sure if those fixes that have been applied to master would settle _most_ issues people are having with 1.17.4.

As far as I understand it, the only thing broken is non-200 requests should still trigger error events. Is there more to this? I believe all it needs is the fixes from master.

It seems like a good idea to pull v1.7.4 from Github and NPM too. Most people are reverting it or can't upgrade to it.

If that's all that has changed I would suggest just including the fixes that have gone into master and ship a 1.17.5 release ASAP. Could you do it? I am going away for holiday (Date.now()).

As removing NPM versions are not considered very "nice", I issued a npm deprecate command for 1.17.4. Not sure about removing the tag either, if some people are relying on it. That's another discussion, though, and I suggest we focus on shipping the fix.

Sounds like a plan. I should be able to get to it this weekend.

I have a proposed fix to this issue if anyone wants to weigh in on #1102.

If anyone needs it, I modified @Standard8's test file a bit and this is what I've been using to get fake xhr closer to browser behavior.

https://dl.dropboxusercontent.com/u/2400/tc/sinon/xhr-browser-test.htm

@Standard8 Thanks again! :clap:

According to the spec onerror event is triggered only when a network level event like a DNS timeout or not responsive server happens. 500 or 404 are just normal HTTP responses and it's up to an application to decide whether an error has accorded. https://www.w3.org/TR/XMLHttpRequest/#event-xhr-error
The spec is too concise as usual. Non 2xx responses are considered to be errors by jQuery, that's why many people are confused by XMLHttpRequest's behavior.

@nyk0r: that's pretty much what Phred's fix does :)

@gil : this should be fixed by @fearphage's excellent work in #1102, #1108 and #1109. As your test case is incomplete (no check/verification), would you care to verify that it indeed has been fixed by the code in current v1.17 branch? If you do so, we can ship a new patch release ASAP.

Alternatively, @wlepinski or @mbarlock could perhaps verify that this has been fixed in the v1.17 branch? Just change the sinon dependency in package.json to read: sinon#v1.17 to use the right branch directly from GitHub.

OK, verified to be fixed by running this test:

"call load handler on non-2xx statuses" : function(){
  var stub = sinon.stub();
  this.xhr.addEventListener("load", stub);
  this.xhr.open("GET", "/");
  this.xhr.send();

  this.xhr.respond(500, { 'Content-Type' : 'application/json' }, JSON.stringify({}));
  assert(stub.called);
}

This did not work with 1.17.4, but does with the latest fixes.

Turns out @fearphage already covered this test in bf709a7f (line 1797), but hey ...

Closing as fixed.

Unfortunately I don't have access to that project to test it anymore, I'm working at a new company now. But I'll let them know, in case they want to update the library. Thanks for the fix!!