Julia: 0.5 `@test_skip` causes CI to fail

Created on 13 Mar 2017 · 3Comments · Source: JuliaLang/julia

per https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628

The following code will cause CI to fail, even though @test_skip should not contribute to CI success/failure:

using LightGraphs
using LightGraphs.LinAlg
using Base.Test

@testset "foo" begin
@test_skip 1 == 2
end

yields

julia> Pkg.test("LightGraphs")
INFO: Testing LightGraphs
Test Summary: | Broken  Total
  foo         |      1      1
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 0 errored, 1 broken.

causing a CI failure.

This is apparently fixed in 0.6, but I haven't been able to confirm that yet.

bug testsystem

Source

sbromberger

👍2

Most helpful comment

fixed in 0.5.2 which will be tagged and uploaded shortly

tkelman on 6 May 2017

👍2

All 3 comments

Bisect identifies 8982605c1998 as fixing this. I'm glad the version of that PR that we merged was bisect-clean.

That commit is a bit too large and disruptive to backport, but a small subset of it would probably be good enough to fix this bug. Likely just the counting of broken tests in determining whether a testset succeeds or fails.

tkelman on 17 Mar 2017

I just came across this myself for @test_broken. Here's the workaround I'm using until 0.5.2:

using Base.Test

if VERSION < v"0.6-"
    # Override Test.@test_broken, which is broken on julia-0.5!
    # See https://github.com/JuliaLang/julia/issues/21008
    macro test_broken(exs...)
        esc(:(@test !($(exs...))))
    end
end