@junit-team/junit-committers any thoughts on a 4.13 release? I have unfortunately been very busy with my day job so I can't drive a 4.13 release, but I can certainly help.

I've associated some open issues with Milestone 4.13 as a starting point, but it appears that milestones cannot have comment threads, so I figured we can discuss scheduling and which features should be included here

One thing I would like to do is aligning Assert.assertThrows/expectThrows with Jupiter's Assertions.assertThrows. I know there was some controversy around that a while back but now that Jupiter's been released I think we should consider migration scenarios and remove Assert.assertThrows and rename expectThrows to assertThrows. Any objections?

The last attempt to remove one of those was #1396

I can see both sides of the debate about merging assertThrows and expectThrows, so I wouldn't object to a pull removing expectThrows and updating assertThrows to return the caught exception. I suggest reaching out the people involved in the original pull adding these methods to 4.x, to inform them of the resolution of this issue. I realize some people would have been upset regardless of what direction we decided to take with the APIs, but still think that it's good to make sure everyone knows their voice is being heard.

While we are at it, it would be great to port assertThrows(Class<T> expectedType, Executable executable, String message) to JUnit4 (see https://github.com/junit-team/junit5/commit/ea67ca0)

any progress on 4.13 release ?

@zjffdu yes. See https://github.com/junit-team/junit4/pulse/monthly for recent work and https://github.com/junit-team/junit4/milestone/3 for the proposed remaining work.

@kcooney
@marcphilipp
It's been almost 4 years after last release 4.12.
Cut 4.13-beta-1 release version for deeper testing.

I (my organization, really) would really like to use the work that was done for TemporaryFolder (particularly issue #1223) so I'm really interested in getting JUnit 4.13. We're contemplating taking a fork of the project (at 4.12), cherry-picking the TemporaryFolder fixes, and using JUnit artifacts generated from that until 4.13 is actually released. (Haven't yet evaluated how sane/insane that is.) Since at least some of the projects in which we'd be using the fork are public/OSS projects, the artifacts produced from the fork will need to be published (under a Maven group we own) to Maven Central. But, if you're planning on cutting 4.13 anytime soon, we can just wait. (We do have efforts underway to migrate to JUnit 5 but that doesn't help us yet with TemporaryFolder.)

Thoughts?

We have built and use 4.13 as it is, with all its changes, for a while now, because we needed to leverage a bugfix. Haven't seen any issues so far.

@cljohnso publishing your own version of JUnit in Maven could be problematic for some of your users because they will likely depend on an official version of JUnit, and having both dependencies would result in duplicate classes on the classpath, which leads to unpredictable behavior.

Unfortunately, JUnit is maintained by volunteers (many of whom work on JUnit 5) so relesses have taken longer than anticipated.

One way to help us get ready would be to fix and review the release notes. Many pull requests were not documented there (see these bugs).

The blocking bugs for 4.13 are here.

Looks like all issues assigned to 4.13 have been resolved.

@junit-team/junit-committers Is there anything else that need to be addressed before releasing 4.13?

@marcphilipp These closed pull requests might not be documented in the release notes: https://github.com/junit-team/junit4/issues?utf8=✓&q=label%3A"needs+release+notes+update"+

I started to complete the release notes. In doing so I created two new pull requests: #1551 and #1552. At least #1552 should be in 4.13.

Hello all,

I was wondering if anyone had a timeline for this release. There are some features that I was hoping to incorporate, and was wondering whether or not we should should table it.

Thanks for all your work!

@johnterrill I don't think we were planning on adding any more new features to 4.13 (just bug fixes)

If there is something you want to add, please feel free to add a feature request. That being said, almost all feature development for JUnit is happening in the junit5 repository.

@kcooney If 4.13 contains only bug fixes, rename it to 4.12.1. Is there still any ongoing work?

It's not onlt bug fixes, there's also a few new features, e.g. ordering. Thus, I think 4.13 make sense.

There are still 5 PRs missing in the release notes. I take care of them the day after tomorrow when I'm back in Germany. AFAIK this is all that is left.

@marcphilipp I think we are ready for JUnit 4.13. I completed the release notes.

Having #1568 released with JUnit 4.13 would be nice (need a review from one of the maintainers) but it is not necessary.

Any update on 4.13 release? I am very excited to see the new release happens. :)

@robinyqiu
I remember they planned to discontinue JUnit and 4.13 had to be the last version. Since there are 103 open issues and several pending pull requests, the versions should continue development. One way would be to cut several Release Candidate versions and we will see what would happen with the statistics regarding the number of bugs and pull requests.

I believe we still plan to release 4.13 (because of the sheer number of improvements and fixes that have been merged).

Sorry, submitted that last comment before I finished my thought.

Apologies that this is taking awhile. The original JUnit maintainers are either busy supporting JUnit 5 or have taken a step back from working on JUnit. For those reasons I think it's likely that 4.13 will be the last release if the 4.x code line.

@Tibor17 @kcooney Thanks for your response! Do you have a rough estimate of when the 4.13 release is going to happen?

@robinyqiu I am not a committer but a long time contributor.
@kcooney I think the community should clearly present the release plan for next weeks or months.

I am still convinced that JUnit4 should be finalized with zero issues and zero pull request due to the fact that all the world is and still will be dependent on JUnit4 long time.
So if you see any feature requests, please close them saying that the possibility for it would be in JUnit5 and no more in JUnit4.
Issues which are ours have to be fixed and pull requests accepted. Old pull requests look to me that we have not reached a consensus with the person who create the PR and therefore they should be closed.

@Tibor17 If there are any bugs that you feel have to be fixed or pull requests that should to be accepted, please add a comment to those issues. At some point we need to decide to cut a release for 4.13 (we released 4.12 almost four years ago).

I'm not sure how I feel about essentially filing bug bankruptcy for JUnit 4.x. I certainly don't have time to go through that process. Perhaps one of the other mainteners does.

@marcphilipp Do you have time to cut a release 4.13? @kcooney, @dsaff any objections?

@kcooney Yes, I feel there are issues since 2015 till 2017 which need to be closed. This would be the first step to do. It gives clear picture and a signal to the users that seriously we won't continue them. The only remaining issues need to be reinvestigated and spent time only with those. And we will again clean our hands and we can finish the project finally.

Do you have time to cut a release 4.13?

Yes, I can do that on the weekend. Should we first ship a 4.13-RC1 or 4.13-beta-1?

4.13-beta-1 is a good idea. It keeps the naming scheme of the old beta releases.

How long should we wait for 4.13 afterwards if 4.13-beta-1 has no issues? 2 weeks?

No objections to cutting a release. No strong opinion on RC1 vs beta-1.

4.13-beta-1 has been released today and I now available on Maven Central.

Thanks @marcphilipp !

It seems Hamcrest is about to release a new version (see hamcrest/JavaHamcrest#224). Release candidates are already available (https://github.com/hamcrest/JavaHamcrest/releases).

It would be great if the dependency on Hamcrest could be updated for the new release. Would this be feasible?

@BlueIce Thanks for bringing this to our attention!

Since JUnit 4 is still compatible with Java 5, I think we cannot upgrade to Hamcrest 2.x because it requires Java 7.

I'm wondering whether we should make our dependency on Hamcrest optional in the POM, though. That would force users to make a conscious choice whether to use Hamcrest and, if so, which artifact and version.

@tumbarumba @junit-team/junit-committers WDYT?

Hi, Hamcrest committer here. I'm currently preparing the 2.1 release of Hamcrest. It would be awesome if the Hamcrest dependency could be updated. I have recently released v2.1-rc3. Despite the increment in the major version number, this release is intended to be backwards compatible with 1.3 (long story).

One thing to be aware of: in version Hamcrest 2.1, we've changed the way that the jars are packaged. hamcrest-core and hamcrest-library have been merged into a single artifact: hamcrest (e.g. hamcrest-2.1.jar). For backwards compatibility, we are still releasing 2.1 versions of the hamcrest-core and hamcrest-library artifacts, but these jars are just empty, and are provided to allow simplifying the upgrade through a transitive dependency on hamcrest-2.1.jar.

An optional dependency would work, too. I haven't tried it, but I believe if you don't use Assertions.assertThat(...), you don't need the dependency at all? If you do use Matchers, then explicitly declaring a depending upon either hamcrest-core-1.3.jar or hamcrest-2.1.jar would work (I have not tested this, though!)

My personal preference: have JUnit 4.13 depend directly upon hamcrest-2.1.jar. I think this would be the least surprising upgrading path to most people.

@tumbarumba
I would like to use Java Hamcrest 2.1 but it does not mean that I would like to see it in JUnit exactly because of the compatibility reasons of Java 1.5 as @marcphilipp mentioned.
According to what you mentioned that hamcrest-core is trasitively dependent on hamcrest there is no problem for me as a user to use higher version in dependencyManagement of my POM. Anyway I am using hamcrest-library:1.3 and changing the version to 2.1 is what I have to do, no big deal. Maybe the project JUnit5 can think about hamcrest:2.1 but that's another story.

Ah, I missed the point about JUnit 4 being compatible with Java 5. Of course, this means that JUnit 4.x can't depend upon Hamcrest 2.0 or greater, at least not directly. JUnit 5 removed all dependencies on third party matchers, and so is completely independent of Hamcrest

One option: JUnit 4.13 keep the dependency on Hamcrest 1.3. If anyone wants to use the new features of Hamcrest 2.1, they can explicitly declare a dependency on hamcrest-core-2.1.jar, which will trigger the version conflict resolution process, and upgrade the library (as long as it is declared first).

Another option: use the pom <optional> attribute. I don't have much experience with that. Is it possible to declare an optional dependency on both hamcrest-core-1.3.jar and hamcrest-2.1.jar. What happens in that case?

@tumbarumba
Using optional dependency hamcrest-2.1.jar in JUnit's POM means that the user is unable to inherit it into his POM. Basically, it would not be transitive dependency which would be the same effect like not declaring hamcrest-2.1.jar at all.

After pondering this for a few days, I've come to the conclusion that we should keep the mandatory dependency on hamcrest-core:1.3 for backwards compatibility. Making it optional would do more harm than good because it would require most users to add another dependency.

We can add a note to the release notes mentioning Hamcrest 2.1. Moreover, I think it would be a good idea to document how to use the new version with JUnit 4.x and the different build tools on Hamcrest's website or wiki. Ideally, we could link to this page from the JUnit 4.13 release notes.

@tumbarumba WDYT?

I agree with your conclusion, @marcphilipp. As I now understand things, the JUnit 4 dependency on Java 1.5 completely rules out Hamcrest 2.x, and using optional dependencies won't work, as far as I can see.

Regarding the docs, I have a pull request open to update the Hamcrest documentation (see hamcrest/JavaHamcrest#237). I'd appreciate any feedback on that. The Hamcrest 2.1 docs can be previewed at https://tumbarumba.github.io/JavaHamcrest (and specifically the note on Upgrading from Hamcrest 1.x). I won't merge that pull request until the real Hamcrest 2.1 release.

I'm about to release another release candidate for Hamcrest 2.1 to fix some problems with deprecated classes. Hopefully this will be the last RC (unless there is any more feedback). Check hamcrest/JavaHamcrest#224 to follow that next Hamcrest release.

It's been 3 weeks since 4.13-beta-1 has been released. Anything blocking the final 4.13 release?

@ijuma https://github.com/junit-team/junit4/milestone/8 lists one issue

We might include #1569

I think we should include #1569 and release 4.13-beta-2. I created a new corresponding milestone to reflect that.

Is there any news on when we can hope for beta-2/GA to be released? I am so excited by 4.13 and am keen to use it for real, but cannot use it in my day-to-day job since it still has 'beta' status :-(

Now that #1586 has been merged, I'll release 4.13-beta-2 in the next few days.

@junit-team/junit-committers Any objections?

No objections 😊

4.13-beta-2 is released and ready for testing.

It's been over a month since 4.13-beta-2 and, as far as I'm aware, there have been no reports of any issues (except for #1593). I'd propose we release 4.13 in the next few days.

@junit-team/junit-committers Any objections?

We've been using 4.13-beta-2 for Apache Kafka with no issues.

@ijuma Thanks for letting us know!

No objections. I'm using it at work and everything works fine.

I just started the process of using 4.13 at Google and ran into two warnings (which are treated as errors by our build system) and sent out two pull requests. I might run into more problems when I start running lots of tests under 4.13

Can we wait a week or so?

Alternatively we could target 4/13 ;-)

@kcooney Am I right that the two PRs you mentioned are #1596 and #1597? 4/13 sounds like a perfect target :-)

That sounds like a good validation, so let's wait for the results.

Fun fact: 4.12 was released on 12/4 😉

@kcooney Am I right that the two PRs you mentioned are #1596 and #1597?

Yes.

I am seeing a regression in our DefaultInternalRunner in Mockito. Having a hard time figuring out what is going on, but the issue is that testFinished is called even though the test failed. However, from the history of ParentRunner I can't spot any obvious issue.

The test that regresses on 4.13 and passes on 4.12 is https://github.com/mockito/mockito/blob/a323b8132de6f6e1c29d738de245469f8ce009b0/src/test/java/org/mockito/internal/runners/DefaultInternalRunnerTest.java#L42-L59 I will continue debugging, but any pointers would be appreciated :smile:

@TimvdLippe The Javadoc for RunListener.testFinished() states:

"Called when an atomic test has finished, whether the test succeeds or fails." I can't speak to why you are seeing different behavior between 4.12 and 4.13, but the behavior you are seeing for 4.13 sounds like it is correct.

I would like to update the behavior so that it works properly with JUnit 4.13, although I fear that we might break our JUnit 4.12 integration. I will try to figure out a solution tomorrow.

I was able to extract a reproduction case from Mockito into #1599. It seems to be related to the handling of withBefores.

As I'm testing out 4.13 in our code base, #1421 is causing some problems.

Does anyone have time to submit a PR that reverts #1421?

Done in #1602 by @panchenko.

@kcooney Shall we do another beta release?

@marcphilipp up to you. The recent changes shouldn't affect people on 4.12 (only people on a prerelease of 4.13).

In our code base I am currently working through the many places that pass null to the FrameworkMethod constructor 🙄 which unfortunately make it hard to see the real problems.

@kcooney Can you estimate how much time you'll need to get through those changes and run a final test run?

@marcphilipp sorry for the late reply (I was on vacation). Currently 2.9% of our tests fail with JUnit 4.13, but some might be flaky failures. A manual sample isn't showing any problems due to changes in 4.13, but I still have many to go through.

@kcooney No worries! So, no estimate, right? 😉

@marcphilipp So far, the remaining failures have been tests that have done questionable things, so I wouldn't block 4.13 on those. I'll look through all the failures tonight.

Some examples of tests failing due to doing questionable things:

We have some tests were a subclass has a @Rule field that is the same name and type as the base class field (which is also annotated with @Rule). Apparently in 4.12 the subclass Rule would execute instead of the base class rule, but in 4.13 both execute. I remember us changing the way shadowed members work. I can't recall if what I'm seeing was an intentional change.

I'm also seeing failures where JUnit 4.13 is producing improved error messages, and the tests were asserting a specific error message.

(Edited to fix the pull request number)

@marcphilipp I'm trying to remember why I made the changes in #1414

While it's true that fields don't shadow the same field from the base class, JUnit 4.x has treated it as if they do for a while now. The (again, large) code base I'm workong on has many examples of code that assumes that the subclass class @Rule fields will replace the behavior of the base class's fields. While it's not all that hard to work around this problem (change the fields to methods, so subclasses can override) I do wonder if the benefit of that pull outweigh the costs for existing users.

I think we should revert it and restore the old, albeit questionable behavior for backwards compatibility.

@stefanbirkner WDYT?

I agree. I don't see any advantage that justifies breaking existing tests.

I created #1605 which finally reverts #1414.

I think we should release another beta release. @kcooney Shall I do that in the next few days or have you discovered additional issues with 4.13-beta-2?

@marcphilipp I patched in the latest change, and while I am still looking through reruns, I don't see any patterns (other than tests asserting on error messages that we improved). I think the remaining issues I am seeing are bad or flaky tests.

Sorry this took so long but I am glad we reverted those changes!

4.13-beta-3 is released and ready for testing.

It would be great if one of you could merge in https://github.com/junit-team/junit4/pull/1608.

I'd also be grateful if you would consider #1609.

I recently upgraded from JUnit 4.12 to JUnit 4.13-beta-3. It is working great.

The reason that I upgraded is because I needed this fix:
https://github.com/junit-team/junit4/commit/faa0e334080cd91f05fc1acbc7c39a525e172256

Any thoughts about 4.13.0 GA?

@sullis The main remaining issue is to fix the recommendations wrt. Hamcrest (see #1608).

The Hamcrest team (a.k.a me) is not planning to make any changes with respect to #1608. Does this mean that there are no other blockers for 4.13?

I didn't mean we were waiting on the Hamcrest team to make changes. I was referring to https://github.com/junit-team/junit4/pull/1608#issuecomment-496492379:

In summary, somebody needs to review all deprecations regarding Hamcrest and ensure that the "deprecation message" does not advise users to use something that is itself already deprecated and/or no longer available.

I'm taking care of the "deprecation messages" from Wednesday on.

I'm taking care of the "deprecation messages" from Wednesday on.

If you get it done by Wednesday night, I might be able to include it in Spring Framework 5.2 GA, thereby allowing me to revert this commit https://github.com/spring-projects/spring-framework/commit/665e8aa51c11992909134d3ad5eebca6c94aa877.

But... no pressure. Officially stating that Spring Framework 5.2 supports JUnit 4.13 is only nice-to-have. JUnit 4.13 will still work just fine with Spring 5.2 once JUnit 4.13 is released. 😉

Hi @sbrannen, I started but I don't get it done tonight. I'm currently reading all the comments and try to understand all the details.

@stefanbirkner, no worries! Like I said, that would have been "nice to have", but there's certainly no requirement for it. JUnit 4.13 still potentially has time to get picked up officially by Spring Boot 2.2. 😉

any thoughts about 4.13.0 GA?

Without some resolution for #1609, JUnit 4.13 risks to be less relevant than it could be. Users of JUnit 4.12 are waiting for bugfixes and possibly some backports from JUnit 5, not for deprecation of the code that is not broken.

are there any blockers for JUnit 4.13 GA?

I don't see any. I'll publish a 4.13-rc-1 to get some more feedback on the latest changes.

JUnit 4.13-rc-1 is now available on Maven Central for testing! Please let us know if you run into any issues.

Tested 4.13 beta and rc and works for us at our large org 👍

I tested JUnit 4.13-rc1 and I didn't encounter any issues.

I tested JUnit 4.13-rc-1 with Stash Pull Request Builder plugin for Jenkins. Deprecations were triggered for files using ExpectedException once per file, as expected.

In many cases, the tests were checking the exception type, the message and the cause. I was able to fit all checks into one expression to avoid having a temporary variable.

I'm not getting any warnings or deprecations about the new code with SpotBugs, sb-contrib and find-sec-bugs.

The only small imperfection is that the coverage for assertThat is shown in yellow in Eclipse 2019-09 on Linux. Still much better than what is was before. Besides, nobody cares about test coverage of the tests themselves.

what is the status of JUnit 4.13?

It's been three weeks now and only one issue has been reported: https://github.com/junit-team/junit4/issues/1636

@kcooney @stefanbirkner Could you please take a look? I think we can close the above issue.

@marcphilipp sorry for the long delay. I haven't had much time to work on JUnit for over a year.

I commented on that bug. I don't see how we can fix it.

I did have one feature request for 4.13. See #1637

Hi, what is the status of JUnit 4.13?

1637 was resolved and I'll release another RC.

JUnit 4.13-rc-2 is now available on Maven Central for testing! Please let us know if you run into any issues.

I tested JUnit 4.13-rc2 and I didn't encounter any issues. LGTM

Likewise, JUnit 4.13-rc-2 is working correctly for me. I'm using the new exception checks.

are there any blockers for JUnit 4.13 GA?

@kcooney Could you please re-test with 4.13-rc-2 now that #1637 is resolved?

any update?

are there any blockers for JUnit 4.13 GA?

Sorry for the late reply; things have been busy these last few weeks.

I haven't had time to integrate the latest JUnit changes into our code
repository. I did look at the tests on our side that cover the behavior of
randomization when classes use FixMethodOrder, and they look almost exactly
the same as the tests I had it my pull. I don't see a reason block a JUnit
release on doing more verification on that fix.

q: are there any blockers for JUnit 4.13 GA?

JUnit 4.13 is released! 🎊
https://twitter.com/junitteam/status/1212403812192964608