Pytest-django: Tests using live_server fixture removing data from data migrations

That's because of the transactional_db fixture being used automatically by live_server (there is no need to mark it with @pytest.mark.django_db). There are several issues / discussions in this regard, but the last time I looked closer at this, there was no easy solution to this problem that comes with using data migrations.
A workaround is to use pytest fixtures to wrap your data migrations / data fixtures.
(btw: better put the tests inline in the issue rather than into an external resource that might change over time.)

I just hit this as well and it took me a long time to track it down, even though I'm reasonably familiar with Django's test framework and migrations.

This is actually a surprising performance tradeoff of Django's TestCase. It's documented here:
https://docs.djangoproject.com/en/1.9/topics/testing/overview/#rollback-emulation

In a regular Django test suite, the workaround consists in setting a serialized_rollback = True class attribute on the test case.

I don't know how to achieve the same effect with pytest-django's dynamically generated test classes.

The following change "solves" the issue at the expense of unconditionally selecting the least efficient behavior.

--- pytest_django/fixtures.py.orig  2016-04-27 17:12:25.000000000 +0200
+++ pytest_django/fixtures.py   2016-04-27 17:21:50.000000000 +0200
@@ -103,6 +103,7 @@

     if django_case:
         case = django_case(methodName='__init__')
+        case.serialized_rollback = True
         case._pre_setup()
         request.addfinalizer(case._post_teardown)

The following technique works, but I can't recommend it for rather obvious reasons...

import pytest
from django.core.management import call_command
from pytest_django.fixtures import transactional_db as _transactional_db


def _reload_fixture_data():
    fixture_names = [
        # Create fixtures for the data created by data migrations
        # and list them here.
    ]
    call_command('loaddata', *fixture_names)


@pytest.fixture(scope='function')
def transactional_db(request, _django_db_setup, _django_cursor_wrapper):
    """
    Override a pytest-django fixture to restore the contents of the database.

    This works around https://github.com/pytest-dev/pytest-django/issues/329 by
    restoring data created by data migrations. We know what data matters and we
    maintain it in (Django) fixtures. We don't read it from the database. This
    causes some repetition but keeps this (pytest) fixture (almost) simple.

    """
    try:
        return _transactional_db(request, _django_db_setup, _django_cursor_wrapper)
    finally:
        # /!\ Epically shameful hack /!\ _transactional_db adds two finalizers:
        # _django_cursor_wrapper.disable() and case._post_teardown(). Note that
        # finalizers run in the opposite order of that in which they are added.
        # We want to run after case._post_teardown() which flushes the database
        # but before _django_cursor_wrapper.disable() which prevents further
        # database queries. Hence, open heart surgery in pytest internals...
        finalizers = request._fixturedef._finalizer
        assert len(finalizers) == 2
        assert finalizers[0].__qualname__ == 'CursorManager.disable'
        assert finalizers[1].__qualname__ == 'TransactionTestCase._post_teardown'
        finalizers.insert(1, _reload_fixture_data)

Would an option to conditionally enable serialized_rollback be a good solution?

I.e. something like

@pytest.mark.django_db(transaction=True, serialized_rollback=True)
def test_foo():
    ...

I think that would be good.

In my use case:

• the transactional_db fixture is triggered by live_server, but I wouldn't mind adding a django_db fixture to specify serialized rollback
• unrelated to pytest-django -- case.serialized_rollback = True still fails because Django attemps to deserialized objects in an order that doesn't respect FK constraints, I think that's a bug in Django

Ok, perfect :)

I don't have time to work on a fix right now, but adding such an option to the django_db marker should and setting case.serialized_rollback = True (like you did above) should be relatively straightforward.

Ticket for the Django bug I mentioned above: https://code.djangoproject.com/ticket/26552

@pelme -- does this pull request implement the "relatively straightforward" approach you're envisionning?

Thanks a lot for the PR. The implementation looks correct and what I imagined. I did not think of having a serialized_rollback fixture but it is indeed useful to be able to force this behaviour from other fixtures.

We would need a test for this too, but the implementation looks correct!

I'll try to find time to polish the patch (I haven't even run it yet). I need to get the aformentionned bug in Django fixed as well before I can use this.

Updated version of the hack above for pytest-django ≥ 3.0:

@pytest.fixture(scope='function')
def transactional_db(request, django_db_setup, django_db_blocker):
    """
    Override a pytest-django fixture to restore the contents of the database.

    This works around https://github.com/pytest-dev/pytest-django/issues/329 by
    restoring data created by data migrations. We know what data matters and we
    maintain it in (Django) fixtures. We don't read it from the database. This
    causes some repetition but keeps this (pytest) fixture (almost) simple.

    """
    try:
        return _transactional_db(request, django_db_setup, django_db_blocker)
    finally:
        # /!\ Epically shameful hack /!\ _transactional_db adds two finalizers:
        # django_db_blocker.restore() and test_case._post_teardown(). Note that
        # finalizers run in the opposite order of that in which they are added.
        # We want to run after test_case._post_teardown() flushes the database
        # but before django_db_blocker.restore() prevents further database
        # queries. Hence, open heart surgery in pytest internals...
        finalizers = request._fixturedef._finalizer
        assert len(finalizers) == 2
        assert finalizers[0].__qualname__ == '_DatabaseBlocker.restore'
        assert finalizers[1].__qualname__ == 'TransactionTestCase._post_teardown'
        finalizers.insert(1, _reload_fixture_data)

Thanks for the update and sorry I couldn't get this into 3.0. I will try to get to this for the next release!

Is there any chance to see it in next release?

Thanks

@Wierrat
Yes, probably.

Can help us with a test for https://github.com/pytest-dev/pytest-django/pull/353 before that then please?

Hi folks,
what is the latest status of this? In particular I'm using pytest as my test runner with the StaticLiveServerTestCase class from Django. I've set the class attribute serialized_rollback = True but that seems to be in effect only when executing the first test from the sequence.

Just got caught by this one. Looking around it seems there have been some PR on this issue but none of them appear to have been merged.
Is there any particular reason this issue remain open?

some PR

Likely/only https://github.com/pytest-dev/pytest-django/issues/329, no?

The last time I've asked about help with a test, and it has several conflicts (maybe only due to black).

I still suggest for anyone affected to help with that - I am not using it myself.

Thanks for the quick reply @blueyed.
My knowledge of pytest / pytest-django is minimal but I'll put that on my somday/maybe list! :D

Updated version of the above hack for pytest 5.2.1, pytest-django 3.5.1:

from functools import partial

@pytest.fixture
def transactional_db(request, transactional_db, django_db_blocker):
    # Restore DB content after all of transactional_db's finalizers have
    # run. Finalizers are run in the opposite order of that in which they
    # are added, so we prepend the restore to the front of the list.
    #
    # Works for pytest 5.2.1, pytest-django 3.5.1
    restore = partial(_restore_db_content, django_db_blocker)
    finalizers = request._fixture_defs['transactional_db']._finalizers
    finalizers.insert(0, restore)

    # Simply restoring after yielding transactional_db wouldn't work because
    # it would run before transactional_db's finalizers which contains the truncate.
    return transactional_db

def _restore_db_content(django_db_fixture, django_db_blocker):
    with django_db_blocker.unblock():
        call_command('loaddata', '--verbosity', '0', 'TODO your fixture')

I had to depend on the original transactional_db fixture instead of calling it as pytest no longer allows calling fixture functions directly. I then get the fixture def of the original fixture and prepend a finalizer to it. While inserting at index 1 still worked, I insert before all finalizers and use django_db_blocker in the restore function as that seems slightly less fragile.

Edit: removed unnecessary try finally.

@lukaszb

https://github.com/pytest-dev/pytest-django/issues/848

I use this version of pytest-django (https://github.com/pytest-dev/pytest-django/pull/721/files),
But still error.