Pytest-django: How to allow database access globally without using django_db marks?

Created on 20 Sep 2016 · 8Comments · Source: pytest-dev/pytest-django

I am struggling to convert my Django test runner based tests to pytest.
I would expect that putting

import pytest
pytestmark = pytest.mark.django_db

into the last module of the stacktrace that belongs to my code (lib/frontend/decorators.py)
would mitigate this, but it didn't. Putting it into all four of my modules on the stacktrace didn't help either.

Is there any way I can globally allow database access in pytest-django?

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ pytest 
== test session starts ==
platform linux2 -- Python 2.7.6, pytest-3.0.2, py-1.4.31, pluggy-0.3.1
Django settings: patient.settings (from ini file)
rootdir: /vagrant, inifile: pytest.ini
plugins: django-3.0.0, pythonpath-0.7.1
collected 265 items / 2 errors 

[...]

_ ERROR collecting lib/tests/report/testfunc.py _
lib/tests/report/testfunc.py:6: in <module>
    from report.func import generate_report_filename
lib/report/func.py:29: in <module>
    from frontend import chart
lib/frontend/chart.py:294: in <module>
    @pytest.mark.django_db
lib/frontend/decorators.py:20: in decorator
    if Site.objects.get_current().id in (allowed_site_ids):
/usr/local/lib/python2.7/dist-packages/django/contrib/sites/models.py:60: in get_current
    return self._get_site_by_id(site_id)
/usr/local/lib/python2.7/dist-packages/django/contrib/sites/models.py:39: in _get_site_by_id
    site = self.get(pk=site_id)
/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py:127: in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
/usr/local/lib/python2.7/dist-packages/django/db/models/query.py:328: in get
    num = len(clone)
/usr/local/lib/python2.7/dist-packages/django/db/models/query.py:144: in __len__
    self._fetch_all()
/usr/local/lib/python2.7/dist-packages/django/db/models/query.py:965: in _fetch_all
    self._result_cache = list(self.iterator())
/usr/local/lib/python2.7/dist-packages/django/db/models/query.py:238: in iterator
    results = compiler.execute_sql()
/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py:838: in execute_sql
    cursor = self.connection.cursor()
/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py:162: in cursor
    cursor = self.make_debug_cursor(self._cursor())
/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py:135: in _cursor
    self.ensure_connection()
E   Failed: Database access not allowed, use the "django_db" mark to enable it.

Source

arne-cl

Most helpful comment

Well you can always create an autouse fixture, like

@pytest.fixture(autouse=True)
def enable_db_access(db):
    pass

in your conftest

enkore on 28 Sep 2016

👍9 ❤2

All 8 comments

Well you can always create an autouse fixture, like

@pytest.fixture(autouse=True)
def enable_db_access(db):
    pass

in your conftest

enkore on 28 Sep 2016

👍9 ❤2

@enkore's answer is the way to go. I've just added a FAQ in the documentation too:

http://pytest-django.readthedocs.io/en/latest/faq.html#how-can-i-give-database-access-to-all-my-tests-without-the-django-db-marker

pelme on 12 Nov 2016

A module I’m importing in a test connects to the DB at import time. How do I enable DB access globally such that it is already allowed during the test collection? Thanks!

Telofy on 24 Feb 2018

👍3

@Telofy did you ever solve the issue of marking imports with db access? Have the same problem...

lmannering on 31 May 2018

Sorry, but the way Django (and pytest-django) works it is not possible in a reliable way. The only proper solution is simply to not do database queries during import time. pytest-django is built around pytests fixture system and there is no way pytest-django can ensure that fixtures are run before imports.

See my comment here for a more detailed explanation of this issue:
https://github.com/pytest-dev/pytest-django/issues/499#issuecomment-322056315

pelme on 31 May 2018

It’s not within my power or time budget to just rewrite the company’s whole platform to avoid those queries. My solution is super ugly monkey-patching in conftest.py:

from pytest_django.plugin import _blocking_manager
from django.db.backends.base.base import BaseDatabaseWrapper

_blocking_manager.unblock()
_blocking_manager._blocking_wrapper = BaseDatabaseWrapper.ensure_connection

I’m also using this decorator (in conjunction with @pytest.mark.django_db) to then make sure the tests are not running on the wrong database, because it has been a bit complicated to predict:

def assert_test_db(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        assert connection.settings_dict['NAME'] == 'test_kf_server', connection.settings_dict
        return func(*args, **kwargs)
    return wrapper

Telofy on 31 May 2018

🎉4 👍4

Thanks for the example @Telofy - that's really useful. I'd also like to avoid re-writing our entire codebase!

lmannering on 31 May 2018

👍1

Make db queries at import time doesn't just lead to bad pytests, it's going to cause issues in other parts that will be hard to track down. You're going to have a bad time.

You don't have to rewrite your whole codebase, you can get around it using lazy methods/properties. Code that doesn't make a connection until asked for the value. Also passing just querysets without evaluating will work. Avoiding stuff like count etc