Django-rest-framework: Support for ordering by nested related fields in OrderingFilter

I'd consider pull requests for this, but it's not something I'd have time to do myself.

Anyone looking to implement this needs to ensure sensible behavior when incorrect field names are used in the filtering. At a minumum the results should still return okay. (Ideally, any incorrect field names should be ignored, but other fields should be left in.)

On consideration I'm going to close this off. If someone issue a pull request and tests that deals with it then I might reconsider, but more complete implementations of each of the basic filtering classes is really something that someone could tackle really nicely in a third party package, that could then be maintained seperatly, and linked to from the main docs.

No pull request because I haven't had time to write tests or docs, but in case this helps other people finding this issue, I'm using:

from django.db.models.fields import FieldDoesNotExist 
from rest_framework.filters import OrderingFilter

class RelatedOrderingFilter(OrderingFilter):
    """ 
    Extends OrderingFilter to support ordering by fields in related models
    using the Django ORM __ notation
    """
    def is_valid_field(self, model, field):
        """
        Return true if the field exists within the model (or in the related 
        model specified using the Django ORM __ notation)
        """
        components = field.split('__', 1)
        try:
            field, parent_model, direct, m2m = model._meta.get_field_by_name(components[0])
            if field.rel and len(components) == 2:
                return self.is_valid_field(field.rel.to, components[1])
            return True
        except FieldDoesNotExist:
            return False

    def remove_invalid_fields(self, queryset, ordering):
        return [term for term in ordering if self.is_valid_field(queryset.model, term.lstrip('-'))]

@rhunwicks your version does not allow reverse relation sorting, only direct fk's here is patched version

class RelatedOrderingFilter(OrderingFilter):
    """
    Extends OrderingFilter to support ordering by fields in related models
    using the Django ORM __ notation
    """
    def is_valid_field(self, model, field):
        """
        Return true if the field exists within the model (or in the related
        model specified using the Django ORM __ notation)
        """
        components = field.split('__', 1)
        try:
            field, parent_model, direct, m2m = \
                model._meta.get_field_by_name(components[0])

            # reverse relation
            if isinstance(field, RelatedObject):
                return self.is_valid_field(field.model, components[1])

            # foreign key
            if field.rel and len(components) == 2:
                return self.is_valid_field(field.rel.to, components[1])
            return True
        except FieldDoesNotExist:
            return False

    def remove_invalid_fields(self, queryset, ordering, view):
        return [term for term in ordering
                if self.is_valid_field(queryset.model, term.lstrip('-'))]

pySilver's snippet worked well for me, can we get this into DRF by any chance?

Where do you import RelatedObject from? Does not seem to be available to me

Python 3.4.3
Django==1.8.5
djangorestframework==3.2.3

@eldamir django.db.models.related.RelatedObject

ImportError: No module named 'django.db.models.related'

ah, seems it's been removed from 1.8, see https://code.djangoproject.com/ticket/21414

Thanks for the tip :+1: . The link let to the fix. This works

from django.core.exceptions import FieldDoesNotExist
from django.db.models.fields.related import ForeignObjectRel
from rest_framework.filters import OrderingFilter


class RelatedOrderingFilter(OrderingFilter):
    """
    Extends OrderingFilter to support ordering by fields in related models
    using the Django ORM __ notation
    """
    def is_valid_field(self, model, field):
        """
        Return true if the field exists within the model (or in the related
        model specified using the Django ORM __ notation)
        """
        components = field.split('__', 1)
        try:
            field, parent_model, direct, m2m = \
                model._meta.get_field_by_name(components[0])

            # reverse relation
            if isinstance(field, ForeignObjectRel):
                return self.is_valid_field(field.model, components[1])

            # foreign key
            if field.rel and len(components) == 2:
                return self.is_valid_field(field.rel.to, components[1])
            return True
        except FieldDoesNotExist:
            return False

    def remove_invalid_fields(self, queryset, ordering, view):
        return [term for term in ordering
                if self.is_valid_field(queryset.model, term.lstrip('-'))]

Did not have the problem with the last version of django rest framework :)

Don't forget to correctly specify your ordering_fields

Just a note - in Django > 1.10 a couple of the methods have changed

Specifically you'll have to change the following:

field, parent_model, direct, m2m = model._meta.get_field_by_name(components[0])
to
field = model._meta.get_field(components[0])

and the signature for remove_invalid_fields has changed to:

def remove_invalid_fields(self, queryset, ordering, view, request):

Which results in the final working version for Django > 1.10:

class RelatedOrderingFilter(OrderingFilter):
    """
    Extends OrderingFilter to support ordering by fields in related models
    using the Django ORM __ notation
    """
    def is_valid_field(self, model, field):
        """
        Return true if the field exists within the model (or in the related
        model specified using the Django ORM __ notation)
        """
        components = field.split('__', 1)
        try:

            field = model._meta.get_field(components[0])

            # reverse relation
            if isinstance(field, ForeignObjectRel):
                return self.is_valid_field(field.model, components[1])

            # foreign key
            if field.rel and len(components) == 2:
                return self.is_valid_field(field.rel.to, components[1])
            return True
        except FieldDoesNotExist:
            return False

    def remove_invalid_fields(self, queryset, fields, view, request):
        return [term for term in fields if self.is_valid_field(queryset.model, term.lstrip('-'))]

Above code will have issue with OneToOneField, adding a fix for that:

from django.core.exceptions import FieldDoesNotExist
from django.db.models.fields.reverse_related import ForeignObjectRel, OneToOneRel

from rest_framework.filters import OrderingFilter


class RelatedOrderingFilter(OrderingFilter):
    """
    Extends OrderingFilter to support ordering by fields in related models.
    """    

    def is_valid_field(self, model, field):
        """
        Return true if the field exists within the model (or in the related
        model specified using the Django ORM __ notation)
        """
        components = field.split('__', 1)
        try:

            field = model._meta.get_field(components[0])

            if isinstance(field, OneToOneRel):
                return self.is_valid_field(field.related_model, components[1])

            # reverse relation
            if isinstance(field, ForeignObjectRel):
                return self.is_valid_field(field.model, components[1])

            # foreign key
            if field.rel and len(components) == 2:
                return self.is_valid_field(field.rel.to, components[1])
            return True
        except FieldDoesNotExist:
            return False

    def remove_invalid_fields(self, queryset, fields, view):
        return [term for term in fields
                if self.is_valid_field(queryset.model, term.lstrip('-'))]        

field.rel and field.rel.to will raise a deprecation warning on Django >= 1.10. They are now respectively:

• field.remote_field
• field.model

@tomchristie considering this patch has been lingering (and being constantly updated) in this issue for 4 years (we have been using this in production since last year), could it be suitable for a PR and merged into DRF (yes, backed up by a proper set of unittests)?

Just my .2 cents

This patch works to me, can we merge it to the release versions?

As said earlier, either make it a 3rd party (prefered) or it needs a proper PR with tests and documentation.

@filiperinaldi I think that in Django >= 1.10 field.rel.to becomes field.related_model. Here's the latest version of the patch that passes our unittests.

n.b. We are using Django 1.11 so ymmw

class RelatedOrderingFilter(filters.OrderingFilter):
    """

    See: https://github.com/tomchristie/django-rest-framework/issues/1005

    Extends OrderingFilter to support ordering by fields in related models
    using the Django ORM __ notation
    """
    def is_valid_field(self, model, field):
        """
        Return true if the field exists within the model (or in the related
        model specified using the Django ORM __ notation)
        """
        components = field.split('__', 1)
        try:
            field = model._meta.get_field(components[0])

            if isinstance(field, OneToOneRel):
                return self.is_valid_field(field.related_model, components[1])

            # reverse relation
            if isinstance(field, ForeignObjectRel):
                return self.is_valid_field(field.model, components[1])

            # foreign key
            if field.remote_field and len(components) == 2:
                return self.is_valid_field(field.related_model, components[1])
            return True
        except FieldDoesNotExist:
            return False

    def remove_invalid_fields(self, queryset, fields, ordering, view):
        return [term for term in fields
                if self.is_valid_field(queryset.model, term.lstrip('-'))]

Ok people, for the 4th birthday of this issue, I decided to give it a try and hack together an external package to be installed alongside DRF in order to support this useful ordering:

https://github.com/apiraino/djangorestframework_custom_filters_ordering

I'll be working on it in the following days to finish the job (namely I need to properly package and factor the code, refine testing and ensure support for actively supported Django versions).

Contributions are welcome, of course!

Cheers

ref #5533.

I am very confused. This appears to work by default?

I came to this issue, read all the comments, and proceeded to implement the solution by @apiraino, but then I discovered I had merely typed the name of my related field wrong.

However, I am now using ?ordering=job__customer__company for a double nested relationship to order the API results and it's working fine.

@halfnibble - I believe this was fixed in #5533.

cc @carltongibson

@halfnibble, what version did u use? I'm on 3.8.2 and its not working for me. These are my versions:

    "install_requires": [
        "django==2.0.3",
        "coreapi==2.3.3",
        "django-filter==1.1.0",
        "djangorestframework-filters==0.10.2.post0",
        "djangorestframework-queryfields==1.0.0",
        "djangorestframework==3.8.2",
        "django-bulk-update==2.2.0",
        "django-cors-headers==2.4.0",
        "django-rest-auth[with_social]==0.9.2",
        "drf-yasg==1.6.0",
        "django-taggit==0.22.2",
        "google-api-python-client==1.6.2",
        "markdown==2.6.11",
        "pygments==2.2.0",
        "xlrd==1.1.0",
        "xlsxwriter==0.9.8",
        "factory-boy==2.10.0",
        "psycopg2-binary==2.7.4",
        "django-admin-tools==0.8.1"
    ]

Sorry, now I understand. By default it will only work if the related ordering parameter is included in ordering_fields. But for a more general solution the patch is still required.

Hi there, I am relatively new to DRF and was looking for a way to add ordering to my order fields without exposing the structure of our database. For example, if I wanted to look for z, I would need to write x__y__z, and passing that into the endpoint as a parameter exposes the structure. Is this what I am looking for if I only wanted to say z and have the function check to see if it is a related field in the ORM?

You could try overriding OrderingFilter.get_ordering. Something like...

class CustomOrderingFilter(OrderingFilter):
    def get_ordering(self, request, queryset, view):
        ordering = super().get_ordering(request, queryset, view)
        field_map = {
            'z': 'x__y__z',
        }
        return [field_map.get(o, o) for o in ordering]

@rpkilby Thank you so much! This is exactly what I was looking for. I appreciate the help.

Here's a solution I put together:

class RelatedOrderingFilter(filters.OrderingFilter):
    _max_related_depth = 3

    @staticmethod
    def _get_verbose_name(field: models.Field, non_verbose_name: str) -> str:
        return field.verbose_name if hasattr(field, 'verbose_name') else non_verbose_name.replace('_', ' ')

    def _retrieve_all_related_fields(
            self,
            fields: Tuple[models.Field],
            model: models.Model,
            depth: int = 0
    ) -> List[tuple]:
        valid_fields = []
        if depth > self._max_related_depth:
            return valid_fields
        for field in fields:
            if field.related_model and field.related_model != model:
                rel_fields = self._retrieve_all_related_fields(
                    field.related_model._meta.get_fields(),
                    field.related_model,
                    depth + 1
                )
                for rel_field in rel_fields:
                    valid_fields.append((
                        f'{field.name}__{rel_field[0]}',
                        self._get_verbose_name(field, rel_field[1])
                    ))
            else:
                valid_fields.append((
                    field.name,
                    self._get_verbose_name(field, field.name),
                ))
        return valid_fields

    def get_valid_fields(self, queryset: models.QuerySet, view, context: dict = None) -> List[tuple]:
        valid_fields = getattr(view, 'ordering_fields', self.ordering_fields)
        if not valid_fields == '__all_related__':
            if not context:
                context = {}
            valid_fields = super().get_valid_fields(queryset, view, context)
        else:
            valid_fields = [
                *self._retrieve_all_related_fields(queryset.model._meta.get_fields(), queryset.model),
                *[(key, key.title().split('__')) for key in queryset.query.annotations]
            ]
        return valid_fields
````

Then I add this to wherever I want to be able to order by all related fields:
```python
filter_backends = (RelatedOrderingFilter,)
ordering_fields = '__all_related__'