Django-filter: [OrderingFilter] Question / Feature request: Allow to specify arbitrary Django ORM expression for ordering

HI @earshinov. Thanks for the report. Very interesting. Let me have a think about it. Ping me in March if I've not replied by then. 🙂

Another idea: it is possible to go a step further and allow a filter to specify not one, but multiple expressions or field names.

Before (option 1):

class OrderOrderingFilter(ddf.OrderingFilter):

    def __init__(self):
        super().__init__(fields={
            'user': functions.Concat(
                models.F('user__first_name'),
                models.Value(' '),
                models.F('user__last_name'),
            ),
        })

After (option 2):

class OrderOrderingFilter(ddf.OrderingFilter):

    def __init__(self):
        super().__init__(fields={
            'user': ('user__first_name', 'user__last_name'),
        })

In both cases ordering=user orders data by user's first name first, user's second name second, but option 2 is easier to write, and perhaps more efficient when it comes to DB queries.

Hello @carltongibson ,

You asked me to ping you in Match. It is May already :)

Returning back to my proposal on ordering by expressions (one or multiple), I really think it is a missing part of the puzzle.

I see four basic table operations:
a. serialization
b. ~pagination~ (irrelevant to this discussion)
c. filtering
d. sorting

Using DRF and django-filter it is possible to implement all of these with support for related models, except for sorting:

a. For serialization, we have got nested serializers. Also, if we need to return an aggregated field (think of count), we can .annotate() the queryset:

from django.db.models import aggregates, functions, F, Value

class ModelViewSet(...):

    def get_queryset(self):
        qs = Model.objects.all()

        if self.action == 'list':
            qs = qs.annotate(
                author_full_name=functions.Trim(functions.Concat(
                    F('author__first_name'),
                    Value(' '),
                    F('author__last_name'),
                )),
                submodel_count=aggregates.Count('submodel'))
            )

        return qs

c. For filtering with Django expressions, one can implement a custom filter (see an example below). Filtering by an aggregated field (submodel_count) is possible using the ordinary scalar filters (filters.NumberFilter).

from django_filters import filters
from django_filters.rest_framework import FilterSet

class ModelFilterSet(FilterSet):

    author = UserFilter(field_name='author', label='author', lookup_expr='icontains')


class UserFilter(filters.Filter):
    """A django_filters filter that implements filtering by user's full name.

    def filter(self, qs: QuerySet, value: str) -> QuerySet:
        # first_name <lookup_expr> <value> OR last_name <lookup_expr> <value>
        return qs if not value else qs.filter(
            Q(**{f'{self.field_name}__first_name__{self.lookup_expr}': value}) |
            Q(**{f'{self.field_name}__last_name__{self.lookup_expr}': value})
        )

d. There is no solution for sorting :-(

Here is the complete implemention of our custom OrderingFilter, which we have been using so far:

class OrderingFilter(filters.BaseCSVFilter, filters.ChoiceFilter):
    """An alternative to :class:`django_filters.filters.OrderingFilter` that allows to specify any Django ORM expression for ordering.

    Usage example:

    .. code-block:: python

      from django.db import models
      from django.db.models import aggregates, expressions, fields

      import ddl

      class OrderOrderingFilter(ddl.OrderingFilter):
        def __init__(self):
          super().__init__(fields={

            # a model field
            'created': 'created'

            # an expression
            'submitted': expressions.ExpressionWrapper(
                models.Q(submitted_date__isnull=False),
                output_field=fields.BooleanField()
            ),

            # multiple fields or expressions
            'user': ('user__first_name', 'user__last_name'),

            # a complete field descriptor with custom field label
            'products': {
              'label': 'Total number of items in the order',
              # if not specified, `desc_label` would be derived from 'label' anyway
              'desc_label': 'Total number of items in the order (descending)',
              'expr': aggregates.Sum('order_lines__quantity'),
              # it is also possible to filter by multiple fields or expressions here
              #'exprs': (...)
            },
          })

    For more information about expressions, see the official Django documentation at
    https://docs.djangoproject.com/en/dev/ref/models/expressions/
    """

    _fields: typing.Mapping[str, 'FieldDescriptor']

    def __init__(self, fields: typing.Mapping[str, typing.Any]):
        self._fields = normalize_fields(fields)
        super().__init__(choices=build_choices(self._fields))

    # @override
    def filter(self, qs: models.QuerySet, value: typing.Union[typing.List[str], None]):
        return qs if value in EMPTY_VALUES else qs.order_by(*(itertools.chain(*(self.__get_ordering_exprs(param) for param in value))))

    def __get_ordering_exprs(self, param) -> typing.Union[None, typing.List[expressions.Expression]]:
        descending = param.startswith('-')
        param = param[1:] if descending else param
        field_descriptor = self._fields.get(param)
        return () if field_descriptor is None else \
            field_descriptor.exprs if not descending else \
            (expr.desc() for expr in field_descriptor.exprs)



def normalize_fields(fields: typing.Mapping[str, typing.Any]) -> typing.Mapping[str, 'FieldDescriptor']:
    return dict((
        param_name,
        FieldDescriptor(param_name, field if isinstance(field, collections.Mapping) else {'exprs': normalize_exprs(field)})
    ) for param_name, field in fields.items())

def normalize_exprs(exprs: typing.Union[
        typing.Union[str, expressions.Expression],
        typing.List[typing.Union[str, expressions.Expression]]
    ]) -> typing.List[expressions.Expression]:
    # `exprs` is either a single expression or a Sequence of expressions
    exprs = exprs if isinstance(exprs, collections.Sequence) and not isinstance(exprs, str) else (exprs,)
    return [normalize_expr(expr) for expr in exprs]

def normalize_expr(expr: typing.Union[str, expressions.Expression]) -> expressions.Expression:
    return models.F(expr) if isinstance(expr, str) else expr


descending_fmt = _('%s (descending)')


class FieldDescriptor:
    exprs: typing.List[models.Expression]

    def __init__(self, param_name: str, data: typing.Mapping[str, typing.Any]):
        exprs = data.get('exprs') or data.get('expr')
        if not exprs:
            raise ValueError("Expected 'exprs' or 'expr'")
        self.exprs = normalize_exprs(exprs)
        self.label = data.get('label', _(pretty_name(param_name)))
        self.desc_label = data.get('desc_label', descending_fmt.format(self.label))


def build_choices(fields: typing.Mapping[str, 'FieldDescriptor']):
    choices = []
    for param_name, field_descriptor in fields.items():
        choices.append((param_name, field_descriptor.label))
        choices.append((f'-{param_name}', field_descriptor.desc_label))
    return choices

_Update_: Included the implemention of normalize_fields and other helper methods.

Good. 😀

Thanks for the ping.

Tbh I haven’t had a moment to think about this.

In all your great thought here do you have a suggestion for a smallish change that we might make? (Maybe it’s just easier to push forward with a PR).

Yeah, it is not immediately obvious how to make such changes backward compatible, I will have to give it a think. Do not expect a PR soon (will be on vacation in places where Internet is scarce).

That’s fine. 🙂

There’s no rush here. Better we’ll thought through, if at all.

@carltongibson , Okay, we contemplated a little bit, but could not find an acceptable way of combining the old and new implementation of OrderingFilter.

If we are going to put everything into one class, the biggest problem is that in the old implementation the fields dict stores model_field-parameter_name pairs, whereas in the new implementation it is the opposite parameter_name-model_field (it needs to be, because instead of model_field an expression can be passed, which can only be stored in a value, but not in a key in a dictionary).

Technically, it is possible to overcome this problem by interpreting the dictionary "in the old way" if it contains only strings, and "in the new way" otherwise. In this case the user will have to "flip" dictionary entries when a need in ordering by expression arises. And be careful to "flip" the dictionary entries back, should the ordering expression be removed... This sounds like a terrible user experience to me. It will also make the implementation convoluted.

Do you think it is possible to introduce the new OrderingFilter along with the old one under a different name, like ExpressionOrderingFilter?

This is great, exactly at the moment I needed it! Thanks @earshinov!

How would I do something like:

MyModel.objects.all().order_by(F('price').desc(nulls_last=True))

With your ordering filter?

    o = ExpressionOrderingFilter(

        fields={
                'price': F('price').desc(nulls_last=True)
        }

    )

seems to work great!
Now the only thing left is figuring out how to combine mutliple filters, ak. 'price' and 'stock' into one.

ach - started responding to this but my computer went kaput.

I think the proposed changes here are sensible. The model_field-parameter_name pair made sense at the time since we typically derive the exposed params/forms/etc from the model, but there's no reason why this is necessary. Swapping the mapping would make sense, and allow us to take advantage of more complex expressions.

Also, I don't think deprecation process would be insanely difficult, and I'd be happy to help throw in on it. Basically, fields and field_labels would transition to params and param_labels. It's easy enough to convert from one to the other, while retaining backwards compatibility, and raising a deprecation notice for users using the old arguments.

One thing to consider is automatic conversion of complex expressions for the descending case (param vs -param). e.g., if the ascending is .asc(nulls_last=True)., should the inverse of this be .desc(nulls_first=True), or should nulls remain last regardless of sort direction?

OK, super @rpkilby.

Happy to see us move forwards here. My main concern is that we document properly what we're doing. Users already find the df docs a little _terse_ shall we say 🙂 — Happy to add API but we need to make sure it's clear.

Such a thing doesn't have to be a one-person effort.

@rpkilby , @carltongibson , If you need my help in integrating my changes into the project, I think I can put in some time. But I need directions. Where and how do I start?

Hi @earshinov. I'd start by drafting the docs. What's the story we tell? From there the code changes. A PR, even if just a draft gives us something to talk about.

Also, test cases if you have them. They may need to be adjusted to match the final changes, but having the various cases thought through would be very helpful (e.g., converting .asc to .desc, handling nulls_fist / nulls_last arguments, etc..).