Jinja: 2.9 regression when assigning a variable inside a loop

While the current behavior is incorrect, the old behavior is definitely incorrect as well. The {% set %} tag was never intended to override outer scopes. I'm surprised it did in some cases. Not sure what to do here.

The behavior I would assume were correct is an output like [-1:0] [-1:1] [-1:2] [-1:3] [-1:4] -1.

In this specific case I resolved it by rewriting it to use groupby.

I have the feeling this is a somewhat common use case - also stuff like {% set found = true %} in a loop and then checking it afterwards. It's surely likely to break things for people when this stops working...

I'm shocked this worked before. Did this always work?

apparently yes:

>>> import jinja2
>>> jinja2.__version__
'2.0'
>>> jinja2.Template('{% for x in range(5) %}[{{ a }}:{% set a = x %}{{ a }}] {% endfor %}{{ a }}').render()
u'[:0] [0:1] [1:2] [2:3] [3:4] '

Great. Because the issue here is that this is absolutely not sound. Which variable is it supposed to override? What if there is a function scope in between. eg: a macro or something like that. I have no idea how to support this now.

hmm maybe similar to python's scoping assuming no nonlocal was used)? ie don't allow overriding something defined in an outer scope (ie if there's a macro inbetween) but do allow overriding it otherise?

Apparently before this was caught down with a template assertion error:

>>> Template('{% set x = 0 %}{% for y in [1, 2, 3] recursive %}{{ x }}{% set x = y %}{% endfor %}').render()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
jinja2.exceptions.TemplateAssertionError: It's not possible to set and access variables derived from an outer scope! (affects: x (line 1)

the behavior seems to be different with/without recursive (2.8.1 gives me an UnboundLocalError error with recursive and '012' without)

The unbound local error should be resolved on master.

I'm not sure how to progress here. I really dislike that it was apparently possible to do this sort of thing.

With the last changes I'm going to close this as "works as intended". If there is further fallout from this we can investigate alternatives again.

Got sent to this issue (see #656) after this change blew up my blog template on upgrading from v.2.8.1 to v2.9.4.

I was using it keep track if various pieces of data were changing between loop iteration. I was able to fix it because I wrote the original template code (see https://github.com/MinchinWeb/seafoam/commit/8eb760816a06e4f0382816f82586204d1e7734fd and https://github.com/MinchinWeb/seafoam/commit/89d555dbd6a2f256471d43e4184f09512694e5f2), but I doubt I would have been able to otherwise. The new code is harder to follow as the comparisons are now done in-line. For example, my old code (v.2.8.1):

{%- set last_day = None -%}

{% for article in dates %}
    {# ... #}
    <div class="archives-date">
        {%- if last_day != article.date.day %}
            {{ article.date | strftime('%a %-d') }}
        {% else -%}
            &mdash;
        {%- endif -%}
    </div>
    {%- set last_day = article.date.day %}
{% endfor %}

and the new code, with in-line comparisons (v2.9.4):

{% for article in dates %}
    {# ... #}
    <div class="archives-date">
        {%- if ((article.date.day == dates[loop.index0 - 1].date.day) and
                (article.date.month == dates[loop.index0 - 1].date.month) and 
                (article.date.year == dates[loop.index0 - 1].date.year)) %}
                    &mdash;
        {% else -%}
                    {{ article.date | strftime('%a %-d') }}
        {%- endif -%}
    </div>
{% endfor %}

So I just wanted to say that the "feature" (or "hack", if you prefer) is used and is already missed.

If the scoping issues are too complex to figure out sensibly at the moment, could something (at a minimum) be added to the changelog so it bites less people unaware?

I was not aware this is so widely abused :-/ Annoyingly there is really no way to make this work in any reliable way. However I wonder if we can isolate the common use cases here and introduce a nicer api.

In particular maybe we want to add something like this (loop.changed_from_last):

{% for article in dates %}
    <div class="archives-date">
        {%- if loop.changed_from_last(article.date.day) %}
            {{ article.date | strftime('%a %-d') }}
        {% else -%}
            &mdash;
        {%- endif -%}
    </div>
{% endfor %}

changed_from_last sounds good - at least functionality-wise, the name itself is a bit awkward IMO. Maybe just changed would be clear enough?

I guess the first element would always be considered "changed" no matter what it is? If that behavior is not OK for someone they could always add a not loop.first check anyway.

Maybe previous_context just holds the entire previous context, rather than trying to decide what the users will do with it.

just previous or prev? "context" sounds rather confusing in this context (no pun intended)

.....and now I can already imagine someone asking for a next :/

Or maybe an explicit statement for writing outside scope: set_outer or something.

i was thinking this:

class LoopContextBase(object):

    def __init__(self, recurse=None, depth0=0):
        ...
        self._last_iteration = missing

    def changed(self, *value):
        if self._last_iteration != value:
            self._last_iteration = value
            return True
        return False

@davidism we cannot do set_outer without breaking the entire scoping system. This would majorly screw up the entire thing.

Yeah, figured that would be the case. I'm anticipating "what if I want to know if the current value is greater than the previous", or something similar. But I like the changed method too.

As an additional data point, it fell on my feet for a case where I need to inject a special class into generated HTML, but only for the first element of the iterated list that does not also have some specific property set. I can't modify the iterated data. I can't use loop.first (as much as I'd love to) or compare with anything from the last element to reliably do what I need to do here, which led to experiment with the evil construct which worked perfectly (and it was not clear to me that it was actually me abusing a bug).

Additionally I offer extension capabilities through third party plugins and have no way to police how the authors structure their templates, causing sudden breakage for end users after update to Jinja 2.9+. I've pinned the latest 2.8 version now in my program for that matter to remain backwards compatible (as my versioning scheme promises), until I can find a way to get authors to update their templates.

Could someone please clarify why jinja2 scopes cannot work exactly as python works? i.e., jinja2 templates correspond to python modules, macros correspond to functions (and have their own scope), for loops don't have their own scope and so forth. While @mitsuhiko says jinja 2.8 and below had lots of scoping bugs, it was more intuitive for me to understand how scoping works.

For python programmers, the behavior from first post (Jinja 2.8) is obvious. Same applies to https://github.com/pallets/jinja/issues/660, I don't understand why python library should implement javascript behavior (I understand that python's handling of default args is not ideal, but any python developer is aware of it).

Alternatively, there should be a document which describes how scoping in jinja works so that we don't have to guess.

Also please don't take my comment negatively, I'm very grateful for jinja.

@roganov a few notes on this:

Could someone please clarify why jinja2 scopes cannot work exactly as python works

Because Python's scoping is in my opinion really bad, particularly in templates because you can easily accidentally override important information.

While @mitsuhiko says jinja 2.8 and below had lots of scoping bugs, it was more intuitive for me to understand how scoping works.

Note that this was the an exception due to a bug and it only worked in some limited situations and purely by accident. Jinja had always the same documented scoping rules and this behavior was never documented. At all times it was said that variables do not propagate to outer scopes. You can easily see this because after the end of the loop the variable was never set.

Alternatively, there should be a document which describes how scoping in jinja works so that we don't have to guess.

I improved the docs already a few days back for this

What if there was a way to explicitly pass a variable into a loop? Perhaps syntax like this (borrowing from text filters):

{%- set last_day = None -%}

{% for article in dates | pass_variable ( last_day ) %}
    {# ... #}
    <div class="archives-date">
        {%- if last_day != article.date.day -%}
            {{ article.date | strftime('%a %-d') }}
        {%- else %}
            &mdash;
        {% endif -%}
    </div>
    {%- set last_day = article.date.day %}
{% endfor %}

Or is there a way to do something similar to scoped as it currently applies to macros?

Another use case would be maintaining some sort of counter between loops. For example, how many rows total? how many rows meet some criteria? The total of some property for selected rows (to print a total row)?

Filter syntax would not be possible here since filters are already possible on the iterable.

A possible syntax for this that wouldn't look to bad could be this:

{% for article in dates with last_day %}

Especially since with already does scoping stuff in Jinja when used e.g. as {% with %}. OTOH, here it would be the opposite since with blocks open a new scope, not make outer scope vars accessible..

Not sure though if that's a feature Jinja needs or not.

No syntax will change this because it would still subvert the scoping rules. You cannot do that with either the current compilation or id tracking system. Even if it works for that simple case, what happens if one has a for loop that is recursive. What if someone defines a macro in a for loop. What if a call block appears in a for loop.

I think it would make more sense to introduce a global object that acts as a storage namespace and can then be modified with . Eg:

{% set foo = namespace() %}
{% set foo.iterated = false %}
{% for item in seq %}
  {% set foo.iterated = true %}
{% endfor %}

However this would require the set tag to fundamentally change.

@mitsuhiko and that sort of pattern is already in use with the do tag: http://stackoverflow.com/a/4880398/400617

FWIW, the solution with do is extremely awful. Just like the workaround people use in Python 2 when they'd need nonlocal...

I completely agree, just pointing out that it's out there. And like Alex pointed out, a lot of these problems can be rewritten, either with filters or by putting some of the logic in Python.

Another (ugly) hack is to use a list, append, and pop: http://stackoverflow.com/a/32700975/4276230

Hi, I have been using this code since about 2 years, now it breaks and from new documentation and this discussion I don't think there is any other way to do what I am trying to do here. If not please correct me.

{% set list = "" %}
{% for name in names %}
{% if name.last is defined %}
{% set list = list + name.last + " " %}
{% if loop.last %}
{{ list.split(' ')|unique }}
{% endfor %}

@pujan14

{{ names|map(attribute="last")|select|unique }}

Although since unique isn't a built-in filter, you could always just add another filter that does the whole thing, since you're already adding the unique filter.

Hi,
I understand that there are a lot of "ugly" examples due to this, but could you please advise how to increment a variable on a for loop in jinja template in elegant/right way? Because my code has been broken too.

Why do you need to do it at all? Also, please include your code.

{% for state in states.sensor -%}
{% if loop.first %}
{% set devnum = 0 %}
{% endif -%}
{%- if state.state == "online" %}
{% set devnum = devnum + 1 %}
{%- endif -%}
{% if loop.last %}
{{ devnum }}
{% endif -%}
{%- endfor -%}

could you please advise how to increment a variable on a for loop in jinja template in elegant/right way?

You were never supposed to do (or be able to do) this at all. So the answer is don't do that. We are however interested why template authors seem to have the need to do that.

Jinja already enumerates the loop. {{ loop.index0 }}

@davidism I need only those that meets a condition

Write a filter that yields devnum, sensor tuples exactly the way you need them. Or calculate it in Python and pass it to the template. Or use the example below. This goes for every other example I've seen.

@Molodax you can do this:

{{ states.sensor|selectattr('state', 'equalto', 'online')|sum }}

didn't you mean |count?

Sorry yes, count.

@mitsuhiko, thank you for your support, unfortunately, it does not work.
Perhaps, jinja2 (filters) has been implemented with some limitations in a project that use (Home assistant). But it did work with the code provided by me before. Pity.

Hi folks. You're asking for example code that works under 2.8, so here's mine:

{% set count = 0 %}
{% if 'anchors' in group_names %}
nameserver 127.0.0.1
{% set count = count+1 %}
{% endif %}
{% for resolver in resolvers %}
{% if count < 3 %}
{% if resolver|ipv6 and ansible_default_ipv6.address is defined %}
nameserver {{ resolver }}
{% set count = count+1 %}
{% elif resolver|ipv4 and ansible_default_ipv4.address is defined %}
nameserver {{ resolver }}
{% set count = count+1 %}
{% endif %}
{% endif %}
{% endfor %}

I don't know how I would do this without a global "count" variable that I can reference in 2 separate loops. Do you have any suggestions that will allow this to work under both 2.8 and 2.9?

@davidism Your solution is great but I am trying to achieve this.
Create 2 lists as following

{% set list1 = name|default()|map(attribute="last")|select|list %}
{% set list2 = name|default()|map(attribute="age")|select|list %}

and then merge them into list3, which should look like following and finally applying unique(ansible filter) on list3
last1-age1
last2-age2
last3-age3
last4-age4
last5-age5
last6-age6

From what I gathered map does not work with multiple attributes #554
I am using jinja2 via ansible and so adding doing something in python beforehand is not a good idea for me.

@pujan14 @aabdnn @Molodax did this pattern come from official Ansible documentation, or is it something you came up with or found elsewhere? Either way, it might be easier to report this to Ansible, since they understand how their product should be used and could possibly come up with more relevant solutions.

@davidism the template I presented above did not come from any Ansible documentation. Ansible doesn't document Jinja2 specifically. I just created this template myself by reading Jinja2 documentation, and it worked, so I put it into production. I assumed that Jinja2 made variables global.

If it's not officially documented then I'm more inclined to close this as it was originally. I'll reiterate that Ansible may be able to help you more in this regard.

@davidism I played around with loops in new jinja 2.9. with 2.9 even when I create/define a new variable inside a loop it gets cleared/deleted at end of every iteration.
Here is an example.

{% for name in names %}
{% if loop.first %}
{% set list = "" %}
{% endif %}
{% if name.first is defined and name.last is defined and not name.disabled %}
{% set list = list + name.first|string + "-" + name.last|string %}
{% if loop.last %}
{% for item in list.split(' ')|unique %}
{{ item }}
{% endfor %}
{% else %}
{% set list = list + " " %}{% endif %}
{% endif %}
{% endfor %}

This might not be the best way to do this but here from my understanding I am not breaking any scoping rules.

Had this issue in 2.8 and superior

Here goes a test case:

import unittest
from jinja2 import Template

TEMPLATE1 = """{% set a = 1 %}{% for i in items %}{{a}},{% set a = a + 1 %}{% endfor %}"""

class TestTemplate(unittest.TestCase):

  def test_increment(self):
    items = xrange(1,10)
    expected='%s,' % ','.join([str(i) for i in items])
    t = Template(TEMPLATE1)
    result = t.render(items=items)
    self.assertEqual(expected,result)

unittest.main()

Use loop.index instead.

I think providing access to the previous loop value through an attribute of the loop object is the only good solution for this. I just discovered this snippet in our project which couldn't be solved by just checking if the last object is different than the current one and groupby also doesn't work there since it's more than just a trivial item/attribute access to get the key:

{% set previous_date = none %}
{% for item in entries -%}
    {% set date = item.start_dt.astimezone(tz_object).date() %}
    {% if previous_date and previous_date != date -%}
        ...
    {% endif %}
    {% set previous_date = date %}
{%- endfor %}

Yeah that sounds like an idea.

What about adding set(key, value) and get(key) methods to the loop object? Then people can store whatever they want across the loop.

Had the same idea, but then couldn't find any non-ugly cases where this would be needed. And I could already see someone asking for setdefault, pop and other dict-like methods.

@davidism @ThiefMaster i was thinking of just having a storage object available. Like this:

{% set ns = namespace() %}
{% set ns.value = 42 %}
...{% set ns.value = 23 %}

Obviously set can't set attributes at the moment but this could easily be extended. Since no reassignment to the namespace itself happens this is quite safe to implement.

Seems fine to me, it's the same solution as nonlocal for Py 2. Alternatively, automatically set up loop.ns, although that wouldn't be available outside the loop.

What I dislike with the namespace is that it'll feel very tempting to do {% set obj.attr = 42 %} with obj being something that is not a namespace() - something that I think shouldn't work.

Otherwise it looks like an interesting idea, even though I think previtem / nextitem / changed() cover "simple" cases nicely without the "noise" of having to define a new object in the template.

@ThiefMaster it would not work. The way I would see this working is that attribute assignments go through a callback on the environment which would only permit modifications to namespace objects.

ok, so at least no risk of templates causing unexpected side-effects on objects passed to them.

still a bit wary of the things people might do...

{% macro do_stuff(ns) %}
    {% set ns.foo %}bar{% endset %}
    {% set ns.bar %}foobar{% endset %}
{% endmacro %}

{% set ns = namespace() %}
{{ do_stuff(ns) }}

Actually, I think a new block like {% namespace ns %} would be better to define one than a callable - a variable named namespace doesn't sound like something very unlikely to be passed to a template, and while it would probably simply prevent you from using the namespace feature in that template (just like shadowing a builtin in Python) it feels a bit dirty...

Do you have a workaround for this issue or do we have to wait for previtem/nextitem in 2.9.6?
Some of my saltstack templates are broken now.

As has been demonstrated to varying degrees above, you might not even need to do what you're doing. Otherwise, yes, you need to wait if you want to use 2.9. It was never supported before, it just happened to work.

We will not be reverting to the old behavior. While it worked in simple cases, it was not correct and was never documented as supported. While I understand that it is a breaking change, it occurred in a feature release (second number change) which is how we've always managed these changes. Pin the version until a fix is released if you need to keep relying on the old behavior.

Locking this because everything that needs to be said has been said. See #676 and #684 for the fixes that are currently being considered.