Numpy: np.left_shift and np.right_shift fail on np.uint64 scalar types (Trac #1931)

_@charris wrote on 2011-08-13_

This is because the shift number is converted as a signed type and there is no signed integer type big enough to hold a uint64. The shift operators need to be special cased, they can also fail on Intel because the hardware masks out the unneeded bits, i.e., for int32 only the lower 5 bits are looked at.

_trac user parejkoj wrote on 2012-07-27_

I was able to code around this with the following lambda:

lshift = lambda x,s: np.uint64(x)*np.uint64(2**s)

This works, but is not ideal. Getting this fixed in numpy would be very useful.

it's five years later... isn't this an important issue? unsigned integers should have no ambiguity about shifts.

The easier work-around is to use an unsigned integer for the shift:

np.uint64(5) << np.uint64(1)

As @charris mentioned above, the problem is in the way dtypes are made uniform. It is not that trivial to fix generically (see also #5668), though perhaps the shift operators can be special-cased. (PR welcome, though, as said, this is not trivial, but requires fair understanding of how the internal loops in ufucs are decided on...).

The easier work-around is to use an unsigned integer for the shift:

np.uint64(5) << np.uint64(1)

...which does work if you know you are operating on numpy integers, but if you are using generic code (esp. something that has already been written in another module and is out of your control) like

def f(x):
    return x | (x >> 1)

then you can't make it work on both regular Python integers and np.uint64 types.

I'm not trying to state that this is trivial (wayyyy above my skillset) but I do think it should be a high priority.

@jason-s - I agree that the problem is very annoying... Sadly, there not that many of us well-enough versed in how the ufuncs work internally...

Perhaps one could at least special-case scalars for the shift in the method, though, instead of relying on the general ufunc machinery. @charris: was this what you had in mind (if you can remember that after 6 years ;-).

See also #8002.

Personally, if we could do everything over again, I would favor redoing all the casting/coercion so that numpy casting essentially behaves like C casting, and follows the spirit of C-casting for cases that don't exist in C. The numpy casting rules are kind of weird and sometimes confusing (eg the conversion of uint64 + int64 to float64). But we've discussed this a lot on the list, it seems hard to change.

When I come across situations like this (see my comment there) I just wrap every single value in np.uint64.