Numpy: genfromtxt won't unpack if dtype=None

Created on 21 Feb 2014 · 6Comments · Source: numpy/numpy

Not sure if this is a bug or a feature but if I use:

f_data = np.genfromtxt('file.dat', dtype=None, unpack=True)

where file.dat is a standard data file with several columns (ie: http://pastebin.com/ihEW2dxS) the result is not transposed as one would expect having set unpack=True.

For example, the line f_data = np.loadtxt('file.dat', unpack=True) works as expected.

00 - Bug numpy.lib good first issue

Source

Gabriel-p

👍2

Most helpful comment

I also experience this. loadtxt is a bit more clever:

if unpack:
        if len(dtype_types) > 1:
            # For structured arrays, return an array for each field.
            return [X[field] for field in dtype.names]
        else:
            return X.T

Why doesn't genfromtxt do this?

moi90 on 22 Dec 2017

👍2

All 6 comments

Looks like a bug, marking it so until informed otherwise.

charris on 25 Feb 2014

I investigated this a little bit.

The result of @Gabriel-p's code is a shape (12,) flex array. The code calls output.T on it, but since it's one dimensional this is a no-op.

The biggest issue that dtype=None specifies that it will be flex array. Each column has a separate inferred type. If we were to truly transpose this result, each row would need to have a separate type. AFAIK, the only way to do that is to have a list or tuple of homogeneous arrays.

Since numCols << numRows in the text file, using a list or tuple shouldn't be a big deal. If implemented that way, it would unpack for x, y, z = np.genfromtxt(...) as specified in the docs, and solve @Gabriel-p's case.

However, genfromtxt is already over 600 lines. Any thoughts?

ddasilva on 3 Mar 2014

Just got bitten by this, too. I made a test case, see

https://github.com/hsgg/numpy.git

hsgg on 16 May 2014

I also experience this. loadtxt is a bit more clever:

if unpack:
        if len(dtype_types) > 1:
            # For structured arrays, return an array for each field.
            return [X[field] for field in dtype.names]
        else:
            return X.T

Why doesn't genfromtxt do this?

moi90 on 22 Dec 2017

👍2

Any update on this issue? It seems like it hasn't changed since 2014.