Numpy: ndarray dump function (and straight cPickle) fails for large arrays (Trac #1803)

I am also facing this problem.

@zhlsk didn't try further or investigated if there is a fix in python, but I think this is not a numpy but a python problem, since: cPickle.dump(a.tostring(), f) also fails for me just the same and thats just a large string then and has nothing to do with numpy. Using the numpy save functions should not have these kind of problems though.

@seberg Thanks for your reply. You're right. It's a python problem : http://bugs.python.org/issue11564

In principle we could probably work around the bug anyway by passing large
arrays to the pickler as a series of objects, each of which fit in the
limits, and reconstruct the array at unpickle time. It sounds like the
limit is on the size of a single byte array.
On 14 Dec 2012 22:11, "zhlsk" [email protected] wrote:

@seberg https://github.com/seberg Thanks for your reply. You're right.
It's a python problem : http://bugs.python.org/issue11564

—
Reply to this email directly or view it on GitHubhttps://github.com/numpy/numpy/issues/2396#issuecomment-11394338.

Any progress on this? It seems that numpy.save and numpy.savetxt both are affected by this issue. Any workarounds?

It's a bug in Python, the fix is to upgrade to a newer version (it's fixed in Python 3.3), and the workaround is to pickle your array yourself in smaller parts, or use some other file format than Python pickles.

np.savetxt is not affected by this, and np.save is only affected for object arrays.

Closing.

I'm getting this when trying to cPickle a 11,314 x 8,463,980,778 sparse matrix with 352,451,719 stored elements in scipy.sparse.csr_matrix format. Python version: Python 2.7.10 |Anaconda 2.3.0 (x86_64)| (default, May 28 2015, 17:04:42)