Numpy: Unable to import numpy in AWS Lambda function

To isolate the issue, I retried the above with:

def main(event, context):

    a = 100
    print(a)
    return

and

import numpy as np

def main(event, context):

    a = np.array(100)
    print(a)
    return

Both files were zipped a package directory containing a targeted installation of numpy according to https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html. After zipping, the files were ~16MB, so they had the numpy installation. The first example that did not import numpy runs fine when testing the Lambda function, but the second errors with the original error message

That AWS guide isn't telling you the whole story. The Amazon Linux distribution isn't compatible with a regular NumPy install like from PyPi or conda-forge.

I suggest you follow one of these guides or use a zip file from one of these repos:
https://medium.com/@samme/setting-up-python-3-6-aws-lambda-deployment-package-with-numpy-scipy-pillow-and-scikit-image-de488b2afca6
https://medium.com/@korniichuk/lambda-with-pandas-fd81aa2ff25e
https://blog.orikami.nl/building-scipy-pandas-and-numpy-for-aws-lambda-python-3-6-cba9355b44e9
https://github.com/pbegle/aws-lambda-py3.6-pandas-numpy
https://github.com/vitolimandibhrata/aws-lambda-numpy

This is not a NumPy bug, so I'll close the issue.

Thanks,

I appreciate your response. FYI I was able to get it working by initiating an EC2 instance with the AWS Linux OS and installed the libraries there. Everything worked great.

Thanks for the help,
Lee

Get Outlook for iOShttps://aka.ms/o0ukef

From: Ralf Gommers notifications@github.com
Sent: Saturday, May 4, 2019 6:52 AM
To: numpy/numpy
Cc: Lee James; Author
Subject: Re: [numpy/numpy] Unable to import numpy in AWS Lambda function (#13465)

That AWS guide isn't telling you the whole story. The Amazon Linux distribution isn't compatible with a regular NumPy install like from PyPi or conda-forge.

I suggest you follow one of these guides or use a zip file from one of these repos:
https://medium.com/@samme/setting-up-python-3-6-aws-lambda-deployment-package-with-numpy-scipy-pillow-and-scikit-image-de488b2afca6https://urldefense.proofpoint.com/v2/url?u=https-3A__medium.com_-40samme_setting-2Dup-2Dpython-2D3-2D6-2Daws-2Dlambda-2Ddeployment-2Dpackage-2Dwith-2Dnumpy-2Dscipy-2Dpillow-2Dand-2Dscikit-2Dimage-2Dde488b2afca6&d=DwMCaQ&c=0hefKdg9jtsMu47wpF0ovg&r=2qmZ2aA-JTvfrp24tI1PaA&m=o4NpaNDNU2z8VYtFVRO3Z-KgjOittvdP3EKrsYfomhs&s=ZKkVD6L0iXq9wAG929BSTikIAEun9eeS4ZdgUw1Q8oY&e=
https://medium.com/@korniichuk/lambda-with-pandas-fd81aa2ff25ehttps://urldefense.proofpoint.com/v2/url?u=https-3A__medium.com_-40korniichuk_lambda-2Dwith-2Dpandas-2Dfd81aa2ff25e&d=DwMCaQ&c=0hefKdg9jtsMu47wpF0ovg&r=2qmZ2aA-JTvfrp24tI1PaA&m=o4NpaNDNU2z8VYtFVRO3Z-KgjOittvdP3EKrsYfomhs&s=PQOBXsoiGwGl3XWjQFqEbAK77Kg9THTSlEZOmLQNiJo&e=
https://blog.orikami.nl/building-scipy-pandas-and-numpy-for-aws-lambda-python-3-6-cba9355b44e9https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.orikami.nl_building-2Dscipy-2Dpandas-2Dand-2Dnumpy-2Dfor-2Daws-2Dlambda-2Dpython-2D3-2D6-2Dcba9355b44e9&d=DwMCaQ&c=0hefKdg9jtsMu47wpF0ovg&r=2qmZ2aA-JTvfrp24tI1PaA&m=o4NpaNDNU2z8VYtFVRO3Z-KgjOittvdP3EKrsYfomhs&s=jm1X8lQ78YL30-NRN2kXnOGAhMS1LKSowIOFQMCpq5Y&e=
https://github.com/pbegle/aws-lambda-py3.6-pandas-numpyhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_pbegle_aws-2Dlambda-2Dpy3.6-2Dpandas-2Dnumpy&d=DwMCaQ&c=0hefKdg9jtsMu47wpF0ovg&r=2qmZ2aA-JTvfrp24tI1PaA&m=o4NpaNDNU2z8VYtFVRO3Z-KgjOittvdP3EKrsYfomhs&s=708JPsyiAUC0pdCf1QwuLzJ4dEWunFNuuZ_aBrMsakc&e=
https://github.com/vitolimandibhrata/aws-lambda-numpyhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_vitolimandibhrata_aws-2Dlambda-2Dnumpy&d=DwMCaQ&c=0hefKdg9jtsMu47wpF0ovg&r=2qmZ2aA-JTvfrp24tI1PaA&m=o4NpaNDNU2z8VYtFVRO3Z-KgjOittvdP3EKrsYfomhs&s=poqmiFo_VPWUjgf2NEI4vBxzANNWtRm3sXSUypHO88w&e=

This is not a NumPy bug, so I'll close the issue.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_numpy_numpy_issues_13465-23issuecomment-2D489329007&d=DwMCaQ&c=0hefKdg9jtsMu47wpF0ovg&r=2qmZ2aA-JTvfrp24tI1PaA&m=o4NpaNDNU2z8VYtFVRO3Z-KgjOittvdP3EKrsYfomhs&s=J7VXklMAkObDeLGzi_DueXIZ7gya0qrVm9tZhLH9dzI&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AHKHI4N5E7XURJJTX7INB6LPTWIJVANCNFSM4HKX25DQ&d=DwMCaQ&c=0hefKdg9jtsMu47wpF0ovg&r=2qmZ2aA-JTvfrp24tI1PaA&m=o4NpaNDNU2z8VYtFVRO3Z-KgjOittvdP3EKrsYfomhs&s=zMXEEFbVt2bm8P0Cn2aJbHDaSdV0ABZ_hgsaB2AWUWg&e=.

@rgommers Do you know of any python3.7 related tacks on this. I've tried a couple from your list of 3.6 and always end up with the same gripe about the _multiarray lib. I need to make an AWS "layer" since the combination I need (numpy,scipy,pandas) exceeds size allowance of a single function.

@iceback if your error was similar to what I had, which was No module named 'numpy.core._multiarray_umath', this is caused by not installing numpy with the correct OS.

AWS lambda is going to run AWS linux at runtime, so any target-installed packages should be installed with the proper OS. If you target install numpy to a package directory via pip install --target ./package numpy, and you are running another OS (such as macOS), then this will fail because the target installation assumes that the OS will not change from your native OS.

To get around this, you have two options:

1. Perform the target installation from a machine running AWS Linux, such as an EC2 instance. This worked great for me.
2. Figure out how to specify the runtime OS in your target install command. If you find a good way to do this, please share 😄

I'll have to revisit all the things I tried last week which included the @korniichuk and github/pbegle and lord knows what else but always ended up in same spot. Building numpy on an EC2 is not going well. Perhaps wrong Cython version (only 0.27 available on aws linux). Can you elaborate on your #1. How do you get from an installed numpy on one instance to a layer on Lambda?

@iceback Ya this was frustrating when I first did it, but it is definitely possible. I currently run Lambda functions with Python 3.7 with the 3 packages you mentioned.

How I performed step 1. above involved following the steps outlined in Updating a Function with Additional Dependencies, described here. For simplicity sake, I am going to assume that your local python file is named _pony.py_ and the lambda_function is invoked with the _ride_pony_ function.

1. Launch and connect to an EC2 instance, such as an EC2 micro instance. Find instructions here. Ensure that the instance runs AWS Linux.
2. Check that python3 is installed. If it is not installed, install it with sudo yum install python3 -y
3. Make a directory for your local packages. mkdir package
4. Install the libraries you want to use in Lambda. FYI I always call pip as a module to avoid confusion between Python2 and Python3 pips. Also, some installation-order may be preferred in these installations and that may be worth researching.
5. python3 -m pip install --target ./package numpy
6. python3 -m pip install --target ./package pandas
7. python3 -m pip install --target ./package scipy
8. Move into installation directory with cd package
9. Zip the installation directory with zip -r9 ../pony.zip .
10. Now, from your local machine, scp _pony.zip_ from your EC2 instance to your local with the directions here.
11. From your local, zip _pony.py_ into _pony.zip_. zip -g pony.zip pony.py. Your zip file now contains your lambda code along with the python modules needed, which were installed to target with an AWS Linux OS.
12. Upload the zip file to S3 (which allows for larger lambda functions, up to 50 MB). If your code is small, you can skip to step 10.
13. Go into your lambda function (or update from the CLI) to use the code in pony.zip for that lambda function

**Ensure that your lambda function points to & executes your filename and main function, _pony.ride_pony_, if it is not the default _lambda_function.lambda_handler_

Thank you very much. I believe I have a shot! (My understanding is that the top of the zip has to be "python" for a Lambda layer so I'll use that instead of "package")

Shout that how-to out loud and proud! Not sure where I went afoul of the other suggestion (though they were for python3.6) but I'm now back to working on my function code. Thanks a ton.

AWS now also published a layer that includes NumPy and SciPy. From https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-programming-language-and-share-common-components/

_Based on our customer feedback, and to provide an example of how to use Lambda Layers, we are publishing a public layer which includes NumPy and SciPy, two popular scientific libraries for Python. This prebuilt and optimized layer can help you start very quickly with data processing and machine learning applications._

Hm, could there be too much AWX documentation? ;)

On Oct 23, 2019, at 4:29 AM, Ralf Gommers notifications@github.com wrote:

AWS now also published a layer that includes NumPy and SciPy. From https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-programming-language-and-share-common-components/ https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-programming-language-and-share-common-components/
Based on our customer feedback, and to provide an example of how to use Lambda Layers, we are publishing a public layer which includes NumPy and SciPy, two popular scientific libraries for Python. This prebuilt and optimized layer can help you start very quickly with data processing and machine learning applications.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/numpy/numpy/issues/13465?email_source=notifications&email_token=AACY7PDHEVUY6IGQ6IXVXPLQQARRNA5CNFSM4HKX25D2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECA5ACQ#issuecomment-545378314, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACY7PFK3DUQOBMT5QA2ID3QQARRNANCNFSM4HKX25DQ.

What worked for me was using a Linux version of the numpy library (I use macOS). I went to https://pypi.org/project/numpy/#files and downloaded the .whl file in the version i was looking for (for me, it was numpy-1.19.0-cp37-cp37m-manylinux1_x86_64.whl). Next go to the terminal and unzip it by doing 'unzip numpy-1.19.0-cp37-cp37m-manylinux1_x86_64.whl'. This should give you the numpy version that will work on Lambda. Then, zip everything up as you were doing before and upload it. In the end an incredibly frustrating problem was solved pretty simply.

I had same problem, the solution that worked for me is that i uninstalled numpy from my pc,(windows 7).
than i added layer in aws lambda function something called AWSlambda scipy and thats it.

I ran into the same issue with Windows but was able to solve it by either of the following approaches:

• Build the Lambda with AWS SAM (Serverless Application Model) running on Ubuntu that runs on WSL (Windows Subsystem for Linux)
• Build the Lambda with AWS SAM using the --use-container flag, which has the following explanation in the docs:

If your functions depend on packages that have natively compiled dependencies, use this flag to build your function inside an AWS Lambda-like Docker container.