python - Deploy containerized lambda with layer using CDK

Question

I'm working on an ML project that utilizes AWS Lambda for building models and generating predictions. Lambdas are written in python and use several ML libraries like pandas, sklearn, numpy, and scikit-learn. These lambdas use shared code that packaged by Lambda's layer. I use AWS CDK for project deployment. CDK code is written in TypeScript, don't ask why I mix Python and Typescript, it's not relevant in this situation.

The size of the package (lambda code + layer) exceeds the maximum allowed size of 250MB because of ML libraries.

After AWS announcement of containerized lambdas support, I decided to try it out to overcome the 250MB limit. However, I didn't find any good example that fit my situation, so I'm trying to build it myself.

The CDK code looks like this:

...
// Create a lambda layer from code
// Code is located in lambda-code/ml directory and it looks 
// like any Python package with main ML and DB connection functions
const mlLayer = new PythonLayerVersion(this, 'mlLayer', {
      entry: './lambda-code/ml/',
})
...
// Lambda function is specified like
const classifyTransactionLambda = new DockerImageFunction(this, 'classifyTransactionLambda', {      
      code: DockerImageCode.fromImageAsset('./lambda-code/classify'),      
      memorySize: 512,      
      layers: [mlLayer],
      tracing: Tracing.ACTIVE,
      environment: {
        BUCKET_NAME: mlModelsBucket.bucketName,
        ENV: env
      }
});
...

The structure of the code looks like this:

Dockerfile in classify lambda:

# Use the python lambda image from AWS ECR
FROM public.ecr.aws/lambda/python:3.7

COPY requirements.txt ./

RUN pip3 install -r requirements.txt

COPY index.py ./

CMD ["index.classify_transaction_handler"]

When I run cdk deploy I'm getting the following error:

This lambda function uses a runtime that is incompatible with this layer (FROM_IMAGE is not in [python3.7])

Does anyone run into problem like this? Is this error mean that mlLayer version is not compatible with lambda classifyTransactionLambda?

Any help would be very appreciated!

question from:https://stackoverflow.com/questions/65855216/deploy-containerized-lambda-with-layer-using-cdk

Answer 1

At this point

Functions defined as container images do not support layers. When you build a container image, you can package your preferred runtimes and dependencies as a part of the image.

https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

So I modified my build to copy all layer/library code to each lambda function before building an image.