aws lambda - Overriding boto3 client with stubbed client

Question

There are a lot of resources out there for using pytest, Moto, and botocore Stubber to write unit tests.

EDIT. I am rephrasing this question after further investigation:

I have a lambda_function python script that I want to test with pytest and the Boto Stubber. Inside of lambda_function I import a ssm_client from another python files (ssm_clt = boto3.client('ssm', region_name=region))

The problem is when I setup the pytest like this:

def test_lambda_handler(ssm_stubber):
    ssm_stubber.activate()
    ssm_stubber.add_response(
        'get_parameters_by_path',
        expected_params={'Path': 'my/ssm/parameter', 'Recursive': 'True'},
        service_response={
            'Parameters': [
                {
                    'Name': 'my/ssm/parameter',
                    'Type': 'String',
                    'Value': 'my_string returned',
                },
            ],
        },
    )
    ssm_stubber.deactivate()

    ssm_stubber.assert_no_pending_responses()

with the ssm_stubber defined as a pytest fixture:

@pytest.fixture(autouse=True)
def ssm_stubber():
    with Stubber(clients.ssm_clt) as stubber:
        yield stubber

It uses the actual boto3 client and not the stubber one because I have an import statement in lambda_function. I'm struggling with how to get past this. I'd like to not put a bunch of code in the regular lambda_function that is only for testing.

It is almost like I need a conditional import, but to my knowledge this is bad practice.

Did I structure my project in a way that makes it almost impossible to use stubber with pytest in this way?

question from:https://stackoverflow.com/questions/65834259/overriding-boto3-client-with-stubbed-client

Answer 1

So I ended up just using the monkeypatch functionality of pytest. This was a lot simpler then trying to patch the Boto 3 client and get it to properly stub. Below is some example code of what I did.

here is the function I want to test. The problem was the AWS API call from within the param_dictionary = setup.get_ssm_parameters() function never got stubbed correctly. This was not being stubbed because it was outside of the function the test was testing. This resulted in it trying to use the real boto3 client call during testing. All other API calls within the lambda_handler were always stubbed correctly.

"""lambda_function.py"""
    import another_function

    # this was the function that has an SSM AWS client call in it that wasn't get properly stubbed because it is outside of the function I am testing but is still executed as part of the script    
    param_dictionary = another_function.get_ssm_parameters()

    def lambda_handler(event, context):
        # other code here that runs fine and AWS API calls that are properly stubbed

This is the file that contained the AWS API call to parameter store.

"""another_function.py"""
import boto3

ssm_clt = boto3.client('ssm', region_name='us-west-2')

def get_ssm_parameters()
    param_dict = {}
    ssm_resp = ssm_clt.get_parameters_by_path(
        Path=f'/{os.environ["teamName"]}/{os.environ["environment"]}/etl',
        Recursive=True
    )
    
    for parameter in ssm_resp["Parameters"]:
        param_dict.update(json.loads(parameter["Value"]))

    return param_dict 

This is my test. You can see I pass in the money patch pytest.fixture which will patch the response from the function get_ssm_parameters() so it does not make an API call.

"""test_lambda_function.py"""
def test_my_func_one(return_param_dict):
    from lambda_function import lambda_handler

    # insert other snubbers and "add_response" code here for AWS API calls that occur inside of the lambda_handler

    lambda_handler('my_event', None)

This is the config file for pytest where I setup the monkeypatching. I use the setattr functionality of monkeypatch to override the return of get_ssm_parameters(). This return is defined in the function param_dict()

"""conftest.py"""
import pytest
import another function
def param_dict():
    param_dict = {"my_key": "my_value"}

    return param_dict


@pytest.fixture(autouse=True)
def return_param_dict(monkeypatch):
    monkeypatch.setattr(another_function, "get_ssm_parameters", param_dict)

Ultimately this was a lot simpler to do than trying to patch a client in another module outside of the function I was testing.