Using pytest fixtures in parametrized tests

Currently i'm working on a project that is supposed to consume all AWS Cloudwatch logs and send them to Datadog. But before sending them to Datadog i need to make series of manipulations with this logs(formatting, sensitive information obfuscating and other nice things). Since Datadog already has serverless forwarder implemented in python we just use it and extend it the way we need it. Workign with logs is pretty challenging because the amount and variaty of data is huge and we need to take into account lot's of aspects. That's why unit tests are playing a very important role in our development process. In order to verify that our code is working properly we have to collect big amount of different logs ( which is basically huge jsons), run them through our code and verify that result is the same what we expect. For this case parametrized tests are idea candidate. But here's the problem:

@pytest.mark.parametrize('incoming_log, expected_log', [
{"@timestamp": "2020-04-22T07:09:20.431Z", "@version": "1", "msg": "Cleaning thread scoped requestTraceId", "logger_name": "com.demo.demoapi.common.RequestTraceIdInterceptor", "thread_name": "http-nio-8080-exec-2171", "levelname": "DEBUG", "level": 10000, "requestTraceId": "L9vyPoRoEeqBFgpYqf6sKg", "appname": "demo-service", "service": "demo-service", "environment": "dev", "context": {"service": "demo-service", "environment": "dev", "requestTraceId": "L9vyPoRoEeqBFgpYqf6sKg"}}, 
{"@timestamp": "2020-04-22T07:09:20.431Z", "@version": "1", "msg": "Cleaning thread scoped requestTraceId", "logger_name": "com.demo.demoapi.common.RequestTraceIdInterceptor", "thread_name": "http-nio-8080-exec-2171", "levelname": "DEBUG", "level": 10000, "requestTraceId": "9vyPoRoEeqBFgpYqf6sKg", "appname": "demo-service", "service": "demo-service", "environment": "dev", "context": {"service": "demo-service", "environment": "dev", "requestTraceId": "9vyPoRoEeqBFgpYqf6sKg"}}
def test_logs_processing(incoming_log, expected_log):
    assert process_log(incoming_log) == expected_log

It looks terrible - it's only one pair of test data and it's already hard to read, hard to understand and really complicated to maintain. Imagive if we have 10 of different log inputs ?

Keeping such logs would be better in a separate file in @pytest.fixture, but here's the problem - you can't use fixtures in parametrized test out of the box. Or can you ? Thanks to Marcel Zaripov we have pytest-lazy-fixture !
This library allows us to do exactly what we need:

@pytest.fixture
def demo_incoming_log():
    return {"@timestamp": "2020-04-22T07:09:20.431Z", "@version": "1", "msg": "Cleaning thread scoped requestTraceId", "logger_name": "com.demo.demoapi.common.RequestTraceIdInterceptor", "thread_name": "http-nio-8080-exec-2171", "levelname": "DEBUG", "level": 10000, "requestTraceId": "L9vyPoRoEeqBFgpYqf6sKg", "appname": "demo-service", "service": "demo-service", "environment": "dev", "context": {"service": "demo-service", "environment": "dev", "requestTraceId": "L9vyPoRoEeqBFgpYqf6sKg"}}

@pytest.fixture
def demo_expecting_log():
    return {"@timestamp": "2020-04-22T07:09:20.431Z", "@version": "1", "msg": "Cleaning thread scoped requestTraceId", "logger_name": "com.demo.demoapi.common.RequestTraceIdInterceptor", "thread_name": "http-nio-8080-exec-2171", "levelname": "DEBUG", "level": 10000, "requestTraceId": "L9vyPoRoEeqBFgpYqf6sKg", "appname": "demo-service", "service": "demo-service", "environment": "dev", "context": {"service": "demo-service", "environment": "dev", "requestTraceId": "L9vyPoRoEeqBFgpYqf6sKg"}}

@pytest.mark.parametrize('incoming_log, expected_log', [
    (pytest.lazy_fixture('demo_incoming_log'), pytest.lazy_fixture('demo_expecting_log')),
])
def test_demo_log(incoming_log, expected_log):
    assert process_log(incoming_log) == expected_log

Now you can put as many inputs in your test as you need and it's gonna be nicely managed, easy to read and easy to understand :)

Happy coding everyone :)