python - Missing DataDog metrics when using ThreadStats for a Celery task in Django

Question

The problem

I have a scheduled Celery task that queries x number of rows, does some processing and upon success (or error) it increments a specific metric using ThreadStats. For each execution of this task, the metric should be incremented by x at a specific time.

The problem is that some of these increments are not posted to DataDog. Ex. if the total number of rows is 100 and the task processes x=10 at a time, some of those task executions fails to increment the metric and it ends up displaying only 60.

Attempts for resolution

This is what I tried to do without success:

1. Manually flushing the metrics in the task by setting flush_in_thread=False and calling the flush() method.
2. Using dogstatsd-collector library to delay the submission

question from:https://stackoverflow.com/questions/65603045/missing-datadog-metrics-when-using-threadstats-for-a-celery-task-in-django

Answer 1

Waitting for answers