Error code 2200 - "payload too large" on data factory copy job of json from Azure Blob to Azure Data Lake Gen2

Question

I have made a data factory copy job, that is supposed to copy JSON-files from blob storage to JSON in Azure Data Lake Gen 2. I have made several other copy jobs that works but not from json to json before, and in this instance I keep getting the error:

Error code: 2200. Failure type: User configuration issue. Details: The payload including configurations on activity/dataset/linked service is too large. Please check if you have settings with very large value and try to reduce its size.

I have tried decreasing the block size on the sink, but that only makes it fail faster, so not sure what is the problem. The json files are pretty big and includes output from forecasting algorithms, so there is both time series, model parameters and other stuff in the same json document.

Here is the first part of the JSON-script for the copy activity until the mappings if that helps:

{
    "name": "BlobStorage_To_DataLakeG2",
    "properties": {
        "description": "This job is intended to perform data copies of json-files from blob storage to ADLS gen2 for selected files.",
        "activities": [
            {
                "name": "TotalLoadForecast_ADLSG2_json",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "JsonSource",
                        "storeSettings": {
                            "type": "AzureBlobStorageReadSettings",
                            "recursive": true,
                            "enablePartitionDiscovery": true,
                            "partitionRootPath": "totalloadforecastoutput/"
                        },
                        "formatSettings": {
                            "type": "JsonReadSettings"
                        }
                    },
                    "sink": {
                        "type": "JsonSink",
                        "storeSettings": {
                            "type": "AzureBlobFSWriteSettings",
                            "blockSizeInMB": 4
                        },
                        "formatSettings": {
                            "type": "JsonWriteSettings"
                        }
                    },
                    "enableStaging": false,
                    "translator": {
                        "type": "TabularTranslator",
                        "mappings": [
′′′

question from:https://stackoverflow.com/questions/66047480/error-code-2200-payload-too-large-on-data-factory-copy-job-of-json-from-azur

Answer 1

It seems that the setting that was causing the problems was "enablePartitionDiscovery": true. Setting it to false made the job succeed. Maybe it does not work with json documents.