I have made a data factory copy job, that is supposed to copy JSON-files from blob storage to JSON in Azure Data Lake Gen 2. I have made several other copy jobs that works but not from json to json before, and in this instance I keep getting the error:
Error code: 2200. Failure type: User configuration issue. Details: The payload including configurations on activity/dataset/linked service is too large. Please check if you have settings with very large value and try to reduce its size.
I have tried decreasing the block size on the sink, but that only makes it fail faster, so not sure what is the problem. The json files are pretty big and includes output from forecasting algorithms, so there is both time series, model parameters and other stuff in the same json document.
Here is the first part of the JSON-script for the copy activity until the mappings if that helps:
{
"name": "BlobStorage_To_DataLakeG2",
"properties": {
"description": "This job is intended to perform data copies of json-files from blob storage to ADLS gen2 for selected files.",
"activities": [
{
"name": "TotalLoadForecast_ADLSG2_json",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "JsonSource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": true,
"enablePartitionDiscovery": true,
"partitionRootPath": "totalloadforecastoutput/"
},
"formatSettings": {
"type": "JsonReadSettings"
}
},
"sink": {
"type": "JsonSink",
"storeSettings": {
"type": "AzureBlobFSWriteSettings",
"blockSizeInMB": 4
},
"formatSettings": {
"type": "JsonWriteSettings"
}
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"mappings": [
′′′
question from:
https://stackoverflow.com/questions/66047480/error-code-2200-payload-too-large-on-data-factory-copy-job-of-json-from-azur 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…