Need to provide dynamic file from Azure blob to the any data activitiy in Azure Data Factory, please share some link

ONLINECAST

2 min read

·

05-10-2024

8

0

Share

Dynamically Accessing Azure Blob Files in Azure Data Factory

Problem: You need to dynamically retrieve files from Azure Blob storage within an Azure Data Factory pipeline. This might involve pulling data from various files based on a date range, specific file names, or other criteria.

Rephrased: Imagine you have a folder in Azure Blob Storage filled with data files. You want to build a data pipeline that can automatically select and process only the relevant files based on your needs.

Solution: Azure Data Factory (ADF) provides several approaches for accessing files dynamically from Azure Blob storage. This article will explore two popular methods and provide links for further exploration:

1. Using Lookup Activity:

• Scenario: You need to get a list of file names from an Azure Blob container based on a certain pattern or criteria.
• Approach: The Lookup Activity in ADF is ideal for this. You can use the AzureBlobStorage dataset with wildcard characters in the file path. For example, you can specify *.csv to retrieve all CSV files in a container.
• Example:

  "activities": [
    {
      "type": "Lookup",
      "name": "LookupBlobFiles",
      "inputs": [
        {
          "referenceName": "BlobDataset"
        }
      ],
      "output": {
        "name": "LookupOutput"
      },
      "dataset": {
        "referenceName": "BlobDataset",
        "type": "AzureBlobStorage"
      },
      "source": {
        "type": "BlobSource",
        "recursive": true,
        "folderPath": "your-container-name/data/2023-08-01/",
        "filePattern": "*.csv"
      }
    },
    ... // subsequent activities that use the retrieved file names
  ]
  

• Links:
  • Azure Data Factory Lookup Activity
  • AzureBlobStorage Dataset

2. Using ForEach Activity:

• Scenario: You want to process a batch of files based on a list of file names.
• Approach: The ForEach Activity in ADF allows you to iterate over a collection of items. You can first use a Lookup Activity to retrieve file names and then use the ForEach Activity to process each file individually.
• Example:

  "activities": [
    {
      "type": "ForEach",
      "name": "ForEachBlobFile",
      "inputs": [
        {
          "referenceName": "LookupOutput"
        }
      ],
      "items": {
        "type": "Expression",
        "value": "@items('LookupOutput')"
      },
      "activities": [
        {
          "type": "Copy",
          "name": "CopyFile",
          "inputs": [
            {
              "referenceName": "BlobDataset",
              "parameters": {
                "filePath": "@item().name"
              }
            }
          ],
          "outputs": [
            {
              "referenceName": "SinkDataset"
            }
          ],
          ... // other copy activity settings
        }
      ]
    }
  ]
  

• Links:
  • Azure Data Factory ForEach Activity
  • Azure Data Factory Copy Activity

Additional Tips:

• You can use ADF expressions (@ symbol) to dynamically construct file names, paths, and other parameters.
• Use the recursive parameter in the BlobSource to fetch files from subfolders.
• If you need to filter files based on timestamps, you can use ADF functions like utcNow() and string manipulation functions in your expressions.

Conclusion:

By leveraging ADF's built-in capabilities, you can build data pipelines that dynamically access Azure Blob storage files based on your specific requirements. This empowers you to automate data ingestion, processing, and analysis for various scenarios.