Create a zip file on S3 from files on S3 in Python

ONLINECAST

2 min read

·

06-10-2024

8

0

Share

Creating a Zip File on S3 from Files on S3 Using Python

Storing and managing files in the cloud is becoming increasingly common, with Amazon S3 being a popular choice. But what if you need to combine multiple files stored on S3 into a single, compressed zip file? This is where Python and the boto3 library come in handy.

The Problem: You have several files scattered across different folders within your S3 bucket and need to package them into a single zip archive for download or further processing.

Solution: This article will guide you through the process of creating a zip file on S3 using Python, directly from other files stored within the same bucket.

Code Breakdown:

import boto3
import io
import zipfile

def create_zip_on_s3(bucket_name, source_prefix, zip_file_name):
    """
    Creates a zip file on S3 from files within a specific prefix.

    Args:
        bucket_name: The name of the S3 bucket.
        source_prefix: The prefix of the files to be included in the zip.
        zip_file_name: The name of the zip file to be created.
    """

    s3 = boto3.client('s3')

    # Create in-memory buffer for the zip file
    zip_buffer = io.BytesIO()
    with zipfile.ZipFile(zip_buffer, 'w') as zip_file:
        # Iterate through all files matching the prefix
        for obj in s3.list_objects_v2(Bucket=bucket_name, Prefix=source_prefix)['Contents']:
            key = obj['Key']
            # Skip the zip file itself if it's included in the prefix
            if key == zip_file_name:
                continue
            # Download the file from S3
            s3.download_fileobj(bucket_name, key, zip_file)
            # Add the file to the zip archive
            zip_file.writestr(key, s3.get_object(Bucket=bucket_name, Key=key)['Body'].read())

    # Upload the zip file to S3
    s3.upload_fileobj(zip_buffer, bucket_name, zip_file_name)

# Example usage
create_zip_on_s3('your-bucket-name', 'data/reports/', 'reports.zip')

Explanation:

1. Import necessary libraries: boto3 for S3 interaction, io for in-memory buffers, and zipfile for zip archive handling.
2. Define the create_zip_on_s3 function:
   • Takes the bucket name, source prefix, and the desired zip file name as arguments.
   • Creates a boto3 S3 client.
   • Initializes an in-memory buffer (zip_buffer) for the zip file.
   • Uses zipfile.ZipFile to create a zip archive within the buffer.
   • Iterates through all objects matching the source_prefix using s3.list_objects_v2.
   • For each object, it downloads the file from S3 using s3.download_fileobj and adds it to the zip archive.
   • Finally, the function uploads the generated zip file to S3 using s3.upload_fileobj.

Key Considerations:

• Prefix Matching: The source_prefix should be carefully chosen to include only the desired files in the zip.
• Avoid Circular Dependencies: Ensure that the zip file name is not included within the source_prefix to avoid an infinite loop.
• Performance: For large files or numerous objects, the performance may be impacted by downloading each file individually. Consider using a streaming approach for better efficiency.

Further Optimization:

• Use s3.generate_presigned_url to generate a pre-signed URL for the generated zip file, allowing temporary download access without exposing the S3 credentials.
• Implement error handling and logging for a more robust solution.

Conclusion:

This Python code demonstrates a simple yet effective approach to create a zip file on S3 from files stored within the same bucket. By leveraging the power of boto3 and zipfile, you can streamline your S3 file management and efficiently package multiple files into a single compressed archive.

Additional Resources:

• boto3 documentation
• zipfile documentation
• S3 Object Listing API