Automated Image Processing Pipeline for a Media Production Startup

At AMJ Cloud Technologies, we transformed digital asset management for MediaFlow, a media production startup, by implementing an automated image processing pipeline on AWS. Leveraging serverless architecture with AWS Lambda and S3, we delivered a scalable, cost-efficient solution that accelerated content delivery and enhanced media workflows.

Situation

MediaFlow, specializing in high-volume video and image content creation, faced challenges with manual image processing, which delayed content delivery and increased costs. The startup needed an automated pipeline to resize uploaded images into thumbnails for previews, ensuring fast processing, scalability, and cost efficiency while maintaining high availability.

Task

Our goal was to design and deploy an automated image processing pipeline using AWS services. Key objectives included:

• Create two S3 buckets for source images and processed thumbnails.
• Develop a Lambda function to resize images upon upload.
• Configure an S3 trigger to invoke Lambda automatically.
• Use Terraform for infrastructure as code.
• Test the pipeline for functionality and performance.
• Complete the project within three months.

Action

To meet MediaFlow’s requirements, we took the following actions:

1. Infrastructure Provisioning with Terraform:

   • Defined two S3 buckets (source-bucket and thumbnail-bucket), an IAM role, and an S3 event trigger using Terraform.

   • Organized code into modules for maintainability.

   • Used an S3 backend with DynamoDB for state management.

     resource "aws_s3_bucket" "source" {
       bucket = "source-bucket"
       versioning {
         enabled = true
       }
     }
      
     resource "aws_s3_bucket" "thumbnail" {
       bucket = "thumbnail-bucket"
     }
      
     resource "aws_lambda_function" "image_processor" {
       function_name = "ImageProcessor"
       handler       = "lambda_function.lambda_handler"
       runtime       = "python3.9"
       role          = aws_iam_role.lambda_role.arn
       filename      = "lambda_function.zip"
       environment {
         variables = {
           THUMBNAIL_BUCKET = "thumbnail-bucket"
           THUMBNAIL_SIZE   = "128"
         }
       }
     }
      
     resource "aws_s3_bucket_notification" "source_trigger" {
       bucket = aws_s3_bucket.source.id
       lambda_function {
         lambda_function_arn = aws_lambda_function.image_processor.arn
         events              = ["s3:ObjectCreated:*"]
       }
     }

2. Amazon S3 Buckets:

   • Configured source-bucket with versioning and thumbnail-bucket with restricted access.
   • Set bucket policies for Lambda access.
   • Uploaded a test image (test.jpg) to validate the pipeline.

3. AWS Lambda Function:

   • Developed a Python Lambda function using Pillow to resize images to 128x128 pixels.

   • Packaged with dependencies and configured with environment variables.

     from PIL import Image
     import boto3
     import os
     import io
      
     s3_client = boto3.client('s3')
     THUMBNAIL_BUCKET = os.environ['THUMBNAIL_BUCKET']
     THUMBNAIL_SIZE = int(os.environ['THUMBNAIL_SIZE'])
      
     def lambda_handler(event, context):
         try:
             bucket = event['Records'][0]['s3']['bucket']['name']
             key = event['Records'][0]['s3']['object']['key']
             obj = s3_client.get_object(Bucket=bucket, Key=key)
             image = Image.open(io.BytesIO(obj['Body'].read()))
             image.thumbnail((THUMBNAIL_SIZE, THUMBNAIL_SIZE))
             buffer = io.BytesIO()
             image.save(buffer, format="JPEG")
             buffer.seek(0)
             s3_client.put_object(
                 Bucket=THUMBNAIL_BUCKET,
                 Key=f"thumbnail/{key}",
                 Body=buffer,
                 ContentType='image/jpeg'
             )
             return {'statusCode': 200, 'message': 'Thumbnail created'}
         except Exception as e:
             print(f"Error: {str(e)}")
             return {'statusCode': 500, 'message': str(e)}

4. S3 Trigger Configuration:

   • Set an S3 event notification on source-bucket for s3:ObjectCreated:* to trigger the Lambda function.
   • Validated integration via Terraform outputs.

5. Testing and Validation:

   • Tested Lambda locally with a dummy S3 event.
   • Uploaded multiple images to source-bucket and verified thumbnails in thumbnail-bucket.
   • Checked CloudWatch logs for errors and performance.

6. Monitoring and Optimization:

   • Enabled CloudWatch metrics for Lambda and S3.
   • Set alarms for failures and costs.
   • Optimized Lambda memory to 256 MB.

Result

The pipeline delivered transformative outcomes for MediaFlow:

• 90% Processing Time Reduction: Automated resizing cut manual effort by 90%.
• 60% Cost Reduction: Serverless architecture reduced costs by 60%.
• Scalability: Handled thousands of daily uploads effortlessly.
• 99.99% Uptime: Ensured high availability with S3 and Lambda.
• Reproducibility: Terraform ensured consistent environments.
• Business Impact: Faster previews improved client satisfaction and scaled production capacity.

This project highlights AMJ Cloud Technologies’ expertise in serverless media solutions.

Technologies Used

• AWS S3: Stored images.
• AWS Lambda: Processed images.
• Terraform: Managed infrastructure.
• Python/Pillow: Resized images.
• AWS IAM: Secured access.
• AWS CloudWatch: Monitored performance.
• AWS CLI: Validated pipeline.

Key Takeaways

This project demonstrates the power of serverless architectures and IaC in media production, enabling efficiency, scalability, and cost savings. AMJ Cloud Technologies continues to drive innovation for startups.

Architectural Diagram