The University of Arizona
Homework 7

CSC 346 - Homework 7 #

In this homework assignment you will deploy two serverless Lambda functions to faciliate image uploading for the Picturegram client. You will create two S3 buckets to hold the uploaded and resized image files. The first Lambda function will be for generating one-time-use signed upload URLs for uploading original images to the upload bucket. The second lambda function will trigger on new objects in the upload bucket, resize the images to standard sizes, and then store the resized images in a second S3 bucket which will be used as the source for images displayed in the Picturegram front end application.

What to turn in #

Turn in a .zip file of a folder containing the following files:

yournetid-hw07/
    resize-bucket-permissions.png (or .jpg)
    upload-bucket-permissions.png (or .jpg)
    image-urls.txt

Points #

This assignment will be worth 50 points. Turning the assignment in early will earn you an additional 5 points extra credit.

  • Early submission: April 11th 11pm MST (UTC-7)
  • Final Due Date: April 13th 11pm MST (UTC-7)

Starting Point #

The starter zip for this assignment contains the python code that will be used for the Lambda functions. While I encourage you to read through the code files (they aren't long), this assignment focuses more on deploying and configuring the Cloud Infrastructure components, and not on the lambda code itself.

https://www2.cs.arizona.edu/classes/cs346/spring24/homework/NETID-hw07.zip

Part 1: S3 buckets #

To start, create two S3 buckets from the provided template. One should be for receiving the uploaded original images. The second will hold the resized images for download by your app.. I'll refer to these as the upload bucket and the download bucket in the assignment.

Refer to the Week 11 slides for creating the S3 buckets based on the supplied CloudFormation template.

https://fischerm-csc346-download.s3.amazonaws.com/s3_template.yaml

Once your S3 buckets are deployed, check them out in the AWS S3 console, and take a screenshot of the Permissions tab for each bucket. Names these images "upload-bucket-permissions" and "download-bucket-permissions". They can be PNG files or JPEG files, what ever your screenshots come in.

Part 2: Signed Upload URL Lambda Function #

The first Lambda function will generate signed URLs that can be used to upload files directly to the upload bucket. This is really useful for us in that we don't need to wory about local disk space on an EC2 instance or anything. Files can go directly from the user's browser to the S3 bucket. You do not need to add the Lambda Layer to this function, as it does not need any additional python modules.

Use generateurl.py from the starter zip file for the code for this function.

Refer to the Week 11 slide 63 for creating this function. The Function URL used to call the Lambda function is found under Configuration → Function URL.

The upload function does not need a trigger configured. It will be invoked from the Function URL you configure.

Environment Variables #

Both Lambda functions need to know infromation about your S3 buckets. We can pass this information into the code through the use of Environment Variables. Select the Confifuration tab of the lambda function, then "Environment variables" in the left sidebar. You will need to create two env vars for the generateurl.py lambda function containing your two S3 bucket names. The generate URL function needs to know both of them. The S3_UPLOAD_BUCKET_NAME is used to create the presigned URL returned from this function, which your app will use to actually upload the image. The S3_DOWNLOAD_BUCKET_NAME is used to construct the URLs of the resulting images that will be created after the resize function runs. These URLs are also returned as part of the response, even though technically they won't be valid for a breif time while the resize function actually gets to work after the image is uploaded.

Testing Signed Upload URL Function #

To test out your first lambda function, find the URL created when you set up the Function URL. Take this URL to Postman or curl and use it to test out the function. Remember that you need to include a query string parameter with the name of the file you intend to upload. For testing this can literally be any string, when we get to integrating this into our PictureGram client, we will take the filename off of whatever file we select to upload.

The query string parameter key is filename and the value is whatever image you want to upload. Grab some tasteful image that's somewhat large to test with, something at least 1500px x 600px so we have something to resize.

Your images are visible to your classmantes and the public. Be mindful of the Student Code of Conduct specifically section F and the prohibition on harassment, intimidation, and threats. Be respectful! 
curl -v https://alw676it2cnnrshan3627wop7m0eampr.lambda-url.us-east-1.on.aws/?filename=oldmain.jpg

You should get a JSON response that looks something like:

{
  "status": "OK",
  "upload_url": "https://fischerm-csc346-upload.s3.amazonaws.com/input/oldmain-d8365602-a16a-43fc-88de-17cb345e1866.jpg?AWSAccessKeyId=ASIAT5UK7QF4Q43HMI54&Signature=p8mVTma3gl94ntFIHBMCbp7vI84%3D&content-type=image%2Fjpeg&x-amz-security-token=IQoJb3JpZ2luX2VjEBUaCXVzLWVhc3QtMSJHMEUCIQCBieHTs0PzOtDQKiwpDqaO9RZ30abeDZLAj19BHoCKbwIgcrAjrTkepINuJVE2iM81aus35dSIeVescPvY9SFjNR8qtQMIPRAEGgwyNjk4MDA2Njk1NjEiDDOc%2B2UTp%2FZrStOJWSqSA3zz%2B%2FdRG2Y19iHSgt8tjXCvVImIEVZXp4TYoFbrRY4citEHQbxY2Qq8A3KTTxUQAZWpOX9tP0ARpC0CURpZEyR373R5dgn%2BXH2mv5zEOuWIhdg4QtidD1pEAq7Wari%2FqRvPzaoAxxWRFwarRExoYyAOYEfZIi21VbLMuLiOyt2vTQaubUg1aqucxHPOxVW32YXkgFk2if2ZCZZlBFbC4z6dlIFT6qOF%2FVKc9K8EcTh3v5aJKeQJ%2FgqRQ492jMgmpOrWirZbSD8Wuk1DEsZTRC6OAlTkeic6%2FV9k0paQdbFTqg7BJky9P4k8EgwXrjtGMPLBpd%2FeUicC5AL4MC%2BkjZgi7liGq6A8r3tLpnhnwwwbJL%2BZry1my6Rtm9SRkUp1DnkNOZy5jXgOyRptvO0tTwZovVruO94sKggTOY%2B66YwRwIrVqHrARD36Cj0gE76bwUEhUU7rdylV6ciI5u9Pm0BskrbIhyIVDGNbmAyr%2FAs5LYCZlpxkee0WA5If0ZU14cUXr1jKYmSX3J9xI6N6StB9CzDx5KiwBjqeAYcvA%2FrUNFCdLFP%2Fm2srf9A6vmN3vNiBfivdFDE8m%2Fcvazu8DgIQg64stYCktFuAGYzxRsuaDHPCW3VuD%2FyTXtMd6tXiIcAExBVNny4l3N7ZyOJ7TSJLFRpqyjjHG%2FNxOvWKKnaGCdaqRq7F%2B3baA6ryUXO%2FJgIYKvjg56NfUGGWMK%2B0WStQ%2FtZZl0CS45QY%2BDxaAfD0yEhqJT1rf3Lf&Expires=1711944606",
  "full_url": "https://fischerm-csc346-download.s3.amazonaws.com/oldmain-d8365602-a16a-43fc-88de-17cb345e1866-scale1500.jpg",
  "thumbnail_url": "https://fischerm-csc346-download.s3.amazonaws.com/oldmain-d8365602-a16a-43fc-88de-17cb345e1866-crop600.jpg"
}

The upload_url attribute in the response contains the URL you will use to actually upload the image to the S3 bucket. This can again be done with either Postman or curl, although I find Postman easier.

Refer to the Week 11 slide 65 for Postman examples.

The presigned upload URLs are only valid for 5 minutes (300 seconds, see the code). You may have to generate a new URL periodically while testing. There's no penalty or cost to calling the Signed Upload URL lambda function. You'll only be charged for the images that actually get uploaded.

Part 3: Image Resizing Lambda Function #

The next Lambda function will resize any image that shows up in the upload bucket. It will make two standard sized images from this, and store them in the download bucket.

Note that you will not test your function in the same way the slides show. The final function code provided assumes event data is coming in that matches an S3 CreateObject event. So you still need to Deploy your function code whenever you make changes, but the testing will be done by just uploading a new image to the upload bucket, and then checking the resize bucket and CloudWatch Logs. See the Troubleshooting section below for more information about CloudWatch Logs.

Use resize_image.py from the starter zip file for the code for this function.

Refer to the Week 11 slides 31 for creating the Lambda function.

Event Trigger #

You need to add an event trigger to the resize function. This is what connects new files showing up in the upload bucket, to triggering this function to run. Refer to the Week 11 slides 41 for adding triggers.

Be sure to trigger on the UPLOAD s3 bucket, and set the S3_DOWNLOAD_BUCKET_NAME environment variable to your DOWNLOAD bucket. If you use the same bucket name in both places you can create an infinite recursive loop!
Go ahead and include the input/ prefix when setting up the trigger. This can help prevent against accidental recursive problems. The generateurl.py function has this prefix defined when it creates the presigned uplaod URLs. This means all your uploaded images should end up in this 'folder'. The resize_image.py code puts all the resized images at the root level of the S3 bucket. So in the unlikely event you get your buckets crossed, hopefully this difference prevents the resized images from triggering additional resize function executions, since they won't be in the input/ folder.

Lambda Layer #

In order to use the imaging modules, you need to have the correct python modules available within your lambda runtime. I've packaged these up in a Lambda Layer that you can reference. Refer to the Week 11 slides 45 for adding triggers.

arn:aws:lambda:us-east-1:269800669561:layer:fischerm-csc346-imagelayer:5

Environment Variables #

The resize function only needs to have your S3_DOWNLOAD_BUCKET_NAME set. The upload file names and bucket names will come in as part of the event object passed to your lambda handler as a result of getting the Event Trigger connected.

Be sure to trigger on the UPLOAD s3 bucket, and set the S3_DOWNLOAD_BUCKET_NAME environment variable to your DOWNLOAD bucket. If you use the same bucket name in both places you can create an infinite recursive loop!

Part 4: Upload an Image #

If all goes well, you should now be able to generate an upload URL, and then use that URL to upload an image to your upload bucket. Once there, your resize function should have been triggered to grab the original, resize it, and save the resized images in the resize bucket.

Create a text file named image-urls.txt. Upload a test image for resizing, and include three URLs in the image-urls.txt text file:

Original URL:
Thumbnail URL:
Large URL:

The Thumbnail and Large URLs should be able to be viewed by anyone in a browser. We will examine these URLs as part of the grading. The Original URL is the file uploaded to your upload bucket. These won't be publically visible, but the TAs and I can look at the contents of your S3 bucket to verify it.

Part 4: Automated Testing #

I have updated the testing app you can use to test out your homework assignment once you get it running. All the tests you run through the app are logged, and we will look at the most recent test results for your NetID when we do the grading.

https://csc346.test.apps.uits.arizona.edu/

You will need to log in through WebAuth to see the URL form.

The input URL will be the Function URL for your Generate Upload URL Lambda function, ie https://xxxxxxxxxxxxxx.lambda-url.us-east-1.on.aws/

The tests take between 15 and 30 seconds to run, so be patient.

Troubleshooting #

CloudFormation Stack Deployment Errors #

If you have problems deploying the S3 CloudFormation stacks, double check that the URL is correct when specifying the S3 template URL. Also check the Events tab of the stack to see what errors show up there. Bring errors to the Discord class channel and we'll work through them.

Lambda Function Execution Errors #

Your Lambda functions will be executed in two ways. The resize function is triggered whenever a new image shows up in the upload bucket. The upload function is executed when you send a GET API request to the Function URL. In any case, you don't have direct access to the Lambda console to see any errors.

AWS Lambda integrates with the CloudWatch Logs service to store all the log files generated. Search for CloudWatch in the services bar, and then select Log groups in the sidebar. You will get one log group for each Lambda function. Look at the individual log streams in there for possible errors.

A screenshot of the AWS CloudWatch Logs Console

A screenshot of the AWS CloudWatch Logs with a python error

Upload URL Errors #

If you get a "Forbidden" error when trying to hit your upload-url Function URL in postman:

{
    "Message": "Forbidden"
}

Check the configuration on your Function URL. Be sure that the Auth setting is "NONE".

File Upload Errors #

If you get a "signature does not match" error when trying to upload the file to your signed upload URL, check and make sure that the HTTP method is PUT. Also maje sure that the image file type matches what you requested in the upload-url call. For example if you requested filename=image.jpg you need to upload an image of type JPEG, with either a .jpg or .jpeg extension. The Content-Type header must match the signed URL.`