AWS S3 Related Notes

AWS
  • Upload and download large files (example code)
  • S3 Transfer Manager
    • Link to documentation and example code
    • S3 file name <= 1024 character unless characters in the name require more than one byte in UTF-8 representation, then it will be less.
    • I personally have problem getting the s3 API getobject call to save the file directly to a local drive.  The transfer manager does this just fine without any problem.  The only thing is that it transfers an entire bucket, not just a single object.
    • Here’s a script to use curl to upload to s3.  Curtesy of https://www.gyanblog.com/ 
      # about the file
file_to_upload=your_file
bucket=your_bucket
filepath="/${bucket}/${file_to_upload}"

# metadata
contentType="application/x-compressed-tar"
dateValue=`date -R`
signature_string="PUT\n\n${contentType}\n${dateValue}\n${filepath}"

#s3 keys
s3_access_key=your_access_key
s3_secret_key=your_secret_key

#prepare signature hash to be sent in Authorization header
signature_hash=`echo -en ${signature_string} | openssl sha1 -hmac ${s3_secret_key} -binary | base64`

# actual curl command to do PUT operation on s3
curl -X PUT -T "${file_to_upload}" \
  -H "Host: ${bucket}.s3.amazonaws.com" \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3_access_key}:${signature_hash}" \
  https://${bucket}.s3.amazonaws.com/${file_to_upload}
    • A good tutorial on access s3 with preset url