Migrating an FTP Process to an AWS S3 Bucket Process

Introduction:

I was working with an entity that pulls a file via FTP from my company. The process became problematic. When the outside entity downloaded from our FTP site, it took a long time, and my company’s Internet connection was impacted. The process was changed so that the file is no longer on an FTP server – it is in an S3 bucket.

Prerequisites:

• Knowledge of AWS S3
• Understanding of AWS Secrity Policies
• Knowledge of Powershell and AWS Tools for PowerShell

Solution:

Although there are many moving parts to this process, I am focusing only on one here: replacing an FTP server with an S3 bucket. I will be using the AWS tolls for PowerShell. The overall steps are:

• Create an S3 Bucket, and two IAM Users
• Create three IAM Policies: one to copy to the S3 bucket, and one to download from it, and one to read it
• Apply the polices to the accounts
• Generate access keys and secret access keys for both IAM accounts
• Create two PowerShell scripts, one to copy to the S3 bucket, and one to download

Here are the details of each step.

Step 1: Create an S3 Bucket, and two IAM Users

A pretty straight forward three line command:

New-S3Bucket -Region us-east-1 -BucketName 'fileextractbucket'
New-IAMUser -UserName 'fileupload'
New-IAMUser -UserName 'filedownload'

Step 2: Create three IAM Policies:

We need to create three policies. The first two will be part of a “read-only” policy. To read an S3 object, an account needs to read the bucket, and the object, hence two policies. One policy permits the IAM user to read the bucket, and the other allows reading the object. I save these into variables.

$ReadBucketPolicy = '{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Effect": "Allow",
       "Action": ["s3:ListBucket"],
       "Resource": ["arn:aws:s3:::fileextract"]
     }
   ]
 }'  
$ReadObjectPolicy = '{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Allow",
             "Action": "s3:GetObject",
             "Resource": "arn:aws:s3:::fileextract/*"
         }
     ]
 }'

Next is the write policy that the upload account will need. Again, it is saved into a variable.

$ReadWritePolicy = '{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Effect": "Allow",
       "Action": [
         "s3:PutObject",
         "s3:GetObject",
         "s3:DeleteObject"
       ],
       "Resource": ["arn:aws:s3:::fileextract/*"]
     }
   ]
 }'

Step 3: Apply the polices to the accounts

For the filedownload account, we create two policies for reading the bucket and the objects.

Write-IAMUserPolicy -UserName 'filedownload' `
    -PolicyName 'fileextract-ReadBucketPolicy-download' `
    -PolicyDocument $ReadBucketPolicy    
Write-IAMUserPolicy -UserName 'filedownload' `
    -PolicyName 'fileextract-ReadObjectPolicy-download' '
    -PolicyDocument $ReadObjectPolicy 

For the fileupload account, we create two policies, one for reading the bucket and one for reading and writing the objects.

Write-IAMUserPolicy -UserName 'fileupload' `
    -PolicyName 'fileextract-ReadBucketPolicy-upload' `
    -PolicyDocument $ReadBucketPolicy   
Write-IAMUserPolicy -UserName 'fileupload' `
    -PolicyName 'fileextract-WriteBucketPolicy-upload' `
    -PolicyDocument $ReadWritePolicy 

Step 4: Create Access Keys for both accounts

You will need to create access keys and secret keys for both the upload and download accounts. The commands below will create these keys. Record the Access Key and Secret Key for both accounts.

New-IAMAccessKey -UserName 'fileupload'
New-IAMAccessKey -UserName 'filedownload' 

After you have the access keys, you can create AWS credential sets as shown in Managing AWS Credentials with PowerShell

Set-AWSCredential -AccessKey <……..> -SecretKey <……..> -StoreAs upload
Set-AWSCredential -AccessKey <……..> -SecretKey <……..> -StoreAs download

Step 5: Create two PowerShell scripts

To copy the file up to the S3 bucket, run the following command, where the $file is a variable containing the full name and path of the local file, and $Key is the name of the file after it is written to the S3 bucket:

Write-S3Object -BucketName fileextractbucket `
    -File $file -Key $Key -ProfileName upload

Copy-s3object -bucketname fileextractbucket `
    -Key $Key -LocalFile $localFilePath -ProfileName download

These statements are wrapped in a try catch statement within a script as shown at Running Powershell SQL Agent Jobs

Summary:

The new process reduced the time for the outside entity by an order of magnitude. YMMV, but this was a win for us (by relieving our internet connection of the stress from FTP), and for the outside entity by reducing the time they needed. Also note that the file should be encrypted regardless of the set up. A public\private key pair is a good choice.

You May Also Like:

• Managing AWS Credentials with PowerShell
• Running Powershell SQL Agent Jobs