AWS S3 – A Beginners Guide To Awesome Storage

Introduction to AWS S3 Buckets

AWS Amazon Simple Storage Service (S3) is a scalable, high-speed, web-based cloud storage service designed for online backup and archiving of data and applications on Amazon Web Services (AWS). S3 allows you to store and retrieve any amount of data from anywhere on the Internet. One of the most fundamental building blocks of S3 is the bucket. A bucket is a container for storing files, similar to a directory or folder on a traditional file system.

Creating a Bucket

1. Log in to the AWS Management Console and navigate to the S3 service.
2. Click on the “Create Bucket” button.
3. Enter a unique name for your bucket and select a region. It’s important to choose a region that is geographically close to your users to minimize latency.
4. Click “Create” to create the bucket.

Uploading and Downloading Files

1. To upload a file, click on the name of the bucket you want to upload to.
2. Click on the “Upload” button and select the file you want to upload.
3. To download a file, navigate to the file in the bucket and click on the “Download” button.

Sample Bucket Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::example-bucket/*"
        }
    ]
}

This policy allows anyone to read objects in the “example-bucket” bucket.

Note: The * on the end of the bucket means allow all files and folders inside that bucket.

Public and Private Buckets

S3 allows you to control access to your buckets and the objects within them. A public bucket can be read and written to by anyone, while a private bucket can only be accessed by authorized users. You can set the access permissions for a bucket by editing the bucket’s ACL (Access Control List) or by using a bucket policy.

Accessing S3 Buckets with the AWS CLI

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. You can use the AWS CLI to access and manage your S3 buckets.

1. First, you need to have the AWS CLI installed and configured on your computer. You can find the installation instructions for your operating system in the AWS CLI documentation.
2. Once the AWS CLI is installed, you can use the aws s3 command to interact with your S3 buckets. To list all the S3 buckets in your account, use the ls command

aws s3 ls

2022-12-01 21:35:01 my-bucket

3. To upload a file to an S3 bucket, use the cp command. For example, to upload the file “example.txt” to the “my-bucket” bucket

aws s3 cp example.txt s3://my-bucket/

4. To download a file from an S3 bucket, use the cp command with the --recursive option. For example, to download the file “example.txt” from the “my-bucket” bucket

aws s3 cp s3://my-bucket/example.txt .

5. To delete a file from an S3 bucket, use the rm command. For example, to delete the file “example.txt” from the “my-bucket” bucket

aws s3 rm s3://my-bucket/example.txt

Accessing S3 Buckets from Python

1. First, you need to have the AWS SDK for Python (Boto3) installed on your computer. You can install it with pip

pip install boto3

2. Once Boto3 is installed, you can use it to interact with your S3 buckets. To list all the S3 buckets in your account, use the list_buckets() method

import boto3

s3 = boto3.client('s3')
response = s3.list_buckets()

for bucket in response['Buckets']:
    print(bucket['Name'])

3. To upload a file to an S3 bucket, use the upload_file() method. For example, to upload the file “example.txt” to the “my-bucket” bucket

s3.upload_file("example.txt", "my-bucket", "example.txt")

4. To download a file from an S3 bucket, use the download_file() method. For example, to download the file “example.txt” from the “my-bucket” bucket

s3.download_file("my-bucket", "example.txt", "example.txt")

5. To delete a file from an S3 bucket, use the delete_object() method. For example, to delete the file “example.txt” from the “my-bucket” bucket

s3.delete_object(Bucket="my-bucket", Key="example.txt")

Also, you should be aware that the S3 service has a few other features and settings such as versioning, cross-region replication, and access logging that you might want to take into account when working with S3 Buckets. It is also important to note that S3 has a cost associated with the storage, data transfer, and requests made to the service. Therefore, it’s important to monitor and optimize your usage to avoid unnecessary costs. In conclusion, S3 is a powerful and versatile storage service that can be easily accessed and managed through the AWS CLI and Boto3 library in Python.

By understanding the basic concepts and features of S3, you can take full advantage of its capabilities and store and manage your data efficiently and cost-effectively.

S3 Storage Costs and Classes

The cost of using S3 storage depends on several factors, including the type of storage class, the amount of data stored, and the number of requests made to the service. The cost of storing 1 TB of data in S3 can vary depending on the storage class and region you choose.

Here’s a comparison of the costs for 1 TB of storage in different S3 storage classes in US East (N. Virginia) region as of January 29, 2023.

Storage Class	Cost per GB/Month	Cost for 1 TB/Month
Standard	$0.023	$23.00
Intelligent-Tiering	$0.024	$24.00
S3 One Zone	$0.0125	$12.50
S3 Glacier	$0.004	$4.00
S3 Glacier Deep Archive	$0.00099	$0.99

It’s worth noting that these are just estimates, and the actual cost may vary based on your specific usage patterns. Also, these costs are for storage only and don’t include costs associated with data transfer, retrieval, or requests made to the service.

It’s also important to note that there are additional costs for data retrieval from S3 Glacier and S3 Glacier Deep Archive, which can vary depending on the retrieval method and the amount of data retrieved.

It’s recommended to use the AWS Pricing Calculator or the S3 pricing page to get an estimate of the costs for your specific usage.

Additionally, you can also save money on storage costs by using S3 Lifecycle policies to automatically transition objects to lower-cost storage classes or to delete them when they are no longer needed. Another way to save money is by using S3 Transfer Acceleration, which allows you to transfer large files to S3 over the Amazon CloudFront content delivery network (CDN) at faster speeds and lower costs.

For more information regarding cost optimizations, budgets, and alerting see my other blog.

Replicating a bucket to another region

Replicating a bucket to another region is a way to create a real-time, exact copy of your data in a different region for disaster recovery or to serve users from a geographically closer location.

Here is an example of how to replicate a bucket to another region using the AWS CLI:

1. First, you need to have the AWS CLI installed and configured on your computer. You can find the installation instructions for your operating system in the AWS CLI documentation.
2. To enable versioning on the source bucket, you need to run the following command:

aws s3api put-bucket-versioning --bucket my-bucket --versioning-configuration Status=Enabled

3. To create a new replication configuration, you can use the following command:

aws s3api put-bucket-replication --bucket my-bucket --replication-configuration file://config.json

The config.json file should contain the replication configuration, like this:

{
    "Role": "arn:aws:iam::111111111111:role/s3-replication-role",
    "Rules": [
        {
            "ID": "replication",
            "Prefix": "",
            "Status": "Enabled",
            "Destination": {
                "Bucket": "arn:aws:s3:::my-bucket-replica",
                "StorageClass": "STANDARD",
                "Account": "222222222222"
            }
        }
    ]
}

The role specified in the configuration file should have permissions to access both the source and destination buckets.

4. Once the replication configuration is created, any new objects added to the source bucket will be automatically replicated to the destination bucket in the specified region.

It’s worth noting that this is just an example, and you can customize it according to your needs, for example, you can filter the objects that you want to replicate based on prefix or tag. Additionally, you can also replicate to multiple regions if you want.

Additionally, you can use the AWS Management Console, the AWS SDKs, or the S3 REST API to enable cross-region replication.

It’s also important to note that cross-region replication incurs data transfer charges between the source and destination regions, so you need to take that into account when considering this feature.

In conclusion, S3 is a powerful and flexible storage service that can be used for a wide range of applications. By understanding the basic concepts of buckets, lifecycle policies, and access control, you can take full advantage of S3’s capabilities and save money on storage costs.