S3 bits

Things might be specific to Hetzner Object-Storage instead of AWS S3.

» Copy Files

» Upload

Using mc:

1
mc cp <LOCAL_PATH> <ALIAS>/<BUCKET>

» Download

Using mc:

1
mc cp <ALIAS>/<BUCKET> <LOCAL_PATH>

» Multipart Uploads

» List

Using mc:

1
mc ls --incomplete <ALIAS>/<BUCKET>

Using aws:

1
aws --profile <PROFILE> s3api list-multipart-uploads --bucket <BUCKET>

» Remove

Using mc:

1
mc rm --incomplete --recursive --force --dangerous <ALIAS>/<BUCKET>

(https://stackoverflow.com/a/39953195/7125878)

Using aws:

1
aws --profile <PROFILE> s3api abort-multipart-upload --bucket <BUCKET> --key <OBJECT> --upload-id <UPLOAD_ID>

» SSE-C

Adjust the secret key (32 bytes).

» Upload

1
aws s3 --profile=<PROFILE> cp <LOCAL_PATH> s3://<BUCKET>/ --sse-c --sse-c-key=32ByteKey_1234567890123456789012

» Download

Using aws with cp:

1
aws s3 --profile=<PROFILE> cp s3://<BUCKET>/<FILE_PATH> <LOCAL_PATH> --sse-c --sse-c-key=32ByteKey_1234567890123456789012

Using aws with get-object (allows for specific object version)

1
aws s3api --profile=<PROFILE> get-object --bucket <BUCKET> --key <FILE_PATH> --sse-customer-algorithm AES256 --sse-customer-key 32ByteKey_1234567890123456789012 --version-id <VERSION_ID> <LOCAL_PATH>

» cURL

» Download

1
2
3
4
5
curl --request GET \
--aws-sigv4 'aws:amz:<REGION>:s3' \
--user "<ACCESS_KEY>:<SECRET_KEY>" \
'https://<ENDPOINT>/<BUCKET>/<FILE_PATH>' \
-O

» Upload

1
2
3
4
5
curl --request PUT \
--aws-sigv4 'aws:amz:<REGION>:s3' \
--user "<ACCESS_KEY>:<SECRET_KEY>" \
'https://<ENDPOINT>/<BUCKET>/' \
--upload-file <FILE_PATH>

» Delete

1
2
3
4
curl --request DELETE \
--aws-sigv4 'aws:amz:<REGION>:s3' \
--user "<ACCESS_KEY>:<SECRET_KEY>" \
'https://<ENDPOINT>/<BUCKET>/<FILE_PATH>'

» Policies

» Get

1
mc anonymous get-json <ALIAS>/<BUCKET>

» Set

1
mc anonymous set-json policy.json <ALIAS>/<BUCKET>

» Deny Access

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
// Deny access to all users accept user p<PROJECT_ID:<ACCESS_KEY>
// Replace 'NotPrincipal' to 'Principal' to deny access only to this user
// and allow all other users.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAccess",
      "Effect": "Deny",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::<BUCKET>",
        "arn:aws:s3:::<BUCKET>/*"
      ],
      "NotPrincipal": {
        "AWS": "arn:aws:iam:::user/p<PROJECT_ID:<ACCESS_KEY>"
      }
    }
  ]
}

» Public Downloads

Allow public listing and downloading of files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicList",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Principal": {
                "AWS": [
                    "*"
                ]
            },
            "Resource": "arn:aws:s3:::<BUCKET>"
        },
        {
            "Sid": "PublicDownload",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Principal": {
                "AWS": [
                    "*"
                ]
            },
            "Resource": [
                "arn:aws:s3:::<BUCKET>/*"
            ]
        }
    ]
}

» Lifecycles

» Set

Using cerl:

curl –request PUT ‘https://..your-objectstorage.com/?lifecycle’ –aws-sigv4 ‘aws:amz::s3’ –user “<ACCESS_KEY>:<SECRET_KEY>” –data “id2logs/Enabled365”

Using mc:

1
mc ilm rule import <PROFILE>/<BUCKET> < lifecycle.json

Example lifecycle.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{
  "Rules": [
    {
      "ID": "AbortIncompleteMultipartUpload",
      "Status": "Enabled",
      "Prefix": "",
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 1
      }
    },
    {
      "ID": "NoncurrentVersionExpiration",
      "Status": "Disabled", // <--
      "Prefix": "",
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 30
      }
    },
    {
      "ID": "Expiration",
      "Status": "Disabled", // <--
      "Prefix": "",
      "Expiration": {
        "Days": 1
      }
    },
    {
      "ID": "DeleteMarkers",
      "Status": "Enabled",
      "Prefix": "",
      "Expiration": {
        "ExpiredObjectDeleteMarker": true
      }
    }
  ]
}

» CORS

» Set

1
aws s3api put-bucket-cors --profile=<PROFILE> --bucket <BUCKET> --cors-configuration file://cors.json

Example cors.json:

1
2
3
4
5
6
7
8
9
{
    "CORSRules": [
      {
        "AllowedOrigins": ["https://www.example.com"],
        "AllowedHeaders": ["*"],
        "AllowedMethods": ["GET", "HEAD"]
      }
    ]
  }