Skip to content

img0

S3 API - Buckets

Note

Operations on buckets or objects are an S3 API activity. We will be using the s3cmd S3 client for this lab. There are many other kinds of S3 clients available on the market.

Configure s3cmd

Let’s configure s3cmd to use the new S3 security credentials of the engineer1 user. Refer to the previous lab Admin API User for the S3 credentials of the engineer1 user.

Instructions

  1. Configure s3cmd 
    s3cmd --configure
  2. Enter Access Key obtained from Admin API User Lab
  3. Enter Secret Key from Admin API User Lab
  4. Enter region1 for Default Region
  5. Enter your region1 S3 Endpoint from your student worksheet
  6. For DNS-Style Enter %(bucket)s.{your region1 S3 Endpoint}
  7. Do not enter an Encryption password
  8. Accept default Path to GPG
  9. Choose no, for HTTPS protocol
  10. Press Enter for Proxy Server Name
  11. Test Access (Y)
  12. Save Settings (Y)

Test S3 Credentials

Check access and confirm engineer1 has no buckets, create a bucket, then check again

Instructions

  1. Check for existing buckets 
    s3cmd ls s3://
  2. There should be no buckets shown. Even though you have created buckets in previous labs, these are not accessible by engineer1.
  3. Let's create a new bucket for engineer1 
    s3cmd mb s3://engineerbucket
  4. You should see your new bucket has been created.
  5. Check for existing buckets again, and this time you should see your new bucket. 
    s3cmd ls s3://

Info

There is a limitation when using most (ALL?) third party s3 tools, and that is you cannot specify a storage policy, as most tools will be created for AWS S3, which has no concept of Storage Policy.

Non Default Storage Policy

Note

In order to create a bucket using a specific storage policy, we need dive a bit deeper. Cloudian supports the header x-gmt-policyid: . We would need to create the entire PUT request and sign using sigV4. This is well beyond the scope of this course, so we will install a helper library and create the request using python.

Instructions

pip install requests-aws4auth

We need to determine what the policyId is of the RF4 policy we created previously

Instructions

curl -s -X GET -k -u sysadmin:$auth_pass "https://$admin_endpoint:19443/bppolicy/bucketsperpolicy" | python -mjson.tool

In this example, the policyId associated to the policyName “RF4” is 0b707b3cc1266754eaaa6179e0c85a0e We need to use this Id in the python script we will be using to create the new bucket.

Create a new file called create_bucketRF4.py

Instructions

echo 'import requests
from requests_aws4auth import AWS4Auth

endpoint = "http://myrf4bucket.s3-region1.trainer1.cloudian.tech"
auth = AWS4Auth(
    "ACCESSKEY",
    "SECRETKEY",
    "region1",
    "s3",
)

headers = {
    "x-gmt-policyid": "POLICYID",
}

response = requests.put(endpoint, auth=auth, headers=headers)
print(response.text)  ' > ~/admin_api/create_bucketRF4.py

Instructions

  1. Edit the create_bucketRF4.py script to reflect your environment.
    endpoint
    engineer1 access Key
    engineer1 secret Key
    policyId
  2. Once updated and saved, execute the pyhon script 
    python ~/admin_api/create_bucketRF4.py
    There is no output for a successful run

Check that your new myrf4bucket has been created by re-running the list buckets per policy API (bppolicy)

Instructions

curl -s -X GET -k -u sysadmin:$auth_pass "https://$admin_endpoint:19443/bppolicy/bucketsperpolicy" | python -mjson.tool

You can also confirm by checking the buckets for engineer1 in the CMC