S3 API - Buckets
Note
Operations on buckets or objects are an S3 API activity. We will be using the s3cmd
S3 client
for this lab. There are many other kinds of S3 clients available on the market.
Configure s3cmd
Let’s configure s3cmd to use the new S3 security credentials of the engineer1 user. Refer to the previous lab Admin API User for the S3 credentials of the engineer1 user.
Instructions
- Configure s3cmd
s3cmd --configure
- Enter Access Key obtained from Admin API User Lab
- Enter Secret Key from Admin API User Lab
- Enter region1 for Default Region
- Enter your region1 S3 Endpoint from your student worksheet
- For DNS-Style Enter %(bucket)s.{your region1 S3 Endpoint}
- Do not enter an Encryption password
- Accept default Path to GPG
- Choose no, for HTTPS protocol
- Press Enter for Proxy Server Name
- Test Access (Y)
- Save Settings (Y)
Test S3 Credentials
Check access and confirm engineer1 has no buckets, create a bucket, then check again
Instructions
- Check for existing buckets
s3cmd ls s3://
- There should be no buckets shown. Even though you have created buckets in previous labs, these are not accessible by engineer1.
- Let's create a new bucket for engineer1
s3cmd mb s3://engineerbucket
- You should see your new bucket has been created.
- Check for existing buckets again, and this time you should see your new bucket.
s3cmd ls s3://
Info
There is a limitation when using most (ALL?) third party s3 tools, and that is you cannot specify a storage policy, as most tools will be created for AWS S3, which has no concept of Storage Policy.
Non Default Storage Policy
Note
In order to create a bucket using a specific storage policy, we need dive a bit deeper. Cloudian supports the header x-gmt-policyid: . We would need to create the entire PUT request and sign using sigV4. This is well beyond the scope of this course, so we will install a helper library and create the request using python.
Instructions
pip install requests-aws4auth
We need to determine what the policyId is of the RF4 policy we created previously
Instructions
curl -s -X GET -k -u sysadmin:$auth_pass "https://$admin_endpoint:19443/bppolicy/bucketsperpolicy" | python -mjson.tool
In this example, the policyId associated to the policyName “RF4” is 0b707b3cc1266754eaaa6179e0c85a0e We need to use this Id in the python script we will be using to create the new bucket.
Create a new file called create_bucketRF4.py
Instructions
echo 'import requests
from requests_aws4auth import AWS4Auth
endpoint = "http://myrf4bucket.s3-region1.trainer1.cloudian.tech"
auth = AWS4Auth(
"ACCESSKEY",
"SECRETKEY",
"region1",
"s3",
)
headers = {
"x-gmt-policyid": "POLICYID",
}
response = requests.put(endpoint, auth=auth, headers=headers)
print(response.text) ' > ~/admin_api/create_bucketRF4.py
Instructions
- Edit the create_bucketRF4.py script to reflect your environment.
endpoint
engineer1 access Key
engineer1 secret Key
policyId - Once updated and saved, execute the pyhon script
There is no output for a successful run
python ~/admin_api/create_bucketRF4.py
Check that your new myrf4bucket has been created by re-running the list buckets per policy API (bppolicy)
Instructions
curl -s -X GET -k -u sysadmin:$auth_pass "https://$admin_endpoint:19443/bppolicy/bucketsperpolicy" | python -mjson.tool

You can also confirm by checking the buckets for engineer1 in the CMC