Using the Object Store Service (Swift & S3)

Knowledge Base How To and Guides
#1

Introduction

OpenStack@Louros offers a Ceph-backed object storage service using Ceph Rados Object Gateway, which provides applications with a RESTful gateway to the Ceph Storage Cluster, supporting two interfaces:

  • An S3-compatible Rest API, usable with any S3-compatible software.
  • A Swift-compatible Rest API, usable with any Swift-compatible software and the OpenStack CLI.

This guide explains how to use the Object Storage service, create a container (bucket), upload and download an example file.

Table Of Contents

Changelog

[2020-11-18]

  • Initial Version

OpenStack Dashboard

Follow the instructions below to create a new bucket and upload an example file.

  1. Login to the OpenStack dashboard at https://ui.cloud.grnet.gr

  2. From the side-bar, navigate to Projects > Object Store > Containers.

1-containers
1-containers767×616 29.1 KB

  1. Click on + Container to create a new empty container (containers also referred to as buckets). Choose a name for the new container, and choose an access mode. A Public container has a public URL that anyone can use to access objects inside that container. For sensitive files, make sure that your container is Not public.

2-create-bucket
2-create-bucket1103×434 35.2 KB

  1. After creating a container, you can search for files, use the + Folder button to create a new folder, or the Upload button to upload a new file. Choose a file from your local filesystem and click Upload File. This may take a long time, depending on the size of the file that you are uploading.

3-upload-file
3-upload-file1360×617 61.9 KB

  1. You can then Download the file, or view details. If the container is Public, then you can also retrieve a public URL that anyone can use to download the file.

4-list-files
4-list-files1311×665 40.6 KB

OpenStack Swift CLI

  1. Generate Application Credentials (see also Usage Of OpenStack CLI).

  2. Source the Application Credentials:

    $ source openstack-credentials.sh

  3. The OpenStack CLI uses the Swift API endpoints to access the user containers and objects. Use openstack container --help to see commands for working with containers, and openstack object --help to see commands for working with objects.

  4. List the available containers with openstack container list. You can see the akolaitis-bucket we created previously.

    $ openstack container list
+------------------+
| Name             |
+------------------+
| akolaitis-bucket |
+------------------+

  5. Use the openstack object list CONTAINER command to list the objects inside CONTAINER.

    $ openstack object list akolaitis-bucket
+--------------------------+
| Name                     |
+--------------------------+
| my-important-file.tar.gz |
+--------------------------+

  6. Use the openstack object save CONTAINER OBJECT command to download a remote object and save it locally:

    $ openstack object save akolaitis-bucket my-important-file.tar.gz

For any of the above commands, append the --debug flag to see the raw HTTP API calls that are made to the Swift API.

S3 API

S3 API credentials

In order to use the S3 API, you will need to generate EC2 compatible credentials, which comprise of an Access Key and a Secret Key. At the time of writing this guide, this is only possible from the OpenStack CLI:

$ openstack ec2 credentials create -f json
{
  "access": "ACCESS_KEY",
  "links": {
    "self": "....."
  },
  "project_id": "......",
  "secret": "SECRET_KEY",
  "trust_id": null,
  "user_id": "......."
}

Make sure to store the ACCESS_KEY and the SECRET_KEY values safely, as they can be used to get read-write access to your objects.

In order to use the S3-compatible API, you can use any S3-compatible software with the following configuration:

Config Value
AWS_SECRET_KEY The SECRET_KEY from the generated credentials
AWS_ACCESS_KEY The ACCESS_KEY from the generated credentials
S3_ENDPOINT https://radosgw-louros.cloud.grnet.gr

Example usage with s3cmd

s3cmd is a CLI tool that can be used to access an S3-compatible API. s3cmd can be installed with pip:

$ virtualenv --python=python3 venv  # optional, create a Python 3 virtual environment
$ . venv/bin/activate
$ pip install s3cmd

  1. Configure interactively with s3cmd --configure. You only need to configure Access Key, Secret Key, S3 Endpoint, and Use HTTPS protocol:

    $ s3cmd --configure
...
Access Key: <enter ACCESS_KEY>
Secret Key: <enter SECRET_KEY>
...

S3 Endpoint: radosgw-louros.cloud.grnet.gr
....
DNS-style bucket+hostname:port template for accessing a bucket: radosgw-louros.cloud.grnet.gr 

Use HTTPS protocol [Yes]: yes
 
...
Test access with supplied credentials? [Y/n] y
....

  2. List available buckets with s3cmd ls

    $ s3cmd ls
2020-11-18 20:23  s3://akolaitis-bucket

$ s3cmd ls s3://akolaitis-bucket
2020-11-18 20:28        43660  s3://akolaitis-bucket/my-important-file.tar.gz

  3. Retrieve object with s3cmd get and upload files with s3cmd put:

    $ s3cmd get s3://akolaitis-bucket/my-important-file.tar.gz
$ s3cmd put another-file.tar.gz s3://akolaitis-bucket/another-file.tar.gz
upload: 'another-file.tar.gz' -> 's3://akolaitis-bucket/another-file.tar.gz'  [1 of 1]
 43660 of 43660   100% in    0s   180.88 KB/s  done

Conclusion

By following the steps above you should be able to have a working object store Swift and/or S3 endpoint. Feel free to reach out in case you encounter any errors. Good luck!

~ Aggelos Kolaitis