Manage S3 Buckets

Concepts

InfinStor maintains a table of buckets. A bucket known to InfinStor may be used for InfinSnap, InfinSlice or MLflow artifact storage. This table of buckets has access information for each bucket, such as:

• access key id and secret access key.
• or role ARN and role external ID.

Buckets can be managed using

• Manage Buckets using the UI
• or the REST API

Manage Buckets using the UI

Buckets can be managed using the Infinstor service dashboard available at <service_dns_name>.<infinstor_service_dns_name>. - For example service.infinstor.yourcompany.com.

See Infinstor Service Dashboard Overview for more information about the Infinstor Service Dashboard.

This bucket management is available only for Administrators of Infinstor service.

Following screenshot shows the Bucket management UI which can be used to

• add a new bucket
• edit an existing bucket
• remove an existing bucket

See Create Bucket For MLFlow artifacts for how to create an AWS Role which allows access to an S3 bucket from a foreign AWS account.

REST API

InfinStor provides an API for adding, modifying and removing buckets from this buckets table

Adding, modifying and removing buckets

URL: https://your_service_server/add-mod-del-bucket

Method: POST

Data:

• op: required. must be add, modify or delete
• bucketname: required for the add operation.
• cloud: required for add operation. Must be aws, azure, minio or other
• endpoint: optional
• sharedSecret: optional. used by minio storage for sending infinsnap events from minio to InfinStor object_store_event lambda
• accessKeyId: optional
• secretAccessKey: optional
• accessRole: optional. See Create Bucket For MLFlow artifacts for how to create an AWS Role which allows access to an S3 bucket from a foreign AWS account.
• accessRoleExt: optional. This is the External ID specified during the creation of the AWS Role

Notes:

Calls to the API must be authenticated. Only the administrator for the service has permissions to make this call.

Example 1:

In this example, a new buckets table entry is created for the bucket named testb3. This bucket uses the default endpoint and has access role called aaaa with external id bbbb. The following is the data sent in the POST call

op=add&bucketname=testb3&endpoint=&accessKeyId=&secretAccessKey=&accessRole=aaaa&accessRoleExt=bbbb

Here are some hints if the bucket testb3 belongs to another AWS account.

The bucket must have CORS enabled

Go to the bucket, click on permissions, scroll down to the CORS section and add a policy that might look like:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3600
    }
]

Role Details

The role described as aaa in the example above is a role ARN. The role must have a permissions policy that enables access to the bucket testb3. For example, the following policy would work

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::testb3"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::testb3/*"
            ]
        }
    ]
}

Example 2:

In this example, an existing buckets table entry is modified for the bucket named testb3. A non default endpoint https://minio.isstage1.bb.com/ is specified in this call. Note that the endpoint URL is urlencoded

op=modify&bucketname=testb3&endpoint=https%3A%2F%2Fminio.isstage1.bb.com%2F

Example 3:

In this example, an existing buckets table entry for the bucket named testb3 is deleted

op=delete&bucketname=testb3

Example 4:

In this example, an existing buckets table entry for the bucket named testb3 is converted from role based access to secret access key based access

op=add&bucketname=testb3&endpoint=&accessKeyId=iiii&secretAccessKey=ssss&accessRole=&accessRoleExt=