Skip to content

S3Proxy for access using aws cli, python, java etc.

InfinSnap for S3 provides fine grained bucket wide snapshots for your S3 buckets. Applications that read multiple files from S3 in the manner of traditional file systems will benefit from bucket wide snapshots over object versioning. Once you have InfinSnap for S3 enabled for a bucket, you can view snapshots of the bucket's contents in the dashboard using the web browser, or using the s3proxy using any s3 client such as the aws cli, aws python sdk, etc.

Accessing InfinSnap snapshots using the InfinStor S3proxy

  1. Get the InfinStor assigned credentials for s3proxy, i.e. the Access Key ID and Secret Key for use with s3proxy
  2. Understand the endpoint configuration that specifies the InfinSnap snapshot
  3. Configure your s3 client application to use the s3proxy endpoint for access to InfinSnap snapshot view of your s3 bucket's state at any time in the past

1. S3Proxy Credentials

The second section of the main pane shows the Access Key ID. Press the 'Show' button to view, copy and save the Secret Access Key for use by your s3 client application

2. Endpoint Config

The endpoint URL is used to direct the s3 client to the InfinStor S3 proxy. Additionally, the hostname includes the InfinSnap encoded in it

  • The endpoint is of the format
  • The tmYYYYMMDDHHMMSS component of the hostname is used to specify the time of the snapshot in UTC
    • YYYY stands for year
    • MM stands for month (Jan is 01)
    • DD is for day
    • HH is for hour (24 hour format)
    • MM is minutes and SS is seconds.

aws s3 --endpoint ls s3://network-ddos-ml/testf1/
2020-02-28 11:49:02   39980244 ec2-imageprocessor (1).log
In the above example, the endpoint URL directs the aws cli to use the InfinStor s3proxy to get a listing of the bucket network-ddos-ml at the prefix (folder) /testf1/. The time as encoded in the URL tm20200229180000 is Feb 29, 2020 at 6 PM UTC.

3. AWS Config Profile

The credentials that you download for access to InfinStor's S3 Proxy are usable only with the s3proxy, and not with other AWS services. In most cases, you may have another set of credentials for other AWS services such as EC2, DynamoDB, etc. In this case configure the s3proxy credentials using a different profile and refer to this profile in S3 requests. For example:

$ cat ~/.aws/credentials
aws_access_key_id = AKIAXXXXXXXXXXXXXXXX
aws_secret_access_key = oegr94390gremmkgio590klmgtmkvfdndl595lgf

aws_access_key_id = AKIAYYYYYYYYYYYYYYYY
aws_secret_access_key = sdlk49t53kngkdk395968nkfnfgdndij9j590690

$ aws s3 --endpoint ls s3://network-ddos-ml/testf1/

An error occurred (SignatureDoesNotMatch) when calling the ListObjectsV2 operation: Unknown
$ aws s3 --profile s3proxy --endpoint ls s3://network-ddos-ml/testf1/
2020-02-28 11:49:02   39980244 ec2-imageprocessor (1).log
In the above example, the s3proxy credentials are configured as profile s3proxy. Following that, using the aws cli without specifying the profile fails, while specifying the s3proxy profile results in success

The infinstor access credentials can be obtained in the following three ways:

  • Login using the Jupyterlab sidebar, the [infinstor] profile is automatically added to the ~/.aws/credentials file
  • Login using the CLI, i.e. by running 'python -m infinstor_mlflow_plugin.login'. The [infinstor] profile is automatically added to the ~/.aws/credentials file
  • Login to the dashboard, i.e. and go to 'Configuration->S3Proxy Access'. The credentials are displayed, and must be copied to the file ~/.aws/credentials by hand