InfinSnap, InfinSlice Access using boto3

InfinSnap and InfinSlice may be accessed using the standard boto3 python client. There are two important considerations for using boto3 to access InfinSnap and InfinSlice:

  • Profile infinstor in ~/.aws/credentials
  • Endpoint Configuration

InfinStor includes an S3 proxy for access to InfinSnap and InfinSlice. The snapshot and slice times are encoded in the endpoint specification. Further, an independent set of credentials recognized only by the s3proxy, and not by Amazon's S3 service, are included. These credentials must be saved in the user's ~/.aws/credentials as profile infinstor

Profile infinstor in ~/.aws/credentials

The credentials used for access to InfinSnap and InfinSlice through the InfinStor s3proxy start with AKII and are written in the user's ~/.aws/credentials file. Here is an example (don't worry - these are invalid credentials):

[default]
aws_access_key_id = AKIAAAABBBCCCDDDEEEF
aws_secret_access_key = kjf589gjnksjnkbh5jngnjfhjfuitg89jnkgtvd

[infinstor]
aws_access_key_id = AKIIXXXXYYYYYZZZZZAA
aws_secret_access_key = vcfire8954jngnjbfnjkgfhjnthtrnhjgfhnjgfh

The infinstor access credentials can be obtained in the following three ways:

  • Login using the Jupyterlab sidebar, the [infinstor] profile is automatically added to the ~/.aws/credentials file
  • Login using the CLI, i.e. by running 'python -m infinstor_mlflow_plugin.login'. The [infinstor] profile is automatically added to the ~/.aws/credentials file
  • Login to the dashboard, i.e. https://service.infinstor.com/ and go to 'Configuration->S3Proxy Access'. The credentials are displayed, and must be copied to the file ~/.aws/credentials by hand

Endpoint Configuration

There are two convenience functions provided in the infinstor pip package for generating endpoint specification strings. These are:

infinsnap

def infinsnap(snaptime=datetime.now()):

This function takes an optional parameter snaptime and returns a string that defines the infinsnap endpoint. The following code snippet demonstrates the use of this function with boto3 s3 'client'

from infinstor import infinsnap, infinslice
from datetime import datetime, timedelta
import boto3

session = boto3.Session(profile_name='infinstor')
client = session.client('s3', endpoint_url=infinsnap())

all_objects = client.list_objects(Bucket='infinstordemo', Prefix='spot-prices/')
print(all_objects)

This function can be used with the 'resource' concept in boto3 in the following manner:

from infinstor import infinsnap, infinslice
from datetime import datetime, timedelta
import boto3

now = datetime.now()

session = boto3.Session(profile_name='infinstor')
s3 = session.resource('s3', endpoint_url=infinsnap(now - timedelta(hours=48)))
bucket = s3.Bucket(name='infinstordemo')

for obj in bucket.objects.filter(Prefix='spot-prices/'):
    print(str(obj))

Here is a screen capture of the above example:

infinslice

def infinslice(start_time, end_time):

This function takes an two parameters - start_time and end_time and returns a string that defines the infinslice endpoint. In the following code snippet, we demonstrate the use of this function with the 'client' access model of boto3

from infinstor import infinsnap, infinslice
from datetime import datetime, timedelta
import boto3

now = datetime.now()

session = boto3.Session(profile_name='infinstor')
client = session.client('s3', endpoint_url=infinslice(now - timedelta(hours=24), now))

all_objects = client.list_objects(Bucket='infinstordemo', Prefix='spot-prices/')
print(all_objects)

And, here's a 'resource' access model example

from infinstor import infinsnap, infinslice
from datetime import datetime, timedelta
import boto3

now = datetime.now()

session = boto3.Session(profile_name='infinstor')
s3 = session.resource('s3', endpoint_url=infinslice(now - timedelta(hours=48), now))
bucket = s3.Bucket(name='infinstordemo')

for obj in bucket.objects.filter(Prefix='spot-prices/'):
    print(str(obj))

Finally, here is a screen capture of the above example: