InfinSnap, InfinSlice Access using boto3¶

InfinStor includes the capability to access snapshots(InfinSnap) and slices(InfinSlice) of data using the boto3 python package. InfinStor provides a intercept mechanism called infin_boto3 that is used for this purpose.

Read InfinSnap¶

import os
import boto3
from infinstor import infin_boto3, infinsnap
from datetime import datetime, timezone

s3client = boto3.client('s3', infinstor_time_spec=infinsnap(datetime(2022, 12, 6, tzinfo=timezone.utc)))
resp = s3client.list_objects_v2(Bucket='test-bucket-xyzzy', Prefix='index1/', Delimiter='/')
if 'Contents' in resp:
    for one in resp['Contents']:
        nm = one['Key']
        if not nm[-1] == '/':
            dnm = os.path.join('/tmp', nm[nm.rindex('/') + 1:])
            print(f"Downloading object {nm} to file {dnm}")
            s3client.download_file('test-bucket-xyzzy', nm, dnm)

First, we create a boto3 s3 client for a specific snapshot time.

from infinstor import infin_boto3, infinsnap

Then, we activate the infin_boto3 intercept mechanism. The chosen snapshot time is December 6, 2022. This must be present in all the python files where InfinSnap or InfinSlice are used. Note that times are specified in UTC

s3client = boto3.client('s3', infinstor_time_spec=infinsnap(datetime(2022, 12, 6)))

Read InfinSlice¶

import os
import boto3
from infinstor import infin_boto3, infinslice
from datetime import datetime, timezone

s3client = boto3.client('s3', infinstor_time_spec=infinslice(datetime(2022, 12, 1, tzinfo=timezone.utc), datetime(2022, 12, 10, tzinfo=timezone.utc)))
resp = s3client.list_objects_v2(Bucket='test-bucket-xyzzy', Prefix='index1/', Delimiter='/')
if 'Contents' in resp:
    for one in resp['Contents']:
        nm = one['Key']
        if not nm[-1] == '/':
            dnm = os.path.join('/tmp', nm[nm.rindex('/') + 1:])
            print(f"Downloading object {nm} to file {dnm}")

In this example, we choose a slice of time between December 1, 2022 and December 10, 2022. This must be present in all the python files where InfinSnap or InfinSlice are used. Note that times are specified in UTC

s3client = boto3.client('s3', infinstor_time_spec=infinsnap(datetime(2022, 12, 6)))

Upload to InfinSnap bucket¶

InfinStor includes a feature where you can use InfinStor login to access InfinSnap enabled buckets for both write and read. In the following example, a file called test1.json is uploaded to the s3 bucket.

import boto3
from infinstor import infin_boto3

s3client = boto3.client('s3')
response = s3client.upload_file('test1.json', 'test-bucket-xyzzy', 'test/test1.json')

The value of this method of uploading files to s3 is that you do not need s3 access key id and secret access key specified in the ~/.aws/credentials file

Note that when an object is created in this manner, the snapshot time associated with this object is the time when this specific python code was run. In case you want to set the infinsnap time to a different time, then take a look at the example below. For example, if you are creating indexes from CloudWatch log streams, perhaps it is useful to set the creation time of the index file to be the same as the creation time of the CloudWatch log stream.

import boto3
from infinstor import infin_boto3

s3client = boto3.client('s3')
response = s3client.upload_file('test1.json', 'test-bucket-xyzzy', 'test/test1.json',
    ExtraArgs={
        "Metadata": {
            "infinsnap": "1672792947"
        }
    }
)

In the above example 1672792947 is the milliseconds since the epoch, specified in UTC.