Skip to content

Configure AWS Services to protect MLflow data from accidental deletion

InfinStor MLflow uses cloud object stores (s3, azure blob store, etc.) for the actual storage of artifacts. This page describes the procedure for protection of data from accidental deletion. Two AWS technologies will be employed for this purpose:

  • S3 Cross Region Replication
  • S3 Object Lock

Details of S3 Object Lock can be found at https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html

Summary of the four steps for configuring AWS S3 for total MLflow data protection

The following four steps must be performed in order to protect MLflow data from cloud outages and accidental deletion.

Step 1: Backup bucket

Create a backup bucket in a different region from your main working region (us-west-2 in this example). This bucket must have versioning enabled and object lock enabled. The following screen capture shows these two options enabled in the bucket creation page:

Step 2: Enable versioning for the MLflow artifacts bucket

The artifacts bucket may be the one created by InfinStor MLflow at installation time, or it can be some other bucket that is utilized for MLflow artifact storage

Step 3: Create a Replication Rule

In this step, we configure the main MLflow artifact bucket with a replication rule that sends a copy of all objects to the backup bucket. As shown in the screen capture below, all objects are replicated, and a new role is creted for the replication task.

Step 4: Enable Object Lock in Backup Bucket

The next step is to enable object lock for the backup bucket as shown below. In this example, the object retention mode is compliance, and the retention period is 1 day. Choose a retention mode that is suitable for your purpose.

Testing configuration

To test this configuration, copy an object into the source bucket, i.e. the MLflow artifacts bucket. After a minute or so, that object should show up in the console for the backup bucket. Now, if you try to 'empty' the backup bucket, S3 should refuse to do so with the following message: