Skip to content

InfinStor Mlflow Projects - Concepts and High Level Architecture

InfinStor Mlflow Projects is a Mlflow Projects plugin for running mlflow projects in the cloud, either in a single VM or in an EMR cluster. The InfinStor Starter service includes functionality to run mlflow projects in a single EC2 VM in the cloud and the InfinStor Premium service enables users to run their projects in an EMR cluster. Project tracking is autmatic with InfinStor Mlflow Tracking service.

Architecture

  • This shows the Mlflow Projects components of the InfinStor Starter service
  • InfinStor supplies a projects backend plugin for the open source mlflow library installed using pip
  • Applications must add the parameter '-b infinstor-backend' to the 'mlflow run' command
  • During InfinStor Starter service setup, the infinstor-config.sh script creates a bucket for storage of mlflow artifacts
  • This bucket is also used for copying the project's source files from the computer where 'mlflow run' is called to the single EC2 VM or EMR cluster in the cloud where the project is actually executed
  • The single EC2 VM is created on demand and shutdown when idle