Connecting to and using an existing artifact store in lineapy is easy and allows artifacts to be shared among team members.

Prerequisites

1. A working lineapy development environment with dependencies
   • Particularly the s3 and postgres extensions if these components are hosted remotely

2. Existing artifact store directory
   • This can be NFS, a local directory, or remote object storage like s3

3. Existing corresponding artifact store DB
   • On-prem or hosted DB instance

4. Credentials to access artifact store directory and DB if hosted remotely

Configurations of interest

artifact_storage_dir: points to filesystem or remote storage holding artifact values

Examples:

'artifact_storage_dir': '/usr/linea_pyrate/.lineapy/linea_pickles'

'artifact_storage_dir': 's3://bucket-name'

database_url: points to database holding artifact metadata

Examples:

'database_url': 'sqlite:////linea_pyrate/.lineapy/db.sqlite'

'database_url': 'postgresql://linea_pyrate:password@pg-db:5432/db-name

storage_options: optional configurations to connect to remote artifact stores

Example:

'storage_options',{'profile':'LINEA_PYRATE_AWS_PROFILE'}

Setting configurations

Below is a summary of setting configurations, which is covered in more detail in the lineapy internal documentation on configurations: https://docs.lineapy.org/en/latest/references/configurations.html

Configurations can be set in one of 3 ways before the session starts

1. Through the CLI on lineapy startup
2. Using environment variables
3. Setting lineapy configuration files

The CLI option is preferred if you can start the lineapy development environment using lineapy ipython. The lineapy configuration files option is preferred if you want to persist a configuration for future use.

Configurations can also be set in the development environment at the start of a session using lineapy.options.set. This is preferred for setting storage_options, and must be done at ​​the beginning of the ipython session. See here for more information.

Tips on using remote S3 hosted artifact store

To connect to a remote artifact store hosted in AWS, you must also set the credentials correctly.

More documentation can be found here:

https://docs.lineapy.org/latest/guides/configuration/storage-location/#storing-artifact-values-in-amazon-s3

An example of how these configurations are set in a lineapy_config.json file here:

https://gist.github.com/andycui97/66f9c3d6a87895d6c15a9741c81380ff