S3 Fuse Plugin

This is a python pip module that creates a virtual fuse layer so that we can easily mount data to an S3 bucket. Since we will use this in the container environment, this project is running inside of Docker container.

How to use

1. Allow SSH from your local machine to your Gihub private repositories

   1. Generating a new SSH key and adding it to the ssh-agent
   2. Adding a new SSH key to your GitHub account

2. Create valut.conf file with the information below

   [DEFAULT]
   vault_data_directory = 
   vault_data_directory_old = 
   vault_storage_das_device = none
   vault_storage_type = s3
   vault_s3_auth_version = DEFAULT
   vault_s3_access_key_id = 
   vault_s3_secret_access_key = 
   vault_s3_region_name = 
   vault_s3_bucket = 
   vault_s3_endpoint_url =
   vault_s3_signature_version = default
   vault_s3_ssl = False
   vault_enable_threadpool = True
   vault_s3_support_empty_dir = False
   
   [s3fuse_sys_admin]
   helper_command = sudo PATH_OF_PRIVSEP_HELPER_BINARY
   

3. Add following entry in nova sudoers file for privsep-helper nova ALL = (root) NOPASSWD: PATH_OF_PRIVSEP_HELPER_BINARY *

4. pip install git+ssh://git@github.com/trilioData/s3-fuse-plugin.git

5. Run it.

   import s3fuse # This is the name of the module created by our pip install command
   
   # mount start
   s3fuse.mount()
   
   # umount and clean all the files and folder generated from the mount
   s3fuse.umount()
   
   """
   Fixed Required
       There is a bug that build one additional mount points.
       For now, this function will remove the files and folders from this mountpoint.
   """
   s3fuse.clean()
   

Update flow

1. Folk the repository and make changes
2. Push them to the folked repository
3. Send the merge request to the original repo(trilioData/s3-fuse-plugin)
4. Travis CI detects the changes in the master branch and runs the test
5. If all the test is passed, Travis CI reports the coverage
6. Now, it's good to merge forked repo into the original repo(trilioData/s3-fuse-plugin)

Things to know

1. Use moto to mock S3
   • In real cases, AWS credentials will be written in a vault.conf file bypassing as environment variables. However, for the testing purpose, I used moto library, which mocks the s3 connection.

Possible Improvements

1. The test codes have not hit 100 percent coverage. Even though the parts that have not covered yet are trivial, they should be covered eventually to avoid the edge cases.

2. Except for the two mount points we assign, the current working directory is automatically set as an additional mount point.