Why Gemfury?
Push, build, and install
RubyGems
npm packages
Python packages
Maven artifacts
PHP packages
Go Modules
Debian packages
RPM packages
NuGet packages
Repository URL to install this package:
|
Version:
4.1.94.1.dev5-4.1 ▾
|
python3-s3fuse-plugin
/
usr
/
lib
/
python3.6
/
site-packages
/
s3fuse
/
__pycache__
/
vaults3.cpython-36.opt-1.pyc
|
|---|
3
¬`^p ã @ sp d Z ddlZddlZddlZddlZddlZddlZddlmZ dZ dZ
dZ dd Z d d
Z
G d d d eZdS )
z Trilio S3 Backend implimentation
This module contains the back end implimentation of all of all S3 specific
support.
é N)Ú ClientErrorÚs3iè z$80bc80ff-0c51-4534-86a2-ec5e719643c2c C s& | j dd}|tjddd j }|S )zè Utility function used to convert a datetime to an OS timestamp.
Args:
modified_time (datetime): Datatime object to convert to a Unix Epoc timestamp.
Returns:
The value of modified_time as a timestamp.
N)Ztzinfoi² é )ÚreplaceÚdatetimeZ
total_seconds)Z
modified_timeZ
naive_timeZ
delta_seconds© r ú /usr/lib/python3.6/vaults3.pyÚ_make_timestamp s r c C sZ yF| j |d tjjddddddd}| j||d d
| j||d W n Y nX d S )
a Validates the S3 credentials / permission on the provided bucket.
Args:
s3_client : Initialized boto3 S3 client object
s3_bucket : User provided S3 bucket
Returns:
Does not return anything, But raises exception if validation fails.
)ÚBucketztvault_config/z.workload_f5190be6-7f80-4856-8c24-149cb40500c5/z.snapshot_f2e5c6a7-3c21-4b7f-969c-915bb408c64f/z+vm_id_e81d1ac8-b49a-4ccf-9d92-5f1ef358f1be/z3vm_res_id_72477d99-c475-4a5d-90ae-2560f5f3b319_vda/z.deac2b8a-dca9-4415-adc1-f3c6598204ed-segments/z0000000000000000.00000000z Test Data)r
ÚKeyÚBody)r
r N)Z head_bucketÚosÚpathÚjoinÚ
put_objectÚ
delete_object)Ú s3_clientZ s3_bucketZlong_keyr r r Úvalidate_s3_client, s
r c @ s e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd'd
dZ dd Z
dd Z d(ddZ dd Z
d)ddZdd Zd d Zd d Zd!d" Zd#d$ Zd%d& ZdS )*Ú S3Backendz´ S3 Backend implimentation.
A Wrapper for the AWS S3 boto3 and botocore API. This class encapsulates all S3
operations and exposes them as a backend storage instance.
c
C sæ d }d }|d dkr2t jj|d t|d d}nt jjt|d d}|d dkrtjd|d d
|d |d |d
|d d ||d d}n4tjd|d d
|d |d |d
|d d |d}t||d || _|d | _d S )NZ s3_signatureÚdefaultZs3_read_timeout)Zsignature_versionÚ read_timeout)r Z s3_ssl_certÚ r Z
os_optionsÚ region_nameZs3_sslÚuserÚkeyZobject_storage_url)r Úuse_sslÚaws_access_key_idÚaws_secret_access_keyÚ endpoint_urlÚconfigZverify)r r r r r r Zbucket)ÚbotocoreZclientZConfigÚintÚboto3r Ú_S3Backend__clientÚ_S3Backend__bucket_name)ÚselfÚoptionsZ
config_objectr r r r Ú__init__R s.
zS3Backend.__init__c C s yzxtt |dkrvg }|dt
}x0|D ](}|jdr@|dd
}|jd|i q&W | jj| jd|id |td
}qW W n tk
r Y nX dS )zß Utility method that takes a list of objects and puts it in the correct format
for the S3 delete_objects() API.
Args:
object_list (list): A list of objects with the correct S3 path.
r Nú/r r ZObjects)r
ZDelete)ÚlenÚMAX_S3_PAGE_SIZEÚ
startswithÚappendr# Zdelete_objectsr$ r )r% Ú object_listZobject_delete_listZ list_partÚobjr r r Z__delete_object_list s
z S3Backend.__delete_object_listc C s y|g }| j jd}|j| j|d}xL|D ]D}d|kr(|jdt|d dkr(x |d D ]}|j|d qVW q(W | j| W n tk
r Y nX dS )z Utility method used to perform a rmdir operation on an object hierarchy
Args:
object_tree (str): Object/Tree path.
Ú list_objects)r
ÚPrefixÚContentsÚKeyCountr r N) r# Ú
get_paginatorÚpaginater$ Úgetr) r, Ú _S3Backend__delete_object_listr )r% Ú object_treer- Úlist_paginatorÚ
page_iteratorÚobjectsr. r r r Z__delete_object_tree s
z S3Backend.__delete_object_treec C s | j | dS )zó Delete a list of objects.
Args:
object_list (list): List of objects. Each with a full object path
in correct format.
options (dic): Dictionary of configuration options.
N)r6 )r% r- r& r r r Údelete_object_list® s z S3Backend.delete_object_listc C sz y^|s
dS dj |}|jdr*|dd
}d|krF| jj| j|d n| jj| j|d d W n tk
rt Y nX dS )a Delete an object from the S3 object store.
The object and any segments stored in the segment directory will be
removed.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Nr( r z -segments)r
r z .manifest)r r+ r# r r$ r )r% Úargsr& Ú object_namer r r r ¸ s
zS3Backend.delete_objectr c C s¤ t d|}y~xxtd|D ]j}y| jj| j|d dS tk
r } z2|jd d dkrn|d |krntjd w W Y dd}~X qX qW W n tk
r Y nX dS )a Utility method used to wait for a S3 object to become available.
This routine will keep performing a head_object() request every second
for "retries" number of times. This was added to work around any potential
eventual consistancey issues when uploading to AWS.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter that defaults to 1.
Returns:
Returns when the object is available or ClientError not found exception.
r r )r
r NÚResponseMetadataÚHTTPStatusCodei ) ÚmaxÚranger# Ú head_objectr$ r ÚresponseÚtimeÚsleep)r% r= ÚretriesÚretryÚerrorr r r Z__wait_for_objectÖ s
zS3Backend.__wait_for_objectc C sÄ |j dd }y*| jj| j|d}tj|d j|d S tk
r¾ } zj|jd d dkr` nNy6| j |d | jj| j|d}tj|d j|d S tk
r¬ Y nX W Y d
d
}~X nX d
S ) zÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
r( z .manifest)r
r r Ú
ContentLengthr> r? i é
N)
Ústripr# Z
get_objectr$ ÚjsonÚloadsÚreadr rC Ú_S3Backend__wait_for_object)r% r= Ú
manifest_nameZresprH r r r Úget_object_manifestõ s
z S3Backend.get_object_manifestc C sR y"dj |}| jj| j||j W n* tk
r8 Y n tk
rL Y nX dS )a? Download a file from the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
r( N)r r# Z
download_filer$ Zout_filer Ú Exception)r% r< r& r= r r r Údownload_object s
zS3Backend.download_objectNTc C sD t }dj|}|d dk rB|d |d }|jdrB|dd
}yât}|dk r^|tk r^|}| jjd}|s|j| jdd|id} n|j| j|d|id} x| D ]v}
d |
koÈ|
jd
t |
d d krÂ|d } xì|
d D ]à} |dk oôt ||kr t
|S t j j
| d jd\}
}|d
kr\t j j
|
\}}t |d krÞ|dkrÞ||krÞ|j| qÞt j j
|\}}||
kr~|
|ksÞ|dksÞ| d | krqÞ|dks¨|
|krÞ|j|j
ddd qÞW d|
kr¤xP|
d D ]D}|dk røt ||krøt
|S |d jtsÔ|j|d qÔW q¤W t
|S tk
r> Y nX dS )a° Return a list of objects based on the provided prefix.
Used to generate a directory listing based on a prefix path
constructed from the object parts in args[] and the options.prefix.
Args:
args (list): List of prefix keyParts
options (dic): Dictionary of configuration options
max_items (int): Maximum number of objects in a list query. Default is
unlimited but when testing for an empty directory a smaller
number can be used to limit the query and size of the list that
is returned.
strict_path_filter (bool): Defaults to True in order to only return
objects with the exact same prefix. Individual segment listings
set this to False in order to get a match on the segment offset.
Returns:
On success a list of unique items to be used as a directory listing.
r( ÚprefixNr r/ ZPageSize)r
Z DelimiterÚPaginationConfig)r
r0 rU r1 r2 r r zx.hiddenr Fz .manifestZCommonPrefixesr0 éÿÿÿÿ)Úsetr Úendswithr* r# r3 r4 r$ r5 r) Úlistr
r ÚsplitÚrstripÚaddr+ Ú_SEGMENT_PREFIXr )r% r< r& Z max_itemsÚstrict_path_filterZ
object_setrT Z page_sizer8 r9 r: Z split_tokenÚitemr r= Z root_pathZsub_dirÚ_Z
object_prefixr r r r/ % sZ
"
zS3Backend.list_objectsc C sf yJg }t jj|j\}}| j||dd}x |D ]}|j|d | q.W |S tk
r` Y nX dS )a& Returns a list of object segments based on a prefix.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A list of segments in the object store that match the supplied prefix.
F)r^ r( N)r
r rZ rT r/ r, r )r% r< r& Z segment_listZ
segments_pathr` r- r_ r r r Ú
list_segments s
zS3Backend.list_segmentsc / C s t d|}yZxRtd|D ]B}y| jj| j|d}|S tk
r^ } z|jd d dkrLy| jj| j|d d}|S tk
rJ } z°|jd d dkr8d|kr y| jj| j|d d}|S tk
r } z6|jd d dkr
|d |kr
tjd w W Y d
d
}~X nX |d |kr8tjd w W Y d
d
}~X nX W Y d
d
}~X q X q W W n tk
r~ Y nX d
S ) a= Utility method that gets an object head from the repository with retry support.
The default is to try once. Admittedly, this is ugly because the
provided object name might be a terminal object or a "directory" so
we need to try both cases for each retry attempt. When retry > 1, there is
a 1 second delay between attempts.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter, default is 1 attempt.
Returns:
Object head dictionary or Boto3 exception.
r r )r
r r> r? i r( z -segmentsz .manifestN) r@ rA r# rB r$ r rC rD rE )r% r= rF rG Z obj_headersrH r r r Z__get_object_headers sB
z S3Backend.__get_object_headersc C s2 ddddd}i |d< i |d d< dj |}yè|sHttjj |d< |S | j|}t|d |d< ||d< |jd
d d ks|d
dkrd|d< x0|d d j D ] \}}d|kr ||d |< q W t|d d jdddkrö|d d jdd|d< n |d
|d< |d |d< |S W n tk
r, Y nX dS )a) Get "operating system like" stat data for the given object.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A stat structure containing object information required by caller.
r r F)Ú timestampÚsizeÚetagÚ directoryZheadersÚMetadatar( rb Z LastModifiedÚ ContentTypez
text/plainzapplication/x-directoryrI Tre z
x-object-metazx-object-meta-segment-countzx-object-meta-total-sizerc zx-account-bytes-usedN) r r r ZnowÚ _S3Backend__get_object_headersr5 Úitemsr! r )r% r< r& Z stat_datar= Z
obj_headerr Úvaluer r r Ú stat_objectÐ s2
zS3Backend.stat_objectc C s dj |}| j|| dS )zÎ Create an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r( N)r Ú_S3Backend__create_folder)r% r< r& Z
new_folderr r r Ú mkdir_objectü s
zS3Backend.mkdir_objectc C sD dj |d }|jdr$|dd
}| j| | jj| j|d dS )zÎ Remove an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r( r N)r
r )r r+ Ú _S3Backend__delete_object_treer# r r$ )r% r< r& r7 r r r Ú rmdir_object s
zS3Backend.rmdir_objectc
C s| y`x*|D ]"}|j d|d< |j dd|d< qW tj|}|jdd }| jj| j|||dd W n tk
rv Y nX d
S ) a? Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added to the object.
manifest_data (dic): The manifest data which becomes the body of file.
r Únamerd r Úhashr( z .manifestz
text/plain)r
r rf r rg N)ÚpoprL ÚdumpsrK r# r r$ r )r% r= Z put_headersZ
manifest_dataZ segment_dataZmanifestrP r r r Úupload_object_manifest s
z S3Backend.upload_object_manifestc ) C sX y| j j| j|d W n: tk
rR } z|jd d dkrB|jd}d}xð|D ]è}|dkrdP || d }y| j j| j|d W qV tk
r< } z|jd d dkr*|d dkrÔ| j j| j|d d
d nVy| j j| j|d d
d W n: tk
r( } z |jd d dkr W Y dd}~X nX wVW Y dd}~X qVX qVW W Y dd}~X nX dS )a< Utility method used to create a object directory structure.
The entire path is split and each directory level is created as an object
if it does not exist.
Args:
folder_name (str): The entire folder name.
options (dic): Dictionary of configuration options.
)r
r r> r? i r( r Zsupport_empty_dirTzx.hiddenz
Do Not Remove)r
r r z TrilioVault directory objectzapplication/x-directory)r
r r rg i N)r# rB r$ r rC rZ r )r% Z folder_namer& rH Ú
path_partsÚnew_pathÚpartr r r Z__create_folder0 s4
zS3Backend.__create_folderc C s¨ |dd
}y|j dkr<| jtjj|d d |d | |jdk rt|d d0}| jj|| j |d d |d dd id
W dQ R X W n t
k
r¢ Y nX dS ) zµ Upload an object to the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r NFr r( r= Úrbrg zapplication/octet-stream)r
r Z ExtraArgs) Z
path_validrl r
r Údirnamer= Úopenr# Zupload_fileobjr$ r )r% r< r& ÚfilesÚdatar r r Ú
upload_object` s
zS3Backend.upload_object)r )NT)r )Ú__name__Ú
__module__Ú __qualname__Ú__doc__r' r6 rn r; r rO rQ rS r/ ra rh rk rm ro rt rl r} r r r r r L s$ .
c
2, 0r )r r rD rL r
r" r Zbotocore.exceptionsr ZBASENAMEr* r] r r Úobjectr r r r r Ú<module> s