Why Gemfury?
Push, build, and install
RubyGems
npm packages
Python packages
Maven artifacts
PHP packages
Go Modules
Debian packages
RPM packages
NuGet packages
Repository URL to install this package:
|
Version:
4.1.70-4.1 ▾
|
python3-s3fuse-plugin
/
usr
/
lib
/
python3.6
/
site-packages
/
s3fuse
/
__pycache__
/
vaults3.cpython-36.opt-1.pyc
|
|---|
3
Ù2Ù_l ã @ sh d Z ddlZddlZddlZddlZddlZddlZddlmZ dZ dZ
dZ dd Z G d d
d
e
ZdS ) z Trilio S3 Backend implimentation
This module contains the back end implimentation of all of all S3 specific
support.
é N)Ú ClientErrorÚs3iè z$80bc80ff-0c51-4534-86a2-ec5e719643c2c C s& | j dd}|tjddd j }|S )zè Utility function used to convert a datetime to an OS timestamp.
Args:
modified_time (datetime): Datatime object to convert to a Unix Epoc timestamp.
Returns:
The value of modified_time as a timestamp.
N)Ztzinfoi² é )ÚreplaceÚdatetimeZ
total_seconds)Z
modified_timeZ
naive_timeZ
delta_seconds© r ú /usr/lib/python3.6/vaults3.pyÚ_make_timestamp s r c @ s e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd'd
dZ dd Z
dd Z d(ddZ dd Z
d)ddZdd Zd d Zd d Zd!d" Zd#d$ Zd%d& ZdS )*Ú S3Backendz´ S3 Backend implimentation.
A Wrapper for the AWS S3 boto3 and botocore API. This class encapsulates all S3
operations and exposes them as a backend storage instance.
c
C sÒ d }|d dkr.t jj|d t|d d}nt jjt|d d}|d dkrtjd|d d
|d |d |d
|d d ||d d| _n6tjd|d d
|d |d |d
|d d |d| _|d | _d S )NZ s3_signatureÚdefaultZs3_read_timeout)Zsignature_versionÚ read_timeout)r Z s3_ssl_certÚ r Z
os_optionsÚ region_nameZs3_sslÚuserÚkeyZobject_storage_url)r Úuse_sslÚaws_access_key_idÚaws_secret_access_keyÚ endpoint_urlÚconfigZverify)r r r r r r Zbucket)ÚbotocoreZclientZConfigÚintÚboto3Ú_S3Backend__clientÚ_S3Backend__bucket_name)ÚselfÚoptionsZ
config_objectr r r Ú__init__2 s*
zS3Backend.__init__c C s yzxtt |dkrvg }|dt
}x0|D ](}|jdr@|dd
}|jd|i q&W | jj| jd|id |td
}qW W n tk
r Y nX dS )zß Utility method that takes a list of objects and puts it in the correct format
for the S3 delete_objects() API.
Args:
object_list (list): A list of objects with the correct S3 path.
r Nú/r ÚKeyZObjects)ÚBucketZDelete)ÚlenÚMAX_S3_PAGE_SIZEÚ
startswithÚappendr Zdelete_objectsr r )r Ú object_listZobject_delete_listZ list_partÚobjr r r Z__delete_object_list] s
z S3Backend.__delete_object_listc C s y|g }| j jd}|j| j|d}xL|D ]D}d|kr(|jdt|d dkr(x |d D ]}|j|d qVW q(W | j| W n tk
r Y nX dS )z Utility method used to perform a rmdir operation on an object hierarchy
Args:
object_tree (str): Object/Tree path.
Ú list_objects)r ÚPrefixÚContentsÚKeyCountr r N) r Ú
get_paginatorÚpaginater Úgetr! r$ Ú _S3Backend__delete_object_listr )r Ú object_treer% Úlist_paginatorÚ
page_iteratorÚobjectsr& r r r Z__delete_object_treev s
z S3Backend.__delete_object_treec C s | j | dS )zó Delete a list of objects.
Args:
object_list (list): List of objects. Each with a full object path
in correct format.
options (dic): Dictionary of configuration options.
N)r. )r r% r r r r Údelete_object_list s z S3Backend.delete_object_listc C sz y^|s
dS dj |}|jdr*|dd
}d|krF| jj| j|d n| jj| j|d d W n tk
rt Y nX dS )a Delete an object from the S3 object store.
The object and any segments stored in the segment directory will be
removed.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Nr r z -segments)r r z .manifest)Újoinr# r Ú
delete_objectr r )r Úargsr Ú object_namer r r r5 s
zS3Backend.delete_objectr c C s¤ t d|}y~xxtd|D ]j}y| jj| j|d dS tk
r } z2|jd d dkrn|d |krntjd w W Y dd}~X qX qW W n tk
r Y nX dS )a Utility method used to wait for a S3 object to become available.
This routine will keep performing a head_object() request every second
for "retries" number of times. This was added to work around any potential
eventual consistancey issues when uploading to AWS.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter that defaults to 1.
Returns:
Returns when the object is available or ClientError not found exception.
r r )r r NÚResponseMetadataÚHTTPStatusCodei ) ÚmaxÚranger Ú head_objectr r ÚresponseÚtimeÚsleep)r r7 ÚretriesÚretryÚerrorr r r Z__wait_for_object³ s
zS3Backend.__wait_for_objectc C sÄ |j dd }y*| jj| j|d}tj|d j|d S tk
r¾ } zj|jd d dkr` nNy6| j |d | jj| j|d}tj|d j|d S tk
r¬ Y nX W Y d
d
}~X nX d
S ) zÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
r z .manifest)r r ÚBodyÚ
ContentLengthr8 r9 i é
N)
Ústripr Z
get_objectr ÚjsonÚloadsÚreadr r= Ú_S3Backend__wait_for_object)r r7 Ú
manifest_nameZresprB r r r Úget_object_manifestÒ s
z S3Backend.get_object_manifestc C sR y"dj |}| jj| j||j W n* tk
r8 Y n tk
rL Y nX dS )a? Download a file from the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
r N)r4 r Z
download_filer Zout_filer Ú Exception)r r6 r r7 r r r Údownload_objectì s
zS3Backend.download_objectNTc C sD t }dj|}|d dk rB|d |d }|jdrB|dd
}yât}|dk r^|tk r^|}| jjd}|s|j| jdd|id} n|j| j|d|id} x| D ]v}
d |
koÈ|
jd
t |
d d krÂ|d } xì|
d D ]à} |dk oôt ||kr t
|S t j j
| d jd\}
}|d
kr\t j j
|
\}}t |d krÞ|dkrÞ||krÞ|j| qÞt j j
|\}}||
kr~|
|ksÞ|dksÞ| d | krqÞ|dks¨|
|krÞ|j|j
ddd qÞW d|
kr¤xP|
d D ]D}|dk røt ||krøt
|S |d jtsÔ|j|d qÔW q¤W t
|S tk
r> Y nX dS )a° Return a list of objects based on the provided prefix.
Used to generate a directory listing based on a prefix path
constructed from the object parts in args[] and the options.prefix.
Args:
args (list): List of prefix keyParts
options (dic): Dictionary of configuration options
max_items (int): Maximum number of objects in a list query. Default is
unlimited but when testing for an empty directory a smaller
number can be used to limit the query and size of the list that
is returned.
strict_path_filter (bool): Defaults to True in order to only return
objects with the exact same prefix. Individual segment listings
set this to False in order to get a match on the segment offset.
Returns:
On success a list of unique items to be used as a directory listing.
r ÚprefixNr r' ZPageSize)r Z DelimiterÚPaginationConfig)r r( rP r) r* r r zx.hiddenr
Fz .manifestZCommonPrefixesr( éÿÿÿÿ)Úsetr4 Úendswithr" r r+ r, r r- r! ÚlistÚosÚpathÚsplitÚrstripÚaddr# Ú_SEGMENT_PREFIXr )r r6 r Z max_itemsÚstrict_path_filterZ
object_setrO Z page_sizer0 r1 r2 Z split_tokenÚitemrV r7 Z root_pathZsub_dirÚ_Z
object_prefixr r r r' sZ
"
zS3Backend.list_objectsc C sf yJg }t jj|j\}}| j||dd}x |D ]}|j|d | q.W |S tk
r` Y nX dS )a& Returns a list of object segments based on a prefix.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A list of segments in the object store that match the supplied prefix.
F)r[ r N)rU rV rW rO r' r$ r )r r6 r Z segment_listZ
segments_pathr] r% r\ r r r Ú
list_segmentse s
zS3Backend.list_segmentsc / C s t d|}yZxRtd|D ]B}y| jj| j|d}|S tk
r^ } z|jd d dkrLy| jj| j|d d}|S tk
rJ } z°|jd d dkr8d|kr y| jj| j|d d}|S tk
r } z6|jd d dkr
|d |kr
tjd w W Y d
d
}~X nX |d |kr8tjd w W Y d
d
}~X nX W Y d
d
}~X q X q W W n tk
r~ Y nX d
S ) a= Utility method that gets an object head from the repository with retry support.
The default is to try once. Admittedly, this is ugly because the
provided object name might be a terminal object or a "directory" so
we need to try both cases for each retry attempt. When retry > 1, there is
a 1 second delay between attempts.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter, default is 1 attempt.
Returns:
Object head dictionary or Boto3 exception.
r r )r r r8 r9 i r z -segmentsz .manifestN) r: r; r r< r r r= r> r? )r r7 r@ rA Z obj_headersrB r r r Z__get_object_headers{ sB
z S3Backend.__get_object_headersc C s2 ddddd}i |d< i |d d< dj |}yè|sHttjj |d< |S | j|}t|d |d< ||d< |jd
d d ks|d
dkrd|d< x0|d d j D ] \}}d|kr ||d |< q W t|d d jdddkrö|d d jdd|d< n |d
|d< |d |d< |S W n tk
r, Y nX dS )a) Get "operating system like" stat data for the given object.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A stat structure containing object information required by caller.
r r
F)Ú timestampÚsizeÚetagÚ directoryZheadersÚMetadatar r_ Z LastModifiedÚ ContentTypez
text/plainzapplication/x-directoryrD Trb z
x-object-metazx-object-meta-segment-countzx-object-meta-total-sizer` zx-account-bytes-usedN) r4 r r ZnowÚ _S3Backend__get_object_headersr- Úitemsr r )r r6 r Z stat_datar7 Z
obj_headerr Úvaluer r r Ú stat_object s2
zS3Backend.stat_objectc C s dj |}| j|| dS )zÎ Create an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r N)r4 Ú_S3Backend__create_folder)r r6 r Z
new_folderr r r Ú mkdir_objectÙ s
zS3Backend.mkdir_objectc C sD dj |d }|jdr$|dd
}| j| | jj| j|d dS )zÎ Remove an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r r N)r r )r4 r# Ú _S3Backend__delete_object_treer r5 r )r r6 r r/ r r r Ú rmdir_objectä s
zS3Backend.rmdir_objectc
C s| y`x*|D ]"}|j d|d< |j dd|d< qW tj|}|jdd }| jj| j|||dd W n tk
rv Y nX d
S ) a? Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added to the object.
manifest_data (dic): The manifest data which becomes the body of file.
rV Únamera r Úhashr z .manifestz
text/plain)r r rc rC rd N)ÚpoprG ÚdumpsrF r Ú
put_objectr r )r r7 Z put_headersZ
manifest_dataZ segment_dataZmanifestrK r r r Úupload_object_manifestõ s
z S3Backend.upload_object_manifestc ) C sX y| j j| j|d W n: tk
rR } z|jd d dkrB|jd}d}xð|D ]è}|dkrdP || d }y| j j| j|d W qV tk
r< } z|jd d dkr*|d dkrÔ| j j| j|d d
d nVy| j j| j|d d
d W n: tk
r( } z |jd d dkr W Y dd}~X nX wVW Y dd}~X qVX qVW W Y dd}~X nX dS )a< Utility method used to create a object directory structure.
The entire path is split and each directory level is created as an object
if it does not exist.
Args:
folder_name (str): The entire folder name.
options (dic): Dictionary of configuration options.
)r r r8 r9 i r r
Zsupport_empty_dirTzx.hiddenz
Do Not Remove)r r rC z TrilioVault directory objectzapplication/x-directory)r r rC rd i N)r r< r r r= rW rq )r Z folder_namer rB Ú
path_partsÚnew_pathÚpartr r r Z__create_folder
s4
zS3Backend.__create_folderc C s¨ |dd
}y|j dkr<| jtjj|d d |d | |jdk rt|d d0}| jj|| j |d d |d dd id
W dQ R X W n t
k
r¢ Y nX dS ) zµ Upload an object to the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r NFr r r7 Úrbrd zapplication/octet-stream)r r Z ExtraArgs) Z
path_validri rU rV Údirnamer7 Úopenr Zupload_fileobjr r )r r6 r ÚfilesÚdatar r r Ú
upload_object= s
zS3Backend.upload_object)r )NT)r )Ú__name__Ú
__module__Ú __qualname__Ú__doc__r r. rk r3 r5 rJ rL rN r' r^ re rh rj rl rr ri r{ r r r r r
, s$ +
c
2, 0r
)r r r> rG rU r r Zbotocore.exceptionsr ZBASENAMEr" rZ r Úobjectr
r r r r Ú<module> s