Why Gemfury?
Push, build, and install
RubyGems
npm packages
Python packages
Maven artifacts
PHP packages
Go Modules
Debian packages
RPM packages
NuGet packages
Repository URL to install this package:
|
Version:
4.2.32-4.2 ▾
|
python3-s3fuse-plugin
/
usr
/
lib
/
python3.6
/
site-packages
/
s3fuse
/
__pycache__
/
vaults3.cpython-36.opt-1.pyc
|
|---|
3
䓍aJ ã @ sl d Z ddlZddlZddlZddlZddlZddlZddlmZ dZ dZ
dZ dZ dd Z
G d
d d eZdS ) z Trilio S3 Backend implimentation
This module contains the back end implimentation of all of all S3 specific
support.
é N)Ú ClientErrorÚs3iè z$80bc80ff-0c51-4534-86a2-ec5e719643c2iô c C s& | j dd}|tjddd j }|S )zè Utility function used to convert a datetime to an OS timestamp.
Args:
modified_time (datetime): Datatime object to convert to a Unix Epoc timestamp.
Returns:
The value of modified_time as a timestamp.
N)Ztzinfoi² é )ÚreplaceÚdatetimeZ
total_seconds)Z
modified_timeZ
naive_timeZ
delta_seconds© r ú /usr/lib/python3.6/vaults3.pyÚ_make_timestamp s r c @ sÆ e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd d
Z d1ddZ
dd Z dd Z dd Z
d2ddZdd Zd3d d Zd d Zd!d" Zd#d$ Zd%d& Zd'd( Zd)d* Zd+d, Zd-d. Zd/d0 ZdS )4Ú S3Backendz´ S3 Backend implimentation.
A Wrapper for the AWS S3 boto3 and botocore API. This class encapsulates all S3
operations and exposes them as a backend storage instance.
c C sò d }d }|| _ |jdr&t|jdnt}|d dkrVtjj|d t|d |d}ntjjt|d |d}|d dkr°tjd |d
d |d |d
|d
d ||d d}n.tjd |d
d |d |d
|d
d |d}|| _|d | _ d S )NZ vault_s3_max_pool_connectionsZ s3_signatureÚdefaultZs3_read_timeout)Zsignature_versionÚ read_timeoutÚmax_pool_connections)r r
Z s3_ssl_certÚ r Z
os_optionsÚ region_nameÚuserÚkeyZobject_storage_url)r Úaws_access_key_idÚaws_secret_access_keyÚ endpoint_urlÚconfigZverify)r r r r r Zbucket)
ÚoptionsÚgetÚintÚDEFAULT_MAX_POOL_SIZEÚbotocoreZclientZConfigÚboto3Ú_S3Backend__clientÚ_S3Backend__bucket_name)Úselfr Z
config_objectZ s3_clientZ
MAX_POOL_SIZEr r r Ú__init__6 s4
zS3Backend.__init__c C sú yæ| j j| jd y^| j j| jd}|d rH|d d dkrHtd| |d rn|d d dkrntd| W n, tk
r } z|d r W Y dd}~X nX |d sätjjdd d
d d d
d}| j j | j|dd | j j
| j|d W n Y nX dS )ah Validates the S3 credentials / permission on the provided bucket.
It also validates against s3 object lock configuration and if the s3 supports
more than 255 object name length
Args:
self : S3 Backend object
Returns:
Does not return anything, But raises exception if validation fails.
)ÚBucketÚbucket_object_lockZObjectLockConfigurationZObjectLockEnabledZEnabledzZConfigration says object lock enabled on the bucket, but s3 API does not agree with it. %sNztvault_config/z.workload_f5190be6-7f80-4856-8c24-149cb40500c5/z.snapshot_f2e5c6a7-3c21-4b7f-969c-915bb408c64f/z+vm_id_e81d1ac8-b49a-4ccf-9d92-5f1ef358f1be/z3vm_res_id_72477d99-c475-4a5d-90ae-2560f5f3b319_vda/z.deac2b8a-dca9-4415-adc1-f3c6598204ed-segments/z0000000000000000.00000000z Test Data)r ÚKeyÚBody)r r" ) r Z head_bucketr Z get_object_lock_configurationZExceptonÚ ExceptionÚosÚpathÚjoinÚ
put_objectÚ
delete_object)r r Z obj_lock_cfgZexZlong_keyr r r Úvalidate_s3_clienth s:
z S3Backend.validate_s3_clientc C sÌ y°xªt |dkr¬g }|dt
}x0|D ](}|jdr@|dd
}|jd|i q&W y| jj| jd|id W n0 x"|D ]}| jj| j|d d qzW Y nX |td
}qW W n tk
rÆ Y nX dS ) zß Utility method that takes a list of objects and puts it in the correct format
for the S3 delete_objects() API.
Args:
object_list (list): A list of objects with the correct S3 path.
r Nú/r r" ÚObjects)r ÚDelete)r r" ) ÚlenÚMAX_S3_PAGE_SIZEÚ
startswithÚappendr Údelete_objectsr r) r )r Ú object_listZobject_delete_listZ list_partÚobjr r r Z__delete_object_list s$
z S3Backend.__delete_object_listc C s y|g }| j jd}|j| j|d}xL|D ]D}d|kr(|jdt|d dkr(x |d D ]}|j|d qVW q(W | j| W n tk
r Y nX dS )z Utility method used to perform a rmdir operation on an object hierarchy
Args:
object_tree (str): Object/Tree path.
Ú list_objects)r ÚPrefixÚContentsÚKeyCountr r" N) r Ú
get_paginatorÚpaginater r r. r1 Ú _S3Backend__delete_object_listr )r Ú object_treer3 Úlist_paginatorÚ
page_iteratorÚobjectsr4 r r r Z__delete_object_tree³ s
z S3Backend.__delete_object_treec C s | j | dS )zó Delete a list of objects.
Args:
object_list (list): List of objects. Each with a full object path
in correct format.
options (dic): Dictionary of configuration options.
N)r; )r r3 r r r r Údelete_object_listÉ s z S3Backend.delete_object_listc C s¬ y|s
dS dj |}|jdr*|dd
}d|krF| jj| j|d nH|jdd }| j|dd}|rxt|dd d
}n|}| jj| j|d W n tk
r¦ Y nX dS ) a Delete an object from the S3 object store.
The object and any segments stored in the segment directory will be
removed.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Nr+ r z -segments)r r" z .manifestT)Ústrict_path_filter)Úreverser ) r' r0 r r) r Ústripr5 Úsortedr )r Úargsr Ú object_nameÚmanifest_prefixÚ manifestsÚ
manifest_namer r r r) Ó s$
zS3Backend.delete_objectr c C s¤ t d|}y~xxtd|D ]j}y| jj| j|d dS tk
r } z2|jd d dkrn|d |krntjd w W Y dd}~X qX qW W n tk
r Y nX dS )a Utility method used to wait for a S3 object to become available.
This routine will keep performing a head_object() request every second
for "retries" number of times. This was added to work around any potential
eventual consistancey issues when uploading to AWS.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter that defaults to 1.
Returns:
Returns when the object is available or ClientError not found exception.
r r )r r" NÚResponseMetadataÚHTTPStatusCodei ) ÚmaxÚranger Ú head_objectr r ÚresponseÚtimeÚsleep)r rF ÚretriesÚretryÚerrorr r r Z__wait_for_objectö s
zS3Backend.__wait_for_objectc C s y|j dd }| jd r0| jj| j|d}n| jj| j|d}|rÖ|}|jdg p`|jdg }| jd rÊx\|D ]T}| jj| j|d |jdd d
}|d d jd
ddkrrtj |d j
|d S qrW |d d }|sât d| jj| j|d}tj |d j
|d S t k
r } zn|j
d d dkr6 nPy6| j|d | jj| j|d}tj |d j
|d S t k
r Y nX W Y dd}~X nX dS )zÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
r+ z .manifestr! )r r6 ÚVersionsr7 r" Ú VersionIdr )r r" rV rJ Ú HTTPHeadersz$x-amz-meta-stamp-trilio-authenticityÚFalseÚTruer# Ú
ContentLengthr zOManifest object with 'x-amz-meta-stamp-trilio-authenticity' attribute not found)r r" rK i é
Néÿÿÿÿ)rC r r Úlist_object_versionsr r5 r Z
get_objectÚjsonÚloadsÚreadr$ r rO Ú_S3Backend__wait_for_object)r rF rG rH rI ÚmanifestÚresprT r r r Úget_object_manifest sH
z S3Backend.get_object_manifestc C sà |j dd }| jd r,| jj| j|d}n| jj| j|d}|rÐ|jdg pX|jdg }g }x |D ]}|jd|d i qdW |rÜy| jj| jd|id W qÜ x"|D ]}| jj | j|d d
qªW Y qÜX n t
d | d S )
zÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
r+ z .manifestr! )r r6 rU r7 r" r, )r r- )r r" z Manifest for %s is not foundN) rC r r r] r r5 r r1 r2 r) r$ )r rF rG rc rH Úobjs_to_deleteÚmanr4 r r r Údelete_object_manifestJ s,
z S3Backend.delete_object_manifestc C sh y8dj |}i }|jr d|ji}| jj| j||j| W n* tk
rN Y n tk
rb Y nX dS )a? Download a file from the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
r+ rV N)r' Z
version_idr Z
download_filer Zout_filer r$ )r rE r rF Z
extra_argsr r r Údownload_objectm s
zS3Backend.download_objectNFc C s t }|jd}y^t}|dk r,|tk r,|}| jjd}|j| j|dd|id}x|D ] }d|krè|jdt|d dkrèxh|d D ]\} |dk r¨t||kr¨t |S | d j
d}
|rÆ|j |
qd
|
kr|j |
j d
d d qW d |krXxt|d D ]h} |dk r t||kr t |S | d
j
tsú|jdrN|j | d
jd qú|j | d
jd qúW qXW t |S tk
r Y nX dS )ae Return a list of objects based on the provided prefix.
Used to generate a directory listing based on a prefix path
constructed from the object parts in prefix
Args:
prefix (string): prefix of the object path. If the prefix is terminated
by "/", then it lists the contents in the "subdir". If the prefix
is not terminated by "/", then it lists the contents in the current
directory that matches the prefix
Ex:
case 1:
=======
prefix - /a/b/c
object store has objects - /a/b/c, /a/b/c1, /a/b/d
return values - /a/b/c and /a/b/c1
case 2:
=======
prefix - /a/b/c/
object store has objects - /a/b/c/d, /a/b/c1, /a/b/c/e
return values - /a/b/c/d and /a/b/c/e
options (dic): Dictionary of configuration options
max_items (int): Maximum number of objects in a list query. Default is
unlimited but when testing for an empty directory a smaller
number can be used to limit the query and size of the list that
is returned.
strict_path_filter (bool): Defaults to True in order to only return
objects with the exact same prefix. Individual segment listings
set this to False in order to get a match on the segment offset.
Returns:
On success a list of unique items to be used as a directory listing.
r+ Nr5 ZPageSize)r r6 Z DelimiterZPaginationConfigr7 r8 r r" z .manifestr ZCommonPrefixesr6 )ÚsetÚlstripr/ r r9 r: r r r. ÚlistÚrstripÚaddÚsplitr0 Ú_SEGMENT_PREFIXÚendswithrC r ) r ÚprefixZ max_itemsrA Z
object_setZ page_sizer= r> r? ÚitemrF Z
object_prefixr r r r5
s@ #
zS3Backend.list_objectsc C sH y,g }t jjdj||j}| j|dd}|S tk
rB Y nX dS )a& Returns a list of object segments based on a prefix.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A list of segments in the object store that match the supplied prefix.
r+ T)rA N)r% r& r' rq r5 r )r rE r Z segment_listZ
segments_pathr3 r r r Ú
list_segmentsØ s
zS3Backend.list_segmentsc
/ C sj t d|}y@x8td|D ](}y| jj| j|d}|S tk
rD } zì|jd d dkr2y| jj| j|d d}|S tk
r0 } z|jd d dkr d|kr yü|jdd }| jd
rà| jj | j|d }n| jj
| j|d }|r|}|j d g p|j d
g }t |dd d}| jd
rxL|D ]D} | jj| j| d | j ddd}|d d j dddkr8|S q8W |d d }|st
d| jj| j|d}|S tk
r } z6|jd d dkrð|d |krðtjd w W Y dd}~X nX |d |kr tjd w W Y dd}~X nX W Y dd}~X q X q W W n tk
rd Y nX dS ) a= Utility method that gets an object head from the repository with retry support.
The default is to try once. Admittedly, this is ugly because the
provided object name might be a terminal object or a "directory" so
we need to try both cases for each retry attempt. When retry > 1, there is
a 1 second delay between attempts.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter, default is 1 attempt.
Returns:
Object head dictionary or Boto3 exception.
r r )r r" rJ rK i r+ z -segmentsz .manifestr! )r r6 rU r7 c S s | d S )NÚ LastModifiedr )Úkr r r Ú<lambda> s z0S3Backend.__get_object_headers.<locals>.<lambda>)r r" rV r )r r" rV rW z$x-amz-meta-stamp-trilio-authenticityrX rY zOManifest object with 'x-amz-meta-stamp-trilio-authenticity' attribute not foundNr\ )rL rM r rN r r rO rC r r] r5 r rD r$ rP rQ )
r rF rR rS Ú obj_headersrT rG rH rI rb r r r Z__get_object_headersí sr
z S3Backend.__get_object_headersc C sl ddddd}i |d< i |d d< dj |}y |sJttjj |d< |S | j|}t|d |d< ||d< |jd
d d ks|d
dkrd|d< x0|d d j D ] \}}d|kr¢||d |< q¢W t|d d jdddkrø|d d jdd|d< nDt|d d jdddkr0|d d jdd|d< n |d
|d< |d |d< |S W n tk
rf Y nX dS )a) Get "operating system like" stat data for the given object.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A stat structure containing object information required by caller.
r r F)Ú timestampÚsizeÚetagÚ directoryZheadersÚMetadatar+ rx rt Ú ContentTypez
text/plainzapplication/x-directoryrZ Tr{ z
x-object-metazx-object-meta-segment-countzx-object-meta-total-sizery z
segment-countz
total-sizezx-account-bytes-usedN) r' r r ZnowÚ _S3Backend__get_object_headersr Úitemsr r )r rE r Z stat_datarF Z
obj_headerr Úvaluer r r Ú stat_object< s<
zS3Backend.stat_objectc C s dj |}| j|| dS )zÎ Create an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r+ N)r' Ú_S3Backend__create_folder)r rE r Z
new_folderr r r Ú mkdir_objectm s
zS3Backend.mkdir_objectc
C sV dj |d }|jdr$|dd
}| j| y| jj| j|d W n Y nX dS )zÎ Remove an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r+ r N)r r" )r' r0 Ú _S3Backend__delete_object_treer r) r )r rE r r< r r r Ú rmdir_objectx s
zS3Backend.rmdir_objectc
C sH y(|j dd }| j|dd}|rRtt|ddd jdd d }d
|d }nd
d }x*|D ]"}|jd |d < |jd
d|d< q`W tj|}|d | } | jj | j
| ||dd |r(g }
x|D ]} |
j d| i qÄW y| jj | j
d|
id W n2 x$|
D ] } | jj
| j
| d d qW Y nX W n tk
rB Y nX dS )a? Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added to the object.
manifest_data (dic): The manifest data which becomes the body of file.
r+ z .manifestT)rA )rB r Ú.r é z%08xr& Únamerz Úhashz
text/plain)r r" r| r# r} r" r, )r r- )r r" Nr\ )rC r5 r rD rn Úpopr^ Údumpsr r( r r1 r2 r) r )
r rF Z put_headersZ
manifest_datarG rH ÚsuffixZ segment_datarb rI re rf r4 r r r Úupload_object_manifest s>
z S3Backend.upload_object_manifestc
C s¦ y|j dd }| j|dd}|rLtt|ddd jdd d }d
| }nd
d }|d | }tjj| j|}| j j
| j|||d d W n t k
r Y nX d
S )a? Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added to the object.
manifest_data (dic): The manifest data which becomes the body of file.
r+ z .manifestT)rA )rB r r r r z%08xZREPLACE)r r" Z
CopySourcer| ZMetadataDirectiveNr\ ) rC r5 r rD rn r% r& r' r r Z copy_objectr )r rF Z
attributesrG rH r rI Zcopy_srcr r r Úupdate_object_attributes· s"
z"S3Backend.update_object_attributesc C sD y(|j d}| jj| j|d|dd W n tk
r> Y nX dS )a? Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added to the object.
manifest_data (dic): The manifest data which becomes the body of file.
r+ Ú
GOVERNANCE)ÚModeÚRetainUntilDate)r r" Ú RetentionN)rC r Úput_object_retentionr r )r rF Ú retainuntilr r r Ú put_object_segment_retentionÔ s
z&S3Backend.put_object_segment_retentionc C s y||j dd }| j|dd}|rLtt|ddd jdd d }d
| }nd
d }|d | }| jj| j|d |d d
W n tk
r Y nX dS )a? Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added to the object.
manifest_data (dic): The manifest data which becomes the body of file.
r+ z .manifestT)rA )rB r r r r z%08xr )r r )r r" r Nr\ ) rC r5 r rD rn r r r r )r rF r rG rH r rI r r r Úput_manifest_retentionè s
z S3Backend.put_manifest_retentionc ) C s* y| j j| j|d W n tk
r$ } zî|jd d dkr|jd}d}xÄ|D ]¼}|dkrbP || d }y| j j| j|d W qT tk
r } zl|jd d dkrüy| j j| j|ddd W n6 tk
rú } z|jd d d
krê W Y d d }~X nX wTW Y d d }~X qTX qTW W Y d d }~X nX d S ) a< Utility method used to create a object directory structure.
The entire path is split and each directory level is created as an object
if it does not exist.
Args:
folder_name (str): The entire folder name.
options (dic): Dictionary of configuration options.
)r r" rJ rK i r+ r z TrilioVault directory objectzapplication/x-directory)r r" r# r} i N)r rN r r rO rn r( )r Z folder_namer rT Ú
path_partsÚnew_pathÚpartr r r Z__create_folder s0
zS3Backend.__create_folderc C sÊ |dd
}y¢|j dkr<| jtjj|d d |d | |jdk r¬t|d dR}| jj|| j |d d |d dd id
| jj
| j |d d |d d }|S Q R X W n t k
rÄ Y nX dS ) zµ Upload an object to the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r NFr r+ rF Úrbr} zapplication/octet-stream)r r" Z ExtraArgs)r r" ) Z
path_validr r% r& ÚdirnamerF Úopenr Zupload_fileobjr rN r )r rE r ÚfilesÚdatarw r r r Ú
upload_object. s"
zS3Backend.upload_object)r )NF)r )Ú__name__Ú
__module__Ú __qualname__Ú__doc__r r* r; r r@ r) ra rd rg rh r5 rs r~ r r r
r r r r r r r r r r r
/ s. 2-
#
5#
S
O1 + *r
)r£ r rP r^ r% r r Zbotocore.exceptionsr ZBASENAMEr/ ro r r Úobjectr
r r r r Ú<module> s