Why Gemfury?
Push, build, and install
RubyGems
npm packages
Python packages
Maven artifacts
PHP packages
Go Modules
Debian packages
RPM packages
NuGet packages
Repository URL to install this package:
|
Version:
6.0.5-6.0 ▾
|
python3-s3fuse-plugin
/
usr
/
lib
/
python3.6
/
site-packages
/
s3fuse
/
__pycache__
/
vaults3.cpython-36.opt-1.pyc
|
|---|
3
©Dg¬¯ ã @ s d Z ddlZddlZddlZddlZddlZddlZddlZddlm Z ddl
m Z dZ dZ
dZdZejeZd d
ZG d d d eZdS )
z Trilio S3 Backend implimentation
This module contains the back end implimentation of all of all S3 specific
support.
é N)Ú ClientError)Úget_retention_modeÚs3iè z$80bc80ff-0c51-4534-86a2-ec5e719643c2iô c C s& | j dd}|tjddd j }|S )zè Utility function used to convert a datetime to an OS timestamp.
Args:
modified_time (datetime): Datatime object to convert to a Unix Epoc timestamp.
Returns:
The value of modified_time as a timestamp.
N)Ztzinfoi² é )ÚreplaceÚdatetimeZ
total_seconds)Z
modified_timeZ
naive_timeZ
delta_seconds© r ú /usr/lib/python3.6/vaults3.pyÚ_make_timestamp" s r
c @ sÞ e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd d
Z dd Z
d7ddZ dd Z dd Z
dd Zdd Zdd Zd8d d Zd!d" Zd9d#d$Zd%d& Zd'd( Zd)d* Zd+d, Zd-d. Zd/d0 Zd1d2 Zd3d4 Zd5d6 Z d S ):Ú S3Backendz´ S3 Backend implimentation.
A Wrapper for the AWS S3 boto3 and botocore API. This class encapsulates all S3
operations and exposes them as a backend storage instance.
c
C sd d }d }|| _ |jdr&t|jdnt}|d dkrVtjj|d t|d |d}ntjjt|d |d}|d s²tjd|d d
|d |d |d d
||d |d d}n||d dkrútjd|d d
|d |d |d d
||d |d d}n4tjd|d d
|d |d |d d
||d d}dd }|jj j
d| || _ |d | _ |d | _
d S )NZ vault_s3_max_pool_connectionsZ s3_signatureÚdefaultZs3_read_timeout)Zsignature_versionÚ read_timeoutÚmax_pool_connections)r
r Z
s3_ssl_verifyr Z
os_optionsÚ region_nameÚuserÚkeyZobject_storage_urlZs3_ssl)r Úaws_access_key_idÚaws_secret_access_keyÚ endpoint_urlÚconfigÚuse_sslZverifyZ s3_ssl_certÚ )r r r r r r c [ s d| j d< d S )Nz
keep-aliveZ
Connection)Úheaders)ZrequestZoperation_nameÚkwargsr r r Úset_connection_header\ s z1S3Backend.__init__.<locals>.set_connection_headerzrequest-created.s3ÚbucketÚretention_mode)ÚoptionsÚgetÚintÚDEFAULT_MAX_POOL_SIZEÚbotocoreZclientZConfigÚboto3ÚmetaZeventsÚregisterÚ_S3Backend__clientÚ_S3Backend__bucket_nameÚ_S3Backend__retention_mode)Úselfr Z
config_objectZ s3_clientZ
MAX_POOL_SIZEr r r r Ú__init__8 sH
zS3Backend.__init__c C s( t | j|d |d |d< |d | _d S )Nr Úbucket_object_lockr )r r% r' )r( r r r r Úset_retention_moded s z S3Backend.set_retention_modec C s" y | j j| jd y| j j| jd}|d rP|jdi jddkrPtd| |d r||jdi jddkr|td| |d r|d std W n, tk
r } z|d r² W Y d
d
}~X nX |d s tjjd d d
dddd}| j j | j|dd | j j
| j|d W n Y nX d
S )af Validates the S3 credentials / permission on the provided bucket.
It also validates against s3 object lock configuration and if the s3 supports
more than 255 object name length
Args:
self : S3 Backend object
Returns:
Does not return anything, But raises exception if validation fails.
)ÚBucketr* ZObjectLockConfigurationZObjectLockEnabledZEnabledzZConfigration says object lock enabled on the bucket, but s3 API does not agree with it. %sz[Configration says object lock disabled on the bucket, but s3 API does not agree with it. %sÚuse_manifest_suffixzR'use_manifest_suffix' must be set to True when 'bucket_object_lock' is set to TrueNztvault_config/z.workload_f5190be6-7f80-4856-8c24-149cb40500c5/z.snapshot_f2e5c6a7-3c21-4b7f-969c-915bb408c64f/z+vm_id_e81d1ac8-b49a-4ccf-9d92-5f1ef358f1be/z3vm_res_id_72477d99-c475-4a5d-90ae-2560f5f3b319_vda/z.deac2b8a-dca9-4415-adc1-f3c6598204ed-segments/z0000000000000000.00000000z Test Data)r, ÚKeyÚBody)r, r. ) r% Z head_bucketr& Z get_object_lock_configurationr Ú ExceptionÚosÚpathÚjoinÚ
put_objectÚ
delete_object)r( r Z obj_lock_cfgÚexZlong_keyr r r Úvalidate_s3_clienti s@
z S3Backend.validate_s3_clientc C sÌ y°xªt |dkr¬g }|dt
}x0|D ](}|jdr@|dd
}|jd|i q&W y| jj| jd|id W n0 x"|D ]}| jj| j|d d qzW Y nX |td
}qW W n tk
rÆ Y nX dS ) zß Utility method that takes a list of objects and puts it in the correct format
for the S3 delete_objects() API.
Args:
object_list (list): A list of objects with the correct S3 path.
r Nú/r r. ÚObjects)r, ÚDelete)r, r. ) ÚlenÚMAX_S3_PAGE_SIZEÚ
startswithÚappendr% Údelete_objectsr& r5 r )r( Ú object_listZobject_delete_listZ list_partÚobjr r r Z__delete_object_list s$
z S3Backend.__delete_object_listc C s y|g }| j jd}|j| j|d}xL|D ]D}d|kr(|jdt|d dkr(x |d D ]}|j|d qVW q(W | j| W n tk
r Y nX dS )z Utility method used to perform a rmdir operation on an object hierarchy
Args:
object_tree (str): Object/Tree path.
Ú list_objects)r, ÚPrefixÚContentsÚKeyCountr r. N) r% Ú
get_paginatorÚpaginater& r r; r> Ú _S3Backend__delete_object_listr )r( Ú object_treer@ Úlist_paginatorÚ
page_iteratorÚobjectsrA r r r Z__delete_object_tree¹ s
z S3Backend.__delete_object_treec C s | j | dS )zó Delete a list of objects.
Args:
object_list (list): List of objects. Each with a full object path
in correct format.
options (dic): Dictionary of configuration options.
N)rH )r( r@ r r r r Údelete_object_listÏ s z S3Backend.delete_object_listc C s¬ y|s
dS dj |}|jdr*|dd
}d|krF| jj| j|d nH|jdd }| j|dd}|rxt|dd d
}n|}| jj| j|d W n tk
r¦ Y nX dS ) a Delete an object from the S3 object store.
The object and any segments stored in the segment directory will be
removed.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Nr8 r z -segments)r, r. z .manifestT)Ústrict_path_filter)Úreverser ) r3 r= r% r5 r& ÚstriprB Úsortedr )r( Úargsr Ú object_nameÚmanifest_prefixÚ manifestsÚ
manifest_namer r r r5 Ù s$
zS3Backend.delete_objectr c C s¤ t d|}y~xxtd|D ]j}y| jj| j|d dS tk
r } z2|jd d dkrn|d |krntjd w W Y dd}~X qX qW W n tk
r Y nX dS )a Utility method used to wait for a S3 object to become available.
This routine will keep performing a head_object() request every second
for "retries" number of times. This was added to work around any potential
eventual consistancey issues when uploading to AWS.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter that defaults to 1.
Returns:
Returns when the object is available or ClientError not found exception.
r r )r, r. NÚResponseMetadataÚHTTPStatusCodei ) ÚmaxÚranger% Ú head_objectr& r ÚresponseÚtimeÚsleep)r( rS ÚretriesÚretryÚerrorr r r Z__wait_for_objectü s
zS3Backend.__wait_for_objectc C sò y^|j dd }| jd r&| jd r<| jj| j|d}n| jj| j|d}|r*|}|jdg pn|jdg }| jd r| jj| j|d}|jdg p¤|jdg }|rºt|dd d
}x\|D ]T}| jj | j|d |jd d
d}|d d jdddkrÀt
j |d j |d S qÀW |d d }n|}|s8t
d| jj | j|d}t
j |d j |d S tk
rì } zn|jd d dkr nPy6| j|d | jj | j|d}t
j |d j |d S tk
rÚ Y nX W Y d d }~X nX d S ) zÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
r8 z .manifestr- r* )r, rC ÚVersionsrD c S s | d S )NÚ LastModifiedr )Úkr r r Ú<lambda>7 s z/S3Backend.get_object_manifest.<locals>.<lambda>)r r. Ú VersionIdr )r, r. rf rW Ú HTTPHeadersz$x-amz-meta-stamp-trilio-authenticityÚFalseÚTruer/ Ú
ContentLengthr zOManifest object with 'x-amz-meta-stamp-trilio-authenticity' attribute not found)r, r. rX i é
Néÿÿÿÿ)rP r r% Zlist_objects_v2r& rB r Úlist_object_versionsrQ Ú
get_objectÚjsonÚloadsÚreadr0 r r\ Ú_S3Backend__wait_for_object) r( rS rT rU rV Úmanifests_dataÚmanifestÚrespra r r r Úget_object_manifest sT
z S3Backend.get_object_manifestc C sÐ |j dd }| jd r dS | jj| j|d}|rÀ|jdg pH|jdg }g }x |D ]}|jd|d i qTW |rÌy| jj| jd |id
W qÌ x"|D ]}| jj| j|d d qW Y qÌX n t d | dS )
zÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
r8 z .manifestr* N)r, rC rb rD r. r9 )r, r: )r, r. z Manifest for %s is not found)
rP r r% rB r& r r> r? r5 r0 )r( rS rT ru rU Úobjs_to_deleteÚmanrA r r r Údelete_object_manifestW s(
z S3Backend.delete_object_manifestc C sh y8dj |}i }|jr d|ji}| jj| j||j| W n* tk
rN Y n tk
rb Y nX dS )a? Download a file from the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
r8 rf N)r3 Ú
version_idr% Z
download_filer& Zout_filer r0 )r( rR r rS Z
extra_argsr r r Údownload_objecty s
zS3Backend.download_objectc C sv yFdj |}|jr*| jj| j||jd}n| jj| j|d}|d j S tk
r\ Y n tk
rp Y nX dS )aB Get object from the S3 object store in buffer.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
r8 )r, r. rf )r, r. r/ N)r3 rz r% rn r& rq r r0 )r( rR r rS r\ r r r rn s
zS3Backend.get_objectc
C s yh|j dkr0| jtjj|d d |d | |jdk rf| jj| j|d d |d ||dd}|S W n t k
r~ Y nX dS )a Put an object to the S3 object store.
Args:
args (list): List of object name parts.
buf: Data to upload in object.
buf_len: Length of data to upload in object.
options (dic): Dictionary of configuration options.
Fr r8 rS Nzapplication/octet-stream)r, r. rj r/ Ú ContentType)
Ú
path_validÚ_S3Backend__create_folderr1 r2 ÚdirnamerS r% r4 r& r )r( rR ZbufZbuf_lenr ru r r r r4 « s
zS3Backend.put_objectNFc C sR t }|jd}y"t}|dk r,|tk r,|}| jd rD| jjd}n | jjd}|j| j|dd|id}xÂ|D ]¸}d|ko|jd t |d d
krxh|d D ]\} |dk rÂt ||krÂt
|S | d j d}
|rà|j |
q¢d |
kr¢|j |
j
d d
d
q¢W d|kr|jd t |d d
krxr|d D ]f} |dk rTt ||krTt
|S | d j d}
|rt|j |
n d |
kr0|j |
j
d d
d
q0W d|krpx|d D ]z} |dk rÐt ||krÐt
|S |jtsî| d jt r¬|jdr|j | d jd n|j | d jd q¬W qpW t
|S tk
rL Y nX dS )ae Return a list of objects based on the provided prefix.
Used to generate a directory listing based on a prefix path
constructed from the object parts in prefix
Args:
prefix (string): prefix of the object path. If the prefix is terminated
by "/", then it lists the contents in the "subdir". If the prefix
is not terminated by "/", then it lists the contents in the current
directory that matches the prefix
Ex:
case 1:
=======
prefix - /a/b/c
object store has objects - /a/b/c, /a/b/c1, /a/b/d
return values - /a/b/c and /a/b/c1
case 2:
=======
prefix - /a/b/c/
object store has objects - /a/b/c/d, /a/b/c1, /a/b/c/e
return values - /a/b/c/d and /a/b/c/e
options (dic): Dictionary of configuration options
max_items (int): Maximum number of objects in a list query. Default is
unlimited but when testing for an empty directory a smaller
number can be used to limit the query and size of the list that
is returned.
strict_path_filter (bool): Defaults to True in order to only return
objects with the exact same prefix. Individual segment listings
set this to False in order to get a match on the segment offset.
Returns:
On success a list of unique items to be used as a directory listing.
r8 Nr* rm rB ZPageSize)r, rC Ú DelimiterZPaginationConfigrD rE r r. z .manifestr rb ÚCommonPrefixesrC )ÚsetÚlstripr< r r% rF rG r& r r; ÚlistÚrstripÚaddÚsplitr= Ú_SEGMENT_PREFIXÚendswithrP r ) r( ÚprefixZ max_itemsrN Z
object_setZ page_sizerJ rK rL ÚitemrS Z
object_prefixr r r rB Ç sX #
" $
zS3Backend.list_objectsc C sH y,g }t jjdj||j}| j|dd}|S tk
rB Y nX dS )a& Returns a list of object segments based on a prefix.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A list of segments in the object store that match the supplied prefix.
r8 T)rN N)r1 r2 r3 r rB r )r( rR r Z segment_listZ
segments_pathr@ r r r Ú
list_segments+ s
zS3Backend.list_segmentsc / C s t d|}ytxltd|D ]\}y| jj| j|d}|S tk
rx } z |jd d dkrfy,| jj| j|d d}|jdpd |d< |S tk
rd } z¶|jd d dkrRd
|kr8y |jdd }| j d rv| jj
| j|d
}|jdg p
|jdg }|r"t |dd d}xL|D ]D}| jj| j|d |jddd}|d d jdddkr(|S q(W |nj| jj | j|d
} | jdg p | jdg } t | dd d} | rÆ| d d }
n|}
| jj| j|
d}|S W nT tk
r6 } z6|jd d dkr$|d |kr$t
jd w W Y d d }~X nX |d |krRt
jd w W Y d d }~X nX W Y d d }~X q X q W W n tk
r Y nX d S ) a= Utility method that gets an object head from the repository with retry support.
The default is to try once. Admittedly, this is ugly because the
provided object name might be a terminal object or a "directory" so
we need to try both cases for each retry attempt. When retry > 1, there is
a 1 second delay between attempts.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter, default is 1 attempt.
Returns:
Object head dictionary or Boto3 exception.
r r )r, r. rW rX i r8 r| zapplication/x-directoryz -segmentsz .manifestr* )r, rC rb rD c S s | d S )Nrc r )rd r r r re i s z0S3Backend.__get_object_headers.<locals>.<lambda>)r r. rf r )r, r. rf rg z$x-amz-meta-stamp-trilio-authenticityrh ri c S s | d S )Nrc r )rd r r r re y s Nrl )rY rZ r% r[ r& r r\ r rP r rm rQ rB r] r^ ) r( rS r_ r` Ú obj_headersra rT rs rt rU rV r r r Z__get_object_headers@ sx
z S3Backend.__get_object_headersc C s ddddd}i |d< i |d d< dj |}yÒ|sJttjj |d< |S y| j|}W n tk
rÞ } zj| jj| jd|d }d
|krÌ|d d d |d
D krÈtjj |d
d d d}d|i d}q n W Y dd}~X nX t|d |d< ||d< |j
dddks|d dkr d|d< xX|d d j D ]D\} }
d| j krd|
|d | j j
dd < n|
|d | j < q2W t|d d j
d ddkr²|d d j
d d|d < n<t|d j
d ddkrâ|d j
d d|d < n |d |d < |d |d!< |S W n tk
r Y nX dS )"a) Get "operating system like" stat data for the given object.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A stat structure containing object information required by caller.
r r F)Ú timestampÚsizeÚetagÚ directoryr ÚMetadatar8 r )r, r rC r c S s g | ] }|d qS )rC r )Ú.0Úxr r r ú
<listcomp>¯ s z)S3Backend.stat_object.<locals>.<listcomp>rW rg Zdatez%a, %d %b %Y %H:%M:%S %Z)rj rc r Nrc r| z
text/plainzapplication/x-directoryrj Tr zx-object-meta-r zx-object-meta-segment-countzx-object-meta-total-sizer z
segment-countz
total-sizezx-account-bytes-used)r3 r
r ZnowÚ _S3Backend__get_object_headersÚ
BaseExceptionr% rB r& Zstrptimer ÚitemsÚlowerr r r ) r( rR r Z stat_datarS Z
obj_headerr6 rL Zmdr Úvaluer r r Ú stat_object s`
zS3Backend.stat_objectc C s dj |}| j|| dS )zÎ Create an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r8 N)r3 r~ )r( rR r Z
new_folderr r r Ú mkdir_objectÜ s
zS3Backend.mkdir_objectc C s^ dj |d }|jdr$|dd
}| j| y| jj| j|d W n tk
rX Y nX dS )zÎ Remove an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r8 r N)r, r. )r3 r= Ú _S3Backend__delete_object_treer% r5 r& r )r( rR r rI r r r Ú rmdir_objectç s
zS3Backend.rmdir_objectc
C sr yR|j dd }| jd rv| j|dd}|r`t|ddd jdd }t|d
}d |d }nd d }|d | }n|}x*|D ]"}|jd |d
< |jdd|d< qW tj|} | j j
| j ||| dd | jd rR|rRg }
x|D ]} |
j d| i qäW y| j j
| j d|
id W n< tk
rP x$|
D ] } | j j| j | d d q,W Y nX W n tk
rl Y nX dS )a Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be
added to the object.
manifest_data (dic): The manifest data which becomes
the body of file.
r8 z .manifestr- T)rN )rO r Ú.r é z%08xr2 Únamer Úhashz
text/plain)r, r. r r/ r| r. r9 )r, r: )r, r. Nrl )rP r rB rQ r r Úpopro Údumpsr% r4 r& r> r? r r5 r )
r( rS Z put_headersZ
manifest_datarT rU ÚsuffixrV Z segment_datart rw rx rA r r r Úupload_object_manifestû sJ
z S3Backend.upload_object_manifestc
C sº y|j dd }| jd rp| j|dd}|rZt|ddd jdd }t|d
}d | }nd d }|d | }n|}tjj| j |}| j
j | j |||d d
W n t k
r´ Y nX dS )a
Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
attributes (dic): The attributes data which becomes
the body of file.
r8 z .manifestr- T)rN )rO r r r r z%08xZREPLACE)r, r. Z
CopySourcer ZMetadataDirectiveNrl )
rP r rB rQ r r r1 r2 r3 r& r% Z copy_objectr )r( rS Z
attributesrT rU r¥ rV Zcopy_srcr r r Úupdate_object_attributes. s*
z"S3Backend.update_object_attributesc C sF y*|j d}| jj| j|| j|dd W n tk
r@ Y nX dS )a Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added
to the object.
manifest_data (dic): The manifest data which becomes the
body of file.
r8 )ÚModeÚRetainUntilDate)r, r. Ú RetentionN)rP r% Úput_object_retentionr& r' r )r( rS Ú retainuntilr r r Ú put_object_segment_retentionO s
z&S3Backend.put_object_segment_retentionc C s® y|j dd }| jd rp| j|dd}|rZt|ddd jdd }t|d
}d | }nd d }|d | }n|}| jj| j|| j |d d
W n t
k
r¨ Y nX dS )aK Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest"
will be added. put_headers (dic): Dictionary of meta data
to be added to the object. manifest_data (dic): The manifest
data which becomes the body of file.
r8 z .manifestr- T)rN )rO r r r r z%08x)r¨ r© )r, r. rª Nrl ) rP r rB rQ r r r% r« r& r' r )r( rS r¬ rT rU r¥ rV r r r Úput_manifest_retentione s(
z S3Backend.put_manifest_retentionc ) C s* y| j j| j|d W n tk
r$ } zî|jd d dkr|jd}d}xÄ|D ]¼}|dkrbP || d }y| j j| j|d W qT tk
r } zl|jd d dkrüy| j j| j|ddd W n6 tk
rú } z|jd d d
krê W Y d d }~X nX wTW Y d d }~X qTX qTW W Y d d }~X nX d S ) a< Utility method used to create a object directory structure.
The entire path is split and each directory level is created
as an object if it does not exist.
Args:
folder_name (str): The entire folder name.
options (dic): Dictionary of configuration options.
)r, r. rW rX i r8 r z TrilioVault directory objectzapplication/x-directory)r, r. r/ r| i N)r% r[ r& r r\ r r4 )r( Z folder_namer ra Ú
path_partsÚnew_pathÚpartr r r Z__create_folder s:
zS3Backend.__create_folderc C sÊ |dd
}y¢|j dkr<| jtjj|d d |d | |jdk r¬t|d dR}| jj|| j |d d |d dd id
| jj
| j |d d |d d }|S Q R X W n t k
rÄ Y nX dS ) zµ Upload an object to the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r NFr r8 rS Úrbr| zapplication/octet-stream)r, r. Z ExtraArgs)r, r. ) r} r~ r1 r2 r rS Úopenr% Zupload_fileobjr& r[ r )r( rR r ÚfilesÚdatar r r r Ú
upload_object· s(
zS3Backend.upload_object)r )NF)r ) Ú__name__Ú
__module__Ú __qualname__Ú__doc__r) r+ r7 rH r rM r5 rr rv ry r{ rn r4 rB r r r r r r¦ r§ r r® r~ r¶ r r r r r 1 s4 ,2
#
<"
d
SI 3!!1r )rº r r] ro r1 Zloggingr" r! Zbotocore.exceptionsr Z s3fuse.utilsr ZBASENAMEr< r r Z getLoggerr· ZLOGr
Úobjectr r r r r Ú<module> s