Why Gemfury?
Push, build, and install
RubyGems
npm packages
Python packages
Maven artifacts
PHP packages
Go Modules
Debian packages
RPM packages
NuGet packages
Repository URL to install this package:
|
Version:
4.2.64.1-4.2 ▾
|
python3-s3fuse-plugin
/
usr
/
lib
/
python3.6
/
site-packages
/
s3fuse
/
__pycache__
/
vaults3.cpython-36.opt-1.pyc
|
|---|
3
D÷c ã @ s d Z ddlZddlZddlZddlZddlmZ ddlZddl Z ddl
m Z ddl m
Z
dZdZdZd ZejeZd
d ZG d d
d
eZdS )z Trilio S3 Backend implimentation
This module contains the back end implimentation of all of all S3 specific
support.
é N)Úlog)Ú ClientError)Úget_retention_modeÚs3iè z$80bc80ff-0c51-4534-86a2-ec5e719643c2iô c C s& | j dd}|tjddd j }|S )zè Utility function used to convert a datetime to an OS timestamp.
Args:
modified_time (datetime): Datatime object to convert to a Unix Epoc timestamp.
Returns:
The value of modified_time as a timestamp.
N)Ztzinfoi² é )ÚreplaceÚdatetimeZ
total_seconds)Z
modified_timeZ
naive_timeZ
delta_seconds© r ú /usr/lib/python3.6/vaults3.pyÚ_make_timestamp" s r c @ sÞ e Zd ZdZdd Zdd Zdd Zdd Zd
d Zd d
Z dd Z
d7ddZ dd Z dd Z
dd Zdd Zdd Zd8d d Zd!d" Zd9d#d$Zd%d& Zd'd( Zd)d* Zd+d, Zd-d. Zd/d0 Zd1d2 Zd3d4 Zd5d6 Z d S ):Ú S3Backendz´ S3 Backend implimentation.
A Wrapper for the AWS S3 boto3 and botocore API. This class encapsulates all S3
operations and exposes them as a backend storage instance.
c C s d }d }|| _ |jdr&t|jdnt}|d dkrVtjj|d t|d |d}ntjjt|d |d}|d dkr°tjd |d
d |d |d
|d
d ||d d}n.tjd |d
d |d |d
|d
d |d}dd }|jj j
d| || _ |d | _ |d | _
d S )NZ vault_s3_max_pool_connectionsZ s3_signatureÚdefaultZs3_read_timeout)Zsignature_versionÚ read_timeoutÚmax_pool_connections)r r Z s3_ssl_certÚ r Z
os_optionsÚ region_nameÚuserÚkeyZobject_storage_url)r Úaws_access_key_idÚaws_secret_access_keyÚ endpoint_urlÚconfigZverify)r r r r r c [ s d| j d< d S )Nz
keep-aliveZ
Connection)Úheaders)ZrequestZoperation_nameÚkwargsr r r
Úset_connection_headerU s z1S3Backend.__init__.<locals>.set_connection_headerzrequest-created.s3ÚbucketÚretention_mode)ÚoptionsÚgetÚintÚDEFAULT_MAX_POOL_SIZEÚbotocoreZclientZConfigÚboto3ÚmetaZeventsÚregisterÚ_S3Backend__clientÚ_S3Backend__bucket_nameÚ_S3Backend__retention_mode)Úselfr Z
config_objectZ s3_clientZ
MAX_POOL_SIZEr r r r
Ú__init__8 s:
zS3Backend.__init__c C s( t | j|d |d |d< |d | _d S )Nr Úbucket_object_lockr )r r% r' )r( r r r r
Úset_retention_mode] s z S3Backend.set_retention_modec C s y | j j| jd yv| j j| jd}|d rJ|d d dkrJtd| |d rp|d d dkrptd| |d r|d stdW n, tk
r¶ } z|d r¦ W Y d d }~X nX |d s tjjd
d d d
ddd}| j j| j|dd | j j | j|d W n Y nX d S )af Validates the S3 credentials / permission on the provided bucket.
It also validates against s3 object lock configuration and if the s3 supports
more than 255 object name length
Args:
self : S3 Backend object
Returns:
Does not return anything, But raises exception if validation fails.
)ÚBucketr* ZObjectLockConfigurationZObjectLockEnabledZEnabledzZConfigration says object lock enabled on the bucket, but s3 API does not agree with it. %sÚuse_manifest_suffixzR'use_manifest_suffix' must be set to True when 'bucket_object_lock' is set to TrueNztvault_config/z.workload_f5190be6-7f80-4856-8c24-149cb40500c5/z.snapshot_f2e5c6a7-3c21-4b7f-969c-915bb408c64f/z+vm_id_e81d1ac8-b49a-4ccf-9d92-5f1ef358f1be/z3vm_res_id_72477d99-c475-4a5d-90ae-2560f5f3b319_vda/z.deac2b8a-dca9-4415-adc1-f3c6598204ed-segments/z0000000000000000.00000000z Test Data)r, ÚKeyÚBody)r, r. )
r% Z head_bucketr& Z get_object_lock_configurationÚ ExceptionÚosÚpathÚjoinÚ
put_objectÚ
delete_object)r( r Z obj_lock_cfgZexZlong_keyr r r
Úvalidate_s3_clientb s@
z S3Backend.validate_s3_clientc C sÌ y°xªt |dkr¬g }|dt
}x0|D ](}|jdr@|dd
}|jd|i q&W y| jj| jd|id W n0 x"|D ]}| jj| j|d d qzW Y nX |td
}qW W n tk
rÆ Y nX dS ) zß Utility method that takes a list of objects and puts it in the correct format
for the S3 delete_objects() API.
Args:
object_list (list): A list of objects with the correct S3 path.
r Nú/r r. ÚObjects)r, ÚDelete)r, r. ) ÚlenÚMAX_S3_PAGE_SIZEÚ
startswithÚappendr% Údelete_objectsr& r5 r )r( Ú object_listZobject_delete_listZ list_partÚobjr r r
Z__delete_object_list s$
z S3Backend.__delete_object_listc C s y|g }| j jd}|j| j|d}xL|D ]D}d|kr(|jdt|d dkr(x |d D ]}|j|d qVW q(W | j| W n tk
r Y nX dS )z Utility method used to perform a rmdir operation on an object hierarchy
Args:
object_tree (str): Object/Tree path.
Ú list_objects)r, ÚPrefixÚContentsÚKeyCountr r. N) r% Ú
get_paginatorÚpaginater& r r: r= Ú _S3Backend__delete_object_listr )r( Ú object_treer? Úlist_paginatorÚ
page_iteratorÚobjectsr@ r r r
Z__delete_object_tree² s
z S3Backend.__delete_object_treec C s | j | dS )zó Delete a list of objects.
Args:
object_list (list): List of objects. Each with a full object path
in correct format.
options (dic): Dictionary of configuration options.
N)rG )r( r? r r r r
Údelete_object_listÈ s z S3Backend.delete_object_listc C s¬ y|s
dS dj |}|jdr*|dd
}d|krF| jj| j|d nH|jdd }| j|dd}|rxt|dd d
}n|}| jj| j|d W n tk
r¦ Y nX dS ) a Delete an object from the S3 object store.
The object and any segments stored in the segment directory will be
removed.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Nr7 r z -segments)r, r. z .manifestT)Ústrict_path_filter)Úreverser ) r3 r< r% r5 r& ÚstriprA Úsortedr )r( Úargsr Ú object_nameÚmanifest_prefixÚ manifestsÚ
manifest_namer r r
r5 Ò s$
zS3Backend.delete_objectr c C s¤ t d|}y~xxtd|D ]j}y| jj| j|d dS tk
r } z2|jd d dkrn|d |krntjd w W Y dd}~X qX qW W n tk
r Y nX dS )a Utility method used to wait for a S3 object to become available.
This routine will keep performing a head_object() request every second
for "retries" number of times. This was added to work around any potential
eventual consistancey issues when uploading to AWS.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter that defaults to 1.
Returns:
Returns when the object is available or ClientError not found exception.
r r )r, r. NÚResponseMetadataÚHTTPStatusCodei ) ÚmaxÚranger% Ú head_objectr& r ÚresponseÚtimeÚsleep)r( rR ÚretriesÚretryÚerrorr r r
Z__wait_for_objectõ s
zS3Backend.__wait_for_objectc C s¬ y|j dd }| jd râ| jd r:| jj| j|d}n| jj| j|d}|ræ|}|jdg pj|jdg }| jd rÔx\|D ]T}| jj| j|d |jd d
d }|d d
jdddkr|tj |d j
|d S q|W |d d }n|}|sòt d| jj| j|d}tj |d j
|d S t k
r¦ } zn|j
d d dkrF nPy6| j|d | jj| j|d}tj |d j
|d S t k
r Y nX W Y dd}~X nX dS )zÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
r7 z .manifestr- r* )r, rB ÚVersionsrC r. Ú VersionIdr )r, r. rb rV Ú HTTPHeadersz$x-amz-meta-stamp-trilio-authenticityÚFalseÚTruer/ Ú
ContentLengthr zOManifest object with 'x-amz-meta-stamp-trilio-authenticity' attribute not found)r, r. rW i é
Néÿÿÿÿ)rO r r% Úlist_object_versionsr& rA r Ú
get_objectÚjsonÚloadsÚreadr0 r r[ Ú_S3Backend__wait_for_object)r( rR rS rT rU ÚmanifestÚrespr` r r r
Úget_object_manifest sL
z S3Backend.get_object_manifestc C sà |j dd }| jd r,| jj| j|d}n| jj| j|d}|rÐ|jdg pX|jdg }g }x |D ]}|jd|d i qdW |rÜy| jj| jd|id W qÜ x"|D ]}| jj | j|d d
qªW Y qÜX n t
d | d S )
zÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
r7 z .manifestr* )r, rB ra rC r. r8 )r, r9 )r, r. z Manifest for %s is not foundN) rO r r% ri r& rA r r= r> r5 r0 )r( rR rS rp rT Úobjs_to_deleteÚmanr@ r r r
Údelete_object_manifestL s,
z S3Backend.delete_object_manifestc C sh y8dj |}i }|jr d|ji}| jj| j||j| W n* tk
rN Y n tk
rb Y nX dS )a? Download a file from the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
r7 rb N)r3 Ú
version_idr% Z
download_filer& Zout_filer r0 )r( rQ r rR Z
extra_argsr r r
Údownload_objecto s
zS3Backend.download_objectc C sv yFdj |}|jr*| jj| j||jd}n| jj| j|d}|d j S tk
r\ Y n tk
rp Y nX dS )aB Get object from the S3 object store in buffer.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
r7 )r, r. rb )r, r. r/ N)r3 ru r% rj r& rm r r0 )r( rQ r rR r[ r r r
rj s
zS3Backend.get_objectc
C s yh|j dkr0| jtjj|d d |d | |jdk rf| jj| j|d d |d ||dd}|S W n t k
r~ Y nX dS )a Put an object to the S3 object store.
Args:
args (list): List of object name parts.
buf: Data to upload in object.
buf_len: Length of data to upload in object.
options (dic): Dictionary of configuration options.
Fr r7 rR Nzapplication/octet-stream)r, r. rf r/ Ú ContentType)
Ú
path_validÚ_S3Backend__create_folderr1 r2 ÚdirnamerR r% r4 r& r )r( rQ ZbufZbuf_lenr rp r r r
r4 ¡ s
zS3Backend.put_objectNFc C sR t }|jd}y"t}|dk r,|tk r,|}| jd rD| jjd}n | jjd}|j| j|dd|id}xÂ|D ]¸}d|ko|jd t |d d
krxh|d D ]\} |dk rÂt ||krÂt
|S | d j d}
|rà|j |
q¢d |
kr¢|j |
j
d d
d
q¢W d|kr|jd t |d d
krxr|d D ]f} |dk rTt ||krTt
|S | d j d}
|rt|j |
n d |
kr0|j |
j
d d
d
q0W d|krpx|d D ]z} |dk rÐt ||krÐt
|S |jtsî| d jt r¬|jdr|j | d jd n|j | d jd q¬W qpW t
|S tk
rL Y nX dS )ae Return a list of objects based on the provided prefix.
Used to generate a directory listing based on a prefix path
constructed from the object parts in prefix
Args:
prefix (string): prefix of the object path. If the prefix is terminated
by "/", then it lists the contents in the "subdir". If the prefix
is not terminated by "/", then it lists the contents in the current
directory that matches the prefix
Ex:
case 1:
=======
prefix - /a/b/c
object store has objects - /a/b/c, /a/b/c1, /a/b/d
return values - /a/b/c and /a/b/c1
case 2:
=======
prefix - /a/b/c/
object store has objects - /a/b/c/d, /a/b/c1, /a/b/c/e
return values - /a/b/c/d and /a/b/c/e
options (dic): Dictionary of configuration options
max_items (int): Maximum number of objects in a list query. Default is
unlimited but when testing for an empty directory a smaller
number can be used to limit the query and size of the list that
is returned.
strict_path_filter (bool): Defaults to True in order to only return
objects with the exact same prefix. Individual segment listings
set this to False in order to get a match on the segment offset.
Returns:
On success a list of unique items to be used as a directory listing.
r7 Nr* ri rA ZPageSize)r, rB Ú DelimiterZPaginationConfigrC rD r r. z .manifestr ra ÚCommonPrefixesrB )ÚsetÚlstripr; r r% rE rF r& r r: ÚlistÚrstripÚaddÚsplitr< Ú_SEGMENT_PREFIXÚendswithrO r ) r( ÚprefixZ max_itemsrM Z
object_setZ page_sizerI rJ rK ÚitemrR Z
object_prefixr r r
rA ½ sX #
" $
zS3Backend.list_objectsc C sH y,g }t jjdj||j}| j|dd}|S tk
rB Y nX dS )a& Returns a list of object segments based on a prefix.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A list of segments in the object store that match the supplied prefix.
r7 T)rM N)r1 r2 r3 r
rA r )r( rQ r Z segment_listZ
segments_pathr? r r r
Ú
list_segments! s
zS3Backend.list_segmentsc
/ C s t d|}yXxPtd|D ]@}y| jj| j|d}|S tk
r\ } z|jd d dkrJy,| jj| j|d d}|jdpd |d< |S tk
rH } z|jd d dkr6d
|kr y|jdd }| j d rò| jj
| j|d
}n| jj | j|d
}|jdg p |jdg }|r®t |dd d}d}| j d rxL|D ]D} | jj| j| d | jddd}|d d jdddkrJ|S qJW |d d }|s²t
d n|}| jj| j|d}|S tk
r } z6|jd d dkr|d |krtjd w W Y dd}~X nX |d |kr6tjd w W Y dd}~X nX W Y dd}~X q X q W W n tk
r| Y nX dS ) a= Utility method that gets an object head from the repository with retry support.
The default is to try once. Admittedly, this is ugly because the
provided object name might be a terminal object or a "directory" so
we need to try both cases for each retry attempt. When retry > 1, there is
a 1 second delay between attempts.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter, default is 1 attempt.
Returns:
Object head dictionary or Boto3 exception.
r r )r, r. rV rW i r7 rw zapplication/x-directoryz -segmentsz .manifestr* )r, rB ra rC c S s | d S )NÚ LastModifiedr )Úkr r r
Ú<lambda>f s z0S3Backend.__get_object_headers.<locals>.<lambda>)r Nr. rb r )r, r. rb rc z$x-amz-meta-stamp-trilio-authenticityrd re zOManifest object with 'x-amz-meta-stamp-trilio-authenticity' attribute not foundrh )rX rY r% rZ r& r r[ r rO r ri rA rP r0 r\ r] )
r( rR r^ r_ Ú obj_headersr` rS rT rU ro r r r
Z__get_object_headers6 s|
z S3Backend.__get_object_headersc
C s ddddd}i |d< i |d d< dj |}yÀ|sJttjj |d< |S y| j|}W nt tk
rÌ | jj| jd|d }d
|krÆ|d d d |d
D krÈtjj |d
d d d}d|i d}n Y nX t|d |d< ||d< |j
dddks|d dkrd|d< xX|d d j D ]D\}} d|j krR| |d |j j
dd < n| |d |j < q W t|d d j
dddkr |d d j
d d|d < n<t|d j
d ddkrÐ|d j
d d|d < n |d |d < |d |d < |S W n tk
r Y nX d!S )"a) Get "operating system like" stat data for the given object.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A stat structure containing object information required by caller.
r r F)Ú timestampÚsizeÚetagÚ directoryr ÚMetadatar7 r )r, r{ rB r| c S s g | ] }|d qS )rB r )Ú.0Úxr r r
ú
<listcomp>ª s z)S3Backend.stat_object.<locals>.<listcomp>rV rc Zdatez%a, %d %b %Y %H:%M:%S %Z)rf r r r rw z
text/plainzapplication/x-directoryrf Tr zx-object-meta-r zx-object-meta-segment-countzx-object-meta-total-sizer z
segment-countz
total-sizezx-account-bytes-usedN)r3 r r ZnowÚ _S3Backend__get_object_headersÚ
BaseExceptionr% rA r& Zstrptimer ÚitemsÚlowerr r r )
r( rQ r Z stat_datarR Z
obj_headerrK Zmdr Úvaluer r r
Ú stat_object s^
zS3Backend.stat_objectc C s dj |}| j|| dS )zÎ Create an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r7 N)r3 ry )r( rQ r Z
new_folderr r r
Ú mkdir_objectÕ s
zS3Backend.mkdir_objectc C s^ dj |d }|jdr$|dd
}| j| y| jj| j|d W n tk
rX Y nX dS )zÎ Remove an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r7 r N)r, r. )r3 r< Ú _S3Backend__delete_object_treer% r5 r& r )r( rQ r rH r r r
Ú rmdir_objectà s
zS3Backend.rmdir_objectc
C sr yR|j dd }| jd rv| j|dd}|r`t|ddd jdd }t|d
}d |d }nd d }|d | }n|}x*|D ]"}|jd |d
< |jdd|d< qW tj|} | j j
| j ||| dd | jd rR|rRg }
x|D ]} |
j d| i qäW y| j j
| j d|
id W n< tk
rP x$|
D ] } | j j| j | d d q,W Y nX W n tk
rl Y nX dS )a Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be
added to the object.
manifest_data (dic): The manifest data which becomes
the body of file.
r7 z .manifestr- T)rM )rN r Ú.r é z%08xr2 Únamer Úhashz
text/plain)r, r. r r/ rw r. r8 )r, r9 )r, r. Nrh )rO r rA rP r r Úpoprk Údumpsr% r4 r& r= r> r r5 r )
r( rR Z put_headersZ
manifest_datarS rT ÚsuffixrU Z segment_dataro rr rs r@ r r r
Úupload_object_manifestô sJ
z S3Backend.upload_object_manifestc
C s¾ y¢|j dd }| jd rt| j|dd}|r^t|ddd jdd }t|d
}d |d }nd d }|d | }n|}tjj| j |}| j
j | j |||d d
W n t k
r¸ Y nX dS )a
Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
attributes (dic): The attributes data which becomes
the body of file.
r7 z .manifestr- T)rM )rN r r r r z%08xZREPLACE)r, r. Z
CopySourcer ZMetadataDirectiveNrh )
rO r rA rP r r r1 r2 r3 r& r% Z copy_objectr )r( rR Z
attributesrS rT r£ rU Zcopy_srcr r r
Úupdate_object_attributes' s*
z"S3Backend.update_object_attributesc C sF y*|j d}| jj| j|| j|dd W n tk
r@ Y nX dS )a Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added
to the object.
manifest_data (dic): The manifest data which becomes the
body of file.
r7 )ÚModeÚRetainUntilDate)r, r. Ú RetentionN)rO r% Úput_object_retentionr& r' r )r( rR Ú retainuntilr r r
Ú put_object_segment_retentionH s
z&S3Backend.put_object_segment_retentionc C s² y|j dd }| jd rt| j|dd}|r^t|ddd jdd }t|d
}d |d }nd d }|d | }n|}| jj| j|| j |d d
W n t
k
r¬ Y nX dS )aK Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest"
will be added. put_headers (dic): Dictionary of meta data
to be added to the object. manifest_data (dic): The manifest
data which becomes the body of file.
r7 z .manifestr- T)rM )rN r r r r z%08x)r¦ r§ )r, r. r¨ Nrh ) rO r rA rP r r r% r© r& r' r )r( rR rª rS rT r£ rU r r r
Úput_manifest_retention^ s(
z S3Backend.put_manifest_retentionc ) C s* y| j j| j|d W n tk
r$ } zî|jd d dkr|jd}d}xÄ|D ]¼}|dkrbP || d }y| j j| j|d W qT tk
r } zl|jd d dkrüy| j j| j|ddd W n6 tk
rú } z|jd d d
krê W Y d d }~X nX wTW Y d d }~X qTX qTW W Y d d }~X nX d S ) a< Utility method used to create a object directory structure.
The entire path is split and each directory level is created
as an object if it does not exist.
Args:
folder_name (str): The entire folder name.
options (dic): Dictionary of configuration options.
)r, r. rV rW i r7 r z TrilioVault directory objectzapplication/x-directory)r, r. r/ rw i N)r% rZ r& r r[ r r4 )r( Z folder_namer r` Ú
path_partsÚnew_pathÚpartr r r
Z__create_folder s:
zS3Backend.__create_folderc C sÊ |dd
}y¢|j dkr<| jtjj|d d |d | |jdk r¬t|d dR}| jj|| j |d d |d dd id
| jj
| j |d d |d d }|S Q R X W n t k
rÄ Y nX dS ) zµ Upload an object to the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
r NFr r7 rR Úrbrw zapplication/octet-stream)r, r. Z ExtraArgs)r, r. ) rx ry r1 r2 rz rR Úopenr% Zupload_fileobjr& rZ r )r( rQ r ÚfilesÚdatar r r r
Ú
upload_object° s(
zS3Backend.upload_object)r )NF)r ) Ú__name__Ú
__module__Ú __qualname__Ú__doc__r) r+ r6 rG r rL r5 rn rq rt rv rj r4 rA r r r r r r¤ r¥ r« r¬ ry r´ r r r r
r 1 s4 %2
#
8#
d
XG 3!!1r )r¸ r r\ rk r1 Zoslo_logr Zloggingr" r! Zbotocore.exceptionsr Z s3fuse.utilsr ZBASENAMEr; r r Z getLoggerrµ ZLOGr Úobjectr r r r r
Ú<module> s