Why Gemfury?
Push, build, and install
RubyGems
npm packages
Python packages
Maven artifacts
PHP packages
Go Modules
Debian packages
RPM packages
NuGet packages
Repository URL to install this package:
|
Version:
5.0.13-5.0 ▾
|
ó
s dc @ sÈ d Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l Z d d l Z d d l
m Z d d l m
Z
d Z d Z d Z d Z e j e Z d
Z d e f d YZ d S(
s Trilio S3 Backend implimentation
This module contains the back end implimentation of all of all S3 specific
support.
iÿÿÿÿN( t log( t ClientError( t get_retention_modet s3iè s$ 80bc80ff-0c51-4534-86a2-ec5e719643c2iô c C s5 | j d d } | t j d d d j } | S( sè Utility function used to convert a datetime to an OS timestamp.
Args:
modified_time (datetime): Datatime object to convert to a Unix Epoc timestamp.
Returns:
The value of modified_time as a timestamp.
t tzinfoi² i N( t replacet Nonet datetimet
total_seconds( t
modified_timet
naive_timet
delta_seconds( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt _make_timestamp" s t S3Backendc B sû e Z d Z d Z d Z d Z d Z d Z d Z d Z d d Z
d
Z d Z d Z
d
Z d Z d e d Z d Z d d Z d Z d Z d Z d Z d Z d Z d Z d Z d Z RS( s´ S3 Backend implimentation.
A Wrapper for the AWS S3 boto3 and botocore API. This class encapsulates all S3
operations and exposes them as a backend storage instance.
c C s d } d } | | _ | j d r9 t | j d n t } | d d k r t j j d | d d t | d d | } n% t j j d t | d d | } | d d k rt j d
d | d d d
| d d | d d | d d d | d | d } nE t j d
d | d d d
| d d | d d | d d d | } d } | j j
j d | | | _ | d | _
| d | _ d S( Nt vault_s3_max_pool_connectionst s3_signaturet defaultt signature_versiont read_timeoutt s3_read_timeoutt max_pool_connectionst s3_ssl_certt R t region_namet
os_optionst aws_access_key_idt usert aws_secret_access_keyt keyt endpoint_urlt object_storage_urlt configt verifyc [ s d | j d <d S( Ns
keep-alivet
Connection( t headers( t requestt operation_namet kwargs( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt set_connection_headerU s s request-created.s3t buckett retention_mode( R t optionst gett intt DEFAULT_MAX_POOL_SIZEt botocoret clientt Configt boto3t metat eventst registert _S3Backend__clientt _S3Backend__bucket_namet _S3Backend__retention_mode( t selfR) t
config_objectt s3_clientt
MAX_POOL_SIZER& ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt __init__8 s: $
c C s2 t | j | d | d | d <| d | _ d S( NR' t bucket_object_lockR( ( R R4 R6 ( R7 R) ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt set_retention_mode] s !c
C se yT| j j d | j y¥ | j j d | j } | d re | d d d k re t d | n | d r | d d d k r t d | n | d r½ | d s½ t d q½ n Wn# t k
rã } | d rä qä n X| d sSt j j d d
d d d
d d } | j j d | j d | d d | j j d | j d | n Wn
n Xd S( sf Validates the S3 credentials / permission on the provided bucket.
It also validates against s3 object lock configuration and if the s3 supports
more than 255 object name length
Args:
self : S3 Backend object
Returns:
Does not return anything, But raises exception if validation fails.
t BucketR< t ObjectLockConfigurationt ObjectLockEnabledt EnabledsZ Configration says object lock enabled on the bucket, but s3 API does not agree with it. %st use_manifest_suffixsR 'use_manifest_suffix' must be set to True when 'bucket_object_lock' is set to Trues tvault_config/s. workload_f5190be6-7f80-4856-8c24-149cb40500c5/s. snapshot_f2e5c6a7-3c21-4b7f-969c-915bb408c64f/s+ vm_id_e81d1ac8-b49a-4ccf-9d92-5f1ef358f1be/s3 vm_res_id_72477d99-c475-4a5d-90ae-2560f5f3b319_vda/s. deac2b8a-dca9-4415-adc1-f3c6598204ed-segments/s 0000000000000000.00000000t Keyt Bodys Test DataN(
R4 t head_bucketR5 t get_object_lock_configurationt Exceptiont ost patht joint
put_objectt
delete_object( R7 R) t obj_lock_cfgt ext long_key( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt validate_s3_clientb s@
#c C sõ yÚ xÓ t | d k rØ g } | t } x> | D]6 } | j d rQ | d } n | j i | d 6 q/ Wy' | j j d | j d i | d 6 Wn8 x2 | D]& } | j j d | j d | d q Wn X| t } q WWn t k
rð n Xd S( sß Utility method that takes a list of objects and puts it in the correct format
for the S3 delete_objects() API.
Args:
object_list (list): A list of objects with the correct S3 path.
i t /i RC R> t Deletet ObjectsN( t lent MAX_S3_PAGE_SIZEt
startswitht appendR4 t delete_objectsR5 RL R ( R7 t object_listt object_delete_listt list_partt obj( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt __delete_object_list s$
c C sÇ y¬ g } | j j d } | j d | j d | } xe | D]] } d | k r= | j d t | d d k r= x& | d D] } | j | d q| Wq= q= W| j | Wn t k
r n Xd S( s Utility method used to perform a rmdir operation on an object hierarchy
Args:
object_tree (str): Object/Tree path.
t list_objectsR> t Prefixt Contentst KeyCounti RC N( R4 t
get_paginatort paginateR5 R* RT RW t _S3Backend__delete_object_listR ( R7 t object_treeRY t list_paginatort
page_iteratort objectsR\ ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt __delete_object_tree² s
.
c C s | j | d S( só Delete a list of objects.
Args:
object_list (list): List of objects. Each with a full object path
in correct format.
options (dic): Dictionary of configuration options.
N( Rd ( R7 RY R) ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt delete_object_listÈ s c C sè yÍ | s
d Sd j | } | j d r8 | d } n d | k rc | j j d | j d | ni | j d d } | j | d t } | rª t | d t d
} n | } | j j d | j d | Wn t k
rã n Xd S( s Delete an object from the S3 object store.
The object and any segments stored in the segment directory will be
removed.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
NRQ i s -segmentsR> RC s .manifestt strict_path_filtert reversei (
RJ RV R4 RL R5 t stripR^ t Truet sortedR ( R7 t argsR) t object_namet manifest_prefixt manifestst
manifest_name( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyRL Ò s$
i c C s¿ t d | } y x t d | D]} } y$ | j j d | j d | d SWq" t k
r } | j d d d k r | d | k r t j d q" n q" Xq" WWn t k
rº n Xd S( s Utility method used to wait for a S3 object to become available.
This routine will keep performing a head_object() request every second
for "retries" number of times. This was added to work around any potential
eventual consistancey issues when uploading to AWS.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter that defaults to 1.
Returns:
Returns when the object is available or ClientError not found exception.
i i R> RC Nt ResponseMetadatat HTTPStatusCodei ( t maxt rangeR4 t head_objectR5 R t responset timet sleep( R7 Rq t retriest retryt error( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt __wait_for_objectõ s
c C s< y| j d d } | j d rF| j d rQ | j j d | j d | } n | j j d | j d | } | rL| } | j d g p | j d g } | j d r2x | D]x } | j j d | j d | d d
| j d
d } | d d
j d d d k r³ t j | d j
| d Sq³ Wn | d d } qLn | } | sat d n | j j d | j d | } t j | d j
| d SWn t k
r7} | j
d d d k rÍ q8yP | j | d | j j d | j d | } t j | d j
| d SWq8t k
r3 q8Xn Xd S( sÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
RQ s .manifestRB R< R> R_ t VersionsR` RC t VersionIdR Ru t HTTPHeaderss$ x-amz-meta-stamp-trilio-authenticityt FalseRn RD t
ContentLengthiÿÿÿÿsO Manifest object with 'x-amz-meta-stamp-trilio-authenticity' attribute not foundRv i i
N( Rm R) R4 t list_object_versionsR5 R^ R* t
get_objectt jsont loadst readRG R Rz t _S3Backend__wait_for_object( R7 Rq Rr Rs Rt t manifestt respR ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt get_object_manifest sL
$
% " "
c C s: | j d d } | j d rA | j j d | j d | } n | j j d | j d | } | r&| j d g p | j d g } g } x& | D] } | j i | d d 6 q W| r6y' | j j d | j d i | d
6 Wq#x2 | D]& } | j j d | j d | d qò Wq#Xq6n t
d | d S(
sÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
RQ s .manifestR< R> R_ R R` RC RR RS s Manifest for %s is not foundN( Rm R) R4 R R5 R^ R* RW RX RL RG ( R7 Rq Rr R Rs t objs_to_deletet manR\ ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt delete_object_manifestL s,
$
c C s yT d j | } i } | j r4 i | j d 6} n | j j | j | | j | Wn' t k
rj n t k
r} n Xd S( s? Download a file from the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
RQ R N( RJ t
version_idR4 t
download_fileR5 t out_fileR RG ( R7 Rp R) Rq t
extra_args( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt download_objecto s
c C s yr d j | } | j rE | j j d | j d | d | j } n | j j d | j d | } | d j SWn' t k
r n t k
r n Xd S( sB Get object from the S3 object store in buffer.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
RQ R> RC R RD N( RJ R R4 R R5 R R RG ( R7 Rp R) Rq Rz ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyR s
c C s° y | j t k rA | j t j j | d d | d | n | j d
k r | j j d | j
d | d d | d d | d | d d } | SWn t k
r« n Xd
S( s Put an object to the S3 object store.
Args:
args (list): List of object name parts.
buf: Data to upload in object.
buf_len: Length of data to upload in object.
options (dic): Dictionary of configuration options.
i RQ Rq R> RC R
RD t ContentTypes application/octet-streamN( t
path_validR t _S3Backend__create_folderRH RI t dirnameRq R R4 RK R5 R ( R7 Rp t buft buf_lenR) R ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyRK ¡ s %
c C s t } | j d } yæt } | d k rB | t k rB | } n | j d rd | j j d } n | j j d } | j d | j d | d d d i | d 6 } xL| D]D} d
| k rt| j d t
| d
d k rtx | d
D] } | d k rt
| | k rt | S| d
j d }
| rA| j
|
qê d |
k rê | j
|
j d d d qê qê Wn d | k r7| j d t
| d d k r7x | d D] } | d k rÛt
| | k rÛt | S| d
j d }
| r| j
|
qd |
k r| j
|
j d d d qqWn d | k r« x© | d D] } | d k r|t
| | k r|t | S| j t s| d j t rN| j d rË| j
| d j d qè| j
| d j d qNqNWq« q« Wt | SWn t k
r n Xd S( se Return a list of objects based on the provided prefix.
Used to generate a directory listing based on a prefix path
constructed from the object parts in prefix
Args:
prefix (string): prefix of the object path. If the prefix is terminated
by "/", then it lists the contents in the "subdir". If the prefix
is not terminated by "/", then it lists the contents in the current
directory that matches the prefix
Ex:
case 1:
=======
prefix - /a/b/c
object store has objects - /a/b/c, /a/b/c1, /a/b/d
return values - /a/b/c and /a/b/c1
case 2:
=======
prefix - /a/b/c/
object store has objects - /a/b/c/d, /a/b/c1, /a/b/c/e
return values - /a/b/c/d and /a/b/c/e
options (dic): Dictionary of configuration options
max_items (int): Maximum number of objects in a list query. Default is
unlimited but when testing for an empty directory a smaller
number can be used to limit the query and size of the list that
is returned.
strict_path_filter (bool): Defaults to True in order to only return
objects with the exact same prefix. Individual segment listings
set this to False in order to get a match on the segment offset.
Returns:
On success a list of unique items to be used as a directory listing.
RQ R< R R^ R> R_ t Delimitert PaginationConfigt PageSizeR` Ra i RC s .manifesti R t CommonPrefixesN( t sett lstripRU R R) R4 Rb Rc R5 R* RT t listt rstript addt splitRV t _SEGMENT_PREFIXt endswithRm R ( R7 t prefixt max_itemsRk t
object_sett page_sizeRf Rg Rh t itemRq t
object_prefix( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyR^ ½ sX #
.
'.
'
(
c C s_ yD g } t j j d j | | j } | j | d t } | SWn t k
rZ n Xd S( s& Returns a list of object segments based on a prefix.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A list of segments in the object store that match the supplied prefix.
RQ Rk N( RH RI RJ R© R^ Rn R ( R7 Rp R) t segment_listt
segments_pathRY ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt
list_segments! s
!
c
C s t d | } yáxÚt d | D]É} y& | j j d | j d | } | SWq" t k
rê} | j d d d k räyC | j j d | j d | d } | j d p® d
| d <| SWqät k
rà} | j d d d k rÚd | k r´yo| j d d } | j d
r3| j j
d | j d | } n | j j d | j d | } | j d g pr| j d g } | r5t | d d } d } | j d
rxi | D]^ } | j j d | j d | d d | j d d } | d d j d d d k rª| SqªWn | d d } | s;t d q;n | } | j j d | j d | } | SWq´t k
r°} | j d d d k rª| d | k rªt j d q" n q´Xn | d | k rÚt j d q" qÚn qäXn q" Xq" WWn t k
r n Xd S( s= Utility method that gets an object head from the repository with retry support.
The default is to try once. Admittedly, this is ugly because the
provided object name might be a terminal object or a "directory" so
we need to try both cases for each retry attempt. When retry > 1, there is
a 1 second delay between attempts.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter, default is 1 attempt.
Returns:
Object head dictionary or Boto3 exception.
i i R> RC Ru Rv i RQ R s application/x-directorys -segmentss .manifestR< R_ R R` R c S s | d S( Nt LastModified( ( t k( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt <lambda>f s R R R s$ x-amz-meta-stamp-trilio-authenticityR Rn iÿÿÿÿsO Manifest object with 'x-amz-meta-stamp-trilio-authenticity' attribute not foundN( Rw Rx R4 Ry R5 R Rz R* Rm R) R R^ Ro R RG R{ R| (
R7 Rq R} R~ t obj_headersR Rr Rs Rt R ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt __get_object_headers6 s|
$
c C sº i d d 6d d 6d d 6t d 6} i | d <i | d d <d j | } yV| so t t j j | d <| Sy | j | } Wn± t k
r5| j j d
| j d d d | } d
| k r/| d g | d
D] } | d ^ qÔ k r2t j j
| d d d d } i d d 6| d 6i d 6} q2q6 n Xt | d | d <| | d <| j d d d k s|| d d k rt | d <n xl | d d j
D]V \ } }
d | j k rà|
| d | j j d d <q|
| d | j <qWt | d d j d d d k r?| d d j d d | d <nM t | d j d d d k r~| d j d d | d <n | d | d <| d | d <| SWn t k
rµ n Xd S( s) Get "operating system like" stat data for the given object.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A stat structure containing object information required by caller.
i t timestampt sizeR t etagt directoryR" t MetadataRQ R> R R_ R Ru R t dates %a, %d %b %Y %H:%M:%S %ZR
R² R s
text/plains application/x-directorys x-object-meta-i s x-object-meta-segment-counts x-object-meta-total-sizes
segment-counts
total-sizes x-account-bytes-usedN( R RJ R R t nowt _S3Backend__get_object_headerst
BaseExceptionR4 R^ R5 t strptimeR* Rn t itemst lowerR¦ R+ R ( R7 Rp R) t stat_dataRq t
obj_headerRh t xt mdR t value( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt stat_object s^
"
+
!$
"
c C s# d j | } | j | | d S( sÎ Create an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
RQ N( RJ R ( R7 Rp R) t
new_folder( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt mkdir_objectÕ s c C st d j | d } | j d r/ | d } n | j | y | j j d | j d | Wn t k
ro n Xd S( sÎ Remove an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
RQ i R> RC N( RJ RV t _S3Backend__delete_object_treeR4 RL R5 R¿ ( R7 Rp R) Re ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt rmdir_objectà s
c
C sá yÆ| j d d } | j d r | j | d t } | r t | d t d j d d } t | d } d
| d } n d } | d | } n | } x7 | D]/ } | j d | d
<| j d d | d <q¥ Wt j | } | j
j d | j d | d | d | d d | j d rÅ| rÅg }
x" | D] } |
j
i | d 6 q5Wy' | j
j d | j d i |
d 6 WqÂt k
r¾x2 |
D]& } | j
j d | j d | d qWqÂXqÅn Wn t k
rÜ n Xd S( s Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be
added to the object.
manifest_data (dic): The manifest data which becomes
the body of file.
RQ s .manifestRB Rk Rl i t .iÿÿÿÿi s %08xi RI t nameR¹ t hashR> RC R» RD R s
text/plainRR RS Nt 00000000( Rm R) R^ Rn Ro R¦ R+ t popR t dumpsR4 RK R5 RW RX R¿ RL R (
R7 Rq t put_headerst
manifest_dataRr Rs t suffixRt t segment_dataR R R R\ ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt upload_object_manifestô sJ
#
c C s yå | j d d } | j d r | j | d t } | r t | d t d j d d } t | d } d
| d } n d } | d | } n | } t j j | j
| } | j j d | j
d
| d | d | d d Wn t
k
rû n Xd S( s
Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
attributes (dic): The attributes data which becomes
the body of file.
RQ s .manifestRB Rk Rl i RÍ iÿÿÿÿi s %08xi R> RC t
CopySourceR» t MetadataDirectivet REPLACENRÐ ( Rm R) R^ Rn Ro R¦ R+ RH RI RJ R5 R4 t copy_objectR ( R7 Rq t
attributesRr Rs RÕ Rt t copy_src( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt update_object_attributes' s*
#
c C sa yF | j d } | j j d | j d | d i | j d 6| d 6 Wn t k
r\ n Xd S( s Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added
to the object.
manifest_data (dic): The manifest data which becomes the
body of file.
RQ R> RC t Retentiont Modet RetainUntilDateN( Rm R4 t put_object_retentionR5 R6 R ( R7 Rq t retainuntil( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt put_object_segment_retentionH s
c C sí yÒ | j d d } | j d r | j | d t } | r t | d t d j d d } t | d } d
| d } n d } | d | } n | } | j j d | j d
| d i | j
d 6| d 6 Wn t k
rè n Xd S( sK Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest"
will be added. put_headers (dic): Dictionary of meta data
to be added to the object. manifest_data (dic): The manifest
data which becomes the body of file.
RQ s .manifestRB Rk Rl i RÍ iÿÿÿÿi s %08xi R> RC Rß Rà Rá NRÐ ( Rm R) R^ Rn Ro R¦ R+ R4 Râ R5 R6 R ( R7 Rq Rã Rr Rs RÕ Rt ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt put_manifest_retention^ s(
#
c
C sJ y | j j d | j d | Wn#t k
rE} | j d d d k rF| j d } d } xá | D]Ö } | d k r{ Pn | | d } y | j j d | j d | Wqe t k
r:} | j d d d k re y, | j j d | j d | d d d
d Wq4t k
r0} | j d d d k r1 q1q4Xqe qe qe Xqe WqFn Xd
S( s< Utility method used to create a object directory structure.
The entire path is split and each directory level is created
as an object if it does not exist.
Args:
folder_name (str): The entire folder name.
options (dic): Dictionary of configuration options.
R> RC Ru Rv i RQ R RD s TrilioVault directory objectR s application/x-directoryi N( R4 Ry R5 R Rz R¦ RK ( R7 t folder_nameR) R t
path_partst new_patht part( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt __create_folder s:
c C s | d } yÞ | j t k rK | j t j j | d d | d | n | j d k rç t | d d u } | j j
| d | j d | d d | d d i d d
6| j j d | j d | d d | d } | SWd QXn Wn t
k
rþ n Xd S( sµ Upload an object to the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
i i RQ Rq t rbR> RC t ExtraArgss application/octet-streamR N( R R R RH RI R Rq R t openR4 t upload_fileobjR5 Ry R ( R7 Rp R) t filest dataRµ ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt
upload_object° s*
N( t __name__t
__module__t __doc__R; R= RP Rd RË Rj RL R R R R R RK R R R^ R± R¾ RÈ RÊ RÌ R× RÞ Rä Rå R Rñ ( ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyR
1 s4 % 2
# 8 # d X G 3 ! ! 1( Rô R R{ R RH t oslo_logR t loggingR0 R- t botocore.exceptionsR t s3fuse.utilsR t BASENAMERU R§ R, t getLoggerRò t LOGR t objectR
( ( ( s2 /usr/lib/python2.7/site-packages/s3fuse/vaults3.pyt <module> s