Why Gemfury?
Push, build, and install
RubyGems
npm packages
Python packages
Maven artifacts
PHP packages
Go Modules
Debian packages
RPM packages
NuGet packages
Repository URL to install this package:
|
Version:
4.1.130 ▾
|
ó
âí`c @ s d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d Z d Z
d Z d Z d Z
d e f d
YZ d S( s Trilio S3 Backend implimentation
This module contains the back end implimentation of all of all S3 specific
support.
iÿÿÿÿN( t ClientErrort s3iè s$ 80bc80ff-0c51-4534-86a2-ec5e719643c2c C s5 | j d d } | t j d d d j } | S( sè Utility function used to convert a datetime to an OS timestamp.
Args:
modified_time (datetime): Datatime object to convert to a Unix Epoc timestamp.
Returns:
The value of modified_time as a timestamp.
t tzinfoi² i N( t replacet Nonet datetimet
total_seconds( t
modified_timet
naive_timet
delta_seconds( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt _make_timestamp s c C s{ yj | j d | t j j d d d d d d d } | j d | d | d
d | j d | d | Wn
n Xd S(
s Validates the S3 credentials / permission on the provided bucket.
Args:
s3_client : Initialized boto3 S3 client object
s3_bucket : User provided S3 bucket
Returns:
Does not return anything, But raises exception if validation fails.
t Buckets tvault_config/s. workload_f5190be6-7f80-4856-8c24-149cb40500c5/s. snapshot_f2e5c6a7-3c21-4b7f-969c-915bb408c64f/s+ vm_id_e81d1ac8-b49a-4ccf-9d92-5f1ef358f1be/s3 vm_res_id_72477d99-c475-4a5d-90ae-2560f5f3b319_vda/s. deac2b8a-dca9-4415-adc1-f3c6598204ed-segments/s 0000000000000000.00000000t Keyt Bodys Test DataN( t head_buckett ost patht joint
put_objectt
delete_object( t s3_clientt s3_buckett long_key( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt validate_s3_client, s
t S3Backendc B s³ e Z d Z d Z d Z d Z d Z d Z d d Z d Z d Z
d e d
Z
d Z d d Z d
Z d Z d Z d Z d Z d Z RS( s´ S3 Backend implimentation.
A Wrapper for the AWS S3 boto3 and botocore API. This class encapsulates all S3
operations and exposes them as a backend storage instance.
c C sM d } d } | d d k rH t j j d | d d t | d } n t j j d t | d } | d d k rÓ t j d d | d
d d | d d
| d d | d d | d
d d | d | d } nO t j d d | d
d d | d d
| d d | d d | d
d d | } t | | d | | _ | d | _ d S( Nt s3_signaturet defaultt signature_versiont read_timeoutt s3_read_timeoutt s3_ssl_certt R t region_namet
os_optionst use_sslt s3_sslt aws_access_key_idt usert aws_secret_access_keyt keyt endpoint_urlt object_storage_urlt configt verifyt bucket( R t botocoret clientt Configt intt boto3R t _S3Backend__clientt _S3Backend__bucket_name( t selft optionst
config_objectR ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt __init__R s. ,
c C s¶ y x t | d k r g } | t } x> | D]6 } | j d rQ | d } n | j i | d 6 q/ W| j j d | j d i | d 6 | t } q WWn t k
r± n Xd S( sß Utility method that takes a list of objects and puts it in the correct format
for the S3 delete_objects() API.
Args:
object_list (list): A list of objects with the correct S3 path.
i t /i R R t Deletet ObjectsN( t lent MAX_S3_PAGE_SIZEt
startswitht appendR2 t delete_objectsR3 R ( R4 t object_listt object_delete_listt list_partt obj( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt __delete_object_list s
c C sÇ y¬ g } | j j d } | j d | j d | } xe | D]] } d | k r= | j d t | d d k r= x& | d D] } | j | d q| Wq= q= W| j | Wn t k
r n Xd S( s Utility method used to perform a rmdir operation on an object hierarchy
Args:
object_tree (str): Object/Tree path.
t list_objectsR t Prefixt Contentst KeyCounti R N( R2 t
get_paginatort paginateR3 t getR; R> t _S3Backend__delete_object_listR ( R4 t object_treeR@ t list_paginatort
page_iteratort objectsRC ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt __delete_object_tree s
.
c C s | j | d S( só Delete a list of objects.
Args:
object_list (list): List of objects. Each with a full object path
in correct format.
options (dic): Dictionary of configuration options.
N( RL ( R4 R@ R5 ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt delete_object_list® s c C s y | s
d Sd j | } | j d r8 | d } n d | k rc | j j d | j d | n | j j d | j d | d Wn t k
r n Xd S( s Delete an object from the S3 object store.
The object and any segments stored in the segment directory will be
removed.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
NR8 i s -segmentsR R s .manifest( R R= R2 R R3 R ( R4 t argsR5 t object_name( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyR ¸ s
i c C s¿ t d | } y x t d | D]} } y$ | j j d | j d | d SWq" t k
r } | j d d d k r | d | k r t j d q" n q" Xq" WWn t k
rº n Xd S( s Utility method used to wait for a S3 object to become available.
This routine will keep performing a head_object() request every second
for "retries" number of times. This was added to work around any potential
eventual consistancey issues when uploading to AWS.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter that defaults to 1.
Returns:
Returns when the object is available or ClientError not found exception.
i i R R Nt ResponseMetadatat HTTPStatusCodei ( t maxt rangeR2 t head_objectR3 R t responset timet sleep( R4 RT t retriest retryt error( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt __wait_for_objectÖ s
c C sñ | j d d } y@ | j j d | j d | } t j | d j | d SWn t k
rì } | j d d d k r qí yP | j | d
| j j d | j d | } t j | d j | d SWqí t k
rè qí Xn Xd S( sÓ Download and return the object manifest as a json array.
Args:
object_name (str): Name of the object.
Returns:
Object manifest as a dictionary.
R8 s .manifestR R R
t
ContentLengthRU RV i i
N(
t stripR2 t
get_objectR3 t jsont loadst readR RZ t _S3Backend__wait_for_object( R4 RT t
manifest_namet respR_ ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt get_object_manifestõ s " "
c C s] y/ d j | } | j j | j | | j Wn' t k
rE n t k
rX n Xd S( s? Download a file from the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
On success, the contents of the object are downloaded to file identidfied
by options.out_file.
R8 N( R R2 t
download_fileR3 t out_fileR t Exception( R4 RS R5 RT ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt download_object s
c C s# t } d j | } | d d k rY | d | d } | j d rY | d } qY n y¯t } | d k r | t k r | } n | j j d } | sÆ | j d | j d d d i | d 6 } n( | j d | j d | d i | d 6 } x | D]}
d
|
k r|
j d t
|
d
d k r| d } xP|
d
D]A} | d k rlt
| | k rlt | St j
j | d
j d \ }
} | d k rõt j
j |
\ } } t
| d k r>| d k r>| | k r>| j | q>q>n t j
j | \ } } | |
k r%|
| k s>| d k s>| d
| k rGq>n | t k s_|
| k r>| j | j d d d q>q>Wn d |
k rõ xd |
d D]U } | d k rËt
| | k rËt | S| d j t s| j | d qqWqõ qõ Wt | SWn t k
r n Xd S( s° Return a list of objects based on the provided prefix.
Used to generate a directory listing based on a prefix path
constructed from the object parts in args[] and the options.prefix.
Args:
args (list): List of prefix keyParts
options (dic): Dictionary of configuration options
max_items (int): Maximum number of objects in a list query. Default is
unlimited but when testing for an empty directory a smaller
number can be used to limit the query and size of the list that
is returned.
strict_path_filter (bool): Defaults to True in order to only return
objects with the exact same prefix. Individual segment listings
set this to False in order to get a match on the segment offset.
Returns:
On success a list of unique items to be used as a directory listing.
R8 t prefixiÿÿÿÿRE R t Delimitert PaginationConfigt PageSizeRF RG RH i R s x.hiddenR s .manifesti t CommonPrefixesN( t setR R t endswithR< R2 RI RJ R3 RK R; t listR R t splitt rstript addt FalseR= t _SEGMENT_PREFIXR ( R4 RS R5 t max_itemst strict_path_filtert
object_setRo t page_sizeRN RO RP t split_tokent itemR RT t root_patht sub_dirt _t
object_prefix( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyRE % sZ
.
% *$'
c C s yg g } t j j | j \ } } | j | | d t } x# | D] } | j | d | qC W| SWn t k
r} n Xd S( s& Returns a list of object segments based on a prefix.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A list of segments in the object store that match the supplied prefix.
R} R8 N( R R Rw Ro RE Rz R> R ( R4 RS R5 t segment_listt
segments_pathR R@ R ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt
list_segments s
c C s t d | } yx|t d | D]k} y& | j j d | j d | } | SWq" t k
r} | j d d d k ry* | j j d | j d | d } | SWqt k
r} | j d d d k r|d | k rVy* | j j d | j d | d
} | SWqVt k
rR} | j d d d k rL| d | k rLt j d q" n qVXn | d | k r|t j d q" q|n qXn q" Xq" WWn t k
r¨ n Xd S( s= Utility method that gets an object head from the repository with retry support.
The default is to try once. Admittedly, this is ugly because the
provided object name might be a terminal object or a "directory" so
we need to try both cases for each retry attempt. When retry > 1, there is
a 1 second delay between attempts.
Args:
object_name (str): Name of the object.
retries (int): Optional parameter, default is 1 attempt.
Returns:
Object head dictionary or Boto3 exception.
i i R R RU RV i R8 s -segmentss .manifestN( RW RX R2 RY R3 R RZ R[ R\ ( R4 RT R] R^ t obj_headersR_ ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt __get_object_headers sB
c C s i d d 6d d 6d d 6t d 6} i | d <i | d d <d j | } y2| so t t j j | d <| S| j | } t | d
| d <| | d <| j d d d
k sÄ | d d k rÑ t | d <n x? | d d j D]) \ } } d | k ræ | | d | <qæ qæ Wt | d d j d d d k rZ| d d j d d | d <n | d | d <| d | d <| SWn t
k
r n Xd S( s) Get "operating system like" stat data for the given object.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
Returns:
A stat structure containing object information required by caller.
i t timestampt sizeR t etagt directoryt headerst MetadataR8 t LastModifiedt ContentTypes
text/plains application/x-directoryRa s
x-object-metas x-object-meta-segment-counts x-object-meta-total-sizes x-account-bytes-usedN( Rz R R
R t nowt _S3Backend__get_object_headersRK t Truet itemsR0 R ( R4 RS R5 t stat_dataRT t
obj_headerR' t value( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt stat_objectÐ s2
"
! &
c C s# d j | } | j | | d S( sÎ Create an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
R8 N( R t _S3Backend__create_folder( R4 RS R5 t
new_folder( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt mkdir_objectü s c C s\ d j | d } | j d r/ | d } n | j | | j j d | j d | d S( sÎ Remove an object that represents a directory in the object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
R8 i R R N( R R= t _S3Backend__delete_object_treeR2 R R3 ( R4 RS R5 RM ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt rmdir_object s
c C s© y x7 | D]/ } | j d | d <| j d d | d <q
Wt j | } | j d d } | j j d | j d | d
| d | d d
Wn t k
r¤ n Xd S( s? Upload a new object manifest to the object store.
Args:
object_name (str): Name of the object. A ".manifest" will be added.
put_headers (dic): Dictionary of meta data to be added to the object.
manifest_data (dic): The manifest data which becomes the body of file.
R t nameR i t hashR8 s .manifestR R R R
R s
text/plainN( t popRd t dumpsRb R2 R R3 R ( R4 RT t put_headerst
manifest_datat segment_datat manifestRh ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt upload_object_manifest s
c
C s y | j j d | j d | Wn\t k
r~} | j d d d k r| j d } d } x| D]} | d k r{ Pn | | d } y | j j d | j d | Wqe t k
rs} | j d d d k re | d t k r | j j d | j d | d d
d qmy, | j j d | j d | d
d d
d Wqmt k
ri} | j d d d k rj qjqmXqe qe qe Xqe Wqn Xd S( s< Utility method used to create a object directory structure.
The entire path is split and each directory level is created as an object
if it does not exist.
Args:
folder_name (str): The entire folder name.
options (dic): Dictionary of configuration options.
R R RU RV i R8 R t support_empty_dirs x.hiddenR
s
Do Not Removes TrilioVault directory objectR s application/x-directoryi N( R2 RY R3 R RZ Rw R R ( R4 t folder_nameR5 R_ t
path_partst new_patht part( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt __create_folder0 s4
c C sÑ | d } y¬ | j t k rK | j t j j | d d | d | n | j d k rµ t | d d C } | j j
| d | j d | d d | d d i d d
6Wd QXn Wn t k
rÌ n Xd S( sµ Upload an object to the S3 object store.
Args:
args (list): List of object name parts.
options (dic): Dictionary of configuration options.
i i R8 RT t rbR R t ExtraArgss application/octet-streamR N(
t
path_validRz R R R t dirnameRT R t openR2 t upload_fileobjR3 R ( R4 RS R5 t filest data( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt
upload_object` s
%
N( t __name__t
__module__t __doc__R7 RL R RR R Rg Rj Rn R R RE R R R R R R¨ R R· ( ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyR L s$ .
c 2 , 0( Rº R R[ Rd R R1 R- t botocore.exceptionsR t BASENAMER< R{ R
R t objectR ( ( ( s2 /usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt <module> s