Why Gemfury? Push, build, and install  RubyGems npm packages Python packages Maven artifacts PHP packages Go Modules Debian packages RPM packages NuGet packages

triliodata-4-0-patch / s3-fuse-plugin deb

Repository URL to install this package:

Version: 
4.0.116  ▾
Details    
s3-fuse-plugin / usr / lib / python2.7 / dist-packages / s3fuse / vaults3.pyc
Size: Mime: 
ó
«ht_c@s“dZddlZddlZddlZddlZddlZddlZddlmZdZ	dZ
dZd„Zde
fd	„ƒYZdS(
s Trilio S3 Backend implimentation

    This module contains the back end implimentation of all of all S3 specific
    support.
iÿÿÿÿN(tClientErrorts3iès$80bc80ff-0c51-4534-86a2-ec5e719643c2cCs5|jddƒ}|tjdddƒjƒ}|S(sè Utility function used to convert a datetime to an OS timestamp.

    Args:
        modified_time (datetime): Datatime object to convert to a Unix Epoc timestamp.

    Returns:
        The value of modified_time as a timestamp.
    ttzinfoi²iN(treplacetNonetdatetimet
total_seconds(t
modified_timet
naive_timet
delta_seconds((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt_make_timestamps	t	S3BackendcBs³eZdZd„Zd„Zd„Zd„Zd„Zdd„Zd„Z	d	„Z
ded
„Z
d„Zdd„Zd
„Zd„Zd„Zd„Zd„Zd„ZRS(s´ S3 Backend implimentation.

    A Wrapper for the AWS S3 boto3 and botocore API. This class encapsulates all S3
    operations and exposes them as a backend storage instance.
    cCs3d}|ddkrBtjjd|ddt|dƒƒ}ntjjdt|dƒƒ}|ddkrÐtjdd	|d
d	d|dd
|dd|dd|d
dd|d|dƒ|_nRtjdd	|d
d	d|dd
|dd|dd|d
dd|ƒ|_|d|_dS(Nts3_signaturetdefaulttsignature_versiontread_timeoutts3_read_timeoutts3_ssl_certtRtregion_namet
os_optionstuse_sslts3_ssltaws_access_key_idtusertaws_secret_access_keytkeytendpoint_urltobject_storage_urltconfigtverifytbucket(RtbotocoretclienttConfigtinttboto3t_S3Backend__clientt_S3Backend__bucket_name(tselftoptionst
config_object((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt__init__2s*,





cCs¶y›x”t|ƒdkr™g}|t }x>|D]6}|jdƒrQ|d}n|ji|d6ƒq/W|jjd|jdi|d6ƒ|t}qWWntk
r±‚nXdS(	sß Utility method that takes a list of objects and puts it in the correct format
            for the S3 delete_objects() API.

        Args:
            object_list (list): A list of objects with the correct S3 path.
        it/itKeytBuckettDeletetObjectsN(tlentMAX_S3_PAGE_SIZEt
startswithtappendR%tdelete_objectsR&R(R'tobject_listtobject_delete_listt	list_parttobj((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt__delete_object_list]s



cCsÇy¬g}|jjdƒ}|jd|jd|ƒ}xe|D]]}d|kr=|jdt|dƒƒdkr=x&|dD]}|j|dƒq|Wq=q=W|j|ƒWntk
r‚nXdS(	s‘ Utility method used to perform a rmdir operation on an object hierarchy

        Args:
            object_tree (str): Object/Tree path.
        tlist_objectsR-tPrefixtContentstKeyCountiR,N(	R%t
get_paginatortpaginateR&tgetR0R3t_S3Backend__delete_object_listR(R'tobject_treeR5tlist_paginatort
page_iteratortobjectsR8((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt__delete_object_treevs
.
cCs|j|ƒdS(só Delete a list of objects.

        Args:
            object_list (list): List of objects. Each with a full object path
                                in correct format.
            options (dic): Dictionary of configuration options.
        N(RA(R'R5R(((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pytdelete_object_list‹scCsŸy„|s
dSdj|ƒ}|jdƒr8|d}nd|krc|jjd|jd|ƒn |jjd|jd|dƒWntk
rš‚nXdS(s Delete an object from the S3 object store.

        The object and any segments stored in the segment directory will be
        removed.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.
        NR+is	-segmentsR-R,s	.manifest(tjoinR2R%t
delete_objectR&R(R'targsR(tobject_name((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyRI•s



icCs¿td|ƒ}y•xŽtd|ƒD]}}y$|jjd|jd|ƒdSWq"tk
rž}|jdddkr˜|d|kr˜tjdƒq"n‚q"Xq"WWntk
rº‚nXdS(	s Utility method used to wait for a S3 object to become available.

        This routine will keep performing a head_object() request every second
        for "retries" number of times. This was added to work around any potential
        eventual consistancey issues when uploading to AWS.

        Args:
            object_name (str): Name of the object.
            retries (int): Optional parameter that defaults to 1.

        Returns:
            Returns when the object is available or ClientError not found exception.
        iiR-R,NtResponseMetadatatHTTPStatusCodei”(	tmaxtrangeR%thead_objectR&Rtresponsettimetsleep(R'RKtretriestretryterror((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt__wait_for_object³s

cCsñ|jdƒd}y@|jjd|jd|ƒ}tj|dj|dƒƒSWn—tk
rì}|jddd	kr‚‚qíyP|j	|d
ƒ|jjd|jd|ƒ}tj|dj|dƒƒSWqítk
rè‚qíXnXdS(sÓ Download and return the object manifest as a json array.

            Args:
                object_name (str): Name of the object.

            Returns:
                Object manifest as a dictionary.
        R+s	.manifestR-R,tBodyt
ContentLengthRLRMi”i
N(
tstripR%t
get_objectR&tjsontloadstreadRRQt_S3Backend__wait_for_object(R'RKt
manifest_nametrespRV((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pytget_object_manifestÒs	"	"
cCs]y/dj|ƒ}|jj|j||jƒWn'tk
rE‚ntk
rX‚nXdS(s? Download a file from the S3 object store.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.

        Returns:
            On success, the contents of the object are downloaded to file identidfied
            by options.out_file.
        R+N(RHR%t
download_fileR&tout_fileRt	Exception(R'RJR(RK((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pytdownload_objectìs 

c	Cs#tƒ}dj|ƒ}|ddk	rY|d|d}|jdƒrY|d }qYny¯t}|dk	rƒ|tkrƒ|}n|jjdƒ}|sÆ|jd|jdddi|d6ƒ}	n(|jd|jd	|di|d6ƒ}	x|	D]}
d
|
kr†|
j	dt
|
d
ƒƒdkr†|d}xP|
d
D]A}|dk	rlt
|ƒ|krlt|ƒStj
j|d
jdƒƒ\}
}|dkrõtj
j|
ƒ\}}t
|ƒdkr>|dkr>||kr>|j|ƒq>q>ntj
j|ƒ\}}||
kr%|
|ks>|dks>|d
|krGq>n|tks_|
|kr>|j|jddƒdƒq>q>Wnd|
krõxd|
dD]U}|dk	rËt
|ƒ|krËt|ƒS|d	jtƒs|j|d	ƒqqWqõqõWt|ƒSWntk
r‚nXdS(s° Return a list of objects based on the provided prefix.

        Used to generate a directory listing based on a prefix path
        constructed from the object parts in args[] and the options.prefix.

        Args:
            args (list): List of prefix keyParts
            options (dic): Dictionary of configuration options
            max_items (int): Maximum number of objects in a list query. Default is
                unlimited but when testing for an empty directory a smaller
                number can be used to limit the query and size of the list that
                is returned.
            strict_path_filter (bool): Defaults to True in order to only return
                objects with the exact same prefix. Individual segment listings
                set this to False in order to get a match on the segment offset.

        Returns:
            On success a list of unique items to be used as a directory listing.
        R+tprefixiÿÿÿÿR:R-t	DelimitertPaginationConfigtPageSizeR;R<R=iR,sx.hiddenRs	.manifestitCommonPrefixesN(tsetRHRtendswithR1R%R>R?R&R@R0tlisttostpathtsplittrstriptaddtFalseR2t_SEGMENT_PREFIXR(R'RJR(t	max_itemststrict_path_filtert
object_setRgt	page_sizeRCRDREtsplit_tokentitemRpRKt	root_pathtsub_dirt_t
object_prefix((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyR:sZ		
.

%*$'

cCs‚ygg}tjj|jƒ\}}|j||dtƒ}x#|D]}|j|d|ƒqCW|SWntk
r}‚nXdS(s& Returns a list of object segments based on a prefix.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.

        Returns:
            A list of segments in the object store that match the supplied prefix.
        RwR+N(RoRpRqRgR:RtR3R(R'RJR(tsegment_listt
segments_pathR~R5R{((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt
list_segmentses


cCs­td|ƒ}yƒx|td|ƒD]k}y&|jjd|jd|ƒ}|SWq"tk
rŒ}|jdddkr†y*|jjd|jd|dƒ}|SWq†tk
r‚}|jdddkr|d	|krVy*|jjd|jd|d
ƒ}|SWqVtk
rR}|jdddkrL|d|krLtjdƒq"n‚qVXn|d|kr|tjdƒq"q|n‚q†Xn‚q"Xq"WWntk
r¨‚nXdS(s= Utility method that gets an object head from the repository with retry support.

        The default is to try once. Admittedly, this is ugly because the
        provided object name might be a terminal object or a "directory" so
        we need to try both cases for each retry attempt. When retry > 1, there is
        a 1 second delay between attempts.

        Args:
            object_name (str): Name of the object.
            retries (int): Optional parameter, default is 1 attempt.

        Returns:
            Object head dictionary or Boto3 exception.
        iiR-R,RLRMi”R+s	-segmentss	.manifestN(	RNROR%RPR&RRQRRRS(R'RKRTRUtobj_headersRV((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt__get_object_headers{sB	




	

cCs–idd6dd6dd6td6}i|d<i|dd<d	j|ƒ}y2|sottjjƒƒ|d<|S|j|ƒ}t|d
ƒ|d<||d<|jddƒd
ksÄ|ddkrÑt|d<nx?|ddjƒD])\}}d|kræ||d|<qæqæWt	|ddjddƒƒdkrZ|ddjddƒ|d<n|d|d<|d|d<|SWnt
k
r‘‚nXdS(s) Get "operating system like" stat data for the given object.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.

        Returns:
            A stat structure containing object information required by caller.
        it	timestamptsizeRtetagt	directorytheaderstMetadataR+tLastModifiedtContentTypes
text/plainsapplication/x-directoryRYs
x-object-metasx-object-meta-segment-countsx-object-meta-total-sizesx-account-bytes-usedN(RtRHR
Rtnowt_S3Backend__get_object_headersR@tTruetitemsR#R(R'RJR(t	stat_dataRKt
obj_headerRtvalue((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pytstat_object­s2
"


!&
cCs#dj|ƒ}|j||ƒdS(sÎ Create an object that represents a directory in the object store.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.
        R+N(RHt_S3Backend__create_folder(R'RJR(t
new_folder((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pytmkdir_objectÙscCs\dj|ƒd}|jdƒr/|d}n|j|ƒ|jjd|jd|ƒdS(sÎ Remove an object that represents a directory in the object store.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.
        R+iR-R,N(RHR2t_S3Backend__delete_object_treeR%RIR&(R'RJR(RB((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pytrmdir_objectäs

cCs©yŽx7|D]/}|jdƒ|d<|jddƒ|d<q
Wtj|ƒ}|jdƒd}|jjd|jd	|d
|d|dd
ƒWntk
r¤‚nXdS(s? Upload a new object manifest to the object store.

        Args:
            object_name (str): Name of the object. A ".manifest" will be added.
            put_headers (dic): Dictionary of meta data to be added to the object.
            manifest_data (dic): The manifest data which becomes the body of file.
        RptnameR‡ithashR+s	.manifestR-R,RŠRXRŒs
text/plainN(tpopR\tdumpsRZR%t
put_objectR&R(R'RKtput_headerst
manifest_datatsegment_datatmanifestR`((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pytupload_object_manifestõs

c
CsLy |jjd|jd|ƒWn%tk
rG}|jdddkrH|jdƒ}d}xã|D]Ø}|dkr{Pn||d}y |jjd|jd|ƒWqetk
r<}|jdddkre|dtkr|jjd|jd|d	d
dƒq6|jjd|jd|d
dd
dƒqeqeqeXqeWqHnXdS(s< Utility method used to create a object directory structure.

        The entire path is split and each directory level is created as an object
        if it does not exist.

        Args:
            folder_name (str): The entire folder name.
            options (dic): Dictionary of configuration options.
        R-R,RLRMi”R+Rtsupport_empty_dirsx.hiddenRXs
Do Not RemovesTrilioVault directory objectRŒsapplication/x-directoryN(R%RPR&RRQRqRRž(R'tfolder_nameR(RVt
path_partstnew_pathtpart((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt__create_folder
s,
 
 


cCsÑ|d}y¬|jtkrK|jtjj|dd|dƒ|ƒn|jdk	rµt|ddƒC}|j	j
|d|jd|dd|ddid	d
6ƒWdQXnWntk
rÌ‚nXdS(sµ Upload an object to the S3 object store.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.
        iiR+RKtrbR-R,t	ExtraArgssapplication/octet-streamRŒN(
t
path_validRtR•RoRptdirnameRKRtopenR%tupload_fileobjR&R(R'RJR(tfilestdata((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt
upload_object3s
%

N(t__name__t
__module__t__doc__R*RAR˜RGRIR_RbRfRRR:R‚RŽR”R—R™R£R•R²(((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyR,s$	+			
			c	2	,				&(RµRRRR\RoR$R tbotocore.exceptionsRtBASENAMER1RuR
tobjectR(((s2/usr/lib/python2.7/dist-packages/s3fuse/vaults3.pyt<module>s	

Products

About

Resources

Contact Gemfury