Why Gemfury? Push, build, and install  RubyGems npm packages Python packages Maven artifacts PHP packages Go Modules Debian packages RPM packages NuGet packages

trilio-internal / idna python

Repository URL to install this package:

Version: 
2.5  ▾
Details    
idna / lib / python2.7 / site-packages / contego / nova / extension / driver / vaults3.pyc
Size: Mime: 
ó
w=4\c@s©dZddlmZmZddlZddlZddlZddlZddlZddl	Z	ddl
mZdZdZ
dZd„Zd	efd
„ƒYZdS(u Trilio S3 Backend implimentation

    This module contains the back end implimentation of all of all S3 specific
    support.
iÿÿÿÿ(tprint_functiontunicode_literalsN(tClientErrorus3ièu$80bc80ff-0c51-4534-86a2-ec5e719643c2cCs5|jddƒ}|tjdddƒjƒ}|S(uè Utility function used to convert a datetime to an OS timestamp.

    Args:
        modified_time (datetime): Datatime object to convert to a Unix Epoc timestamp.

    Returns:
        The value of modified_time as a timestamp.
    ttzinfoi²iN(treplacetNonetdatetimet
total_seconds(t
modified_timet
naive_timet
delta_seconds((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt_make_timestamps	t	S3BackendcBs³eZdZd„Zd„Zd„Zd„Zd„Zdd„Zd„Z	d	„Z
ded
„Z
d„Zdd„Zd
„Zd„Zd„Zd„Zd„Zd„ZRS(u´ S3 Backend implimentation.

    A Wrapper for the AWS S3 boto3 and botocore API. This class encapsulates all S3
    operations and exposes them as a backend storage instance.
    cCsÄd}|ddkrBtjjd|ddt|dƒƒ}ntjjdt|dƒƒ}tjdd|dd	d
|dd|d
d|dd|ddd|ƒ|_|d|_dS(Nus3_signatureudefaulttsignature_versiontread_timeoutus3_read_timeoutus3tregion_nameu
os_optionsuregion_nametuse_sslus3_ssltaws_access_key_iduusertaws_secret_access_keyukeytendpoint_urluobject_storage_urltconfigubucket(RtbotocoretclienttConfigtinttboto3t_S3Backend__clientt_S3Backend__bucket_name(tselftoptionst
config_object((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt__init__2s,


cCs¶y›x”t|ƒdkr™g}|t }x>|D]6}|jdƒrQ|d}n|ji|d6ƒq/W|jjd|jdi|d6ƒ|t}qWWntk
r±‚nXdS(	uß Utility method that takes a list of objects and puts it in the correct format
            for the S3 delete_objects() API.

        Args:
            object_list (list): A list of objects with the correct S3 path.
        iu/iuKeytBuckettDeleteuObjectsN(tlentMAX_S3_PAGE_SIZEt
startswithtappendRtdelete_objectsRR(Rtobject_listtobject_delete_listt	list_parttobj((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt__delete_object_listSs



cCsÇy¬g}|jjdƒ}|jd|jd|ƒ}xe|D]]}d|kr=|jdt|dƒƒdkr=x&|dD]}|j|dƒq|Wq=q=W|j|ƒWntk
r‚nXdS(	u‘ Utility method used to perform a rmdir operation on an object hierarchy

        Args:
            object_tree (str): Object/Tree path.
        ulist_objectsR tPrefixuContentsuKeyCountiuKeyN(	Rt
get_paginatortpaginateRtgetR"R%t_S3Backend__delete_object_listR(Rtobject_treeR'tlist_paginatort
page_iteratortobjectsR*((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt__delete_object_treels
.
cCs|j|ƒdS(uó Delete a list of objects.

        Args:
            object_list (list): List of objects. Each with a full object path
                                in correct format.
            options (dic): Dictionary of configuration options.
        N(R0(RR'R((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pytdelete_object_listscCsŸy„|s
dSdj|ƒ}|jdƒr8|d}nd|krc|jjd|jd|ƒn |jjd|jd|dƒWntk
rš‚nXdS(u Delete an object from the S3 object store.

        The object and any segments stored in the segment directory will be
        removed.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.
        Nu/iu	-segmentsR tKeyu	.manifest(tjoinR$Rt
delete_objectRR(RtargsRtobject_name((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyR9‹s



icCs¿td|ƒ}y•xŽtd|ƒD]}}y$|jjd|jd|ƒdSWq"tk
rž}|jdddkr˜|d|kr˜tjdƒq"n‚q"Xq"WWntk
rº‚nXdS(	u Utility method used to wait for a S3 object to become available.

        This routine will keep performing a head_object() request every second
        for "retries" number of times. This was added to work around any potential
        eventual consistancey issues when uploading to AWS.

        Args:
            object_name (str): Name of the object.
            retries (int): Optional parameter that defaults to 1.

        Returns:
            Returns when the object is available or ClientError not found exception.
        iiR R7NuResponseMetadatauHTTPStatusCodei”(	tmaxtrangeRthead_objectRRtresponsettimetsleep(RR;tretriestretryterror((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt__wait_for_object©s

cCsñ|jdƒd}y@|jjd|jd|ƒ}tj|dj|dƒƒSWn—tk
rì}|jddd	kr‚‚qíyP|j	|d
ƒ|jjd|jd|ƒ}tj|dj|dƒƒSWqítk
rè‚qíXnXdS(uÓ Download and return the object manifest as a json array.

            Args:
                object_name (str): Name of the object.

            Returns:
                Object manifest as a dictionary.
        u/u	.manifestR R7uBodyu
ContentLengthuResponseMetadatauHTTPStatusCodei”i
N(
tstripRt
get_objectRtjsontloadstreadRR?t_S3Backend__wait_for_object(RR;t
manifest_nametrespRD((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pytget_object_manifestÈs	"	"
cCs]y/dj|ƒ}|jj|j||jƒWn'tk
rE‚ntk
rX‚nXdS(u? Download a file from the S3 object store.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.

        Returns:
            On success, the contents of the object are downloaded to file identidfied
            by options.out_file.
        u/N(R8Rt
download_fileRtout_fileRt	Exception(RR:RR;((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pytdownload_objectás 

c	Cs#tƒ}dj|ƒ}|ddk	rY|d|d}|jdƒrY|d }qYny¯t}|dk	rƒ|tkrƒ|}n|jjdƒ}|sÆ|jd|jdddi|d6ƒ}	n(|jd|jd	|di|d6ƒ}	x|	D]}
d
|
kr†|
j	dt
|
d
ƒƒdkr†|d}xP|
d
D]A}|dk	rlt
|ƒ|krlt|ƒStj
j|d
jdƒƒ\}
}|dkrõtj
j|
ƒ\}}t
|ƒdkr>|dkr>||kr>|j|ƒq>q>ntj
j|ƒ\}}||
kr%|
|ks>|dks>|d
|krGq>n|tks_|
|kr>|j|jddƒdƒq>q>Wnd|
krõxd|
dD]U}|dk	rËt
|ƒ|krËt|ƒS|djtƒs|j|dƒqqWqõqõWt|ƒSWntk
r‚nXdS(u° Return a list of objects based on the provided prefix.

        Used to generate a directory listing based on a prefix path
        constructed from the object parts in args[] and the options.prefix.

        Args:
            args (list): List of prefix keyParts
            options (dic): Dictionary of configuration options
            max_items (int): Maximum number of objects in a list query. Default is
                unlimited but when testing for an empty directory a smaller
                number can be used to limit the query and size of the list that
                is returned.
            strict_path_filter (bool): Defaults to True in order to only return
                objects with the exact same prefix. Individual segment listings
                set this to False in order to get a match on the segment offset.

        Returns:
            On success a list of unique items to be used as a directory listing.
        u/uprefixiÿÿÿÿulist_objectsR t	DelimitertPaginationConfiguPageSizeR,uContentsuKeyCountiuKeyux.hiddenuu	.manifestiuCommonPrefixesuPrefixN(tsetR8RtendswithR#RR-R.RR/R"tlisttostpathtsplittrstriptaddtFalseR$t_SEGMENT_PREFIXR(RR:Rt	max_itemststrict_path_filtert
object_settprefixt	page_sizeR2R3R4tsplit_tokentitemRYR;t	root_pathtsub_dirt_t
object_prefix((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pytlist_objects÷sZ		
.

%*$'

cCs‚ygg}tjj|jƒ\}}|j||dtƒ}x#|D]}|j|d|ƒqCW|SWntk
r}‚nXdS(u& Returns a list of object segments based on a prefix.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.

        Returns:
            A list of segments in the object store that match the supplied prefix.
        R`u/N(RXRYRZRbRjR]R%R(RR:Rtsegment_listt
segments_pathRhR'Re((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt
list_segmentsZs


cCs­td|ƒ}yƒx|td|ƒD]k}y&|jjd|jd|ƒ}|SWq"tk
rŒ}|jdddkr†y*|jjd|jd|dƒ}|SWq†tk
r‚}|jdddkr|d	|krVy*|jjd|jd|d
ƒ}|SWqVtk
rR}|jdddkrL|d|krLtjdƒq"n‚qVXn|d|kr|tjdƒq"q|n‚q†Xn‚q"Xq"WWntk
r¨‚nXdS(u= Utility method that gets an object head from the repository with retry support.

        The default is to try once. Admittedly, this is ugly because the
        provided object name might be a terminal object or a "directory" so
        we need to try both cases for each retry attempt. When retry > 1, there is
        a 1 second delay between attempts.

        Args:
            object_name (str): Name of the object.
            retries (int): Optional parameter, default is 1 attempt.

        Returns:
            Object head dictionary or Boto3 exception.
        iiR R7uResponseMetadatauHTTPStatusCodei”u/u	-segmentsu	.manifestN(	R<R=RR>RRR?R@RA(RR;RBRCtobj_headersRD((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt__get_object_headerspsB	




	

cCsˆidd6dd6dd6td6}i|d<i|dd<d	j|ƒ}y$|sottjjƒƒ|d<|S|j|ƒ}t|d
ƒ|d<||d<|ddks¼|d
dkrÉt|d<nx?|ddjƒD])\}}d|krÞ||d|<qÞqÞW|ddjddƒdkrL|ddjddƒ|d<n|d
|d<|d|d<|SWnt	k
rƒ‚nXdS(u) Get "operating system like" stat data for the given object.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.

        Returns:
            A stat structure containing object information required by caller.
        iu	timestampusizeuuetagu	directoryuheadersuMetadatau/uLastModifieduContentTypeuapplication/x-directoryu
ContentLengthu
x-object-metaux-object-meta-segment-countux-object-meta-total-sizeux-account-bytes-usedN(
R]R8RRtnowt_S3Backend__get_object_headerstTruet	iteritemsR/R(RR:Rt	stat_dataR;t
obj_headertkeytvalue((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pytstat_object¢s2
"


! 
cCs#dj|ƒ}|j||ƒdS(uÎ Create an object that represents a directory in the object store.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.
        u/N(R8t_S3Backend__create_folder(RR:Rt
new_folder((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pytmkdir_objectÎscCs\dj|ƒd}|jdƒr/|d}n|j|ƒ|jjd|jd|ƒdS(uÎ Remove an object that represents a directory in the object store.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.
        u/iR R7N(R8R$t_S3Backend__delete_object_treeRR9R(RR:RR1((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pytrmdir_objectÙs

c	Cs£yˆx7|D]/}|jdƒ|d<|jddƒ|d<q
Wtj|ƒ}|jdƒd}|jjd|jd	|d
|d|ƒWntk
rž‚nXdS(
u? Upload a new object manifest to the object store.

        Args:
            object_name (str): Name of the object. A ".manifest" will be added.
            put_headers (dic): Dictionary of meta data to be added to the object.
            manifest_data (dic): The manifest data which becomes the body of file.
        upathunameuetagiuhashu/u	.manifestR R7tMetadatatBodyN(tpopRHtdumpsRFRt
put_objectRR(RR;tput_headerst
manifest_datatsegment_datatmanifestRL((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pytupload_object_manifestês

c
CsLy |jjd|jd|ƒWn%tk
rG}|jdddkrH|jdƒ}d}xã|D]Ø}|dkr{Pn||d}y |jjd|jd|ƒWqetk
r<}|jdddkre|dtkr|jjd|jd|d	d
dƒq6|jjd|jd|d
ddd
ƒqeqeqeXqeWqHnXdS(u< Utility method used to create a object directory structure.

        The entire path is split and each directory level is created as an object
        if it does not exist.

        Args:
            folder_name (str): The entire folder name.
            options (dic): Dictionary of configuration options.
        R R7uResponseMetadatauHTTPStatusCodei”u/uusupport_empty_dirux.hiddenRu
Do Not RemovetContentTypeuapplication/x-directoryN(RR>RRR?RZRrR‚(Rtfolder_nameRRDt
path_partstnew_pathtpart((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt__create_folders,
 
 


cCsÑ|d}y¬|jtkrK|jtjj|dd|dƒ|ƒn|jdk	rµt|ddƒC}|j	j
|d|jd|dd|ddid	d
6ƒWdQXnWntk
rÌ‚nXdS(uµ Upload an object to the S3 object store.

        Args:
            args (list): List of object name parts.
            options (dic): Dictionary of configuration options.
        iiu/uobject_nameurbR R7t	ExtraArgsuapplication/octet-streamuContentTypeN(
t
path_validR]RyRXRYtdirnameR;RtopenRtupload_fileobjRR(RR:Rtfilestdata((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt
upload_object's
%

N(t__name__t
__module__t__doc__RR0R|R6R9RKRNRRRRrRjRmRqRxR{R}R‡RyR•(((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyR,s$	!			
			c	2	,				&(R˜t
__future__RRRR@RHRXRRtbotocore.exceptionsRtBASENAMER#R^RtobjectR(((sZ/home/tvault/.virtenv/lib/python2.7/site-packages/contego/nova/extension/driver/vaults3.pyt<module>s	

Products

About

Resources

Contact Gemfury