Gemfury

triliodata-4-1-beta / python3-s3fuse-plugin rpm

Repository URL to install this package:
Details
python3-s3fuse-plugin / usr / lib / python3.6 / site-packages / s3fuse / __pycache__ / s3vaultfuse.cpython-36.pyc
3

Ù2Ù_X:ãK@södZddlZddlZddlZddlZddlZddlZddlZddlZddl	Z	ddl	m
Z
ddl	mZddlm
Z
yddlmZmZWn$ek
r¨ddlmZmZYnXddlmZddlmZydd	lmZmZmZWn dd	lmZmZmZYnXdd
lTddlZyddlmZWn"ek
r>ddlmZYnXddl m!Z"dd
l#m$Z$ddl#m%Z%ddl&m'Z'dZ(dZ)dZ*dZ+ej,dddddej,dddddgZ-ej.ddddej.d dd!dej.d"dd#dej.d$dd%d&d'ej.d(e(d)d*d'ej.d+dd,d-d.ej.d/d0d1d2ej,d3dd4d5ej.d6d7d8d5g	Z/ej,d9d:d;d5ej,d<dd=d5ej,d>dd?d5ej,d@ddAd5ej0dBdCdDd5gZ1ej0dEdFdGd5ej0dHdIdJd5ej.dKdLdMd5ej.dNdOdPd5ej.dQdRdSd5ej.dTdUdVd5ej.dWdXdYd5ej.dZd[d\d5ej2d]d^d_d`dadbdcgddd5ej,deddfd5ej,dgddhd5ej.didjdkd5ej.dldjdmd5ej0dnddod5gZ3ej.dpdqdrd5ej.dsdtduej.dvdwdtd5ej.dxdydzd5ej.d{d|d}d5ej.d~ddd5ej.dddd5ej.dddd5ej.dddd5ej.dddd5ej.dddd5ej.dddd5ej.dddd5ej.dddd5ej.dddd5ej.dddd5ej0de*dd5ej0d e+d¡d5ej0d¢d£d¤d5ej.d¥d¦d§d5ej0d¨dd«d5ej0d¬ddd5ej.d®d¯dd°d'ej.d±d²d³d5ej.d´dµd¶d5ej.d·dµd¸d5ej.d¹dµdºd5ej.d»dµd¼d5ej.d½dµd¾d5ej.d¿dÀdÁd5ej.dÂdµdÃd5ej.dÄddÅd5ej.dÆdÇdÈd5ej.dÉdÊdËd5ej.dÌdÀdÍd5ej.dÎdÀdÏd5ej.dÐdÑdÒd5ej.dÓdÔdÕd5ej.dÖd×dØd5g'Z4ej5Z5e5j6e4e5j7e/e5j7e1e5j7e3e5j7e-ye8ej9dÙZ:e5e:gWne5ej;dCdYnXe"j<ej5e5j=j>e"j?e5j=j>Z@edÚdۍZAyre5jBj>dkr6e@jCjDe"jEnNe5jBj>dÜkrVe@jCjDe"jFn.e5jBj>dÝkrve@jCjDe"jGne@jCjDe"jHWn&eIk
r¬e@jCjDe"j"jHYnXddCgdÞddddddddddddddߜ
d:ddµdµej9jJdàej9jJdáej9jJdâdµdµdd:dddµdµdµddddgddd:ddd:dãdãdãdãdddä&ZKdådæZLdãZMdÚZNddlOZOeOjPZQdçZRiaSdèZTdédêZUdëdìZVdídîZWGdïdðdðZXGdñdòdòeYZZGdódôdôeZZ[GdõdödöeYZ\Gd÷ddeZ]ddùdúZ^dûdüZ_dýdþZ`eadÿkròe`dS(ad Trilio Fuse plugin implimentation

    This module is based on the vaultfuse.py module and will eventually
    become the new vaultfuse.py module once integration and refactoring
    of the existing vaultswift.py is complete.

    Currently this module is temporary for the 2.6 release and will be merged
    into a new vaultfuse.py for the next release.
éN)ÚThread)ÚBoundedSemaphore)Úmkstemp)ÚQueueÚEmpty)Úgetpwnam)ÚLRUCache)ÚFUSEÚFuseOSErrorÚ
Operations)Ú*)Úcfg)Úlog)Úutils)Úvaults3)Úfsz%Y-%m-%d %H:%M:%Sz$80bc80ff-0c51-4534-86a2-ec5e719643c2édéÚdebugÚdFzUPrint debugging output (set logging level to DEBUG instead of default WARNING level).)ZshortÚdefaultÚhelpÚverboseÚvzWPrint more verbose output (set logging level to INFO instead of default WARNING level).z
log-configÚPATHzßIf this option is specified, the logging configuration file specified is used and overrides any other logging options specified. Please see the Python logging module documentation for details on logging configuration files.)Úmetavarrzlog-config-appendz(Optional) Log Appendzwatch-log-filez(Optional) Watch logz
log-formatZFORMATzÞA logging.Formatter log message format string which may use any of the available logging.LogRecord attributes. This option is deprecated.  Please use logging_context_format_string and logging_default_format_string instead.)rrrzlog-date-formatZDATE_FORMATzCFormat string for %%(asctime)s in log records. Default: %(default)szlog-fileZlogfilezZ(Optional) Name of log file to output to. If no default is set, logging will go to stdout.)rÚdeprecated_namerzlog-dirZlogdirz@(Optional) The base directory used for relative --log-file paths)rrz
use-syslogzUse syslog for logging.)rrzsyslog-log-facilityZLOG_USERz$syslog facility to receive log linesZ
use_stderrTzLog output to standard errorZuse_journalz%Enable Systemd native journal supportZuse_eventlogz Log output to Windows Event Log.Zuse_jsonz4Enables JSON formatting in the logs when set to TrueZrate_limit_burstézBurst limitZlog_rotate_intervaléz0Interval, number of hours, of log rate limiting.Zmax_logfile_counté<zNumber of log files to retain.Zlog_rotate_interval_typeZHourszUnits of log rotation intervalZlog_rotation_typeZintervalztype of log retentionZlogging_context_format_stringzw%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(request_id)s %(user)s %(tenant)s] %(instance)s%(message)sz2format string to use for log messages with contextZlogging_default_format_stringzV%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)sz5format string to use for log messages without contextZlogging_debug_format_suffixz$%(funcName)s %(pathname)s:%(lineno)dz0data to append to log format when level is DEBUGZlogging_exception_prefixz?%(asctime)s.%(msecs)03d %(process)d TRACE %(name)s %(instance)sz5prefix each line of exception output with this formatZdefault_log_levelszamqplib=WARNzsqlalchemy=WARNz	boto=WARNz	suds=INFOz
keystone=INFOzeventlet.wsgi.server=WARNzlist of logger=LEVEL pairsZpublish_errorszpublish error eventsZfatal_deprecationszmake deprecations fatalZinstance_formatz[instance: %(uuid)s] zBIf an instance is passed with the log message, format it like thisZinstance_uuid_formatzGIf an instance UUID is passed with the log message, format it like thisZrate_limit_intervalz2Interval, number of seconds, of log rate limiting.Úvault_storage_typeÚs3z'Storage type: nfs, swift-i, swift-s, s3Úvault_data_directoryz'Location where snapshots will be stored)rÚvault_data_directory_oldz/var/triliovaultÚtmpfs_mount_pathÚtmpfszLLocation with respect to CONF.vault_data_directory_oldwhere tmpfs is mountedZvault_storage_nfs_exportZlocalz
NFS ExportZvault_storage_nfs_optionsZnolockzNFS OptionsZvault_swift_auth_versionZKEYSTONE_V2z KEYSTONE_V2 KEYSTONE_V3 TEMPAUTHZvault_swift_auth_urlzhttp://localhost:5000/v2.0zKeystone Authorization URLZvault_swift_tenantZadminzSwift tenantZvault_swift_usernamezSwift usernameZvault_swift_passwordZpasswordzSwift passwordZvault_swift_region_nameZ	RegionOnezSwift Region NameZvault_swift_domain_idrzSwift domain idZvault_swift_domain_nameZDefaultzSwift domain nameZvault_swift_container_prefixÚTrilioVaultzSwift Container PrefixZvault_swift_segment_sizeZ33554432zDefault segment size 34MBÚqueue_depthz-The number of writes to be queued (writeback)Úworker_pool_sizez3Number of worker threads to process the write queueZvault_retry_countéz(The number of times we retry on failuresZvault_swift_url_templatez,http://localhost:8080/v1/AUTH_%(project_id)szThe URL of the Swift endpointÚvault_segment_sizeé izvault object segmentation sizeÚvault_cache_sizez6Number of segments of an object that need to be cachedZ
rootwrap_confz/etc/nova/rootwrap.confzrootwrap config fileÚvault_s3_auth_versionÚDEFAULTzS3 Authentication typeÚvault_s3_access_key_idÚz	S3 Key IDÚvault_s3_secret_access_keyzS3 Secret Access KeyÚvault_s3_region_namez	S3 RegionÚvault_s3_bucketz	S3 BucketÚvault_s3_endpoint_urlzS3 Endpoint URLÚvault_s3_sslÚTruezUse SSLÚvault_s3_ssl_certzUse SSL certificate bundleÚvault_s3_signature_versionzS3 signature version to useÚvault_s3_support_empty_dirÚFalsez,S3 backend needs empty directory work aroundÚvault_s3_read_timeoutZ120z3Time in seconds to wait for a response to API callsZvault_enable_threadpoolzEnable backend thread poolÚvault_threaded_filesystemz"Allow multiple file system threadsZmax_uploads_pendingZ20zNumber of file uploads.Zvault_cache_usernameZnovazSystem username.Úvault_logging_levelÚerrorz0Logging level filter (debug, info, warn, error).ZFLAGé)ÚvalueÚinfoÚwarnz1.0)
Zproject_nameÚregion_nameZuser_domain_nameZ
endpoint_typeÚobject_storage_urlZproject_domain_idÚuser_idZuser_domain_idZ	tenant_idZservice_typeZ
project_idZ
auth_tokenZproject_domain_nameZ	OS_CACERTZOS_CERTZOS_KEYé
)&Zsync_torÚheaderÚauth_versionÚ
os_optionsZssl_compressionZos_storage_urlZos_usernameZos_passwordZ	os_cacertZos_certZos_keyZos_tenant_nameZos_auth_urlZ
os_auth_tokenZinsecureZsnetZsync_keyZauthÚuserÚkeyZread_aclrAZretriesZ	write_aclÚmetarZuse_sloZchecksumZchangedÚleave_segmentsZskip_identicalZsegment_threadsZobject_dd_threadsZobject_uu_threadsZcontainer_threadsZyes_allÚobject_namecCstjjdksttjdks tdtd<tjtd<tjtd<tjtd<tj	td<tj
td	<tjjd
krrdtd<nd
td<tjjd
krdtd<nd
td<tj
r®tj
tdd<tjrÂtjtdd<tjjdkrÚd
td<n2tjjtjrôtjtd<ntjdtjdtd<dS)Nr!r.z1.0rHrJrKZbucketZs3_signatureZs3_read_timeoutÚtrueTZsupport_empty_dirFZs3_sslrIrDrCZfalseZs3_ssl_certzcInvalid S3 SSL cert file path: %sUsing Default path, set the value to False for insecure connectionr0)ÚCONFr ÚlowerÚAssertionErrorr-Úoptionsr/r1r3r8r;r9r5r4r2r7ÚosÚpathÚexistsÚLOGrB©rXrXú!/usr/lib/python3.6/s3vaultfuse.pyÚ
option_config¢s4








rZz
%016x.%08xiècstjfdd}|S)Ncs~ytjjtjWn$tk
r6tjjtjjYnX||}ytjjtjWn$tk
rxtjjtjjYnX|S)N)ÚloggingÚdisableÚERRORÚ
BaseExceptionZNOTSET)ÚargsÚkwargsÚresult)ÚfuncrXrYÚwrapperÒs
z disable_logging.<locals>.wrapper)Ú	functoolsÚwraps)rbrcrX)rbrYÚdisable_loggingÑsrfcCsVtjj|\}}d}x8|dkrL|dkr6tjj||}n|}tjj|\}}qW||fS)Nr0ú/)r0rg)rTrUÚsplitÚjoin)rUÚheadÚtailÚprefixrXrXrYÚsplit_head_tailás
rmcCstjj|\}}|S)N)rTrUrh)rUrjrkrXrXrYÚget_headîsrnc@s&eZdZd	ddZddZddZdS)
Ú	tmpfsfileFcCs||_||_dS)N)ÚremoveÚpersist_exit)ÚselfrprqrXrXrYÚ__init__ôsztmpfsfile.__init__cCsLtjjtjtj}t|d\}|_tj||j	r>tj	|jt
j|jS)N)Údir)rTrUrirPr#r$rÚ	open_fileÚcloserpÚ
tmpfs_semaÚacquire)rrÚtmpfs_mountpathÚfhrXrXrYÚ	__enter__ýs
ztmpfsfile.__enter__cGs|jstjtj|jdS)N)rqrwÚreleaserTrpru)rrr_rXrXrYÚ__exit__sztmpfsfile.__exit__N)FF)Ú__name__Ú
__module__Ú__qualname__rsr{r}rXrXrXrYroós
	roc@seZdZddZddZddZddZd	d
ZddZd
dZ	ddZ
dddZdddZddZ
ddZddZddZdS) ÚObjectRepositorycKs
||_dS)N)Úroot)rrrr`rXrXrYrsszObjectRepository.__init__cCs.|jdr|dd}tjjt|}|j|S)Nrgr)Ú
startswithrTrUriÚ_SEGMENT_PREFIXÚ
_full_path)rrÚpartialrUrXrXrYÚ_full_segment_paths
z#ObjectRepository._full_segment_pathcCs*|jdr|dd}tjj|j|}|S)Nrgr)rrTrUrir)rrrrUrXrXrYrs
zObjectRepository._full_pathcCsdS)NrX)rrrNÚflagsrXrXrYÚobject_open szObjectRepository.object_opencCsdS)NrX)rrrNÚoffÚbufrzrXrXrYÚ
object_upload#szObjectRepository.object_uploadcCsdS)NrX)rrrNÚoffsetrzrXrXrYÚobject_download&sz ObjectRepository.object_downloadcCsdS)NrX)rrrNrXrXrYÚ
object_delete)szObjectRepository.object_deletecCsdS)NrX)rrrNrzrXrXrYÚobject_close,szObjectRepository.object_closeNcCsdS)NrX)rrrNÚlengthrzrXrXrYÚobject_truncate/sz ObjectRepository.object_truncatecCsdS)NrX)rrrNrzrXrXrYÚobject_getattr2szObjectRepository.object_getattrcCsdS)NrX)rrrUrzrXrXrYÚobject_readdir5szObjectRepository.object_readdircCsdS)NrX)rrrUÚmoderXrXrYÚ
object_access8szObjectRepository.object_accesscCsdS)NrX)rrrUrXrXrYÚ
object_unlink;szObjectRepository.object_unlinkcCsdS)NrX)rrrUrXrXrYÚ
object_statfs>szObjectRepository.object_statfs)N)N)r~rrrsrrrrrrrrrrrrrrXrXrXrYrs

rcs\eZdZfddZGdddeZGdddZddZd	d
ZddZ	d
dZ
ddZifddZddZ
ddZddZddZddZddZdd Zed!d"Zd#d$ZdMd&d'Zd(d)Zd*d+Zd,d-Zd.d/ZdNd0d1Zd2d3Zd4d5ZdOd7d8Zd9d:Z dPd;d<Z!d=d>Z"d?d@Z#dAdBZ$dCdDZ%dEdFZ&dGdHZ'dQdIdJZ(dKdLZ)Z*S)RÚBackendRepositorycsTtt|j|f|ttj|_ttj|_i|_	t
jt|_
i|_i|_i|_dS)N)ÚsuperrrsrÚ	FUSE_USERÚpw_uidrEÚpw_gidÚgroup_idÚmanifestrÚ	S3BackendrSÚ_BackendRepository__backendÚ!_BackendRepository__manifest_lockÚ_BackendRepository__worker_poolÚ_BackendRepository__pending_ios)rrrr`)Ú	__class__rXrYrsCszBackendRepository.__init__c@s0eZdZdZddZddZddZdd	Zd
S)zBackendRepository.BackendWorkerzU A thread used to asyncronously perform backend jobs placed in the job_queue
        cCs8tj|tj||_||_tj|_d|_	|j
dS)NT)rrsrr Ú_BackendWorker__backendÚ_BackendWorker__job_queueÚ	threadingZEventÚ_stop_eventZdaemonÚstart)rrÚ	job_queuerSrXrXrYrscs

z(BackendRepository.BackendWorker.__init__cCs|jjdS)N)r©Úset)rrrXrXrYÚstopmsz$BackendRepository.BackendWorker.stopcCs
|jjS)N)r©Zis_set)rrrXrXrYÚstoppedpsz'BackendRepository.BackendWorker.stoppedcCs¤tjdtjjx|jsy<|jjdd\}}}z||jf||Wd|jj	XWqt
k
rpwYqtk
r}ztj|WYdd}~XqXqWdS)zO Start up the worker and block until there is an item in the queue
            zStarting worker thread[%x].TrN)
rWrAr¨Úcurrent_threadÚidentr®r§Úgetr¦Z	task_donerÚ	ExceptionÚ	exception)rrrbr_ÚkargsÚerXrXrYÚrunss
z#BackendRepository.BackendWorker.runN)r~rrÚ__doc__rsrr®r¶rXrXrXrYÚ
BackendWorker_s

r¸c@s8eZdZdZddZddZddZdd	Zd
dZdS)
z#BackendRepository.BackendWorkerPoolzU Pool of backend worker threads that consume selected tasks from a job queue
        cCs>ttj|_g|_x&t|D]}|jjtj|j|qWdS)N)	rrPr'r«ÚworkersÚrangeÚappendrr¸)rrZnum_threadsrSÚ_rXrXrYrss
z,BackendRepository.BackendWorkerPool.__init__cCs4x|jD]}|jqWx|jD]}|jq WdS)N)r¹rri)rrÚtrXrXrYÚ__del__sz+BackendRepository.BackendWorkerPool.__del__cOs|jj|||fdS)z+ Add a job to the worker queue
            N)r«Zput)rrrbr_r´rXrXrYÚadd_jobsz+BackendRepository.BackendWorkerPool.add_jobcCsx|D]}|j||qWdS)z4 Add a list of jobs to the worker queue
            N)r¿)rrrbZ	args_listr_rXrXrYÚmap s
z'BackendRepository.BackendWorkerPool.mapcCs|jjdS)z> Wait for completion of all the jobs in the queue
            N)r«ri)rrrXrXrYÚwait_completion¦sz3BackendRepository.BackendWorkerPool.wait_completionN)	r~rrr·rsr¾r¿rÀrÁrXrXrXrYÚBackendWorkerPoolsrÂcCsVtjj|\}}d}x8|dkrL|dkr6tjj||}n|}tjj|\}}qW||fS)Nr0rg)r0rg)rTrUrhri)rrrUrjrkrlrXrXrYrm«s
z!BackendRepository.split_head_tailcCstjj|\}}|S)N)rTrUrh)rrrUrjrkrXrXrYÚ	_get_head·szBackendRepository._get_headcCs.|jdr|dd}tjjt|}|j|S)Nrgr)rrTrUrirÚ
_get_cache)rrrrUrXrXrYÚ_get_segment_cache»s
z$BackendRepository._get_segment_cachecCs*|jdr|dd}tjj|j|}|S)Nrgr)rrTrUrir)rrrrUrXrXrYrÄÁs
zBackendRepository._get_cachecCs>|j|\}}i}i}tj}||d<t|}|jj|}|S)NrN)rmrSÚcopyÚbunchifyr¡Zget_object_manifest)rrrNÚ	containerrlÚput_headersÚmrÚ_optsrrXrXrYÚ_read_object_manifestÇsz'BackendRepository._read_object_manifestcCsl|j|\}}i}i}d|d<tj}||d<x |jD]\}	}
|
|d|	<q8Wt|}|jj|||dS)NrOzx-static-large-objectrNzx-object-meta-)rmrSrÆÚitemsrÇr¡Zupload_object_manifest)rrrNÚobject_manifestÚmetadatarÈrlrÉrÊrËrKr@rXrXrYÚ_write_object_manifestÛs
z(BackendRepository._write_object_manifestcCsÈ|j|\}}tj}|dkr$g}n|g}d|d<d|d<d|d<d|d<d|d<t|}i}|dkrz||d<|j|nd}|jj||}i}x2|djD]"\}	}
d	|	kr|
||	jd
d<qW|S)Nr0Ú	delimiterFÚhumanÚtotalsÚlongrlÚheadersz
x-object-metazx-object-meta-r)	rmrSrÆrÇr»r¡Ústat_objectrÍrh)rrrNrÈrlrËr_rÚstrÏrKr@rXrXrYÚ_get_object_metadataôs,z&BackendRepository._get_object_metadatac<CsÄ|j|\}}|j|}|jj|tj|j|<|j|Ri|j|<|tjkpp|t	ddt	ddt	ddfk|j|d<|tjksà|t	ddt	ddt	ddfksà|t	ddt	ddfksà|t	ddt	d	dt	d
dfkr*y|j
|}WnFtk
r4}z(tj
d|tjd|j
|}WYdd}~XnXxJ|D]B}|d
jddjdd}	t	|	d}
d|d<||j||
<q<W|j|}|d|j|d<y"|j|j|d}tj|Wntk
rÖYnXtjjtjj|stjtjj|t|d}
|
jtj|WdQRXn`tjjtjj|sRtjtjj||t	ddt	ddt	ddfks|tjkrØy|j|Wntk
r¤YnXyt|d}
WdQRXWntk
rÖYnXy|j|d}tj|Wntk
r
YnXtj||}z$|jtj t!|j"|<|j#||Wd|j"j$|dtj%|Xyt|d}
WdQRXWntk
rYnXWdQRXtj||}|jtj t!|j"|<t&|j'|<|S)NZ8000éZ8800Zc000ÚreadonlyZ8401Z8402Z8002Z8802Zc002z+Manifest for %s not found. Trying in 5 secsrÚnamez
-segments/rÚ.rFÚmodifiedzsegments-dirÚwZ8001Z8801Z0001z	-segments)(rmrr¢r±r¨ÚRLockrrTÚO_RDONLYÚintrÌr²rWrAÚtimeÚsleeprhrØÚmakedirsr^rUrVÚdirnameÚopenÚwriteÚjsonÚdumpsÚO_RDWRrrrÂrPr(rSr£Ú
_object_closeÚpoprvr¬r¤)rrrNrrÈrlÚ	full_pathrÚexÚsegZoffstrrrÏZsegment_dirÚfrzrXrXrYrs


*#
 "


 
zBackendRepository.object_opencCsxyFg}x |D]}|jtjj||qWtj}t|}|j||Wn,tk
rr}zt	j
|WYdd}~XnXdS)a4 Utility method to be dispatched to a worker thread in order to clean up old segments.

        Args:
            backend (class instance): Instance of the backend plugin.
            container (str): Container name, a.k.a root directory.
            segment_list (list): List of segments to remove.
        N)r»rTrUrirSrÆrÇÚdelete_object_listr²rWr³)rrÚbackendrÈÚsegment_listZdelete_listÚobjrËrîrXrXrYZ__purge_old_segments_task¤s
z+BackendRepository.__purge_old_segments_taskcCs|j|j|j|ry:|j|jddt|d}|j|\}}g}i}d}|j|dsVd}	d}
d}x |	|j|krP|j||	dr*y|j|j||	d|}Wn0tj	d|j|j||	d|}YnXt
|}
t|
jt
j}|j|j||	d|
j||
jd	nP|j||	d
}|j|j||	d|j||	d||j||	dd	|	t
j7}	|d
7}|
|7}
qxW|j|||t|t|
ddd}	xdrT|rT|j|||	}t|dkrèPt|}|j|j||	d\}}tt|t|g}t|dkrF|j|j|j|||	t
j7}	q¾WdStk
r}ztj|WYdd}~XnXWdQRXdS)zé Handle object_close by updating the manifest.

         Waits for all of the background jobs to complete prior to updating
         manifest using the main thread. Also queues up tasks to clean up
         any old segments.
        zsegments-dirrgz	-segmentsrrÚrÝrÛr)rUÚetagÚ
size_bytesÚcontent_typeröÚhashr÷r)zsegments-dirz
segment-countz
total-size)rÏTN)r£rÁr¢rr±rrmrrârãrÇÚminÚst_sizerPr*r»rõr÷rÐÚstrróÚlenÚsortedÚlistr¬r¿Ú+_BackendRepository__purge_old_segments_taskr²rWr³)rrrNrzÚsegments_dirrÈrlrÎZ
segments_listZsegmentsrZ
total_sizeÚsegment_sizer×ÚstatZobjectsÚcÚpZ
purge_listrîrXrXrYrë·s




zBackendRepository._object_closecCsv|j|||jj||jj|ytj|Wntk
rFYnXytj|j|Wntk
rpYnXdS)N)	rër£rìr¤rTrvr^rpr)rrrNrzrXrXrYr1szBackendRepository.object_closec#Cs|j|y*||j|krJ|j||ddkrJ|j||d}n8|j||}|j|jddt|d}tjj||}tj	}d|j|k|d<|j
|\}	}
tdd	2}t|d
}|j
|WdQRXt||d<WdQRX|
jd|d<t|}|	|g}
|j|j||j|j|j|
|||||WnRtk
r}z4tj|||j|kslt|j|j|WYdd}~XnXWdQRXdS)
NrÝTrÛzsegments-dirrgz	-segmentsÚ0Z
path_valid)rqÚwbrrN)r¢rÚ,_BackendRepository__next_segname_from_offsetr±rrTrUrirSrÆrmrorærçrüÚrstriprÇr¤Úaddr£r¿Ú&_BackendRepository__object_upload_taskr²rWr³rRrp)rrrNrrrzÚseg_fullnameZsegnamerrËrÈrôÚtempfsrðÚargs1rîrXrXrYr?s8

zBackendRepository.object_uploadc	*CsXtjd||tjjf|j|||j|ks8tWdQRXz|y|j||Wnft	k
rº}zJtj
||j|(||j|kst|j|j|WdQRXWYdd}~XnXWdtj
tj|dX|j|l||j|kri|j||<||j||d<d|j||d<||j|ks:t|j|j|WdQRXdS)zP Method run by a worker in the thread pool to upload an object segment.
        z6Object [%s] segment [%s] upload running in thread[%x].NrrÛTrÝ)rWrAr¨r¯r°r¢r¤rRZ
upload_objectr²r³rprwr|rTr)	rrròr
rËrNrrrzrîrXrXrYZ__object_upload_taskgs,
z&BackendRepository.__object_upload_taskc
CsHxvtjd|j|X||j|kr(w||j|krVtd|tt|tdf|j||d}PWdQRXqW|j|j	d\}}t
dd¨}tj}d|d<d|d	<|j	d|d
<t
|}||g}	y:|jj|	|t|j	dd}
|
j}WdQRXt|Stk
r8}ztj|WYdd}~XnXWdQRXdS)Ng¹?zobject %s not foundrrÛrgT)rprlZ
out_directoryZout_fileÚrb)rârãr¢r¤rr²ÚSEGMENT_FORMATrármrrorSrÆrÇr¡Zdownload_objectræÚreadÚ	bytearrayrWr³)
rrrNrrzrrÈrôrrËr
rðrrîrXrXrYrs6

z!BackendRepository.object_downloadcCs|j||j|\}}tj}t|}|g}|dkrJ|dkrJ|j|y|jj||Wn,tk
r}zt	j
|WYdd}~XnXWdQRXdS)Nr0rg)r¢rmrSrÆrÇr»r¡Ú
delete_objectr²rWr³)rrrNrÈrôrËr
rîrXrXrYrªs

zBackendRepository.object_deleteNcCs0|rdS|j|tj}|jj|tj|dS)N)rrTÚO_WRONLYr£rìrv)rrrNrrzrXrXrYrºs
z!BackendRepository.object_truncatecCsptj}d|d<d|d<d|d<d|d<tjj|dt||d<g}|dkrTg}n|g}t|}|jj||S)	NrÑFrÒrÓrÔz%016xrlr0)	rSrÆrTrUrirárÇr¡Z
list_segments)rrrÈrrrËr_rXrXrYróÁszBackendRepository.segment_listc	CsÀ|j|\}}|j|jdd}|rT|jdtr@|j|\}}|j|\}}|}n|d}|j|||}t|dkrtt|tdfStt|tt	|dj
|dj
ddddfS)	Nzsegments-dirrgz	-segmentsrrrÜrÙéÿÿÿÿ)rmrr±rrrórürrárýrh)	rrrUrrÈrlrrrÚfilesrXrXrYZ__next_segname_from_offsetÓs$z,BackendRepository.__next_segname_from_offsetc
Csò|j||j|\}}|jj|drV||j|krV|j||d}|jddS|j|jdd}|r|j|\}}|}n|d}|j|||}	WdQRXt|	dkrÀtt|tdfStt|tt	|	d	j|djdddfS)
NrÛz
-segments/rzsegments-dirz	-segmentsrrÜrÙr)
r¢rmrr±rhrórürrárý)
rrrUrrÈrlrïrrrrrXrXrYÚcurr_segname_from_offsetès z*BackendRepository.curr_segname_from_offsetcstjj|r~y@tj|tfdddD}d|d<t|d	d|d
<|Stk
rx}ztj|WYdd}~XnXdSdS)aÆ Utility to check the tmpfs cache and return stat() from that.

         This method looks to see if the object (directory) exists in the
         local cache and will perform a stat() locally instead of calling
         all the way to the backend.

        Args:
            object_cache_path (str): Cache path to the object we need stat infor on.

        Returns:
            Dictionary of stat() information or None if object does not exist.
        c3s|]}|t|fVqdS)N)Úgetattr)Ú.0rK)r×rXrYú	<genexpr>sz0BackendRepository._stat_cache.<locals>.<genexpr>Úst_atimeÚst_ctimeÚst_gidÚst_modeÚst_mtimeÚst_nlinkrúÚst_uidiÚ
st_blksizeÚ	st_blocksN)rrrrrrrúr )	rTrUÚisdirÚlstatÚdictrár²rWr³)rrZobject_cache_pathÚ	stat_inforµrX)r×rYÚ_stat_caches(



zBackendRepository._stat_cachec
$sR|j|}|j|}|dk	r |S|j|\}}tj}|dkrDg}n|g}d|d<d|d<d|d<d|d<d|d<t|}i}|dkr||d<|j|nd}yN|jj|||j	|d<|j
|d	<d
jdd|d<d
jd
d|d<td|d<td|d<td|d<d|d<d|d<|dk	rHd|krHd|d<td|d<|ddkrp|dks®|ddkr|dks®|ddkr |dks®ddkrêd|d<d|d<d |d<|j
|}	tjj|	sêtj|	d!WnBtk
r0}
z"|dkr|}|j
tjj||}|j|}ytj|Wntk
rÜ|}t|dkrj|jyVtjj|dd|d<|jj||tdkr´tjtjj|d!n
tj|Wntk
rÖYnXYnX|d"ksü|dd)jd#rdStj|tfd$d%d*D}WYdd}
~
XnXd&|d'<t|dd&|d(<|S)+Nr0rÑFrÒrÓrÔrlrr rÕZETagrõZContentTyper÷Z	timestamprrrrriírZauthorized_keyiÚsizerúrZ	directoryTéiiýAiéZ4913ú~c3s|]}|t|fVqdS)N)r)rrK)r×rXrYrysz3BackendRepository.object_getattr.<locals>.<genexpr>ir!r"r)rrrrrrrúr )rr'rmrSrÆrÇr»r¡rÖrrEr±rárÄrTrUrVÚmkdirr²rirÃr$r^rürìÚlist_objectsrårWrAÚendswithr%)
rrrNrzrírrÈrlrËr_Zcontainer_pathrîÚmkdirsr
rX)r×rYr*s








z BackendRepository.object_getattrccsLg}|j|\}}tj}d|d<d|d<d|d<d|d<d|d<g}|dkrTg}n"|g}|dkrv|dk	rv|d|d<g}t|}||jj||7}x|D]}	|j|	\}
}|
dks¼|dkrøtjj||j	|	}ytj
|d	d
Wntk
röYnX|
dk	r|
dkrd|
krd|
kr|
|kr|j|
qWxt
|D]}
|
Vq8WdS)
NrÑFrÒrÓrÔrlr0rgié)rz	-segmentsZ	_segments)rmrSrÆrÇr¡r,rTrUrirÃrär^r»rþ)rrrNrzZlistingrÈrlrËr_ÚdirentsZlstZ	componentÚrestr.ÚrrXrXrYrsB
z BackendRepository.object_readdircCsdS)NrX)rrrNrrXrXrYr¼szBackendRepository.object_accessFcCs:|td<tj}t|}|sÆy||j|}|j|}g}x|D]}|j|dq<W|jj|||jd}|dk	r|g}	|jj	|	|t
j|j|Wn,t
k
rÄ}
ztj|
WYdd}
~
XnX|j|\}}|g}
|dkrô|dkrô|
j|y|jj|
|Wn.t
k
r4}
ztj|
WYdd}
~
XnXdS)NrMrÛzsegments-dirr0rg)rSrÆrÇrÌrØr»r¡rñr±Úrmdir_objectrTÚrmdirrÄr²rWr³rmr)rrrNrMrËZobj_manifestZobj_metadataróÚmanrr_rîrÈrôr
rXrXrYr¿s8






zBackendRepository.object_unlinkc	CsÐtj}t|}g}y|jj||}Wn6tk
r\}ztj|tt	j
WYdd}~XnXd|dkrt|d}t|dt|d}nd}t|d}i}||d<||d<||d<d	|d
<d	|d<|S)
Nzx-account-meta-quota-bytesrÕzx-account-bytes-usedrr(Úf_blocksÚf_bfreeÚf_bavailrÚf_favailÚf_frsizer)rSrÆrÇr¡rÖr²rWr³r
ÚerrnoÚENOENTrá)	rrrNrËr
Zstvrîr5r7ZdtrXrXrYrßs*
zBackendRepository.object_statfsc
CsÚtjd||j|\}}tj}t|}||g}y|jj||Wn*tk
rp}ztj	|dSd}~XnX|j
|}	|dkr|j
tjj
||}	ytj|	|Wn,tk
rÔ}ztj	|WYdd}~XnXdS)Nz	mkdir, %srF)rWrrmrSrÆrÇr¡Zmkdir_objectr²r³rÄrTrUrirä)
rrrUrZistrÈrôrËr_rîÚ
cache_pathrXrXrYr+þs$

zBackendRepository.mkdirc
:CsÖ|j|\}}tj}d|d<t|}|g}|dkrF|dkrF|j|g}y||jj||dd7}Wn,tk
r}ztj	|WYdd}~XnXt
|dkr¨ttj
y|jj||Wn,tk
ræ}ztj	|WYdd}~XnX|jdty||jj||dd7}Wn.tk
r<}ztj	|WYdd}~XnXt
|dkry|jj||Wn.tk
r}ztj	|WYdd}~XnX|j|}tjj|r°tj||j|}	tjj|	rÒtj|	SdS)Nrlr0rgr))Z	max_itemsr)rmrSrÆrÇr»r¡r,r²rWr³rür
r:Z	ENOTEMPTYr2ÚinsertrrÅrTrUr#r3rÄ)
rrrUrÈrôrËr
Z	dir_itemsrîZsegment_cache_pathr<rXrXrYr3sH






zBackendRepository.rmdircCsVtjd|y.|j|\}}|jtjj||}tj||Stk
rPYnXdS)Nz	chmod, %s)	rWrrmrÄrTrUriÚchmodr^)rrrUrrÈrlr<rXrXrYr>MszBackendRepository.chmodcCsXtjd|y0|j|\}}|jtjj||}tj|||Stk
rRYnXdS)Nz	chown, %s)	rWrrmrÄrTrUriÚchownr^)rrrUÚuidÚgidrÈrlr<rXrXrYr?VszBackendRepository.chowncCstddS)NzNot Applicable)r²)rrrÛÚtargetrXrXrYÚsymlink_szBackendRepository.symlinkc
[CsØtjd||f|d}y|j|ddWn,tk
rX}ztj|WYdd}~XnXyJ|j|}|j|}|jd|d|jd|dkr¢|j|dddSWn,tk
rÐ}ztj|WYdd}~XnXd}y¸y|j|}Wn.tk
r}ztj|WYdd}~XnxX|j|}|jd|d}	g}
x2|D]*}|
j|d	|d
|d|dd
q:W|j	||
|	|d|dddd}Wnt
k
r¦YnXzàyÂ|j|}|j|}|jd|d}	g}x2|D]*}|j|d	|d
|d|dd
qÚW|j	|||	|d|dddy|j|ddWn.tk
rh}ztj|WYdd}~XnXWnt
k
rYnXWd|rÒy|j|ddWn.tk
rÐ}ztj|WYdd}~XnXXdS)Nzrename, %s -> %sr*T)rMzsegments-dirz	-segmentsrFrÛrørör÷)rUrõrör÷z
segment-countz
total-size)zsegments-dirz
segment-countz
total-size)rÏ)rWrrr²r³rØr±rÌr»rÐr^)
rrÚoldÚnewZnewbak_namerîZnew_metadataZold_metadataZnewfile_existsZnew_manifestrZnewbak_manifestr4Zold_manifestrXrXrYÚrenamebs








 zBackendRepository.renamecCsZtjd|t|\}}|jtjj||}t|\}}|jtjj||}tj||S)Nzlink, %s)rWrrmrÄrTrUriÚlink)rrrBrÛrÈrlZcache_path_targetZcache_path_namerXrXrYrGÉszBackendRepository.linkcCsNtjd|t|\}}|j|}ytj||Wntk
rHdSXdS)Nzutimens, %sr)rWrrmrÄrTÚutimer^)rrrUÚtimesrÈrlr<rXrXrYÚutimensÑs
zBackendRepository.utimenscCsXy>tjjtjtj}tjj|r<tjj|r<tj	|dddWnt
k
rRYnXdS)Né)Ztimeoutr)rTrUrirPr#r$r#ÚismountrÚumountr^)rrrUryrXrXrYÚdestroyÝs
zBackendRepository.destroy)N)N)F)F)N)+r~rrrsrr¸rÂrmrÃrÅrÄrÌrÐrØrrÿrërrr
rfrrrrórrr'rrrrrr+r3r>r?rCrFrGrJrNÚ
__classcell__rXrX)r¥rYrBsL)#z(!"
)
c/
 
 /		g
rc@speZdZddZddZdddZdd	ZdddZd
dZddZ	ddZ
ddZddZddZ
ddZd
S)Ú	FuseCachecCsb||_||_t|_ttj|_ttj|_t	j
j|jsy4tj
|jtj|jt|jdt|jgWqÎtk
rYqÎXnHt	j|j}|j|jksª|j|jkrÎtjtjt|jdt|jgyvt	j
jtjtj}t	j
j|søtj|t	j
j|sBdttjttj}dd|d}tjdd|d|gWntk
r\YnXdS)Nú:r?zsize=%dM,mode=0777r)ir%z-o)rÚ
repositoryÚlrucacherrrrErrrTrUr#rr+r?rûr^rr rrPr#rir$rZensure_treerLrár*r,Zmount)rrrrRr&ryZ
tmpfs_sizeZtmpfs_buffer_cfgrXrXrYrsÿs6(
$

zFuseCache.__init__cCsL|jj||}|jj|dr(|jj|ttjdtj	|t
d|j|<|S)N)Úmaxsize)rSÚlrulockrNÚwrites_before_update_manifest)rRrrSr±rìrrPr,r¨ZLockÚNUMBER_UPLOADS_BEFORE_FLUSH)rrrNrrzrXrXrYr$s
zFuseCache.object_openTcCs®tjd|tjjf|j|}|d|ks2t|d}|dPy4x.|j\}}|drH|jj	|||d|qHWWnt
k
rYnXWdQRX|rª|jj||dS)NzCache object_flush [%s] [%x]rNrSrUrÝÚdata)rWrr¨r¯r°rSrRÚpopitemrRrÚKeyErrorrë)rrrNrzÚupdate_manifestÚitemÚcacherrXrXrYÚobject_flush1s"

zFuseCache.object_flushcCsFtjd|tjjf|j||dd|jj|||jj	|dS)NzCache object_close [%s] [%x]F)r[)
rWrr¨r¯r°r^rRrrSrì)rrrNrzrXrXrYrHszFuseCache.object_closeNcCs|jj|||dS)N)rRr)rrrNrrzrXrXrYrRszFuseCache.object_truncateccsVxP|dkrP|tjtj}||}t|tj|}|||fV||7}||8}qWdS)Nr)rPr*rù)rrrrZ
seg_offsetÚbaseÚseg_lenrXrXrYÚ_walk_segmentsUs
zFuseCache._walk_segmentsc!Cs|j|d|kstt}|j|dTxJ|j||D]8\}}}y|j|d|d}	Wnütk
rby¸|j|d}
t|
tjkr|
j\}}|dr|j	j
|||d||j|dd8<|j|ds|j	j||t|j|d<|j	j
|||}	d|	d	|j|d|<Wn,tk
r\}
ztj|
d
Sd}
~
XnXYnX||	|||7}q>WWdQRXt|S)NrNrUrSrXrÝrVrF)rÝrXr)rSrRrrar^rürPr,rYrRrrërWrr²rWr³Úbytes)rrrNrrrzZ
output_bufÚ	segoffsetr_r`Úsegdatar]rr\rîrXrXrYÚobject_read`s@


"zFuseCache.object_readcCsÖtjd|tjjft|}|j|d|ks6td}|j|d~xt|j||D]b\}}}	|j|d}
||
kr\t|
t	j
krø|
j\}}|drø|jj
|||d||j|dd	8<|j|dsø|jj||t|j|d<y4|jj|||}
|
dkrtd
d|
d|
|<Wn,tk
rXtd	}
d
|
d|
|<YnXn|
|d}
t|
|kr|
jd||	t|
||||	|
|||	<d
|
|d<||	7}q\WWdQRXt|S)NzCache object_write [%s] [%x]rNrrUrSrÝrXrVrzObject not foundF)rÝrXTó)rWrr¨r¯r°rürSrRrarPr,rYrRrrërWrr²r^rÚextend)rrrNrrrzrZbufptrrcr_r`r]rr\rdrXrXrYÚobject_writesN


zFuseCache.object_writecCstj|jj||S)N)rTr+rRr)rrrUrrXrXrYr+íszFuseCache.mkdircCs|jj|}tj|S)N)rRrrTr3)rrrUrírXrXrYr3ðszFuseCache.rmdircCs|jj|}tj||S)N)rRrrTr>)rrrUrrírXrXrYr>ôszFuseCache.chmodcCs|jj|}tj|||S)N)rRrrTr?)rrrUr@rArírXrXrYr?øszFuseCache.chown)T)N)r~rrrsrr^rrrarerhr+r3r>r?rXrXrXrYrPþs%


9TrPc@sÚeZdZd2ddZddZddZdd	Zed3d
dZdd
Z	ddZ
ddZddZddZ
ddZddZddZddZddZd4d d!Zd"d#Zd5d$d%Zd&d'Zd(d)Zd6d*d+Zd,d-Zd.d/Zd0d1ZdS)7r&NcCs&||_|pt||_t||j|_dS)N)rrrRrPr])rrrrRrXrXrYrsýszTrilioVault.__init__cCs|jj||dS)N)rRr)rrrUrrXrXrYÚaccessszTrilioVault.accesscCs|jj||S)N)rRr>)rrrUrrXrXrYr>szTrilioVault.chmodcCs|jj|||S)N)rRr?)rrrUr@rArXrXrYr?szTrilioVault.chowncCs|jj||S)N)rRr)rrrUrzrXrXrYrszTrilioVault.getattrccs4ddg}|j|jj||x|D]
}|Vq"WdS)NrÜz..)rgrRr)rrrUrzr/r1rXrXrYÚreaddirs
zTrilioVault.readdircCs2tj|j|}|jdr*tjj||jS|SdS)Nrg)rTÚreadlinkrrrUÚrelpathr)rrrUÚpathnamerXrXrYrks
zTrilioVault.readlinkcCstj|jj|||S)N)rTÚmknodrRr)rrrUrZdevrXrXrYrn*szTrilioVault.mknodcCs|jj|S)N)rRr3)rrrUrXrXrYr3-szTrilioVault.rmdircCs|jj||S)N)rRr+)rrrUrrXrXrYr+0szTrilioVault.mkdircCs|jj|S)N)rRr)rrrUrXrXrYÚstatfs3szTrilioVault.statfscCs|jj|S)N)rRr)rrrUrXrXrYÚunlink6szTrilioVault.unlinkcCs|jj||S)N)rRrC)rrrÛrBrXrXrYrC9szTrilioVault.symlinkcCs|jj||S)N)rRrF)rrrDrErXrXrYrF<szTrilioVault.renamecCs|jj||S)N)rRrG)rrrBrÛrXrXrYrG?szTrilioVault.linkcCs|jj||S)N)rRrJ)rrrUrIrXrXrYrJBszTrilioVault.utimenscCs|jj||S)N)r]r)rrrUrrXrXrYræHszTrilioVault.opencCs|j|tjS)N)rærTÚO_CREAT)rrrUrZfirXrXrYÚcreateKszTrilioVault.createcCs|jj||||}|S)N)r]re)rrrUrrrzrrXrXrYrNszTrilioVault.readcCs|jj||||S)N)r]rh)rrrUrrrzrXrXrYrçRszTrilioVault.writecCs|jj|||ddS)N)rz)r]r)rrrUrrzrXrXrYÚtruncateUszTrilioVault.truncatecCs|jj||dS)N)r]r^)rrrUrzrXrXrYÚflushXszTrilioVault.flushcCs|jj||dS)N)r]r)rrrUrzrXrXrYr|[szTrilioVault.releasecCstjd||jj|S)Nzdestroy, %s)rWrrRrN)rrrUrXrXrYrNbszTrilioVault.destroy)N)N)N)N)N)r~rrrsrir>r?rfrrjrkrnr3r+rorprCrFrGrJrærrrrçrsrtr|rNrXrXrXrYr&üs2



ú/etc/fuse.confcCsÐd}tjj|rJt|d*}x"|D]}d|kr"d|kr"d}Pq"WWdQRX|dkrÌd}tjj|rt|d}|jdd}WdQRXtd	d
}|j|WdQRXtjd	|tjd	tj	ddgdS)
Nrr1Zuser_allow_otherú#rzuser_allow_other 
ZrtÚ
z/tmp/fuse.conf.tmpZwtz/etc/fuse.confz	root:root)
rTrUrVrærrçrrÆrpr?)Z	conf_fileÚfoundrðÚlineÚsZoutfrXrXrYÚ	fuse_confgs"

r{cCs yytj|dgWntk
r(YnXtjj|rPtj|dtdtgnHtj|dgtj|dgtj|tdtgtj|tdtgWn,t	k
rÆ}zt
j|WYdd}~XnXddtjt
jDt|t|d}d}t
jjd	krd
}t|||ddddddS)Nz-lz-RrQz-pcSs&g|]}tjtjjtj|ddqS)T)Ú
ignore_errors)ÚshutilZrmtreerTrUrirPr#)rÚxrXrXrYú
<listcomp>szmain.<locals>.<listcomp>)rRTrOF)Z	nothreadsZ
foregroundZnonemptyZ
big_writesZallow_other)rrMr^rTrUr#r?rr+r²rWr³ÚlistdirrPr#r&rr<rQr	)Z
mountpointZ	cacherootrîZtvaultpluginZdisable_fuse_threadsrXrXrYÚmainzs8rcCs.ttjtjksttttjtjdS)z$Main routine that drives fuse mount.N)	rZrPr'r(rRr{rr"r#rXrXrXrYr¶¦sr¶Ú__main__ii)ru)br·rTÚsysr:rârèr}rdÚ
subprocessr¨rrZtempfilerrrÚImportErrorZqueueÚpwdrZ
cachetoolsrZfuser	r
rZfusepyZs3fuse.bunch_p3ZastZoslo_configr
Zoslo.configZoslo_logrr[Zs3fuserrZs3fuse.privseprZ_DEFAULT_LOG_DATE_FORMATrZQUEUE_DEPTHZWORKER_POOL_SIZEZBoolOptZcommon_cli_optsZStrOptZlogging_cli_optsZIntOptZgeneric_log_optsZListOptZlog_optsZcontego_vault_optsrPZ
register_optsZregister_cli_optsrûÚenvironÚflagÚargvZsetupr rQZ	getLoggerrWrwr=ZloggerZsetLevelÚDEBUGÚINFOZWARNr]r^r±rSrZZCACHE_LOW_WATERMARKZCACHE_HIGH_WATERMARKZgetpassZgetuserrrrSrWrfrmrnroÚobjectrrrPr&r{rr¶r~rXrXrXrYÚ<module>s¼



















#
4Ek,
triliodata-4-1-beta / python3-s3fuse-plugin rpm

Products

About

Resources

Contact Gemfury
