Metadata-Version: 2.1
Name: gmr-ingest
Version: 0.17.0
Summary: GEOMAR ingest component. Pulls data from sensors, pushes data to internal & external targets
Home-page: UNKNOWN
Author: Claas Faber, GEOMAR data management
License: NOT FOR PUBLIC USE
Requires-Dist: pandas
Requires-Dist: click
Requires-Dist: postgis-import
Requires-Dist: waveglider-connect
Requires-Dist: eddy-cv-connect
Requires-Dist: gmr-underway-connect

do not distribute w/o prior consent of author(s)

Description: # gmr_ingest
        This is the central control component which manages the communication with
        data sources and data sinks.
        
        It is installed on a dedicated Virtual Machine dm-ingest-www in the GEOMAR
        network. Cron jobs of the dedicated user datamanagement are running reularly
        to collect and publish data.
        
        # Installation
        
        ## Release version
        ```bash
        pip install --index-url https://pypi.fury.io/cfgmr/ gmr_ingest
        ```
        
        ## From source
        ```bash
        git clone git@git.geomar.de:moses/moses_tools/gmr_ingest.git
        cd gmr_ingest
        pip install -e .
        ```
        
        # Configuration
        For the ingest itself, you can configure log settings:
        ```
        export INGEST_LOG_PATH=/srv/bin/logs
        export INGEST_LOG_FILENAME=gmr_ingest.log
        ```
        
        Each of the data source and data sink connectors needs it's own configuration.
        Hava a look at the 
        [MOSES tools documentation](https://git.geomar.de/moses/moses_tools/docu) 
        for an overview of available connectors. Then look at the documentation of the 
        respective repos for details. See also 
        [configuration of operative installation](#connector-configuration) below.
        
        Here, you need to make sure that the connectors you want to use are installed.
        An example for a requirements file could be:
        ```
        # requirements.txt
        --index-url https://pypi.fury.io/cfgmr/
        --extra-index-url https://pypi.org/simple/
        ocean-eddies-postgis
        waveglider-connect
        eddy-cv-connect
        hzg-sos-connect
        gmr-underway-connect
        po-minijson-connect
        gmr-ingest
        ```
        
        Now run `pip install --upgrade -r requirements.txt`  
        
        *Note*: If you have an virtual environment, you need to activate it first!  
        `source ./venv/bin/activate`
        
        # Usage
        gmr_ingest coes with a simple command line interface.
        *Please Note*: Most connectors need to be configured before use (e.g. database 
        settings, passwords, ...)!
        
        ```bash
        [datamanagement@dm-ingest-www bin]$ cd /srv/bin/
        [datamanagement@dm-ingest-www bin]$ source ./venv/bin/activate
        (venv) [datamanagement@dm-ingest-www bin]$ gmr_data_ingest --help
        Usage: gmr_data_ingest [OPTIONS]
        
        Options:
          --data-source [WaveGlider|SlocumGlider|Ships|EddyCV|HZGSOS|UnderwayNMEA|PoMiniJson]
                                          Source of data  [required]
          --data-destination [OceanEddiesPostGIS]
                                          Destination of data  [required]
          --help                          Show this message and exit.
        (venv) [datamanagement@dm-ingest-www bin]$ gmr_data_ingest --data-source WaveGlider --data-destination OceanEddiesPostGIS
        ```
        
        # Operation
        The ingest module is installed on the VM `dm-ingest-www`. All important
        files are located in `/srv/bin/` there. Ingest is performed as 
        user `datamanagement`.
        
        
        ## `/srv/bin`
        ```
        /srv/bin/
        +-- ingest_geoserver.sh     # Script for data ingest into Geoserver
        +-- ingest_store/           # folder where keys are stored
        |   +-- SECRET_KEY
        +-- ingest_wg4_position.sh # Script for Waveglider position ingest (called more frequently)
        +-- logs/
        |   +-- gmr_ingest.log     # log file written by ingest
        +-- requirements.txt       # requirements file used by pip
        +-- venv/                  # python virtual environment used by ingest
            +-- [...]
        ```
        
        
        ## Connector configuration
        Connectors are configured using environmental variables. The script files
        run by the cron jobs will export all the needed secrets and configurations
        before calling the ingest.
        
        
Platform: UNKNOWN
Requires-Python: >= 3.6