Metadata-Version: 2.4
Name: dship2postgis
Version: 1.9.2
Summary: package for importing dship exports to postgis
Home-page: https://git.geomar.de/dm/dship2postgis
Author: Claas Faber
Author-email: cfaber@geomar.de
License: Copyright (c) 2018, Claas Faber, GEOMAR Data Management
        
        All rights reserved.
        
        Redistribution and use in source and binary forms, with or without modification, are NOT PERMITTED
        without prior permission by GEOMAR data management team. INTERNAL USE ONLY.
        
        Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
        Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
        THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
        
License-File: LICENSE
Requires-Dist: click
Requires-Dist: pandas
Requires-Dist: pyproj
Requires-Dist: geoalchemy2
Requires-Dist: sqlalchemy
Requires-Dist: psycopg2-binary
Requires-Dist: geopandas
Requires-Dist: xmltodict
Requires-Dist: chardet
Requires-Dist: python-dateutil
Requires-Dist: gmr-underway-connect
Requires-Dist: dshipparser
Dynamic: author
Dynamic: author-email
Dynamic: description
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: summary

[![pipeline status](https://git.geomar.de/dm/dship2postgis/badges/master/pipeline.svg)](https://git.geomar.de/dm/dship2postgis/commits/master)
[![coverage report](https://git.geomar.de/dm/dship2postgis/badges/master/coverage.svg)](https://git.geomar.de/dm/dship2postgis/commits/master)

# maps infrastructure overview
```mermaid
graph LR;
  nmea_mail -- read --> dship2postgis;
  daily_mail -- read --> dship2postgis;
  dship_archives -- read --> dship2postgis;
  
  dship2postgis -- import into --> postgis_db;
  postgis_db -- reads --> maps_app;
  
  maps_docu -. uses .-> maps_app
  maps_startpage -. uses .-> maps_app
  
  style dship2postgis fill:#f9f,stroke:#333,stroke-width:4px
  legend[you are here]
  style legend fill:#f9f,stroke:#333,stroke-width:4px
  
  click dship2postgis "https://git.geomar.de/dm/dship2postgis" "dshi2postgis git repo"
  click maps_app "https://git.geomar.de/dm/maps" "maps app git repo"
  click maps_docu "https://git.geomar.de/dm/maps_docu" "maps docu git repo"
  click maps_startpage "https://git.geomar.de/dm/pages/startpage_maps" "maps app git repo"
```

DSHIP to POSTGIS
================

This utility reads in DSHIP data exports (data inventory/ data
extraction) and imports the data into a POSTGIS database.

```mermaid
graph LR;
    subgraph dship2postgis
    nmea(NMEA telegrams)-- dship2postgis -->db(PostGIS DB)
    daily(Daily underway export)-- dship2postgis -->db
    dhsip(DShip export)-- dship2postgis -->db
    end
    
    subgraph PostGIS
    db-->qa(quality check) 
    qa-->db
    end 
    
    subgraph /dm/maps + /dm/pages/startpage_maps
    db-->app(WebApp)
    app-->maps(maps.geomar.de)
    end
```


Limitations
-----------

Currently, only ship **positions** from data inventory, data extractions
are read. Further parameters (e.g. meteorological and ctd data) are
currently ignored. These parameters will probably be imported in the
future, but for now the focus is on importing positions as a base for
the underway/ map application.

This import does not handle Stationbook/ ActionLog imports!

Capabilities
------------

The import understands old (DSHIP2) and new (DSHIP3) data. It examines
all of the files from the DSHIP export (.sys/.txt/.xml) to determine
export metadate such as column separators and decimal symbols.

The imported data is enriched with leg informatn provided by
OSIS/metadata.

Underway nmea telegrams sent every 10 minutes by the ships can be processed.
Daily exports with 1hz resolution sent by mai by the ships can also 
be processed.
Cron jobs doing this automatically are set up on dm-apps-node1 as user root.

The import implements a command line interface making it easy to use
(see below).

Requirements
------------

### Data

All files of a DSHIP export are expected in the same folder, one folder
per export.

For **dship2 exports**, the folder has to contain - *.txt fiel with
export meta-info -*.dat file with the actual data

For **dhsip3 exports**, the directory has to contain - *.sys file with
meta-information - [order]()*.xml file with format and column
specification - \*.dat file the actual data

The export **has to contain columns for lat/ lon** (see Configuration
for recognized columns) and a *timestamp*. The .dat file is **expected
to be headerless**.

### Python

The program needs python >3.6 in order to work. A virtualenv/ conda
environment is strongly encouraged. dship2postgis needs to be installed
(using pip, see below) in the environment

Installation
------------

-   Clone this repository
    git clone git@git.geomar.de:dm/dship2postgis.git
-   in your working directory, activate your conda env/ virtualenv
    source activate my\_env
-   use pip to install
    pip install -e /path\_to\_git\_repo/dship2postgis/ The -e option
    means that dship2postgis is installed *editable*, meaning that you
    don't have to install it again if you pull changes from the repo.

Configuration
-------------

### Application

The application comes with default values that should work fine for all
DM personell. If something does not work, pay special attetion to DB
configuration (see below) as well as configuration of LAT/LON columns
and NA-Values.

Configuration can be done in two ways: - editing
ship2postgis/config.pyor by - setting environment variables

The config will try to read DB passwords for the configured user/ DB
from your .pgpass file, so make sure to have valid credentials for the
configured DB connection.

Database
---------
THIS SECTION NEEDS UPDATE
- all views/ functions
- QA
- cron jobs n dm-pgsql-train2

By default, all imports go to DB host dm-pgsql-train-2, db ifm-geomar
schema postgis\_testbed, a db-user postgis\_import (using pwd:
Maui0,proof) is configured there.

The **import-user needs** - full access to and owndership of the
configured schema - read-only access to DB metadata (for generation of
views)

The following **views** need to be present: - leg\_departure\_return
generated from OSIS data to determine leg of a position - tracks
grouping positions into tracks (linestrings) (should probably be
materialized view, too) - materialized view event\_tracks generated from
OSIS event data (not used in frontend yet)

The following **functions** are defined -
get\_simplified\_track(leg, epsilon) returns GEOJSON for a leg track,
simplified using DRP algorithm

In the future, I would like to create cli-calls which setup DB during
instalation, but not today...

Usage
-----

Once the application is installed, configured and DB is set up (note: if
you use dafaults, you only need to install, since the db on
dm-pgsql-train-2 is already set up properly), you can start the import
on the command line (remember to activate your environment first):

dship2postgis /path/to/dship/data/ -l import\_label The program will do
the following: - print a short message with the most important
configurations - parse meta-info and read data (throwing out all rows
without data (e.g. '9999') - assign a leg label to each imported row
using OSIS port calls as basis - re-format the data to geospatial data
types using a geodataframe - import the data into a temporary table -
merge/ update positions table with data from temp table - delete
temporary table

Each imported row has an import\_label attached to mark it's origin. If
the import\_label is not be passed as an argument, it will be
automatically generated. Call dship2postgis --help for short
instructions.

An example run could look like this

``` {.sourceCode .bash
 dship2postgis input_data/dship_extracts/dship_AL_2009_60s/
 2018-03-05 10:22:02,867 - root - INFO - Importing data from input_data/dship_extracts/dship_AL_2009_60s/ into postgresql://postgis_import:XXXX@dm-pgsql-train2:5432/ifm-geomar postgis_testbed.positions
 2018-03-05 10:22:02,867 - root - INFO - Parsing dship-data found in input_data/dship_extracts/dship_AL_2009_60s/
 2018-03-05 10:22:52,243 - root - INFO - Getting leg-label for each row from OSIS
 2018-03-05 10:22:56,256 - root - INFO - Generating geodataframe from parsed data
 2018-03-05 10:23:35,897 - root - INFO - Writing data to database
 2018-03-05 10:29:42,037 - root - INFO - wrote data to temporary table temp_positions_dhsip_import_dship_al_2009_60s_1520238123_1520238220
 2018-03-05 10:31:47,710 - root - INFO - merged temporary table into positions
 2018-03-05 10:31:47,804 - root - INFO - dropped temporary table ['temp_positions_dhsip_import_dship_al_2009_60s_1520238123_1520238220']}
```

What else?
----------

If something goes wrong, please submit an issue to
<https://git.geomar.de/dm/dship2postgis/issues> . I will try to keep the
planned enhancements/ bugs up to date there.