Browse Source

docs: adding docs

master
Marius Isken 1 month ago
parent
commit
4a081628b2
7 changed files with 280 additions and 2 deletions
  1. +5
    -2
      README.md
  2. +20
    -0
      docs/Makefile
  3. +35
    -0
      docs/make.bat
  4. +52
    -0
      docs/source/conf.py
  5. +14
    -0
      docs/source/index.rst
  6. +10
    -0
      docs/source/processing.rst
  7. +144
    -0
      docs/source/setup.rst

+ 5
- 2
README.md View File

@ -60,8 +60,11 @@ plugins:
# A Telegram bot to keep up-to-date with the process
- !idas_convert.telegram_bot.TelegramBotConfig
enabled: false
token: Telegram Token
chat_id: Telegram Chat ID, e.g. -456413218
# Telegram API Token
token: 9e98b8c0567149eb861838a1d770be7d
# Telegram Chat ID
chat_id: -1237123123
# A status message will be dispatched every 3600 s
status_interval: 3600.0
```


+ 20
- 0
docs/Makefile View File

@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = source
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

+ 35
- 0
docs/make.bat View File

@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd

+ 52
- 0
docs/source/conf.py View File

@ -0,0 +1,52 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- Project information -----------------------------------------------------
project = 'iDAS Convert'
copyright = '2021, Marius Paul Isken'
author = 'Marius Paul Isken'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

+ 14
- 0
docs/source/index.rst View File

@ -0,0 +1,14 @@
iDAS Convert
============
Convert and downsample distribute acousting sensing data (DAS) acquired by `Silixa iDAS <https://silixa.com/products/idas/>`_ to seismological data formats.
Main purpose is to quickly convert and downsample massive amounts of high-resolution TDMS data to MiniSeed.
Contents
--------
.. toctree::
:maxdepth: 2
setup
processing

+ 10
- 0
docs/source/processing.rst View File

@ -0,0 +1,10 @@
Processing details
==================
The command line tools take Silixa iDAS TDMS data as input. The data are loaded and converted to MiniSeed in an efficient manner.
Downsampling
------------
Quality Control (QS)
--------------------

+ 144
- 0
docs/source/setup.rst View File

@ -0,0 +1,144 @@
Setup
=============
Installation
------------
Python 3 requirements:
* `pyrocko <https://pyrocko.org>`_
* telebot (optional)
Installation using pip
.. code-block :: bash
pip3 install git+https://git.pyrocko.org/pyrocko/idas-convert
Or installation from source:
.. code-block :: bash
git clone https://git.pyrocko.org/pyrocko/idas-convert
cd idas-convert
pyhon3 setup.py install
Check out the help with:
.. code-block :: bash
idas_convert -h
Quickstart
----------
Before starting the convertion edit the config file. See details of the YAML config in the :doc:`config`.
.. code-block :: bash
idas_convert dump_config > my_config.yml
idas-convert my_config.yml
In case of an abort or error the conversion can be resumed with:
.. code-block :: bash
idas-convert --resume my_config.yml
Configuration
-------------
The conversion tools are configured with a `YAML <https://en.wikipedia.org/wiki/YAML>`_ file. This file is then **executed** by ``idas_convert`` CLI programm.
.. code-block :: YAML
--- !idas.iDASConvertConfig
# Loading TDMS in parallel and process
nthreads_loading: 1
nthreads_processing: 8
queue_size: 32
processing_batch_size: 8
# Threads used for downsampling the data
nthreads: 8
# Input paths
paths:
- /home/isken/src/idas-convert
# Out path, see pyrocko.io for details
outpath: '%(tmin_year)s%(tmin_month)s%(tmin_day)s/%(network)s.%(station)s_%(tmin_year)s%(tmin_month)s%(tmin_day)s.mseed'
# Overwrite mseed meta information
new_network_code: ID
new_channel_code: HSF
downsample_to: 200.0
# MiniSeed record length
record_length: 4096
# MiniSeed STEIM compression
steim: 2
plugins:
# A plugin handling the communication with the GFZ tage file system
- !idas_convert.gfz_tapes.GFZTapesConfig
enabled: false
bytes_stage: 1T
waterlevel: 0.6
wait_warning_interval: 600.0
release_files: true
path_tapes_mount: /projects/ether/
path_tapes_prefix: /archive_FO1/RAW/
# A Telegram bot to keep up-to-date with the process
- !idas_convert.telegram_bot.TelegramBotConfig
enabled: false
token: Telegram Token
chat_id: Telegram Chat ID, e.g. -456413218
status_interval: 3600.0
Plugins
-------
The following plugins can be configured in the ``plugins`` list in the YAML file.
Telegram bot
^^^^^^^^^^^^
A Telegram bot can be configured to keep up-to-date with the processing progress.
This plugin is forwarding the log levels ``INFO`` and ``WARNING`` to the chat bot, simply add the bot to a Telegram chat group.
Details about the Telegram ``token`` and ``chat_id`` can be found `here <https://core.telegram.org/bots>`_.
.. code-block :: YAML
- !idas_convert.telegram_bot.TelegramBotConfig
enabled: false
# Telegram API Token
token: 9e98b8c0567149eb861838a1d770be7d
# Telegram Chat ID
chat_id: -1237123123
status_interval: 3600.0
GFZ Tape interaction
^^^^^^^^^^^^^^^^^^^^
The `GFZ German Research Centre for Geosciences <https://gfz-potsdam.de>`_ maintains a tape storage system, details about the system `here <https://www.golem.de/news/bandlaufwerke-als-backupmedium-ein-bisschen-tetris-spielen-1906-141575.html>`_ (in German).
This plugin enables seamless inteaction with the SAMFS RPC call procedures, needed to stage and release the files from the hot-storage.
.. code-block :: YAML
- !idas_convert.gfz_tapes.GFZTapesConfig
enabled: false
bytes_stage: 1T
waterlevel: 0.6
wait_warning_interval: 600.0
release_files: true
path_tapes_mount: /projects/ether/
path_tapes_prefix: /archive_FO1/RAW/

Loading…
Cancel
Save