@ -0,0 +1,20 @@ | |||
# Minimal makefile for Sphinx documentation | |||
# | |||
# You can set these variables from the command line, and also | |||
# from the environment for the first two. | |||
SPHINXOPTS ?= | |||
SPHINXBUILD ?= sphinx-build | |||
SOURCEDIR = source | |||
BUILDDIR = build | |||
# Put it first so that "make" without argument is like "make help". | |||
help: | |||
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) | |||
.PHONY: help Makefile | |||
# Catch-all target: route all unknown targets to Sphinx using the new | |||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). | |||
%: Makefile | |||
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) |
@ -0,0 +1,35 @@ | |||
@ECHO OFF | |||
pushd %~dp0 | |||
REM Command file for Sphinx documentation | |||
if "%SPHINXBUILD%" == "" ( | |||
set SPHINXBUILD=sphinx-build | |||
) | |||
set SOURCEDIR=source | |||
set BUILDDIR=build | |||
if "%1" == "" goto help | |||
%SPHINXBUILD% >NUL 2>NUL | |||
if errorlevel 9009 ( | |||
echo. | |||
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx | |||
echo.installed, then set the SPHINXBUILD environment variable to point | |||
echo.to the full path of the 'sphinx-build' executable. Alternatively you | |||
echo.may add the Sphinx directory to PATH. | |||
echo. | |||
echo.If you don't have Sphinx installed, grab it from | |||
echo.http://sphinx-doc.org/ | |||
exit /b 1 | |||
) | |||
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% | |||
goto end | |||
:help | |||
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% | |||
:end | |||
popd |
@ -0,0 +1,52 @@ | |||
# Configuration file for the Sphinx documentation builder. | |||
# | |||
# This file only contains a selection of the most common options. For a full | |||
# list see the documentation: | |||
# https://www.sphinx-doc.org/en/master/usage/configuration.html | |||
# -- Path setup -------------------------------------------------------------- | |||
# If extensions (or modules to document with autodoc) are in another directory, | |||
# add these directories to sys.path here. If the directory is relative to the | |||
# documentation root, use os.path.abspath to make it absolute, like shown here. | |||
# | |||
# import os | |||
# import sys | |||
# sys.path.insert(0, os.path.abspath('.')) | |||
# -- Project information ----------------------------------------------------- | |||
project = 'iDAS Convert' | |||
copyright = '2021, Marius Paul Isken' | |||
author = 'Marius Paul Isken' | |||
# -- General configuration --------------------------------------------------- | |||
# Add any Sphinx extension module names here, as strings. They can be | |||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom | |||
# ones. | |||
extensions = [ | |||
] | |||
# Add any paths that contain templates here, relative to this directory. | |||
templates_path = ['_templates'] | |||
# List of patterns, relative to source directory, that match files and | |||
# directories to ignore when looking for source files. | |||
# This pattern also affects html_static_path and html_extra_path. | |||
exclude_patterns = [] | |||
# -- Options for HTML output ------------------------------------------------- | |||
# The theme to use for HTML and HTML Help pages. See the documentation for | |||
# a list of builtin themes. | |||
# | |||
html_theme = 'alabaster' | |||
# Add any paths that contain custom static files (such as style sheets) here, | |||
# relative to this directory. They are copied after the builtin static files, | |||
# so a file named "default.css" will overwrite the builtin "default.css". | |||
html_static_path = ['_static'] |
@ -0,0 +1,14 @@ | |||
iDAS Convert | |||
============ | |||
Convert and downsample distribute acousting sensing data (DAS) acquired by `Silixa iDAS <https://silixa.com/products/idas/>`_ to seismological data formats. | |||
Main purpose is to quickly convert and downsample massive amounts of high-resolution TDMS data to MiniSeed. | |||
Contents | |||
-------- | |||
.. toctree:: | |||
:maxdepth: 2 | |||
setup | |||
processing |
@ -0,0 +1,10 @@ | |||
Processing details | |||
================== | |||
The command line tools take Silixa iDAS TDMS data as input. The data are loaded and converted to MiniSeed in an efficient manner. | |||
Downsampling | |||
------------ | |||
Quality Control (QS) | |||
-------------------- |
@ -0,0 +1,144 @@ | |||
Setup | |||
============= | |||
Installation | |||
------------ | |||
Python 3 requirements: | |||
* `pyrocko <https://pyrocko.org>`_ | |||
* telebot (optional) | |||
Installation using pip | |||
.. code-block :: bash | |||
pip3 install git+https://git.pyrocko.org/pyrocko/idas-convert | |||
Or installation from source: | |||
.. code-block :: bash | |||
git clone https://git.pyrocko.org/pyrocko/idas-convert | |||
cd idas-convert | |||
pyhon3 setup.py install | |||
Check out the help with: | |||
.. code-block :: bash | |||
idas_convert -h | |||
Quickstart | |||
---------- | |||
Before starting the convertion edit the config file. See details of the YAML config in the :doc:`config`. | |||
.. code-block :: bash | |||
idas_convert dump_config > my_config.yml | |||
idas-convert my_config.yml | |||
In case of an abort or error the conversion can be resumed with: | |||
.. code-block :: bash | |||
idas-convert --resume my_config.yml | |||
Configuration | |||
------------- | |||
The conversion tools are configured with a `YAML <https://en.wikipedia.org/wiki/YAML>`_ file. This file is then **executed** by ``idas_convert`` CLI programm. | |||
.. code-block :: YAML | |||
--- !idas.iDASConvertConfig | |||
# Loading TDMS in parallel and process | |||
nthreads_loading: 1 | |||
nthreads_processing: 8 | |||
queue_size: 32 | |||
processing_batch_size: 8 | |||
# Threads used for downsampling the data | |||
nthreads: 8 | |||
# Input paths | |||
paths: | |||
- /home/isken/src/idas-convert | |||
# Out path, see pyrocko.io for details | |||
outpath: '%(tmin_year)s%(tmin_month)s%(tmin_day)s/%(network)s.%(station)s_%(tmin_year)s%(tmin_month)s%(tmin_day)s.mseed' | |||
# Overwrite mseed meta information | |||
new_network_code: ID | |||
new_channel_code: HSF | |||
downsample_to: 200.0 | |||
# MiniSeed record length | |||
record_length: 4096 | |||
# MiniSeed STEIM compression | |||
steim: 2 | |||
plugins: | |||
# A plugin handling the communication with the GFZ tage file system | |||
- !idas_convert.gfz_tapes.GFZTapesConfig | |||
enabled: false | |||
bytes_stage: 1T | |||
waterlevel: 0.6 | |||
wait_warning_interval: 600.0 | |||
release_files: true | |||
path_tapes_mount: /projects/ether/ | |||
path_tapes_prefix: /archive_FO1/RAW/ | |||
# A Telegram bot to keep up-to-date with the process | |||
- !idas_convert.telegram_bot.TelegramBotConfig | |||
enabled: false | |||
token: Telegram Token | |||
chat_id: Telegram Chat ID, e.g. -456413218 | |||
status_interval: 3600.0 | |||
Plugins | |||
------- | |||
The following plugins can be configured in the ``plugins`` list in the YAML file. | |||
Telegram bot | |||
^^^^^^^^^^^^ | |||
A Telegram bot can be configured to keep up-to-date with the processing progress. | |||
This plugin is forwarding the log levels ``INFO`` and ``WARNING`` to the chat bot, simply add the bot to a Telegram chat group. | |||
Details about the Telegram ``token`` and ``chat_id`` can be found `here <https://core.telegram.org/bots>`_. | |||
.. code-block :: YAML | |||
- !idas_convert.telegram_bot.TelegramBotConfig | |||
enabled: false | |||
# Telegram API Token | |||
token: 9e98b8c0567149eb861838a1d770be7d | |||
# Telegram Chat ID | |||
chat_id: -1237123123 | |||
status_interval: 3600.0 | |||
GFZ Tape interaction | |||
^^^^^^^^^^^^^^^^^^^^ | |||
The `GFZ German Research Centre for Geosciences <https://gfz-potsdam.de>`_ maintains a tape storage system, details about the system `here <https://www.golem.de/news/bandlaufwerke-als-backupmedium-ein-bisschen-tetris-spielen-1906-141575.html>`_ (in German). | |||
This plugin enables seamless inteaction with the SAMFS RPC call procedures, needed to stage and release the files from the hot-storage. | |||
.. code-block :: YAML | |||
- !idas_convert.gfz_tapes.GFZTapesConfig | |||
enabled: false | |||
bytes_stage: 1T | |||
waterlevel: 0.6 | |||
wait_warning_interval: 600.0 | |||
release_files: true | |||
path_tapes_mount: /projects/ether/ | |||
path_tapes_prefix: /archive_FO1/RAW/ |