Browse Source

docs and tests

develop
Marius Isken 9 months ago
parent
commit
f381d827d5
  1. 5
      docs/source/index.rst
  2. 14
      docs/source/setup.rst
  3. 1
      requirements.txt
  4. 10
      src/trace_stats.py
  5. 0
      test/test_conversion.py
  6. 42
      test/test_convert.py
  7. 17
      test/test_various.py

5
docs/source/index.rst

@ -4,6 +4,11 @@ iDAS Convert
Convert and downsample distribute acousting sensing data (DAS) acquired by `Silixa iDAS <https://silixa.com/products/idas/>`_ to seismological data formats.
Main purpose is to quickly convert and downsample massive amounts of high-resolution TDMS data to MiniSeed.
To handle the massive amount of data generated by DAS interogators, the conversion tool is leveraging parallel IO and parallel processing.
On production systems a throughput of 200 MB/s while converting and downsampling (1 kHz -> 200 Hz) has been archived.
The seismological signal processing routines are based on the `Pyrocko <https://pyrocko.org>`_ framework, are mature and well tested.
Contents
--------

14
docs/source/setup.rst

@ -8,6 +8,7 @@ Python 3 requirements:
* `pyrocko <https://pyrocko.org>`_
* telebot (optional)
* `numpy <https://numpy.org>`_
Installation using pip
@ -51,7 +52,7 @@ In case of an abort or error the conversion can be resumed with:
Configuration
-------------
The conversion tools are configured with a `YAML <https://en.wikipedia.org/wiki/YAML>`_ file. This file is then **executed** by ``idas_convert`` CLI programm.
The conversion tools are configured with a `YAML <https://en.wikipedia.org/wiki/YAML>`_ file. This file is then _executed_ by ``idas_convert`` CLI programm.
.. code-block :: YAML
@ -98,8 +99,11 @@ The conversion tools are configured with a `YAML <https://en.wikipedia.org/wiki/
# A Telegram bot to keep up-to-date with the process
- !idas_convert.telegram_bot.TelegramBotConfig
enabled: false
token: Telegram Token
chat_id: Telegram Chat ID, e.g. -456413218
# Telegram API Token
token: 9e98b8c0567149eb861838a1d770be7d
# Telegram Chat ID
chat_id: -1237123123
# A status message will be dispatched every 3600 s
status_interval: 3600.0
Plugins
@ -107,7 +111,7 @@ Plugins
The following plugins can be configured in the ``plugins`` list in the YAML file.
Telegram bot
Telegram Bot
^^^^^^^^^^^^
A Telegram bot can be configured to keep up-to-date with the processing progress.
@ -126,7 +130,7 @@ Details about the Telegram ``token`` and ``chat_id`` can be found `here <https:/
status_interval: 3600.0
GFZ Tape interaction
GFZ Tape Interaction
^^^^^^^^^^^^^^^^^^^^
The `GFZ German Research Centre for Geosciences <https://gfz-potsdam.de>`_ maintains a tape storage system, details about the system `here <https://www.golem.de/news/bandlaufwerke-als-backupmedium-ein-bisschen-tetris-spielen-1906-141575.html>`_ (in German).

1
requirements.txt

@ -1,2 +1,3 @@
numpy
pyrocko
telebot

10
src/trace_stats.py

@ -1,10 +0,0 @@
import os
import time
import logging
import subprocess
from pyrocko.guts import Bool, Float
from .plugin import Plugin, PluginConfig, register_plugin
from .meta import Path, DataSize
logger = logging.getLogger(__name__)

0
test/test_conversion.py

42
test/test_convert.py

@ -0,0 +1,42 @@
import numpy as num
import pytest
from hypothesis import given
from hypothesis.strategies import floats, integers
from hypothesis.extra.numpy import arrays
from pyrocko.trace import Trace
from idas_convert.idas_convert import split, process_data
NSAMPLES = 1000
@given(arrays(num.int32, 1000, elements=integers(-1000, 1000)))
def test_split(data):
tr = Trace(
ydata=data,
deltat=0.01)
tmin_half = (tr.tmax - tr.tmin) / 2
traces = split(tr, tmin_half)
assert len(traces) == 2
assert sum(t.ydata.size for t in traces) == tr.ydata.size
assert traces[0].tmin == tr.tmin
assert traces[1].tmax == tr.tmax
@given(arrays(num.int32, 10000, elements=integers(-1000, 1000)))
def test_process_data(data):
tr = Trace(
ydata=data,
deltat=0.001)
tmin = tr.tmin + .1
tmax = tr.tmax - .1
deltat = 0.005
chunk = (tr, deltat, tmin, tmax)
ptr = process_data(chunk)
assert ptr.deltat == deltat

17
test/test_various.py

@ -0,0 +1,17 @@
from pyrocko.guts import Object
from idas_convert.meta import DataSize
from idas_convert.plugin import Plugin
def test_data_size():
class Test(Object):
data_size = DataSize.T()
t = Test(data_size=10000)
assert t.data_size == 10000
def test_plugin_base():
p = Plugin()
p.set_parent('123')
Loading…
Cancel
Save