rdconverter

A Lieb-ib project to convert raw data from different formats into Acron CSV format.

Tasks

ToDo

- make a small “User Guide”
- extract column configuration to a separate .properties-file
- use multiple configuration files
- “application.properties” for the application
- “modulexyz.properties” for each module
- implement handling for all clients
- - Bad Urach
- - …
- implement splitting of files
- implement unit tests

Done

+ start
+ …
+ extend routes with possibility to read/write all files in given directory
+ convert date/time fields
x call one route from another
+ make one full route instead of two (“row → preprocessed” and “preprocessed → acron”)
+ implement running from different directories
+ implement properly stopping
- + use application class
  - + rename to RDConverterApplication
  - + change batch files to start it
+ implement Excel format
+ implement merging

Projects with customer data

The following file contains information about status of all projects:
S:\Projekte\RÜB-Strangbetrachtungen\Bearbeitungstand Strangbetrachtungen 2023.xlsx
The projects that should be completed are marked with color in the second column (the column “B”).

The following is the list of “projects” in S:\Datenmanagement\Datendownload. The projects with x should not be processed (at least now).

+ Aalen
- implemented before xx.03.2024
+ Altensteig / Altensteig 1 / Altensteig 2
- implemented before xx.03.2024
+ Althengstett
- implemented before xx.03.2024
x AV Pfinz- und Rennachtal
- AV Unteres Echaztal Härten
x AZV Hagegarten
x AZV Oberes Ilmtal (Bayern)
x Bad Abbach
+ Baden-Baden
+ Bad Urach
- implemented before xx.03.2024
x Baiersbronn
x Bergisch Gladbach
+ Besigheim
- implemented before xx.03.2024
x Birkenfeld
+ Buchen
- implemented before xx.03.2024
x Böblingen
x Bönnigheim
+ Calw
- Eningen u Achalm
+ Erdmannhausen
+ Ettlingen
- implemented at xx.03.2024
- Eutingen im Gäu
+ Freiberg am Neckar
x Freiburg
+ Freudenstadt
+ Freudenstadt Dornstetten
- implemented at 18.03.2024
- Freudenstadt-Aach
+ Friedrichshafen
x Gemmrigheim
+ Gomaringen
- implemented before xx.03.2024
+ Großbottwar
- implemented before xx.03.2024
- Gundelfingen
+ Gundelsheim
- implemented before xx.03.2024
+ Heidelsheim
+ Heidenheim
- implemented before xx.03.2024
x Herbolzheim
x Hüfingen
+ Illingen
- implemented before xx.03.2024
x Ingersheim
x KA Ittersbach
+ Kammerforst
x Karlsruhe
x Kenzingen
x Kirchheim am Neckar
x Krauchenwies
x Krautheim
+ Kressbronn
- implemented before xx.03.2024
x Kronau Bad Schönborn
x Kusterdingen
+ Lahr
- implemented before xx.03.2024
+ Laupheim 1 / Laupheim 2
- implemented before xx.03.2024
+ Leonberg
- implemented before xx.03.2024
+ Löchgau
- implemented before xx.03.2024
x Marbach a. N
- Metzingen
+ Mundelsheim
+ Nagold
- implemented before xx.03.2024
x Neckarsulm
x Neuenstadt-Stein am Kocher
x Neuhausen ob Eck
+ Nuertingen (09.01.2024 - stop at first (because of customer))
- - Convert all remaining variables
  - - …
    - - adjust preprocessor.properties
    - - test
    - - verify
      - - spot test
      - - in Acron import
- - Import and verify variables in Acron (because these variables are configured now)
  - - df_nackarwie
  - - rs_nackarwie
  - - wsbe_nackarwie
  - - wszu_nackarwie
  - - GDS_entlas
  - - GDS_ober1
  - - GDS_ober2
  - - GDS_unter
  - - GDS_weiter
  - - GF_df
  - - GF_ds
  - - GF_wsbe
  - - REH_swp1
  - - REH_swp2
  - - REH_wsbe1
  - - REH_wstb1
+ Oberes Mettertal
- implemented before xx.03.2024
+ Öhringen (oehringen and oehringen1)
- implemented on 22.08.2024
x Pforzheim-ESP
+ Pfullingen
- implemented before xx.03.2024
x Rottweil
+ Sachsenheim
- implemented before xx.03.2024
x Schönmünzach
x Schützingen
+ SES Stuttgart
+ Sindelfingen
- implemented at 14.03.2024
x Stein am Kocher
x Stockach
+ Vaihingen
- implemented before xx.03.2024
+ Waghäusel / WaghäuselFlowChief
x Waldachtal
x Waldenburg
+ Weingarten-AV Am Walzbach
- implemented before xx.03.2024
+ Weissach
- implemented at 19.03.2024
+ Winnenden
- implemented at 19.08.2024

Reading Excel data

Library :
- https://github.com/ozlerhakan/poiji

Some articles:

Also:
- Apache POI - the Java API for Microsoft Documents
  - https://poi.apache.org/components/spreadsheet/quick-guide.html

Splitting messages

Documentation:
- https://camel.apache.org/components/4.0.x/eips/split-eip.html

Examples:
- org.apache.camel.example.cafe.CafeRouteBuilder

Input data

The input data is mostly in the subfolders of S:\Datenmanagement\Datendownload\<project_name>.

For example the data from year 2023 for the project waghaeusel is in :

S:\Datenmanagement\Datendownload\Waghäusel\2023\

Output data

The output data is mostly in the subfolders of S:\Projekte\Acron-Projekte\Importdaten\<project_name>.

For example the data from year 2023 for the project waghaeusel is in :

S:\Projekte\Acron-Projekte\Importdaten\Waghäusel\2023

BUT :

explicitly in the case of “Waghäusel” it is not so simple… There are 2 projects: waghaeusel and waghaeuselFlowChief. But it is not very difficult to guess the name of the subfolder with the data for every project :) Normally the name of the project is enough to find the data.

Example data

Multiple values in one file
- S:\Datenmanagement\Datendownload\Metzingen\2021 15-Minutenwerte\RÜB Freibad 2021.csv

Making release

There is a Maven script for this. But… it does not really work… (⇒ make it work with the next release!)

Dry run:

mvn release:prepare -DdryRun

Do it:

mvn release:perform

- make the x.y release
- - commit
- - tag
- - make a package for S:\ and copy it there (at the moment to S:\tmp\rdconverter)
- - start with x.(y+1)_SNAPSHOT

vwiki

Table of Contents