Differences

This shows you the differences between two versions of the page.

--- rdconverter [2024/07/02 10:33] – [Input data] kalaschnikow
+++ rdconverter [2024/08/22 09:07] (current) – [Projects with customer data] kalaschnikow
@@ Line 1: / Line 1: @@
+====== rdconverter ======
+A [[Lieb-ib]] project to convert raw data from different formats into Acron CSV format.
+===== Tasks =====
+==== ToDo ====
+  * **-** make a small "User Guide"
+  * **-** extract column configuration to a separate .properties-file
+  * **-** use multiple configuration files
+    * "application.properties" for the application
+    * "modulexyz.properties" for each module
+  * **-** implement handling for all clients
+    * **-** Bad Urach
+    * **-** ...
+  * **-** implement splitting of files
+  * **-** implement unit tests
+==== Done ====
+  * **+** start
+  * **+** ...
+  * **+** extend routes with possibility to read/write all files in given directory
+  * **+** convert date/time fields
+  * **x** call one route from another
+  * **+** make one full route instead of two ("row -> preprocessed" and "preprocessed -> acron")
+  * **+** implement running from different directories
+  * **+** implement properly stopping
+    * **+** use application class
+      * **+** rename to RDConverterApplication
+      * **+** change batch files to start it
+  * **+** implement Excel format
+  * **+** implement merging
+==== Projects with customer data ====
+The following file contains information about status of all projects:\\
+''S:\Projekte\RÜB-Strangbetrachtungen\Bearbeitungstand Strangbetrachtungen 2023.xlsx''\\
+The projects that should be completed are marked with color in the second column (the column "B").
+The following is the list of "projects" in ''S:\Datenmanagement\Datendownload''. The projects with ''x'' should not be processed (at least now).
+  * **+** Aalen
+    * implemented before xx.03.2024
+  * **+** Altensteig / Altensteig 1 / Altensteig 2
+    * implemented before xx.03.2024
+  * **+** Althengstett
+    * implemented before xx.03.2024
+  * **x** AV Pfinz- und Rennachtal
+  * **-** AV Unteres Echaztal Härten
+  * **x** AZV Hagegarten
+  * **x** AZV Oberes Ilmtal (Bayern)
+  * **x** Bad Abbach
+  * **+** Baden-Baden
+  * **+** Bad Urach
+    * implemented before xx.03.2024
+  * **x** Baiersbronn
+  * **x** Bergisch Gladbach
+  * **+** Besigheim
+    * implemented before xx.03.2024
+  * **x** Birkenfeld
+  * **+** Buchen
+    * implemented before xx.03.2024
+  * **x** Böblingen
+  * **x** Bönnigheim
+  * **+** Calw
+  * **-** Eningen u Achalm
+  * **+** Erdmannhausen
+  * **+** Ettlingen
+    * implemented at xx.03.2024
+  * **-** Eutingen im Gäu
+  * **+** Freiberg am Neckar
+  * **x** Freiburg
+  * **+** Freudenstadt
+  * **+** Freudenstadt Dornstetten
+    * implemented at 18.03.2024
+  * **-** Freudenstadt-Aach
+  * **+** Friedrichshafen
+  * **x** Gemmrigheim
+  * **+** Gomaringen
+    * implemented before xx.03.2024
+  * **+** Großbottwar
+    * implemented before xx.03.2024
+  * **-** Gundelfingen
+  * **+** Gundelsheim
+    * implemented before xx.03.2024
+  * **+** Heidelsheim
+  * **+** Heidenheim
+    * implemented before xx.03.2024
+  * **x** Herbolzheim
+  * **x** Hüfingen
+  * **+** Illingen
+    * implemented before xx.03.2024
+  * **x** Ingersheim
+  * **x** KA Ittersbach
+  * **+** Kammerforst
+  * **x** Karlsruhe
+  * **x** Kenzingen
+  * **x** Kirchheim am Neckar
+  * **x** Krauchenwies
+  * **x** Krautheim
+  * **+** Kressbronn
+    * implemented before xx.03.2024
+  * **x** Kronau Bad Schönborn
+  * **x** Kusterdingen
+  * **+** Lahr
+    * implemented before xx.03.2024
+  * **+** Laupheim 1 / Laupheim 2
+    * implemented before xx.03.2024
+  * **+** Leonberg
+    * implemented before xx.03.2024
+  * **+** Löchgau
+    * implemented before xx.03.2024
+  * **x** Marbach a. N
+  * **-** Metzingen
+  * **+** Mundelsheim
+  * **+** Nagold
+    * implemented before xx.03.2024
+  * **x** Neckarsulm
+  * **x** Neuenstadt-Stein am Kocher
+  * **x** Neuhausen ob Eck
+  * **+** Nuertingen (09.01.2024 - stop at first (because of customer))
+    * **-** Convert all remaining variables
+      * **-** ...
+        * **-** adjust preprocessor.properties
+        * **-** test
+        * **-** verify
+          * **-** spot test
+          * **-** in Acron import
+    * **-** Import and verify variables in Acron (because these variables are configured now)
+      * **-** df_nackarwie
+      * **-** rs_nackarwie
+      * **-** wsbe_nackarwie
+      * **-** wszu_nackarwie
+      * **-** GDS_entlas
+      * **-** GDS_ober1
+      * **-** GDS_ober2
+      * **-** GDS_unter
+      * **-** GDS_weiter
+      * **-** GF_df
+      * **-** GF_ds
+      * **-** GF_wsbe
+      * **-** REH_swp1
+      * **-** REH_swp2
+      * **-** REH_wsbe1
+      * **-** REH_wstb1
+  * **+** Oberes Mettertal
+    * implemented before xx.03.2024
+  * **+** Öhringen (oehringen and oehringen1)
+    * implemented on 22.08.2024
+  * **x** Pforzheim-ESP
+  * **+** Pfullingen
+    * implemented before xx.03.2024
+  * **x** Rottweil
+  * **+** Sachsenheim
+    * implemented before xx.03.2024
+  * **x** Schönmünzach
+  * **x** Schützingen
+  * **+** SES Stuttgart
+  * **+** Sindelfingen
+    * implemented at 14.03.2024
+  * **x** Stein am Kocher
+  * **x** Stockach
+  * **+** Vaihingen
+    * implemented before xx.03.2024
+  * **+** Waghäusel / WaghäuselFlowChief
+  * **x** Waldachtal
+  * **x** Waldenburg
+  * **+** Weingarten-AV Am Walzbach
+    * implemented before xx.03.2024
+  * **+** Weissach
+    * implemented at 19.03.2024
+  * **+** Winnenden
+    * implemented at 19.08.2024
+===== Reading Excel data =====
+  * Library :
+    * https://github.com/ozlerhakan/poiji
+  * Some articles:
+    * https://stackoverflow.com/questions/38946437/apache-camel-split-json-array-and-call-post-to-the-rest-service-iteratively
+    * https://www.baeldung.com/java-microsoft-excel
+    * https://erpsolutions.oodles.io/developer-blogs/Store-data-from-excel-file-to-database-using-Apache-camel/
+  * Also:
+    * [[https://poi.apache.org/|Apache POI - the Java API for Microsoft Documents]]
+      * https://poi.apache.org/components/spreadsheet/quick-guide.html
+===== Splitting messages =====
+  * Documentation:
+    * https://camel.apache.org/components/4.0.x/eips/split-eip.html
+  * Examples:
+    * ''org.apache.camel.example.cafe.CafeRouteBuilder''
+===== Input data =====
+The input data is mostly in the subfolders of ''S:\Datenmanagement\Datendownload\<project_name>''.
+For example the data from year 2023 for the project ''waghaeusel'' is in :
+  * ''S:\Datenmanagement\Datendownload\Waghäusel\2023\''
+===== Output data =====
+The output data is mostly in the subfolders of ''S:\Projekte\Acron-Projekte\Importdaten\<project_name>''.
+For example the data from year 2023 for the project ''waghaeusel'' is in :
+  * ''S:\Projekte\Acron-Projekte\Importdaten\Waghäusel\2023''
+**BUT :**
+  * explicitly in the case of "Waghäusel" it is not so simple... There are 2 projects: ''waghaeusel'' and ''waghaeuselFlowChief''. But it is not very difficult to guess the name of the subfolder with the data for every project :) Normally the name of the project is enough to find the data.
+===== Example data =====
+  * Multiple values in one file
+    * S:\Datenmanagement\Datendownload\Metzingen\2021 15-Minutenwerte\RÜB Freibad 2021.csv
+===== Making release =====
+There is a Maven script for this. But... it does not really work... (=> make it work with the next release!)
+Dry run:
+<code>
+mvn release:prepare -DdryRun
+</code>
+Do it:
+<code>
+mvn release:perform
+</code>
+See also: https://maven.apache.org/guides/mini/guide-releasing.html
+  * **-** make the x.y release
+    * **-** commit
+    * **-** tag
+    * **-** make a package for S:\ and copy it there (at the moment to ''S:\tmp\rdconverter'')
+    * **-** start with x.(y+1)_SNAPSHOT