====== rdconverter ====== A [[Lieb-ib]] project to convert raw data from different formats into Acron CSV format. ===== Tasks ===== ==== ToDo ==== * **-** make a small "User Guide" * **-** extract column configuration to a separate .properties-file * **-** use multiple configuration files * "application.properties" for the application * "modulexyz.properties" for each module * **-** implement handling for all clients * **-** Bad Urach * **-** ... * **-** implement splitting of files * **-** implement unit tests ==== Done ==== * **+** start * **+** ... * **+** extend routes with possibility to read/write all files in given directory * **+** convert date/time fields * **x** call one route from another * **+** make one full route instead of two ("row -> preprocessed" and "preprocessed -> acron") * **+** implement running from different directories * **+** implement properly stopping * **+** use application class * **+** rename to RDConverterApplication * **+** change batch files to start it * **+** implement Excel format * **+** implement merging ==== Projects with customer data ==== The following file contains information about status of all projects:\\ ''S:\Projekte\RÜB-Strangbetrachtungen\Bearbeitungstand Strangbetrachtungen 2023.xlsx''\\ The projects that should be completed are marked with color in the second column (the column "B"). The following is the list of "projects" in ''S:\Datenmanagement\Datendownload''. The projects with ''x'' should not be processed (at least now). * **+** Aalen * implemented before xx.03.2024 * **+** Altensteig / Altensteig 1 / Altensteig 2 * implemented before xx.03.2024 * **+** Althengstett * implemented before xx.03.2024 * **x** AV Pfinz- und Rennachtal * **-** AV Unteres Echaztal Härten * **x** AZV Hagegarten * **x** AZV Oberes Ilmtal (Bayern) * **x** Bad Abbach * **+** Baden-Baden * **+** Bad Urach * implemented before xx.03.2024 * **x** Baiersbronn * **x** Bergisch Gladbach * **+** Besigheim * implemented before xx.03.2024 * **x** Birkenfeld * **+** Buchen * implemented before xx.03.2024 * **x** Böblingen * **x** Bönnigheim * **+** Calw * **-** Eningen u Achalm * **+** Erdmannhausen * **+** Ettlingen * implemented at xx.03.2024 * **-** Eutingen im Gäu * **+** Freiberg am Neckar * **x** Freiburg * **+** Freudenstadt * **+** Freudenstadt Dornstetten * implemented at 18.03.2024 * **-** Freudenstadt-Aach * **+** Friedrichshafen * **x** Gemmrigheim * **+** Gomaringen * implemented before xx.03.2024 * **+** Großbottwar * implemented before xx.03.2024 * **-** Gundelfingen * **+** Gundelsheim * implemented before xx.03.2024 * **+** Heidelsheim * **+** Heidenheim * implemented before xx.03.2024 * **x** Herbolzheim * **x** Hüfingen * **+** Illingen * implemented before xx.03.2024 * **x** Ingersheim * **x** KA Ittersbach * **+** Kammerforst * **x** Karlsruhe * **x** Kenzingen * **x** Kirchheim am Neckar * **x** Krauchenwies * **x** Krautheim * **+** Kressbronn * implemented before xx.03.2024 * **x** Kronau Bad Schönborn * **x** Kusterdingen * **+** Lahr * implemented before xx.03.2024 * **+** Laupheim 1 / Laupheim 2 * implemented before xx.03.2024 * **+** Leonberg * implemented before xx.03.2024 * **+** Löchgau * implemented before xx.03.2024 * **x** Marbach a. N * **-** Metzingen * **+** Mundelsheim * **+** Nagold * implemented before xx.03.2024 * **x** Neckarsulm * **x** Neuenstadt-Stein am Kocher * **x** Neuhausen ob Eck * **+** Nuertingen (09.01.2024 - stop at first (because of customer)) * **-** Convert all remaining variables * **-** ... * **-** adjust preprocessor.properties * **-** test * **-** verify * **-** spot test * **-** in Acron import * **-** Import and verify variables in Acron (because these variables are configured now) * **-** df_nackarwie * **-** rs_nackarwie * **-** wsbe_nackarwie * **-** wszu_nackarwie * **-** GDS_entlas * **-** GDS_ober1 * **-** GDS_ober2 * **-** GDS_unter * **-** GDS_weiter * **-** GF_df * **-** GF_ds * **-** GF_wsbe * **-** REH_swp1 * **-** REH_swp2 * **-** REH_wsbe1 * **-** REH_wstb1 * **+** Oberes Mettertal * implemented before xx.03.2024 * **+** Öhringen (oehringen and oehringen1) * implemented on 22.08.2024 * **x** Pforzheim-ESP * **+** Pfullingen * implemented before xx.03.2024 * **x** Rottweil * **+** Sachsenheim * implemented before xx.03.2024 * **x** Schönmünzach * **x** Schützingen * **+** SES Stuttgart * **+** Sindelfingen * implemented at 14.03.2024 * **x** Stein am Kocher * **x** Stockach * **+** Vaihingen * implemented before xx.03.2024 * **+** Waghäusel / WaghäuselFlowChief * **x** Waldachtal * **x** Waldenburg * **+** Weingarten-AV Am Walzbach * implemented before xx.03.2024 * **+** Weissach * implemented at 19.03.2024 * **+** Winnenden * implemented at 19.08.2024 ===== Reading Excel data ===== * Library : * https://github.com/ozlerhakan/poiji * Some articles: * https://stackoverflow.com/questions/38946437/apache-camel-split-json-array-and-call-post-to-the-rest-service-iteratively * https://www.baeldung.com/java-microsoft-excel * https://erpsolutions.oodles.io/developer-blogs/Store-data-from-excel-file-to-database-using-Apache-camel/ * Also: * [[https://poi.apache.org/|Apache POI - the Java API for Microsoft Documents]] * https://poi.apache.org/components/spreadsheet/quick-guide.html ===== Splitting messages ===== * Documentation: * https://camel.apache.org/components/4.0.x/eips/split-eip.html * Examples: * ''org.apache.camel.example.cafe.CafeRouteBuilder'' ===== Input data ===== The input data is mostly in the subfolders of ''S:\Datenmanagement\Datendownload\''. For example the data from year 2023 for the project ''waghaeusel'' is in : * ''S:\Datenmanagement\Datendownload\Waghäusel\2023\'' ===== Output data ===== The output data is mostly in the subfolders of ''S:\Projekte\Acron-Projekte\Importdaten\''. For example the data from year 2023 for the project ''waghaeusel'' is in : * ''S:\Projekte\Acron-Projekte\Importdaten\Waghäusel\2023'' **BUT :** * explicitly in the case of "Waghäusel" it is not so simple... There are 2 projects: ''waghaeusel'' and ''waghaeuselFlowChief''. But it is not very difficult to guess the name of the subfolder with the data for every project :) Normally the name of the project is enough to find the data. ===== Example data ===== * Multiple values in one file * S:\Datenmanagement\Datendownload\Metzingen\2021 15-Minutenwerte\RÜB Freibad 2021.csv ===== Making release ===== There is a Maven script for this. But... it does not really work... (=> make it work with the next release!) Dry run: mvn release:prepare -DdryRun Do it: mvn release:perform See also: https://maven.apache.org/guides/mini/guide-releasing.html * **-** make the x.y release * **-** commit * **-** tag * **-** make a package for S:\ and copy it there (at the moment to ''S:\tmp\rdconverter'') * **-** start with x.(y+1)_SNAPSHOT