User Tools

Site Tools


31.07.2024

<html> <div style=“font-size:xx-large; font-weight: 900;”>31.07.2024 Mittwoch</div> </html>

Kalender, Juli, KW31

Appointments

  • None

Health

  • - Exercises
    • - Rücken
      • - Morgens
      • - Abends
    • - Augen
      • - Morgens
      • - Abends
    • - Zunge
      • - Morgens
      • - Abends
    • - Pr
      • - Morgens
      • - Abends

Tasks for the day

    • - find out what “RPM” (by Toni Robbins) is
      • our day should look like:
        • Results-oriented,
        • purpose-driven,
        • built around a “massive action plan.”
      • You have to consistently ask yourself what you want, what your purpose is, and what you need to do to achieve it.
    • - extend Rdconverter user doc
      • - extend git info
      • - …
    • - create / change following projects because of new vars / new PLS
      • + Stuttgart (389 vars)
      • + Baden-Baden
      • + Freudenstadt (21 vars)
      • + Friedrichshafen Pannach (11 vars)
      • + Nürtingen Acron (81 vars)
      • + Winnenden (76 vars)
        • S:\Datenmanagement\Datendownload\Winnenden
          • Daten nur aus dem Ordner “Protokolle” nehmen
        • - don't convert invalid data (?)
      • - Eutingen im Gäu (20 vars)
        • S:\Datenmanagement\Datendownload\Eutingen im Gäu
          • Daten für 2022 + 2023 + 2024 (bis 23.07.2024 00:00)
        • Notes (ask Annette?):
          • some data does not contain “last line” / contains “too much”
            • 2022 = “01.01.2023;…” → line too much?
            • 2023 = “01.01.2024;…” → line too much?
            • 2027 = “23.07.2024;…” → sometimes not contained
  • - write a “User Guide” for rdconverter
  • - make a small “Project Developer Guide”
  • - Clean up
    • - nuertingen
      • - use “data/in” as input directory
      • - use “vars” directory for configs
    • - nagold
      • - use “data/in” as input directory (make use of recursion!)
      • - check what can be done additionally
  • - extend documentation
    • - CsvPreprocessor
    • - implement Einstaugrenze
      • - Weissach_wsbe_flachtsued_both.csv
        • - add a check for “Einstaugrenze” during calculation of “isPositiveOutlier”
          • - implement isPositiveOutlierStart
            • + implement isFirstPointWithNegativeSlope
          • - implement isPositiveOutlierEnd
            • + implement isLastPointWithNegativeSlope
    • - export single point outlier correction for Freudenstadt project
      • - check corrected data month per month (full year will not work because of small memory)
        • whether valueKorr == value_korr or not
        • - wsbe_rueb3
        • - wsbe_rueb4
        • - wsbe_rueb5
        • - wsbe_rueb6
          • 1 diff:
            • my correction:
              • 2022-06-25 15:40:00;wsbe_rueb6;3,3
            • manual correction:
              • 2022-06-25 15:40:00;wsbe_rueb6;0
        • - wsbe_rueb11
        • - wsbe_zwiesel
    • - check outliers examples
      • - RÜB Flacht Süd wsbe_flachtsued 2023.csv
      • - Gomaringen_wsbe_wittel.csv
      • - Calw_wsbe_wimno.csv → only 1 outlier
      • - Gomaringen_wsbe_mittel.csv
      • - Leonberg_wszu_rueb26.csv
      • - RÜB 4 Glashüttenweg wsbe_rueb4 2022.csv
    • - calculate different maxAllowedSlope for different time intervals in GetSlopeFromOneRowUDF
      • - try to make a “matrix” instead of “only minimums”
    • - try to configure Spark interpreter memory for Freudenstadt project
      • -Xmx16g worked, but: “Output is truncated to 64048000 bytes. Learn more about ZEPPELIN_INTERPRETER_OUTPUT_LIMIT”
        • ⇒ change zeppelin.interpreter.output.limit value in zeppelin-site.xml
    • - refactor GetSlopeFromOneRowUDF
      • - move maxAllowedSlope member to MarkSlopeFromOneRowUDF class
    • - make a Transformer for “Blockdistanz” (see “1.2 Blockdistanz / Einstau Sonde”)
      • - implement HadSlopeMarkerTransformer
        • - check in current implementation
        • - implement MarkAsBetweenNegativeSlopeStartAndPositiveSlopeEndUDF
          • - test
            • - test with data of one year
            • - tests with real data
        • - clean up in HadSlopeMarkerTransformer and in all UDFs used there
          • - simplify getting/marking slopes using only
            • SLOPE_TO_PREVIOUS_COLUMN_NAME_DEFAULT
              • ⇒ getSlopeToPrevious using GetSlopeFromOneRowUDF()
            • SLOPE_TO_NEXT_COLUMN_NAME_DEFAULT
              • ⇒ getSlopeToNext using GetSlopeFromOneRowUDF()
      • - implement SlopeBasedBlankingDistanceDataTransformer
        • - implement ReplaceValuesWithinEntlastungEventUDF properly
          • ??? use gradient (as first solution) ⇒ GradientBasedBlankingDistanceDataTransformer
            • (⇒ it should be not necessary because it is already calculated in HadSlopeMarkerTransformer.BETWEEN_NEGATIVE_SLOPE_START_AND_POSITIVE_SLOPE_END_MARKER_COLUMN_NAME_DEFAULT)
            • ??? check how gradient can be calculated
              • proposal 1:
                • if (start point is near of the maxValue)
                  • if(t2-t1 < X min)
                    • if(slope > 0 && slope > maxSlope)
                      • ⇒ mark as “probableBlockingDistance”
              • proposal 2 (older?):
                • if (start point is near of the maxValue)
                  • if(there was a slope with “speed” ⇐ maxSpeed)
      • - define scenarios
        • Scenario 1. (simple)
          • runter und nach einer Weile wieder hoch
            • Beispiel: Bild 1, zweite Lücke
        • Scenario 2.
          • runter, dann hoch, aber nicht bis zum Anschlag, dann ein paar mal hoch und runter, aber jedes Mal nicht bis zum Anschlag und nur ziemlich später wieder hoch bis zum Anschlag
            • Beispiel: Bild 1, erste Lücke
        • Scenario 3
          • runter, dann relativ stabil, dann aber noch stark nach unten
            • Besipel: Bild 3
        • Scenario 4
          • ask Benjamin
            • letztes Bild im Abschnitt 1.2, letzte Lücke
              • es ist unklar wie die rechte Seite von der Lücke korrigiert wurde
      • - test all defined scenarios
    • - optimize building
      • set jar to “target” of sparktest
    • - create a new project for transformers
    • - test “1.1. Einfache Ausreißer”
      • test data is in /home/vkalaschnikow/Desktop/buero/Entwicklung/Plausibilitätsprogramm/Beispiele Messfehler Datenlücken
    • - implement a graph for our data
      • - find out which graph libs exist
      • - decide how to handle “big data problem”
        • - find out which solutions exist
        • - implement 2-3 options
    • - “data gaps analysis” → SparkTester.getStatsAboutMultipleIntervals()
      • - check statistics possibilities
      • - check reporting possibilities
  • - Anleitung für neue Mitarbeiter
  • - Acron kennenlernen
    • - Acron-Anleitung.docx
    • - Beschreibung Datentransfer.docx
    • Projekte zur Einarbeitung
      • SES
      • Sindelfingen Q3 2023
      • Aalen Q3 2023
  • …
  • …
  • …
    • - Entry Part 5 into my wiki
    • - Summarize Part 6

Task pool

  • - make a plan for Avril
    • - development
    • - sales and marketing

Goals for the week

31.07.2024.txt · Last modified: by kalaschnikow