====== Apache Spark ======

[[https://spark.apache.org/|Apache Spark]] (Framework)

  * Framework for:
      * data profiling
      * data cleansing
      * data validation
      * data standardization
      * data enrichment

  * Installation on my computer: 
    * ''C:\v\pgm\spark-3.5.0-bin-hadoop3''

  * Code repo: 
    * https://github.com/apache/spark
  * Docs
    * https://spark.apache.org/docs/latest/
    * https://spark.apache.org/docs/latest/quick-start.html
    * https://spark.apache.org/docs/latest/sql-getting-started.html
    * https://spark.apache.org/docs/latest/sql-programming-guide.html
      * CSV -> https://spark.apache.org/docs/latest/sql-data-sources-csv.html
    * https://spark.apache.org/docs/latest/api/java/index.html -> Java API

  * Articles
    * https://ssmertin.com/articles/into-to-data-quality-with-apache-spark/
    * https://www.toptal.com/spark/introduction-to-apache-spark
    * https://www.ibm.com/blog/apache-spark-monitoring-using-listeners-and-data-quality-libraries/


  * APIs (https://spark.apache.org/docs/latest/api/java/index.html)
    * Data profiling
      * ''describe()''
      * ''summary()''
      * ...
    * Data cleansing
      * [[https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/Dataset.html#na--|na()]] - Returns a DataFrameNaFunctions for working with missing data. 
        * ''drop()''
          * remove rows
        * ''fill()''
          * replaces NULLS with ''valXYZ''
        * ''replace()''
          * replaces ''valX'' with ''valY''
        * ''dropDuplicates()''
          * removes identical rows
    * Statistics
      * ''summary()''
      * ''agg()'' -> See https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/RelationalGroupedDataset.html
        * ''min()''
        * ''max()''
        * ''avg()''
        * ''stddev()''
        * ''count()''
        * ''pivot()''
        * ''sum()''
      * [[https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/Dataset.html#stat--|stat()]]
        * List of all "stat" functions: [[https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameStatFunctions.html]]


  * Libraries built for / on top of Apache Spark
    * Deequ -> https://github.com/awslabs/deequ
      * library for defining "unit tests for data", which measure data quality in large datasets.

===== sparktest project =====

My IntelliJ project to test Spark: ''C:\v\IdeaProjects\sparktest''

  * "Outliers" test -> ''SparkTester.getInvalidData()''
  * "Diff" test -> ''SparkTester.getWithDiffColumn()''




===== Misc =====

==== Troubleshooting ====

=== ...module java.base does not export sun.nio.ch... ===

Important:

To not to have the problems like ''cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch ...'' it is necessary to add some parameters for Java VM:
<code>
--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED 
--add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED 
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED 
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED 
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
</code>
See: https://stackoverflow.com/questions/73465937/apache-spark-3-3-0-breaks-on-java-17-with-cannot-access-class-sun-nio-ch-direct