Apache Spark (Framework)

• Framework for:
  • data profiling
  • data cleansing
  • data validation
  • data standardization
  • data enrichment

• Installation on my computer:
  • C:\v\pgm\spark-3.5.0-bin-hadoop3

• Code repo:
  • https://github.com/apache/spark
• Docs
  • https://spark.apache.org/docs/latest/
  • https://spark.apache.org/docs/latest/quick-start.html
  • https://spark.apache.org/docs/latest/sql-getting-started.html
  • https://spark.apache.org/docs/latest/sql-programming-guide.html
    • CSV → https://spark.apache.org/docs/latest/sql-data-sources-csv.html
  • https://spark.apache.org/docs/latest/api/java/index.html → Java API

• Articles
  • https://ssmertin.com/articles/into-to-data-quality-with-apache-spark/
  • https://www.toptal.com/spark/introduction-to-apache-spark
  • https://www.ibm.com/blog/apache-spark-monitoring-using-listeners-and-data-quality-libraries/

• APIs (https://spark.apache.org/docs/latest/api/java/index.html)
  • Data profiling
    • describe()
    • summary()
    • …
  • Data cleansing
    • na() - Returns a DataFrameNaFunctions for working with missing data.
      • drop()
        • remove rows
      • fill()
        • replaces NULLS with valXYZ
      • replace()
        • replaces valX with valY
      • dropDuplicates()
        • removes identical rows
  • Statistics
    • summary()
    • agg() → See https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/RelationalGroupedDataset.html
      • min()
      • max()
      • avg()
      • stddev()
      • count()
      • pivot()
      • sum()
    • stat()
      • List of all “stat” functions: https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameStatFunctions.html

• Libraries built for / on top of Apache Spark
  • Deequ → https://github.com/awslabs/deequ
    • library for defining “unit tests for data”, which measure data quality in large datasets.

My IntelliJ project to test Spark: C:\v\IdeaProjects\sparktest

• “Outliers” test → SparkTester.getInvalidData()
• “Diff” test → SparkTester.getWithDiffColumn()

Important:

To not to have the problems like cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch … it is necessary to add some parameters for Java VM:

--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED 
--add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED 
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED 
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED 
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED

See: https://stackoverflow.com/questions/73465937/apache-spark-3-3-0-breaks-on-java-17-with-cannot-access-class-sun-nio-ch-direct