====== Open Source Data Quality Tools ====== ===== All ===== * [[Apache Spark]] (Java framework) * [[Apache NiFi]] * [[Apache Zeppelin]] * volume location: ''\\wsl.localhost\docker-desktop-data\data\docker\volumes\z-vol'' * starting docker with mount: * ''docker run -p 8080:8080 --rm --name zeppelin --mount source=z-vol,target=/z-vol apache/zeppelin:0.11.0'' * https://kontext.tech/article/243/install-zeppelin-073-in-windows * [[https://rdm-software.pages.ufz.de/saqc/|SaQC]] -> Python framework + command line tools * [[OpenRefine Test|OpenRefine]] ===== Checked ===== ==== Maybe ==== These tools may be good, but not really suitable for our situation now. So we will not use them at the moment: * [[Apache Spark]] (Framework) * [[https://spark.apache.org/|Apache Spark]] * https://github.com/apache/spark * Installation on my computer: ''C:\v\pgm\spark-3.5.0-bin-hadoop3'' * [[Apache Spark]] * Docs * https://spark.apache.org/docs/latest/ * https://spark.apache.org/docs/latest/quick-start.html * Articles * https://www.toptal.com/spark/introduction-to-apache-spark * [[Apache NiFi]] * [[https://nifi.apache.org/|Apache NiFi]] * https://github.com/apache/nifi * Local installation under: * ''C:\v\pgm\nifi-2.0.0-M2'' * Start with ''C:\v\pgm\nifi-2.0.0-M2\bin\run-nifi.bat'' * After that open https://localhost:8443/nifi/ * User / Pass: admin / adminadminadmin * Old: * Generated Username [41504541-3493-405f-94f8-0b0102b7be26] * Generated Password [Ac9XWFj0+4chb+1x3tLySMvbyZxZqRtm] * Docs * https://nifi.apache.org/documentation/v2/ * [[https://rdm-software.pages.ufz.de/saqc/|SaQC]] -> Python framework + command line tools * Repo: https://git.ufz.de/rdm-software/saqc * Installation on my computer: ''...'' * Docs * https://rdm-software.pages.ufz.de/saqc/cookbooks/CookBooksPanels.html * Conclusion * Good: * Good documentaion * Supported by Helmholz institute * Not so good: * Python * not very big community * OpenRefine * https://openrefine.org/ * https://github.com/OpenRefine/OpenRefine * [[https://openrefine.org/docs|OpenRefine User Manual]] * https://github.com/OpenRefine/OpenRefine/issues/5315 * Demo bei mir unter ''C:\v\pgm\openrefine-3.7.7'' * [[OpenRefine Test]] * Conclusion: * Good: * "grosse Lösung" * gut dokumentiert * big community * Not good: * not good to program with built in tools * not very nice UI ==== Won't use ==== * Data Cleaner Community Edition * https://datacleaner.github.io/ * Demo bei mir auf dem Rechner unter ''C:\v\pgm\DataCleaner'' * [[DataCleaner Test]] * Features: * "kleine Lösung" * gut dokumentiert * Fazit: * für unsere Zwecke nicht so gut, da * die Anwendung wohl nicht mehr leben wird, da der Hersteller aufgekauft wurde(?) und jetzt eine andere (nicht Open Source) Lösung entwickelt * [[https://opendatadiscovery.org/|Open Data Discovery]] * https://github.com/opendatadiscovery/odd-platform * https://demo.oddp.io/ * Features: * "grosse Lösung" * web-basiert * dokumentiert, aber nicht besonders gut (nicht vollständig) * Demo vorhanden -> https://demo.oddp.io/ * Fazit: * für unsere Zwecke nicht so gut, da * die eigentlichen Tests in anderen Frameworks gemacht werden sollen * z.B. Great Expectations oder Pandas... (beide Python?) * die Community nicht so gross ist * die Dokumentation nicht so gut ist * [[https://github.com/dqops/dqo|DQOps]] * [[https://dqops.com/docs/|DQOps Docs]] * Features: * gross * sogar mit https://open-metadata.org vergleichbar (sogar manchmal besser, da mehr auf Data Quality etc zugeschnitten ist) * Python-basiert * Fazit: * für unsere Zwecke nicht so gut, da * kleine Community * vieles basiert auf Python (obwohl 70% des Codes angeblich in Java geschrieben) * Apache Griffin * https://griffin.apache.org/ * https://github.com/apache/griffin/tree/master * https://github.com/apache/griffin/blob/master/griffin-doc/ui/user-guide.md * Features: * populär * nicht besonders gut dokumentiert * veraltet? (letztes Release 0.6.0 aus November 2020) * Fazit: * für unsere Zwecke nicht so gut, da * veraltet ==== Not yet checked ==== * https://open-metadata.org/ * https://github.com/open-metadata/OpenMetadata * Docs: * https://docs.open-metadata.org/v1.3.x * https://docs.open-metadata.org/v1.3.x/connectors/ingestion/workflows/data-quality/custom-tests * Features: * gross * web-basiert * Java / Typescript / Python * Open Source Data Quality and Profiling * https://sourceforge.net/projects/dataquality/ * seems to be outdated (last update at 2021) * Great Expectations * https://github.com/great-expectations/great_expectations * Features: * "grosse Lösung" * Python-basiert * mobydq * https://github.com/ubisoft/mobydq * https://ubisoft.github.io/mobydq/pages/demo/ * veraltet? (last update 2 years ago) * [[https://github.com/awslabs/deequ/tree/master|Deequ]] (library) * Deequ is a (Scala) //library// built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets. (Built by Amazon.) * https://www.luminis.eu/blog/data-quality-testing-with-deequ-in-spark/ * https://github.com/tsegall/fta * Metadata/data identification Java library. Identifies Base Type (e.g. Boolean, Double, Long, String, LocalDate, LocalTime, ...) and Semantic Type (e.g. Gender, Age, Color, Country, ...). Extensive country/language support. Extensible via user-defined plugins. Comprehensive Profiling support. * Java * Check these (?): * Apache Flink * Apache Beam * Spring Cloud Data Flow * Some links * https://sourceforge.net/directory/data-quality/java/ * https://github.com/WeBankFinTech/Qualitis * https://github.com/topics/data-profiling * https://github.com/cleanlab/cleanlab * https://www.datastackhub.com/top-tools/open-source-data-profiling-tools/ * https://www.datastackhub.com/top-tools/open-source-data-cleaning-tools/