At a glance
| Supported data platforms | Anything Spark can load: files, tables and databases on any Spark cluster |
| Anomaly detection | Yes, on stored metrics over time |
| Freshness and volume monitoring | Volume through size metrics and anomaly checks; freshness only through checks you write |
| Schema checks | Not documented as a schema change monitor; column-level checks only |
| Validation rules | Scala or Python checks in a VerificationSuite, or DQDL |
| Lineage and root cause | Not provided |
| Alerting | Not built in; act on results in your job or orchestrator |
| Deployment | Library in your Spark jobs (Scala or Java on the JVM, Python via PyDeequ) |
| Licensing | Apache 2.0, free |
Deequ suits data engineers who run Spark and want data quality checks as code inside their jobs, at a scale where pandas-based tools do not fit. It is a library, not a platform: you get checks, metrics and anomaly detection on metric history, but no UI, scheduling or alerting, and you must match the release to your Spark and Java versions.
Overview
Deequ is an open source library from AWS Labs, written in Scala on Apache Spark, for measuring data quality on large datasets. You declare checks in a VerificationSuite, such as completeness, uniqueness, value ranges, containment or quantiles, and Deequ computes the metrics in Spark and reports which constraints failed. It can also profile columns, suggest constraints from the data, store metrics in a repository, compute metrics incrementally on growing or partitioned data and run anomaly detection on metric history. It also supports DQDL, the rules language used by AWS Glue Data Quality, with row-level results for some rules.
Deequ 2.x targets Spark 3.1 to 3.5 (Spark 3.0 and earlier need Deequ 1.x), with artifacts named by Spark version, and 2.1.0 requires Java 11. PyDeequ, a separate Python wrapper, needs the Deequ JAR on the Spark session; PyDeequ 1.7.0 and later support Spark 3.5 and 4.1, and 1.6.0 is the last release for Spark 3.1 to 3.4. AWS states that its managed AWS Glue Data Quality is built on Deequ.
Key features
- Declarative constraint checks on Spark DataFrames: completeness, uniqueness, ranges, containment, quantiles and more
- Column profiling and automatic constraint suggestion for large datasets
- Metrics repository to persist and query computed metrics over time
- Anomaly detection on metric history, such as sudden changes in row counts
- Incremental metric computation on growing and partitioned data
- DQDL support, with row-level results for rules such as IsComplete, IsUnique and ColumnValues
- PyDeequ Python interface for PySpark users
Integrations and ecosystem
Pros and cons
Pros
- Free and Apache 2.0, maintained under AWS Labs
- Runs inside Spark, so it scales to very large datasets
- Metrics repository and anomaly detection go beyond pass or fail checks
- Shares DQDL with AWS Glue Data Quality, so rules can move between them
Cons
- No UI, scheduler or alerting: everything is code in your jobs
- Releases are tied to Spark and Java versions (Java 11 from 2.1.0), which complicates upgrades
- PyDeequ needs the Deequ JAR on the Spark session and supports fewer Spark versions per release
Pricing
Deequ and PyDeequ are free open source libraries under the Apache 2.0 licence. You pay only for the Spark compute you run them on. AWS offers a managed service built on Deequ, AWS Glue Data Quality, which is billed separately in AWS Glue DPU-hours.
Pricing checked on the vendor's official page on 7 November 2026. Prices change; confirm before buying.
Who it is best for
Data engineers running Spark pipelines who want scalable, code-based data quality checks and metric history without a commercial platform.
Typical use cases
- Validating completeness and uniqueness of a daily data lake partition before publishing it
- Tracking row counts and null rates over time and flagging sudden changes with anomaly detection
- Profiling a new large dataset and generating suggested constraints as a starting point
- Running the same DQDL rules in your own Spark jobs and in AWS Glue Data Quality
Who should look elsewhere
- Teams that work in SQL inside a warehouse: see dbt data tests or Soda.
- Teams that want a managed experience on AWS: see AWS Glue Data Quality.
Verdict
Deequ is a solid, free choice for data quality at Spark scale, with metric history and anomaly detection that simple assertion libraries lack. It assumes Spark and JVM skills and gives you no interface or alerting, so it fits engineering teams that will wire results into their own pipelines, or that want an open path alongside AWS Glue Data Quality.
Frequently asked questions
Is Deequ free?
Yes. Deequ and PyDeequ are open source under the Apache 2.0 licence. You pay only for the Spark compute you run them on.
Which Spark versions does Deequ support?
Deequ 2.x supports Spark 3.1 to 3.5 with version-specific artifacts, and Spark 3.0 or earlier needs Deequ 1.x. PyDeequ 1.7.0 and later support Spark 3.5 and 4.1. Deequ 2.1.0 requires Java 11.
How is Deequ related to AWS Glue Data Quality?
AWS states that AWS Glue Data Quality is built on Deequ, and both use the DQDL rules language. Glue Data Quality adds a managed, serverless service with recommendations, scheduling and EventBridge integration.
Sources
- Deequ repository and README (GitHub)
- Deequ releases (GitHub)
- PyDeequ repository (GitHub)
- AWS Glue Data Quality (built on Deequ)
Checked November 2026.
Listed in: Data Quality & Observability.
How we research reviews and what star ratings mean: our editorial method.