Performs the test for unique values in one or more columns of a DataFrame.

Parameters:
  • data_frame (DataFrame) –

    Input DataFrame to perform Data Quality checks on.

  • primary_key_columns (list) –

    List of primary key column names.
Returns:
  • bool( bool ) –

    True if checks pass, False otherwise.
Raises:
  • Exception –

    If Data Quality checks fail.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, StringType
>>> spark = SparkSession.builder.getOrCreate()
>>> data = [
...     ("123.456.789-00", "Maria Silva", "maria@example.com"),
...     ("123.456.789-00", "Joao Souza", "joao.souza@example.com"),
...     (None, "Carlos Pereira", "carlos.pereira@example.com")
... ]
>>> schema = StructType([
...     StructField("cpf", StringType(), True),
...     StructField("full_name", StringType(), True),
...     StructField("email", StringType(), True)
... ])
>>> df = spark.createDataFrame(data, schema)
>>> primary_key_columns = ["cpf"]
>>> try:
...     result = uniqueness_test(df, primary_key_columns)
... except Exception as e:
...     print(e)
Data Quality Check Failed: Found 1 duplicate records!

Logs:

  • ERROR: Data Quality Check Failed: Found 1 duplicate records!

  • INFO: Data Quality Check: Successfully processed duplicate record test. # noqa : E501