Validates if a column in a DataFrame contains only accepted values.

Parameters:
  • dataframe (DataFrame) –

    Input DataFrame.

  • column_name (str) –

    Name of the column to validate.

  • valid_values (list) –

    List of valid values for the column.
Returns:
  • bool( bool ) –

    True if all values in the column are valid, false otherwise.
Raises:
  • ValueError –

    If the specified column does not exist in the DataFrame.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, StringType
>>> spark = SparkSession.builder.getOrCreate()
>>> schema = StructType([
...     StructField("id", StringType(), True),
...     StructField("value", StringType(), True)
... ])
>>> data = [("A", "PT"), ("B", "EN"), ("C", "PT")]
>>> df = spark.createDataFrame(data, schema)
>>> valid_values = ["FR", "EN"]
>>> try:
...     result = accepted_values_test(df, "value", valid_values)
... except ValueError as e:
...     print(e)
Invalid values found in column 'value': ['PT']

Logs:

  • INFO: Invalid values found in column 'value': ['value']

  • INFO: Data Quality Check: Successfully processed freshness test.