Validates that the values in a numeric column fall within a specified range.

Parameters:
  • dataframe (DataFrame) –

    Input DataFrame.

  • column_name (str) –

    Name of the column to validate.

  • min_value (float) –

    Minimum acceptable value.

  • max_value (float) –

    Maximum acceptable value.
Returns:
  • bool( bool ) –

    True if all values are within the range, False otherwise.
Raises:
  • ValueError –

    If the column does not exist in the DataFrame or is not numeric.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, IntegerType
>>> spark = SparkSession.builder.getOrCreate()
>>> schema = StructType([
...     StructField("id", IntegerType(), True),
...     StructField("score", IntegerType(), True)
... ])
>>> data = [(1, 10), (2, 15), (3, 25), (4, 5)]
>>> df = spark.createDataFrame(data, schema)
>>> try:
...     range_test(df, "score", 0, 20)
... except ValueError as e:
...     print(e)
False

Logs:

  • INFO: Invalid values found in column columns: value

  • INFO: Data Quality Check: Successfully processed range test.