Performs the freshness (timeliness) test on a DataFrame. Ensures that the data in your DataFrame is recent or within the expected date.

Parameters:	dataframe (DataFrame) – Input DataFrame. updated_at_column (str) – Name of the updated at column in the DataFrame. warn_after (int) – Threshold in days after which to warn about data staleness. error_after (int) – Threshold in days after which to error about data staleness.

Returns:	bool( bool ) – True if checks pass, False otherwise.

Raises:	Exception – If Data Quality checks fail.

Examples:

>>> from pyspark.sql import SparkSession

>>> from pyspark.sql.functions import to_timestamp

>>> from pyspark.sql.types import StructType, StructField, StringType

>>> spark = SparkSession.builder.getOrCreate()

>>> schema = StructType([
...     StructField("id", StringType(), True),
...     StructField("timestamp", StringType(), True)
... ])

>>> data = [
...     ('A', '2024-07-26 19:31:39'),
...     ('B', '2024-07-26 19:31:39'),
...     ('C', '2024-07-26 19:31:39')
... ]

>>> df = spark.createDataFrame(data, schema)

>>> df = df.withColumn(
...     "timestamp",
...     to_timestamp("timestamp", "yyyy-MM-dd HH:mm:ss")
... )

>>> try:
...     freshness_test(df, "timestamp", 2, 5)
... except Exception as e:
...     print(f"Exception: {e}")
Exception: Data Quality Check Failed: Data is older than 5 days!

Logs:

• ERROR: If data is older than error_after days.

• WARNING: If data is older than warn_after days.

• INFO: Data Quality Check: Successfully processed freshness test.