Performs the test for non-null and non-blank values in one or more columns of a DataFrame.

Parameters:	data_frame (DataFrame) – Input DataFrame to perform Data Quality checks on. required_columns (dict) – Dictionary where keys are column names and values are "error" or "warning" indicating the action to take if the column has null or blank values.

Returns:	boolean( bool ) – True if all checks pass, False otherwise.

Raises:	Exception – If any of the checks fail.

Examples:

>>> from pyspark.sql import SparkSession

>>> from pyspark.sql.types import StructType, StructField, StringType

>>> spark = SparkSession.builder.getOrCreate()

>>> data = [
...     ("123.456.789-00", "Maria Silva", ""),
...     ("987.654.321-00", "Joao Souza", "joao.souza@example.com"),
...     (None, "Carlos Pereira", "carlos.pereira@example.com")
... ]

>>> schema = StructType([
...     StructField("cpf", StringType(), True),
...     StructField("full_name", StringType(), True),
...     StructField("email", StringType(), True)
... ])

>>> df = spark.createDataFrame(data, schema)

>>> required_columns = {
...     "cpf": "error",
...     "email": "warning"
... }

>>> try:
...     result = not_null_test(df, required_columns)
... except Exception as e:
...     print(e)
Not Null Test Failed: Columns with NULL or blank values found!

Logs:

• ERROR: Not Null Test Failed: cpf has 1 NULL or blank values!

• WARNING: Not Null Test Warning: email has 1 NULL or blank values!

• Exception: Not Null Test Failed: Columns with NULL or blank values found! # noqa : E501