Validates that the columns in a DataFrame match specified data types.

This function checks if the columns in the provided DataFrame conform to the expected types.

It can handle basic data types (e.g., string, int, float) as well as date and timestamp formats.

Parameters:	dataframe (DataFrame) – The input DataFrame containing the data to be validated. column_specs (dict) – A dictionary where each key is a column name and each value is the expected data type. The expected type can be a string (e.g., "string", "int", "float") or a tuple for date and timestamp types (e.g.,("date", "yyyy-MM-dd"), ("timestamp", "yyyy-MM-dd HH:mm:ss"))

Returns:	bool( bool ) – Returns True if all columns are of the expected type. Returns False if any column fails the type validation, logging the details of the failure.

Raises:	ValueError – If an unsupported type is provided or if the column specified in the column_specs does not exist in the DataFrame.

Examples:

>>> from pyspark.sql import SparkSession

>>> from pyspark.sql.types import StringType, StructType, StructField

>>> spark = SparkSession.builder.getOrCreate()

>>> schema = StructType([
...     StructField("id", StringType(), True),
...     StructField("date_column", StringType(), True),
...     StructField("name", StringType(), True)
... ])

>>> data = [
...     ("123", "2023-07-15", "Alice"),
...     ("456", "2023-07-16", "Bob"),
...     ("789", "Invalid Date", "Charlie")
... ]

>>> df = spark.createDataFrame(data, schema)

>>> column_specs = {
...     "id": "string",
...     "date_column": ("date", "yyyy-MM-dd"),
...     "name": "string"
... }

>>> result = columns_type_test(df, column_specs)
>>> result
False

Logs:

• ERROR: Column 'date_column' failed date validation with format 'yyyy-MM-dd'. # noqa : E501

• INFO: Data Quality Check: Successfully validated column types.