Validate Column

Validate columns in the DataFrame based on specified patterns.

Parameters:	`df` (`DataFrame`) – Input DataFrame to perform validation. `patterns` (`dict`) – Dictionary with column names as keys and validation patterns as values. Example: {"CustomerRoute": "^BR\d{4}$"}

Returns:	`DataFrame`( `DataFrame` ) – DataFrame with validation results containing: - Column: Name of the column. - TotalValid: Number of valid entries. - TotalInvalid: Number of invalid entries. - PercentValid: Percentage of valid entries.

Raises:	`ValueError` – If a specified column does not exist in the DataFrame.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, StringType

>>> spark = SparkSession.builder.getOrCreate()

>>> schema = StructType([
...     StructField("CustomerRoute", StringType(), True),
...     StructField("OrderID", StringType(), True)
... ])

>>> data = [
...         ("BR1234", "ORD001"),
...         ("BR5678", "ORD002"),
...         ("US9876", "ORD003")
... ]

>>> df = spark.createDataFrame(data, schema)

>>> patterns = {"CustomerRoute": "^BR\d{4}$"}

>>> try:
...     result_df = validate_column_patterns(df, patterns)
... except ValueError as e:
...     print(e)

Logs: