Validates that the values in a column match a specified regular expression pattern.

Parameters:
  • dataframe (DataFrame) –

    Input DataFrame.

  • column_name (str) –

    Name of the column to validate.

  • pattern (str) –

    Regular expression pattern to match.
Returns:
  • bool( bool ) –

    True if all values match the pattern, False otherwise.
Raises:
  • ValueError –

    If the column does not exist in the DataFrame.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, StringType
>>> spark = SparkSession.builder.getOrCreate()
>>> schema = StructType([
...     StructField("id", StringType(), True),
...     StructField("email", StringType(), True)
... ])
>>> data = [
...     ("1", "test@example.com"),
...     ("2", "invalid-email"),
...     ("3", "user@domain.com")
... ]
>>> df = spark.createDataFrame(data, schema)
>>> pattern = "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
>>> try:
...     regex_pattern_test(df, "email", pattern)
... except ValueError as e:
...     print(e)
Invalid values found in 'email': ['invalid-email']
False

Logs:

  • INFO: Invalid values found in column 'email': ['invalid-email']

  • INFO: Data Quality Check: Successfully processed regex pattern test.

Pattern examples:

  • CPF: "^(?:\d{3}.\d{3}.\d{3}-\d{2}|\d{11})$" # noqa : W605

  • CNPJ: "^(?:\d{2}.\d{3}.\d{3}\/\d{4}-\d{2}|\d{14})$"

  • E-mail: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+.[a-zA-Z]{2,}$"

  • Birth date dd/mm/yyyy: "^(?:0[1-9]|[12][0-9]|3[01])/(?:0[1-9]|1[0-2])/\d{4}$"

  • Date format yyyy-mm-dd: "^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])$"

  • Brazil phone: "^(?\d{2})?\s?\d{4,5}-\d{4}$"