Retrieve distinct values for the provided columns and return a DataFrame with these values as lists.

Parameters:
  • df (DataFrame) –

    Input DataFrame.

  • columns (list) –

    List of column names to get distinct values for.
Returns:
  • DataFrame( DataFrame ) –

    DataFrame containing: - Column: Name of the column. - DistinctValues: List of distinct values as a comma-separated string.
Raises:
  • ValueError –

    If any of the specified columns do not exist in the DataFrame.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, StringType
>>> spark = SparkSession.builder.getOrCreate()
>>> schema = StructType([
...     StructField("Category", StringType(), True),
...     StructField("Region", StringType(), True)
... ])
>>> data = [
...         ("Electronics", "North"),
...         ("Furniture", "South"),
...         ("Electronics", "West"),
...         ("Furniture", "North")
... ]
>>> df = spark.createDataFrame(data, schema)
>>> try:
...     result_df = get_distinct_values_as_list(
...         df, ["Category", "Region"]
...     )
... except ValueError as e:
...     print(e)

Logs:

  • INFO: Processing distinct values for column Category
  • INFO: Processing distinct values for column Region
  • INFO: Data Quality Check: Successfully retrieved distinct values.