Perform statistical analysis on columns of the DataFrame.

Parameters:
  • df (DataFrame) –

    The input DataFrame for analysis.

  • numeric_columns (list, default: None ) –

    List of column names that are considered numeric. If not provided, all columns are treated as categorical. - For numeric columns, the function calculates the minimum and maximum values. - For non-numeric columns, it calculates the count of distinct values.
Returns:
  • DataFrame( DataFrame ) –

    DataFrame with the statistics columns: - Column: Name of the column. - DistinctCount: Count of distinct values (for non-numeric columns) - NullCount: Number of null values in the column. - NullPercentage: Percentage of null values in the column. - MinValue: Minimum value (for numeric columns). - MaxValue: Maximum value (for numeric columns).
Raises:
  • ValueError –

    If any specified numeric column does not exist in the DataFrame.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, IntegerType, StringType  # noqa : E501
>>> spark = SparkSession.builder.getOrCreate()
>>> schema = StructType([
...     StructField("id", IntegerType(), True),
...     StructField("score", IntegerType(), True),
...     StructField("category", StringType(), True)
... ])
>>> data = [(1, 10, "A"), (2, 15, "B"), (3, 25, "A"), (4, 5, "C")]
>>> df = spark.createDataFrame(data, schema)
>>> try:
...     analysis_df = standard_analyses(df, numeric_columns=["score"])
... except ValueError as e:
...     print(e)

Logs:

  • INFO: Total rows in the DataFrame: rows_number
  • INFO: Processing column id
  • INFO: Processing column score
  • INFO: Processing column category
  • INFO: Data Quality Check: Successfully processed standard analysis.