Analyze numeric values in specified columns of a DataFrame.

This function calculates the count of negative, zero, and positive values, along with the overall average for each specified column.

Parameters:
  • df (DataFrame) –

    The input PySpark DataFrame.

  • columns (list) –

    List of column names to perform the count and average operations.
Returns:
  • DataFrame( DataFrame ) –

    A DataFrame with counts and averages for each column.
Raises:
  • ValueError –

    If any column in the list is not numeric.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, IntegerType
>>> spark = SparkSession.builder.getOrCreate()
>>> schema = StructType([
...     StructField("id", IntegerType(), True),
...     StructField("score", IntegerType(), True)
... ])
>>> data = [(1, -10), (2, 0), (3, 15), (4, -5), (5, 10)]
>>> df = spark.createDataFrame(data, schema)
>>> result_df = analyses_numeric_values(df, ["score"])

Logs:

  • INFO: Analyzing numeric values for column: score