Encrypts specified columns in a DataFrame using the provided key.

Parameters:
  • data_frame (DataFrame) –

    Input DataFrame with data to be encrypted.

  • columns_to_encrypt (list) –

    List of column names to encrypt.

  • key (bytes) –

    Encryption key to use for encryption.
Returns:
  • DataFrame( DataFrame ) –

    A DataFrame with encrypted columns.
Raises:
  • Exception –

    If any of the columns to encrypt are not found in the DataFrame.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, StringType
>>> spark = SparkSession.builder.getOrCreate()
>>> schema = StructType([
...     StructField("id", StringType(), True),
...     StructField("name", StringType(), True)
... ])
>>> data = [("1", "Alice"), ("2", "Bob")]
>>> df = spark.createDataFrame(data, schema)
>>> key = generate_encryption_key()
>>> encrypted_df = encrypt_dataframe(df, ["name"], key)

Logs:

  • ERROR: Encryption Failed: column is not present in the DataFrame!

  • INFO: DataFrame columns columns_to_encrypt encrypted successfully.