Performs the relationship test (referential integrity) between two DataFrames based on a foreign key of the first DataFrame and the primary key of the target DataFrame.

Parameters:
  • foreign_dataframe (DataFrame) –

    Input DataFrame that foreign key will be checked for referential integrity.

  • target_dataframe (DataFrame) –

    Input DataFrame against which your primary key will be checked for referential integrity.

  • df_foreign_key (str) –

    Foreign key of the first DataFrame.

  • target_dataframe_key (str) –

    Primary key of the second DataFrame.
Returns:
  • bool( bool ) –

    True if checks pass, False otherwise.
Raises:
  • Exception –

    If Data Quality checks fail.

Examples:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import ( # noqa : F401
...     StructType,
...     StructField,
...     IntegerType,
...     StringType,
... )
>>> spark = SparkSession.builder.getOrCreate()
>>> foreign_data = [
...     (1, "Record1"),
...     (2, "Record2"),
...     (3, "Record3"),
...     (4, "Record4")
... ]
>>> foreign_columns = ["id", "value"]
>>> target_data = [
...     (1, "Target1"),
...     (2, "Target2"),
...     (3, "Target3")
... ]
>>> target_columns = ["id_target", "target_value"]
>>> foreign_df = spark.createDataFrame(foreign_data, foreign_columns)
>>> target_df = spark.createDataFrame(target_data, target_columns)
>>> try:
...     result = relationship_test(
...         foreign_df,
...         target_df,
...         'id',
...         'id_target'
...     )
... except Exception as e:
...     print(e)
Data Quality Check Failed: Found 1 unmatched records!

Logs:

  • ERROR: Data Quality Check Failed: Found 1 unmatched records!

  • INFO: Data Quality Check: Successfully processed relationship test.