Data Fake

The Data Fake section of Phoenix Data Tools is designed to provide a versatile toolset for generating synthetic data using the Faker library in PySpark DataFrames. This functionality allows users to create realistic and diverse datasets for testing, development, and data quality assurance, ensuring that pipelines and algorithms can be robustly evaluated without exposing real, sensitive information.

Why Use Data Fake?

Data Fake is essential for simulating real-world data scenarios in a controlled and safe environment. By generating synthetic data, you can:

• Test Data Pipelines: Ensure your ETL processes work correctly with data that mimics real-world scenarios.

• Protect Sensitive Information: Avoid using real data in non-production environments, thus mitigating risks associated with data breaches or leaks.

• Enhance Development Flexibility: Create diverse datasets on demand, allowing for extensive testing across different data patterns and structures.

• Perform Data Quality Assurance: Validate and verify data processing logic by using datasets that include common data types and formats from various regions and contexts.

When to Use Data Fake?

Use the functions in the - Data Fake section: in the following situations:

• Generate Synthetic Data for Testing: When developing or testing data pipelines, it’s crucial to use data that mimics the complexity and variety of real-world data. This function can generate DataFrames with specified column types such as CPF, SSN, CURP, city, and more.

• Data Quality Testing: When you need to validate data processing and quality checks, this function allows you to create controlled datasets that include a variety of data formats, such as names, addresses, dates, and identifiers.

• Avoid Using Real Data: When working in development or testing environments where using actual data is not advisable, generate synthetic data to simulate scenarios without exposing sensitive information.

• Training and Demonstrations: When you need to demonstrate functionality or train users on data processing tasks, synthetic data provides a safe and customizable way to create examples and exercises.

Data Mapping and Possibilities

The Data Fake tool supports a wide range of column types, allowing you to generate various data attributes that are commonly used in real-world datasets. Below is the mapping of the possible column types and the corresponding data they generate:

Column Specifications

column_specs = [
    {'name': 'CPF', 'type': 'cpf'},  # CPF Number (Brazil)
    {'name': 'SSN', 'type': 'ssn'},  # Social Security Number (USA)
    {'name': 'CURP', 'type': 'curp'},  # Unique Population Registry Code (Mexico)
    {'name': 'DNI', 'type': 'dni'},  # National Identity Document (Spain)
    {'name': 'NIT', 'type': 'nit'},  # Tax Identification Number (Colombia)
    {'name': 'Codice_Fiscale', 'type': 'codice_fiscale'},  # Fiscal Code (Italy)
    {'name': 'Name', 'type': 'first_name'},  # First Name
    {'name': 'Last_name', 'type': 'last_name'},  # Last Name
    {'name': 'Birthdate', 'type': 'date_of_birth'},  # Date of Birth
    {'name': 'Email', 'type': 'email'},  # Email
    {'name': 'CEP', 'type': 'postcode'},  # Postal Code
    {'name': 'OptIn', 'type': 'boolean'},  # Opt-In (True/False)
    {'name': 'Language', 'type': 'language'},  # Language (PT, EN, etc.)
    {'name': 'City', 'type': 'city'},  # City
    {'name': 'Neighborhood', 'type': 'street_name'},  # Neighborhood (Using street name)
    {'name': 'Complement', 'type': 'secondary_address'},  # Address Complement
    {'name': 'Is_Winner', 'type': 'boolean'},  # Is Winner (True/False)
    {'name': 'Phone_Number', 'type': 'phone_number'},  # Phone Number
    {'name': 'Street_Address', 'type': 'street_address'},  # Street Address
    {'name': 'Country', 'type': 'country'},  # Country
    {'name': 'Company', 'type': 'company'},  # Company
    {'name': 'Job_Title', 'type': 'job'},  # Job Title
    {'name': 'Credit_Card_Number', 'type': 'credit_card_number'},  # Credit Card Number
    {'name': 'Credit_Card_Provider', 'type': 'credit_card_provider'},  # Credit Card Provider
    {'name': 'License_Plate', 'type': 'license_plate'}  # Vehicle License Plate
]