Senior Data Engineer, Data Lake

About Us
Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be paid.

At Visa, you'll have the opportunity to create impact at scale - tackling meaningful challenges, growing your skills and seeing your contributions impact lives around the world.

Join Visa and do work that matters - to you, to your community, and to the world. Progress starts with you.

Job Description

Summary

We're looking for a Data Engineer at the Analyst level to join our Data Lake team. You'll build and maintain data ingestion and transformation pipelines using Spark, Databricks, and Airflow, contributing to the reliability and quality of the corporate data lake. This is an intermediate role where you'll work under moderate supervision on defined tasks while building deeper expertise in lakehouse architecture.

At Pismo, the Data Lake team is responsible for centralizing and organizing data into a single, trusted platform that supports decision-making across the company and for external clients. We work on challenges such as scaling global data infrastructure, delivering high-quality reporting, and enabling secure, self-service access to data-helping teams move faster while avoiding information silos.

• Develop, test, and maintain data pipelines (ingestion, transformation, quality checks) using PySpark/SparkSQL on Databricks .

• Build and modify Airflow DAGs (MWAA) for pipeline orchestration.

• Write and optimize SQL queries for data transformation and validation.

• Support data quality by implementing and monitoring quality checks (Great Expectations or equivalent).

• Participate in code reviews - both giving and receiving feedback.

• Investigate pipeline failures and data quality issues with guidance from senior engineers.

• Write and maintain documentation for datasets and pipelines you build.
• Participate in sprint planning, estimation, and retrospectives.

• Follow CI/CD, testing, and governance standards defined by the team.

• Proficiency in English at B2 level or above (Upper-Intermediate)

• 3+ years of relevant work experience with a Bachelor's or Associate's Degree OR 5+ years of relevant work experience.
• Python for automation and data processing
• Data pipeline concepts (batch, ETL/ELT patterns)

• Apache Spark basics (PySpark or SparkSQL)
• Databricks (jobs, workflows, cluster management, tuning)
• SQL (intermediate-advanced: joins, window functions, CTEs)
• Amazon S3 data lake design (partitioning, layout, lifecycle)
• Basic cloud concepts (AWS: S3, IAM, CloudWatch)
• Data modeling (dimensional / Kimball, medallion layers)
• Git/GitHub workflow and code review

• Delta Lake (ACID tables, OPTIMIZE, VACUUM, schema evolution, MERGE)
• Apache Airflow / MWAA (DAG design, retries, backfills, SLAs)
• CDC patterns (DMS, incremental processing, MERGE upserts)
• Data quality concepts (validation, testing)
• Terraform basics
• BI integration (Superset, dashboarding)