Lead Software Engineer - Pyspark data bricks Data Engineer

Job Description

Join us as we embark on a journey of collaboration and innovation, where your unique skills and talents will be valued and celebrated. Together we will create a brighter future and make a meaningful difference.

As a Lead Data Engineer at JPMorgan Chase within the Compliance Technology Team which is aligned to Corporate Technology division , you are an integral part of an agile team that works to enhance, build, and deliver data collection, storage, access, and analytics solutions in a secure, stable, and scalable way. As a core technical contributor, you are responsible for maintaining critical data pipelines and architectures across multiple technical areas within various business functions in support of the firm's business objectives.

Job responsibilities

• Provides recommendations and insight on data management, governance procedures, and intricacies applicable to the acquisition, maintenance, validation, and utilization of data
• Designs and delivers trusted data collection, storage, access, and analytics data platform solutions in a secure, stable, and scalable way
• Defines database back-up, recovery, and archiving strategy
• Generates advanced data models for one or more teams using firm wide tooling, linear algebra, statistics, and geometrical algorithms
• Approves data analysis tools and processes
• Creates functional and technical documentation supporting best practices
• Advises junior engineers and technologists
• Evaluates and reports on access control processes to determine effectiveness of data asset security
• Adds to team culture of diversity, equity, inclusion, and respect

• Formal training or certification on Data Engineeringconcepts and 5+ years applied experience
• Demonstrated leadership abilities, including leading and mentoring a team of data engineers.
• Expertise in designing and implementing scalable data architectures and pipelines.
• Strong proficiency in programming languages such as Python and PySpark.
• Experience with AWS cloud platform including their data services.
• In-depth knowledge of data warehousing solutions and ETL processes.
• Ability to work collaboratively with cross-functional teams, including data scientists, analysts, and business stakeholders.
• Experience with data governance and ensuring data quality and integrity.
• Proficiency in Big Data technologies, with a strong focus on Performance Optimization using best practices
• Excellent communication skills, with the ability to convey complex technical concepts to non-technical audiences.
• Experience with version control systems like Git and CI/CD pipelines for data engineering workflows.

• Experience with Databricks using PySpark
• Solid understanding of the Parquet file format.
• Experience with ETL tools, and experience in ETL process
• Familiarity with machine learning frameworks and integrating them into data pipelines.