EPAM Systems

Senior Big Data Engineer

2 months agoKharkiv, Ukraine

EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.

DESCRIPTION
As a Senior Big Data Engineer, you will live on the front lines of the products we create, and build features used by millions of people every day.

Our customer is America's largest online lending marketplace. It connects borrowers with multiple lenders so they can find the best deals on loans, credit cards, deposit accounts, insurance and more. Our project is about developing a new ETL pipelines and support existing ones.
Project technologies and tools

  • Cloud: AWS(EMR, S3, Athena, Glue etc.)
  • Tools and Frameworks: Hortonworks DataPlatform and DataFlow, Apache Yarn, Maven
  • Databases: Apache HBase, Apache Hive, Microsoft SQL Server
  • Processing: Apache Spark, Apache Kafka, Apache NiFi
  • Languages: Scala, Bash, Groovy
  • CI/CD: Jenkins
Responsibilities
  • Writing new and optimizing performance/codebase of existing ETL Spark jobs
  • Planning and developing modules for integration of external data warehouses (from acquired external businesses)
  • Writing new and optimizing existing Jenkins pipelines
  • Data validation and data discrepancy reporting
  • Bug reporting
  • Writing new and refactoring existing NiFi data flows
  • Re-writing batch NiFi flows into Spark jobs to improve performance
  • Providing overall improvements to development process like relentless refactoring of growing codebase and applying new approaches in time-consuming development operations
  • Developing and maintaining project documentation
  • Investigation and requirements clarification
Requirements
  • 3+ years of experience in Java or Python
  • Experience with HIVE, Spark will be a plus
  • Experience with Apache Hadoop will be a plus
  • Strong communication skills
  • Good spoken and written English
We offer
  • Competitive compensation depending on experience and skills
  • Individual career path
  • Unlimited access to LinkedIn learning solutions
  • Social package - medical insurance, sports
  • Compensation for sick lists and regular vacations
  • English classes with native speakers (certified English teachers)
  • Flexible work hours
Apply