Data Engineer

• Must have:
  • Scala development and design using Scala 2.10+ or Python
  • Hadoop, Spark/Pyspark, Hive, YARN
  • Data Modelling: A good data engineer should be able to design, implement and maintain data models that can support the organization's data storage and analysis needs, and demonstrate good use of associated tools
  • Knowledge of operating systems such as windows and linux / unix
  • Experience with Jira, Confluence, GitHub or other similar Agile Scrum technologies and automated deployment tools (Ansible & Jenkins)
  • Knowledge and experience of test methodologies, developing test plans and execution of test cases (smoke/unit/integration/performance testing)
  • Scheduling Pipelines using Control M/Airflow
• Should have:

• Work effectively within a pod structure, often unaided
• Attend daily scrums, meeting or exceeding sprint targets
• Providing L3 support for developed and deployed assets quickly and effectively, responding to tickets efficiently to minimise MTTR (Mean Time To Repair)
• Quick learner, Team player

PySpark