Lead Engineer QA

Collaborate with stakeholders to understand data requirements and translate them into efficient and scalable data engineering solutions.
Optimize data models, database schemas, and data processing algorithms to ensure efficient and high-performance data storage and retrieval.
Implement and maintain data quality and data governance processes, including data cleansing, validation, and metadata management.
Work closely with data scientists, analysts, and business intelligence teams to support their data needs and enable data-driven decision-making.
Develop and implement data security and privacy measures to ensure compliance with regulations and industry best practices.
Monitor and troubleshoot data pipelines, identifying and resolving performance or data quality issues in a timely manner.
Stay up to date with emerging technologies and trends in the data engineering field, evaluating and recommending new tools and frameworks to enhance data processing and analytics capabilities.
Collaborate with infrastructure and operations teams to ensure the availability, reliability, and scalability of data systems and infrastructure.
Mentor and provide technical guidance to junior data engineers, promoting best practices and knowledge sharing.

Qualifications:

• Bachelor's degree in computer science or software engineering

• 5+ years of progressive experience in delivering technology solutions in a production environment
• 5+ years of experience in the software industry as a developer, with a proven track record of shipping high quality products
• 3+ years working with customers (internal and external) on developing requirements and working as a solutions architect to deliver

'Required:
'- Bachelor Degree in Computer Science or Software Engineering or Information Technoogy
- Experience on Cloud Development Platforms - Azure & AWS and their associated data storage options
- Cloud based Analytics (AWI, REST API, Microservices)
- Knowledge of IoT technologies, including cloud processing, like Azure IoT Hub.
- Experience on CI/CD (Continuous Integration/Delivery) i.e. Jenkins, GIT, Travis-CI
- Virtual build environments (Container, VMs and Microservices) and Container orchestration - Docker Swarm, Kubernetes/Red Hat Openshift.
- Relational & non-relational database systems - SQL, Postgres SQL, NoSQL, MongoDB, CosmosDB, DocumentDB
- Data Warehousing & ETL - Write complex queries that are accessible, secure and perform in optimized manner that outputs to different consumers and systems
- ETL on Big Data Technologies - Hive, Impala
- Progamming Knowledge - Java and/or Python and associated IDE's (Eclipse, IntelliJ, PyCharm etc.)
- Data pipelining, scripting, reporting
- In-depth knowledge of SOA (Service Oriented Architecture)
- Experience in Azure Tools - Blob, SQL, Data Lake, Hive, Hadoop, Data Factory , Databricks, Azure Functions
- SW Development life-cycle process & tools
- Agile development methodologies and concepts including handson with Jira, bitbucket and confluence.
- Knowledge of streaming technologies like Apache Kafka, AWS Kinesis, Azure EventHubs
- Knowledge of data analysis tools, like Apache Presto, Hive, Azure Data Lake Analytics, AWS Athena, Zeppelin
- Ability to specify and write code that is accessible, secure, and performs in an optimized manner with an ability to output to different types of consumers and systems
- Proven experience working as a Data Engineer, with at least 6+ years of experience in data engineering, data warehousing, or related fields.
- Strong proficiency in SQL and experience with relational databases like MySQL, PostgreSQL, or similar.
- Hands-on experience with big data technologies such as Apache Hadoop, Spark, Airflow or similar frameworks.
- Expertise in data modeling, data integration, and ETL processes.
- Proficiency in programming languages like Python, Java, or Scala, with experience in building data pipelines and automation scripts.
- Familiarity with cloud-based data platforms and services such as AWS, Azure, or Google Cloud Platform.
- Experience with data visualization tools like Tableau, Power BI, or similar.
- Knowledge of data security and privacy principles, including GDPR and data protection regulations.
- Strong problem-solving and analytical skills, with the ability to troubleshoot and resolve complex data engineering challenges.
- Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams and stakeholders.
- Experience with agile development methodologies and version control systems is preferred.
Desired:

- Experience in Design Thinking or human-centered methods to identify and creatively solve customer needs, through a holistic understanding of customer's problem area
- Knowledgeable in leveraging multiple data transit protocols and technologies (MQTT, Rest API, JDBC, etc)
- Knowledge of Hadoop and MapReduce/Spark or related frameworks
- Knowledge of Scala

Skills:

• Excellent verbal and written communication skills including the ability to effectively communicate technical concepts as a part of virtual, global teams
• Good interpersonal, negotiation and conflict resolution skills
• Ablity to understand academic research and apply new data science techniques
• Experience being part of larger teams with established big data platform practices, as well as smaller teams where they made a bigger impact in terms of scope.
• Experience of working with global teams work Experience and awareness, Strong communication skills to interact with global teams.
• Innate curiosity
• Self-directed and hungry to learn - a person, who with time in his/her hands, will independently find interesting ways to push the envelope, learning new skills and growing Self and the team.
• Team player- we work in small, fast moving teams.