Lead Data Engineer

• Work with a team of experts in deep learning, machine learning, distributed systems, program management, and product teams, and work on all aspects of design, development and delivery of deep learning-enabled end-to-end pipelines and solutions.
• Lead the development of technical solutions and implement architectures for projects and products across data engineering and data science teams
• Lead the architecture, design, and development of new intelligent power technology products and production-quality end-to-end systems
• Experience in ML/AI Software development environment
• Work with your team and others, defining the architecture, design, and management of secure, large-scale, highly distributed, geo-redundant, multi-tenant data stores
• Knowledge of microservices, cloud APIS (e.g. AWS and MS Azure)
• Is accountable for end-to-end delivery of solutions from requirements gathering to production
• Author high-quality, highly-performance, unit-tested code to extract and transform data based on business and data science needs

Key Responsibilities:

1. Data Architecture and Design:
- Design and implement scalable and robust data architectures to support business requirements.
- Collaborate with stakeholders to understand data needs and translate them into technical specifications.

2. Data Modeling and ETL Development:
- Develop and maintain efficient ETL processes for large volumes of data.
- Design and implement data models, ensuring data accuracy, consistency, and integrity.

3. Data Integration:
- Integrate data from various sources, both internal and external, to create a unified and comprehensive view of business data.
- Implement real-time and batch data integration solutions.

4. Data Quality and Governance:
- Establish and enforce data quality standards and best practices.
- Implement data governance processes to ensure data accuracy and security.

5. Performance Optimization:
- Identify and resolve performance bottlenecks in data pipelines and queries.
- Implement optimizations for data storage, retrieval, and processing.

6. Team Leadership and Collaboration:
- Lead a team of data engineers, providing mentorship and guidance.
- Collaborate with cross-functional teams to understand business requirements and ensure data solutions align with overall business goals.

7. Technology Evaluation and Implementation:
- Stay current with industry trends and emerging technologies.
- Evaluate and recommend new tools and technologies to enhance the data engineering stack."

Qualifications:

• PhD, Masters or Bachelor's degree in computer science or software engineering
"• 8+ years of progressive experience in delivering technology solutions in a production environment
• 6+ years of experience in the software industry as a developer, with a proven track record of shipping high quality products
• 6+ years working with customers (internal and external) on developing requirements and working as a solutions architect to deliver"

'- PhD, Masters or Bachelor's degree in Computer Science or Software Engineering or Information Technology

Skills:

- Experience on Cloud Development Platforms - Azure & AWS, releated services on data storage, machine learning pipeline services, etc.
- Cloud based Analytics (AWI, REST API, Microservices)
- Knowledge of IoT technologies, including cloud processing, like Azure IoT Hub.
- Experience on CI/CD (Continuous Integration/Delivery) i.e. i.e. Github Actions, Azure DevOps, GIT, Terraform
- Virtual build environments (Container, VMs and Microservices) and Container orchestration - Docker Swarm, Kubernetes/Red Hat Openshift.
- Relational & non-relational database systems - SQL, Postgres SQL, NoSQL, MongoDB, CosmosDB, DocumentDB and Hands-on SQL experience
- Data Warehousing & ETL - Write complex queries that are accessible, secure and perform in optimized manner that outputs to different consumers and systems
- ETL on Big Data Technologies - HDFS, Hive, Impala, SQOOP
- Progamming Knowledge - Python and/or Java and associated IDE's (Eclipse, IntelliJ, PyCharm etc.)
- APIs development to support data consumption needs of internal and external stakeholders
- Data visualization tools such as Power BI, QlikView and/or Tableau
- Data pipelining, scripting, reporting
- In-depth knowledge of SOA (Service Oriented Architecture)
- Experience in Azure Tools - Blob, SQL, Data Lake, Hive, Hadoop, Data Factory , Databricks, Azure Functions
- SW Development life-cycle process & tools
- Agile development methodologies and concepts including handson with Jira, bitbucket and confluence.
- Knowledge of streaming technologies like Apache Kafka, AWS Kinesis, Azure EventHubs
- Knowledge of Cloudera Hadoop, ML ops
- Knowledge of data analysis tools, like Apache Presto, Hive, Azure Data Lake Analytics, AWS Athena, Zeppelin
- Ability to specify and write code that is accessible, secure, and performs in an optimized manner with an ability to output to different types of consumers and systems
Desired:

- Experience in Design Thinking or human-centered methods to identify and creatively solve customer needs, through a holistic understanding of customer's problem area
- Knowledgeable in leveraging multiple data transit protocols and technologies (MQTT, Rest API, JDBC, etc)
- Knowledge of Hadoop and MapReduce/Spark or related frameworks
- Knowledge of Scala"

"• Excellent verbal and written communication skills including the ability to effectively communicate technical concepts as a part of virtual, global teams
• Good interpersonal, negotiation and conflict resolution skills
• Ablity to understand academic research and apply new data science techniques
• Experience being part of larger teams with established big data platform practices, as well as smaller teams where they made a bigger impact in terms of scope.
• Experience of working with global teams work Experience and awareness, Strong communication skills to interact with global teams.
• Innate curiosity
• Self-directed and hungry to learn - a person, who with time in his/her hands, will independently find interesting ways to push the envelope, learning new skills and growing Self and the team.
• Team player- we work in small, fast moving teams."