Data Engineer

Overview A Data Engineer is responsible for designing, building, and maintaining data pipelines and infrastructure that enable efficient data collection, storage, and processing. This role ensures that data is accessible, reliable, and optimized for analytics and machine learning applications. Key Responsibilities

• Design and implement scalable ETL (Extract, Transform, Load) processes.
• Build and maintain data pipelines for batch and real-time data processing.
• Develop and optimize data models for relational and NoSQL databases.
• Manage data warehouses and data lakes for structured and unstructured data.
• Integrate data from multiple sources into centralized systems.
• Collaborate with data scientists and analysts to ensure data availability and quality.
• Optimize data systems for speed, reliability, and scalability.
• Implement data governance, security, and compliance standards.
• Monitor data workflows and troubleshoot issues.
• Automate processes to reduce manual intervention.

• Proficiency in programming languages (Python, Java, or Scala).
• Experience with big data frameworks (Apache Spark, Databrics, Hadoop).
• Strong knowledge of SQL and database technologies (PostgreSQL, MySQL, NoSQL).
• Familiarity with cloud platforms (AWS, Azure, GCP) and data services.
• Understanding of ETL tools and workflow orchestration.
• Strong problem-solving and analytical skills.

• Experience with containerization (Docker, Kubernetes).
• Familiarity with data governance and compliance standards.
• Understanding of MLOps and integration with machine learning pipelines.