Lead, Data Platform Engineer

Position Overview:

Data Analytics & AI department at RCG is looking for a highly skilled and experienced Data Platform Lead to design, build, and manage our data platform on Azure and Databricks. The successful candidate will be responsible for managing the end-to-end data platform development process, including data ingestion, processing, storage, and analysis. This role requires a strong technical background in data platform architecture, cloud computing, and big data technologies.

This role carries primary responsibilities as a Platform Lead / Platform Engineer and Azure Platform Admin, while also contributing to ML Ops.

Responsibilities:

• Design, build and configure a highly scalable and robust data platform on Azure and Databricks.
• Data platform shuld have capabilities of seamless integration & extensibility of existing cloud services to any new program/project initiative.

• Develop and implement data ingestion and processing pipelines using Azure services such as Azure Data Factory, Azure Stream Analytics, and Databricks
• Collaborate with cross-functional teams to ensure data platform meets business requirements and industry best practices
• Manage cloud infrastructure on Azure platform and maintain Azure environment to ensure maxium efficiency, scalability and security.
• Develop & implement cloud automation and orchestration using tools auch as Azure resource manager, Azure DevOps etc.
• Collaborate with offshore team members with development & deployment activities, per project requirements.

• Proficiency in Python, Scala, or SQL.

• Expertise in Apache Spark, Hadoop, or Apache Kafka.
• Cloud experience in Azure (preferred) or AWS.
• Deployment and management of Databricks clusters.
• Mastery of Azure Data Factory (ADF) for integration.
• Expertise with Databricks workspace, runtime, and Databricks Connect.

• Understanding of Databricks Unity Catalog.
• Knowledge of Delta Live Tables and Delta Engine.
• Familiarity with Delta Lake architecture.
• Strong SQL skills.

• Management of Azure Cloud Services, including VMs, Storage Accounts, and networks.
• Azure Security Management for policies and access controls.
• Knowledge of Azure Active Directory and network security.
• Performance monitoring, resource optimization, backup, and recovery.

• High Availability (HA), Disaster Recovery (DR), and failover expertise.
• Azure Services/Resources cost optimization and monitoring.
• Azure DevOps and CI/CD proficiency.
• Configure Azure resources, native services, security, and monitoring.
• Design platform architecture with Azure Cloud Services and Databricks Delta Lake.

• Create and publish guidelines for Data Integration services.
• Participate in platform solution architecture, including data architecture.
• Implement Databricks workspace, runtime, and Databricks Connect.
• Configure Kafka integration services for real-time streaming and batch processing.
• Enable Databricks Unity Catalog (UC) for lineage and governance.

• Design and implement ML Ops for model deployments and inference pipelines.
• Configure Azure DevOps, CI/CD for seamless deployment.
• Implement monitoring and logging for Azure Service Groups and system metrics.
• Ensure platform security, compliance, and data privacy.

• Proficient in Python, R, or Java for ML algorithms.
• Experience with TensorFlow, PyTorch, and other ML frameworks.
• Efficient code for data processing, model training, and deployment.
• Cloud platform experience, preferably Azure.

• Familiarity with Azure ML Services and Azure DevOps.
• Strong data analytics skills for model training.

• Bachelor's degree in Computer Science, Information Technology, or related field.

• 7+ years of experience in cloud infrastructure management, Azure services, and infrastructure architecture
• Strong understanding of Azure services such as Virtual Machines, Virtual Networks, Storage Accounts, and App Services
• Experience in monitoring and optimizing Azure infrastructure for performance and efficiency
• Proven experience and expertise in the respective role's technical skillsets.
• Strong problem-solving skills and ability to work collaboratively.

• Excellent communication skills for effective cross-team collaboration.