Data Engineering Consultant - Cloud, Databricks & AI

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

We are seeking a highly experienced and innovative Senior Data Engineer with deep expertise in cloud-native data engineering, particularly using Databricks, Python, and AI-driven data solutions. This role will be pivotal in designing and building scalable data platforms that support advanced analytics, machine learning, and business intelligence across the organization.

Primary Responsibilities:

• Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, and Python on Azure Cloud
• Architect and implement cloud-based data solutions leveraging Azure Data Lake, and Azure Data Factory
• Collaborate with data scientists and AI teams to operationalize machine learning models and integrate them into data workflows
• Build and maintain feature stores, model input pipelines, and real-time data processing frameworks
• Ensure data quality, governance, and security across all stages of the data lifecycle
• Develop reusable frameworks and automation scripts to accelerate data engineering and AI deployment
• Participate in Agile development processes, including sprint planning, code reviews, and CI/CD using GitHub or Azure DevOps
• Mentor junior engineers and contribute to knowledge sharing across teams
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regard to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

• 7+ years of experience in data engineering, with a solid focus on cloud platforms and big data technologies
• Hands-on experience with Databricks, Apache Spark, and Delta Lake
• Experience with Azure Cloud Services including Data Factory, Blob Storage
• Experience with CI/CD tools and version control systems (e.g., Git, GitHub Actions, Azure DevOps)
• Expert-level proficiency in Python for data processing, automation, and integration
• Solid understanding of ETL/ELT, data modeling, and distributed computing
• Familiarity with AI/ML concepts, model deployment, and integration into data pipelines
• Proficiency in SQL and working with relational and NoSQL databases
• Proven excellent problem-solving, communication, and collaboration skills

• Azure certifications (e.g., Azure Data Engineer Associate, Azure AI Engineer Associate) are a plus
• Experience with MLflow, Feature Store, or other MLOps tools
• Exposure to Snowflake, Power BI, or other BI platforms
• Knowledge of data governance, privacy, and compliance standards