Overview
The
Data & Insights organization at Insight Global is responsible for delivering
Trusted Data & Verified Insights to drive business decisions across the company. As a
Senior Data Engineer, you will be part of the Data Engineering & Operations team responsible for enhancing and maintaining scalable data pipelines, supporting and improving platform operations processes, and enabling analytics, data science, and AI use cases.
This is a hands-on role reporting to the
Data Operations Manager with direct impact on Insight Global's ability to deliver high-quality, actionable data to the business.
Responsibilities
- Design, develop, and enhance scalable data pipelines supporting analytics, reporting, AI/ML, and operational business processes.
- Own the reliability and stability of enterprise data platforms by proactively monitoring pipelines, troubleshooting production issues, and ensuring adherence to SLAs and operational standards.
- Lead root cause analysis and incident resolution efforts, driving permanent fixes, automation opportunities, and preventative controls to reduce recurring issues.
- Develop and maintain data solutions using Databricks, Azure Data Factory, SQL, and PySpark, following enterprise architecture and engineering standards.
- Drive DataOps best practices, including CI/CD implementation, automated testing, code reviews, deployment automation, release management, and environment governance.
- Improve data quality, performance, observability, and operational efficiency through monitoring, alerting, lineage tracking, logging, and automation initiatives.
- Work with structured, semi-structured, and streaming data, building scalable ingestion and transformation frameworks.
- Collaborate with business stakeholders, analytics teams, architects, product owners, and application teams to deliver reliable and business-aligned data solutions.
- Identify opportunities for continuous improvement in processes, architecture, governance, security, cost optimization, and platform scalability.
Qualifications
Required Skills- 5-7 years of experience in Data Engineering, DataOps, Analytics Engineering, or Enterprise Data Platforms.
- Strong expertise in advanced SQL development, query optimization, data analysis, and troubleshooting.
- Proficiency in Python and PySpark for large-scale data transformation, pipeline development, and automation.
- Hands-on experience with Databricks, Delta Lake, Azure Data Factory, and Azure data services.
- Experience designing and supporting batch, near real-time, and streaming data solutions.
- Strong understanding of CI/CD pipelines, Git-based source control, pipeline testing, and DevOps/DataOps practices.
- Experience with monitoring, observability, logging, alerting, and production support processes.
- Knowledge of data modeling, data warehousing concepts, and dimensional design principles
Preferred Skills- Experience with Databricks Unity Catalog.
- Knowledge of data governance, data quality frameworks, security, and compliance requirements.
- Proven experience managing production incidents, performing root cause analysis, and driving long-term remediation plans.
- Ability to prioritize operational stability while balancing enhancement and project delivery commitments.
- Experience working in Agile, Scrum, or Scrumban environments.
This Role Is Ideal If You:- Are excited to be part of a data team supporting data pipelines in Databricks.
- Thrive in Agile environments and enjoy working across disciplines.
- Take ownership of operational responsibilities and ability to fix data issues.
- Are passionate about delivering reliable, high-quality data solutions.
- Embody Insight Global's shared values and contribute to a collaborative, high-performance culture.
