ABOUT US
Infosys BPM (www.infosysbpm.com), the business process outsourcing subsidiary of Infosys, was set up in April 2002. Infosys BPM focuses on integrated end-to-end outsourcing and delivers transformational benefits to its clients through reduced costs, ongoing productivity improvements, and process re-engineering. Infosys BPM operates in India, Poland, the Czech Republic, the Netherlands, Ireland, South Africa, Brazil, Mexico, Costa Rica, the United States, Puerto Rico, China, the Philippines, Singapore, and Australia.
Infosys BPM has been consistently ranked among the leading BPM companies and has received multiple awards and recognition from key industry bodies and associations. Infosys BPM also has very robust people practices, as substantiated by the various HR-specific awards it has won over the years. The company has consistently been ranked among the top employers of choice, on the basis of its industry-leading HR best practices. The company's senior leaders contribute widely to industry forums as BPM strategists.
Want more jobs like this?
Get Data and Analytics jobs in Alice Springs, Australia delivered to your inbox every week.
Role: - Associate Consultant (Data Engineer)
Location: - Sydney
Status - 6 Months fixed term, extendable based on project requirement
NOTE - Only applicants with Australia full work right, i.e., holding a valid visa will be considered for the role.
JOB DESCRIPTION: -
We are seeking a Data Engineer with deep expertise in real-time, big data processing to join a growing data platform team. If you're passionate about streaming technologies, scalable architecture, and cloud-native data pipelines, we want to hear from you.
As a key contributor, you'll design, build, and maintain high-throughput data pipelines leveraging Apache Spark Structured Streaming, Kafka, and cutting-edge tools within the AWS Big Data services. This role involves hands-on development with technologies like Apache Iceberg, SCD logic, and CI/CD automation.
Key Responsibilities:
- Develop and maintain real-time data pipelines using Apache Spark Structured Streaming with Kafka.
- Parse and process complex data formats, including XML, JSON, and CSV.
- Apply advanced Spark optimization techniques such as broadcast joins, partitioning, and caching.
- Implement SCD Type 1 & Type 2 logic within data warehouse models.
- Design and orchestrate ETL workflows using:
- AWS Glue (trigger-based jobs)
- EventBridge, Lambda, and Step Functions
- Work with AWS Big Data ecosystem, including:
- AWS Glue for serverless ETL
- Amazon S3 and Athena for data lake querying
- Lambda and EventBridge for event-driven workflows
- DynamoDB for real-time lookup tables
- Manage Apache Iceberg internals, including Compaction, Schema Evolution, Time Travel, and Snapshot Management.
- Utilize PyTest for writing unit and integration tests for PySpark pipelines.
- Perform local development & testing of Glue jobs using Docker & PySpark.
- Deploy and manage data jobs through CI/CD pipelines, leveraging Jenkins, GitHub, and JFrog Artifactory.
- Collaborate with architects and stakeholders to define scalable and efficient solutions aligned with business needs.
Key Requirements:
- Strong expertise in Apache Spark Structured Streaming, Kafka, and PySpark.
- Proven experience with Spark job performance tuning.
- Proficiency in parsing & transforming XML data.
- Deep understanding of AWS services, including Glue, Lambda, S3, DynamoDB, and Athena.
- Hands-on experience with Apache Iceberg internals, covering compaction, schema evolution, and metadata management.
- Familiarity with Docker-based local Glue development.
- Strong testing discipline using PyTest.
- Experience implementing SCD1/2 logic in data pipelines.
- Solid background in CI/CD environments using Jenkins, GitHub, and JFrog.
- Excellent communication and problem-solving skills.
- Experience in the Banking or Financial Services domain.
- Familiarity with data governance, lineage, or compliance frameworks.
- Exposure to Terraform, CloudFormation, or Airflow.
Infosys BPM is committed to creating a diverse and inclusive working environment. We actively support and encourage people of all backgrounds; gender, age, ethnicity, race, cultural background, disability, religion and sexual orientation to submit an application and aim to ensure that the recruitment and employee experience is as accessible and inclusive as possible.
We provide adjustments including alternate formats to the recruitment process for candidates including individuals with disability. If you require an adjustment to be made during the recruitment process (including alternate formats) or have an enquiry about the support provided, please contact HR Team on 02 8913 5900 or email to bpm.anz.careers@infosys.com
Please click on the Apply link below to upload your resume before the closing date.
We reserve the right to withdraw this advertisement prior to the closing date.
We respectfully request agencies not to forward unsolicited resumes unless requested.