Data Engineer - Bigdata - Sydney
Yesterday• Sydney, Australia
Introduction
At IBM, work is more than a job - it's a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better, but to attempt things you've never thought possible. Are you ready to lead in this new era of technology and solve some of the world's most challenging problems? If so, lets talk.
Your role and responsibilities
We are currently hiring a Data Engineer for full-time position, based in Sydney, Australia.
The Data Engineers in the Nexus-Ops team are responsible for supporting the Data Harbour program, through working with stakeholders including Architects, technical teams, Business users, and data producers, to understand the target state requirements including upstream and downstream use cases. As a senior member of the Data Engineer team, the Lead Data Engineer will be accountable for end-to-end architecture, design, and development process, including providing technical and architecture advisory support to the implementation team and the wider Macquarie stakeholders. As the Lead Data Engineer your responsibilities include -
Want more jobs like this?
Get jobs in Sydney, Australia delivered to your inbox every week.
Architect, Design and Solutioning cloud based big data engineering and analytics projects.
Architect, Design data pipeline for batch, Streaming, Realtime data using AWS and Spark.
Creating architecture, design, technical specifications artefacts.
Participate in depth architectural discussions to build confidence and ensure customer success when building new solutions and migrating existing data applications on the AWS platform.
Apply in depth experience in AWS cloud services and hands on implementation experience using data-based services on AWS.
Work with a variety of cloud-based storage services: AWS S3, Elastic Storage, RDS, AWS Data pipeline, AWS Glue and Data analytics services.
Abide by team coding standards, development processes and best practices.
Required education
Bachelor's Degree
Preferred education
Bachelor's Degree
Required technical and professional expertise
To ensure success in the role you will possess the following skills -
Must have been through several full-lifecycle Big Data engineering implementations and involved in availability, scalability and performance related design aspects in Data processing and Reporting.
Strong hands-on experience working in AWS Serverless (Lambda), Coding against API and SQL.
Must be hands-on with writing Python or Scala Scripts development.
Hands on experience designing of orchestration using Airflow.
Well versed with Spark development (SQL and/or Spark).
Hands on experience in AWS Service EC2, S3, SNS, SQS, Lambda, Hue, Presto, Spark, Glue, Kinesis, EMR, ECS, EKS.
Sound knowledge in Hive, Redshift, AWS RDS, My SQL and in any NoSQL database e.g., Cassandra, MongoDB.
Experience in architecting and designing batch or real time data streaming.
Ability to dynamically adapt to conventional big data frameworks and opensource tools if project demands.
Knowledge of design strategies for developing scalable, resilient, always-on data lake.
Strong development and automation skills.
Good understanding in data architecture principles, including data access patterns, data modelling and strong in Data base concepts.
Previous experience in Managed Services related reporting in a financial services organization is an advantage.
ABOUT BUSINESS UNIT
IBM Consulting is IBM's consulting and global professional services business, with market leading capabilities in business and technology transformation. With deep expertise in many industries, we offer strategy, experience, technology, and operations services to many of the most innovative and valuable companies in the world. Our people are focused on accelerating our clients' businesses through the power of collaboration. We believe in the power of technology responsibly used to help people, partners and the planet.
YOUR LIFE @ IBM
In a world where technology never stands still, we understand that, dedication to our clients success, innovation that matters, and trust and personal responsibility in all our relationships, lives in what we do as IBMers as we strive to be the catalyst that makes the world work better.
Being an IBMer means you'll be able to learn and develop yourself and your career, you'll be encouraged to be courageous and experiment everyday, all whilst having continuous trust and support in an environment where everyone can thrive whatever their personal or professional background.
Our IBMers are growth minded, always staying curious, open to feedback and learning new information and skills to constantly transform themselves and our company. They are trusted to provide on-going feedback to help other IBMers grow, as well as collaborate with colleagues keeping in mind a team focused approach to include different perspectives to drive exceptional outcomes for our customers. The courage our IBMers have to make critical decisions everyday is essential to IBM becoming the catalyst for progress, always embracing challenges with resources they have to hand, a can-do attitude and always striving for an outcome focused approach within everything that they do.
Are you ready to be an IBMer?
ABOUT IBM
IBM's greatest invention is the IBMer. We believe that through the application of intelligence, reason and science, we can improve business, society and the human condition, bringing the power of an open hybrid cloud and AI strategy to life for our clients and partners around the world.
Restlessly reinventing since 1911, we are not only one of the largest corporate organizations in the world, we're also one of the biggest technology and consulting employers, with many of the Fortune 500 companies relying on the IBM Cloud to run their business.
At IBM, we pride ourselves on being an early adopter of artificial intelligence, quantum computing and blockchain. Now it's time for you to join us on our journey to being a responsible technology innovator and a force for good in the world.
IBM is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, neurodivergence, age, or other characteristics protected by the applicable law. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.
OTHER RELEVANT JOB DETAILS
For additional information about location requirements, please discuss with the recruiter following submission of your application.
Client-provided location(s): Sydney, Australia
Job ID: IBM-63792
Employment Type: OTHER
Posted: 2025-10-08T18:49:53
Perks and Benefits
Health and Wellness
Parental Benefits
Work Flexibility
Office Life and Perks
Vacation and Time Off
Financial and Retirement
Professional Development
Diversity and Inclusion
Company Videos
Hear directly from employees about what it is like to work at IBM.