Staff, Software Engineer - Data Platform

3+ months agoSan Francisco, CA

Because you belong at Twilio

The Who, What, Where and Why


Twilio's Data Platform org is looking for Software Engineers to join our Data Platform (Storage and Compute) team. The team manages Twilio’s GDPR compliant data lake at PB scale and runs Twilio’s complex data pipelines. To achieve this, we leverage technologies that include AWS cloud infrastructure and APIs, open source projects like Apache Spark, Apache Kafka, Apache Hudi etc., Rest APIs and also build our own services or contribute to open source projects. This data lake is an integral part of some of Twilio’s core products and also enables business teams to do analytics and make well informed data driven decisions.

We’re looking for an Engineer to help us scale and provide rock-solid service on our critical data infrastructure, build a data pipeline for Twilio's myriad engineering teams, and uplevel the ecosystem of how Twilio thinks about reliability and availability at large.

  • Your experience includes:

    • 6+ years of experience developing a globally distributed data services platform
    • 3+ years of experience building data pipelines using technologies like Spark, Kafka, Airflow, s3, Hudi etc.
    • Experience in Java or Scala, Python familiarity is also useful
    • Experience in leading and mentoring engineers to execute medium to large scale projects
    • Understanding and experience in Agile/Scrum methodologies
    • Experience in AWS cloud infrastructure to own and operate production services
    • Strong communication skills and desire to make an impact and thrive in small, collaborative, energetic teams
    • Degree in Computer Science or a related field, or equivalent work experience
    • Bonus : Comitter or contributor in open source projects related to distributed storage and computation. Ex. Apache Spark, Apache Hudi, Presto etc.
    • Bonus : Experience with Docker and Kubernetes
    • Bonus : Knowledge/experience building GDPR, SOX compliant systems


As a Staff Engineer, you will design, implement and operate services that will process hundreds of million records daily. You will live the Twilio Magic values. Here are some examples:

  • WEAR THE CUSTOMER’S SHOES: You are driven more by customer impact than technology.
  • BE AN OWNER & BE BOLD: You will own the design, implementation and the operations of a data pipeline and storage, scaling up to hundreds of millions of new records daily. 
  • DRAW THE OWL: Not every problem has a textbook solution. You will be expected to research the unknowns, implement and iterate. Educate your team as you go.
  • DON’T SETTLE: Live and champion mature, high-quality engineering through writing tests, valuing code maintainability, documenting your work, etc
  • WRITE IT DOWN: Twilio is a global, fast-growing company that builds complex systems. To be successful in joining these globally built complex parts of the system, You will be expected to write down the requirements, design, implementation, etc.
  • EMPOWER OTHERS: Mentor junior engineers and help them grow. Collaborate with other teams and work cross-functionally for product launches.


Twilio’s Data Platform processes billions of records per day and operates at the petabyte scale. Our engineers deal with a wide variety of data technologies, distributed systems, and design considerations to provide data as a platform to the rest of Twilio Engineering.

Twilio is a company that is empowering the world’s developers with modern communication in order to build better applications. Twilio's mission is to fuel the future of communications. Developers and businesses use Twilio to make communications relevant and contextual by embedding messaging, voice and video capabilities directly into their software applications. The Engineering team plays an integral role in building out the products that allow our developer community to meet their communication needs.

Twilio is truly unique; we are a company committed to your growth, your learning, your development and your entire employee experience.  We only win when our employees succeed and we're dedicated to helping you develop your strengths. We invest in weeks dedicated to tackling hard problems and creating your own ideas. We have a cultural foundation built on diversity, inclusion and innovation and we want you and your ideas to thrive at Twilio. Come join us. 

About us:

Millions of developers around the world have used Twilio to unlock the magic of communications to improve any human experience. Twilio has democratized communications channels like voice, text, chat, video and email by virtualizing the world’s communications infrastructure through APIs that are simple enough for any developer to use, yet robust enough to power the world’s most demanding applications. By making communications a part of every software developer’s toolkit, Twilio is enabling innovators across every industry — from emerging leaders to the world’s largest organizations — to reinvent how companies engage with their customers.

Twilio is proud to be an equal opportunity workplace and is an affirmative action employer. We are committed to equal opportunity regardless of race, color, ancestry, religion, gender, gender identity, parental or pregnancy status, national origin, sexual orientation, age, citizenship, marital status, disability, or Veteran status and operate in compliance with the San Francisco Fair Chance Ordinance.