Senior Software Engineer, Scale & Transport

Do you have insights into how to improve spark? Are you thinking about how to move from batch to realtime? Do you get excited about data layout? Distributed fault tolerance and consistency? If so, this might be the team for you.

At Quantcast, data is king, and the Scale & Transport team knows how to process it. The unique features of our custom map-reduce implementation include a revolutionary petabyte sort mechanism, an efficient and flexible petabyte join and lookup, based on innovative data indexing and partitioning mechanisms, dynamic task size adjustment in running jobs, and a self-tuning system that learns and improve job performance without developer intervention. In the last 5 years it has allowed Quantcast to scale from processing 400 to 40,000 terabytes of data per day. For a workload consisting primarily of ETL, analytics, and data mining, our processing platform is on average 4 to 5 times more efficient than the most popular open source map-reduce system.

Responsibilities

  • You will design and develop high performance distributed computing systems
  • Diagnose and produce system solutions to big data problems
  • Perform design, architecture, and code reviews
  • Mentor and grow more junior team members
  • Innovate in the big data space: lead, not follow
  • Provide technical feedback into product roadmaps for the team

Requirements

  • BS Computer Science or equivalent experience
  • 3 years of industry experience in software development
  • Experience working on large-scale distributed systems
  • Experience optimizing the performance of large-scale distributed systems
  • Deep understanding of distributed systems
  • Proficient in C++ and/or Java
  • Strong software development skills (version control, code review, testing, release, monitoring, debugging)
  • Proficiency and high level of insight into Linux OS and running software systems on Linux

Quantcast owns and operates the world’s largest audience insights and measurement platform on the open internet. Fueled by live data drawn from more than 100 million web and mobile destinations, Quantcast applies machine learning technology to help marketers, publishers, and agencies grow their brands by better understanding and predicting consumer interactions in real-time. 

Founded in 2006, Quantcast is headquartered in San Francisco and employs more than 700 employees in over 20 offices across 10 countries. We are committed to building an inclusive and diverse environment where everyone can be their authentic self.

Back to top