Cluster Interfaces is responsible for all of the entry points into Quantcast’s big data services. We develop a dashboard through which data pipelines are scheduled and executed. The dashboard provides a holistic view into the company’s data pipelines, including their dependency trees, capacity needs, and execution times. It also allows users to define data retention policies, monitoring, and alerting. The dashboard is similar to the open source project Azkaban.
The team also develops and operates a SQL-On-MapReduce platform along with a data catalog. Quantcast’s SQL-On-MapReduce platform is very similar to Hive, providing company wide easy access to our core data assets.
Cluster Interfaces is looking for an individual to drive improvements to the systems used to schedule, debug and analyze map/reduce pipelines while facilitating the management of shared cluster resources and capacity planning.
- Consolidate the interfaces to Quantcast’s massive compute cluster
- Design rich data-visualizations to aid with analysis and resource management tasks
- Mentor and grow team members
- Work to understand the needs of customers in order to serve them better
- Contribute to the technical roadmap of the team
- BS in computer science or equivalent experience
- Experience leading platform development for big data companies
- Knowledge of authentication/authorization best practices in web frameworks
- Depth in Java
- Depth in a scripting language (python or ruby)
- Proven track record of operational excellence
Quantcast is a fast-growing, late-stage, pre-IPO startup headquartered in San Francisco with offices around the world. We are committed to creating an inclusive and diverse environment where everyone can confidently be their authentic self.