Senior Big Data Engineer - Search Ml Data Platform

3+ months agoPalo Alto, CA


The Amazon Search team builds the search engine that powers Amazon's worldwide shopping experience. Whenever a customer searches or browses using an Amazon website or application, we connect them to the products and services they are looking for.

Search ML Data Platform team is responsible for delivering high quality and fresh ML model training data, and providing seamless access to all ML artifacts through managed Federated Data Lake infrastructure. This big-data platform provides the ML training data to Amazon search ranking, matching quality, search economics and also powers live-site features, including search suggestions, query understanding, spelling, search result ranking, and personalization. More than 350 teams across Amazon consume our datasets. We are located in downtown Palo Alto, a short walk from numerous shops and restaurants, and right across from the Caltrain station.

As a Senior Software Development Engineer you will:

Design & Develop
• Lead the design, get your hands dirty and write code, and ultimately deploy big data and machine learning services. These services define the foundation of our search R&D processes, supporting science, product development and production of the world's largest product search engine.
• Possess expert knowledge in performance, large scale distributed system scalability, system architecture, and engineering best practices.
• Mentor other engineers, defining our challenging technical culture, and helping to build a team
Operational Excellence
• Obsess over operational excellence, evaluate system performance, security, design system metrics and driving quality improvements
• Obsess over customer needs and satisfaction

In this role, you'll help establish technical standards and drive the Search Data organization's overall System architecture and Engineering practices. You'll work on the hardest problems, building high quality, architecturally sound systems that are aligned with our business needs and built to handle Amazon's world wide scale. Your expertise is deep and broad; you're hands on, producing both detailed technical work and high-level architectural designs.


• 4+ years of professional software development experience
• 3+ years of programming experience with at least one modern language such as Java, C++, or C# including object-oriented design
• 2+ years of experience contributing to the architecture and design (architecture, design patterns, reliability and scaling) of new and current systems
At least 10 years of experience in all of the following:
• Software development and System design.
• Writing production code using Java, Scala, C++, and Python
At least 5 years of recent experience in:

• Big data and machine learning applications, for example: Hadoop, Spark, Kubernetes, MXNet, Tensorflow, etc.
• Implementation and tuning experience in the Big Data Ecosystem, (such as Hadoop, Spark, Presto, Hive), Database (such as Oracle, MySQL, PostgreSQL) and Data Warehouses (such as Redshift, Teradata, Vertica)
• Leading delivery of projects requiring work from multiple organizations
• Experience with public cloud infrastructure


• Graduate degree in computer science or related field (MS or Ph.D.)
• AWS Certification, e.g. AWS Solutions Architect, Developer, or SysOps Associate/Professional
• Data-driven and "quantitative" mentality. Grounded, detail-oriented, always backs up ideas with facts
• Ability to understand complex application data flows and bridge the gap between technical and business app requirement
• Strategic and thinks at a high level, but also hands-on, execution-oriented, and capable of getting things done quickly
• Entrepreneurial with an inherent sense of urgency and accountability
• Track record of implementing AWS services in a variety of business such as large enterprises and start-ups

Job ID: Amazon-1345221