Software Development Engineer - Big Data (Level 5)

3+ months agoSunnyvale, CA


AmazonBot is a smart, fast and polite Web crawling service, that is constantly learning about the Web and passing collected information to customers exploring Web content. Our crawler is smart, in that, it understands and organizes the crawled Web content to provide Amazon wide organizations fresh, relevant data required to improve their services in a timely manner. Our team deals with massive scale of data in the order of Petabytes and we need someone who is excited to solve complex challenging problems at scale.
A day in the life
We deal with web scale data which means querying, parsing and processing them via big data pipelines (EMR, Glue, etc). Our crawler runs on the AWS infrastructure (SQS, Lambda, Fargate, Dynamo, etc) and we mainly program in Java and Python
About the hiring group
Our team tries to have a healthy balance between work and play. We celebrate our successes and milestones with awesome parties and we are not afraid to take risks, even if it causes unintentional mistakes along the way. We believe in learning from our mistakes and moving forward.

Why should you join the Alexa team? Here are a few reasons:
• We ship software frequently, get fast feedback from real customers around the globe, and see the results of our work come to fruition.
• There are real-world problems to solve that you won't find ready-made answers for. For example, how do you process massive data in the least time possible. How can you improve Alexa to to answer more questions?
• We are a small, tightly-knit team that is defining the future of Alexa.
• We look for ways to make the our service configurable, maintainable and visible as easy, intuitive and time efficient as possible.

Role Responsibilities:

• Design & Write code: • Write code primarily in Java, Scala, Python and Rust and design scalable, fault tolerant and distributed applications running on linux environments.
• Test, test, test: Our software engineers don't just write code, they also test the heck out of it. You will write unit tests, integration tests and regression tests and find ways to automate them.
• Troubleshoot and fix operational problems: You will work with systems engineers to troubleshoot operational problems and fix them. If you are so inclined, you will also automate troubleshooting procedures and write tools. And when the issues concern specific customers, you will also have the opportunity to talk to them and understand their space better.
• Influence product direction: You will propose your ideas for the future of the space to product management, and get to shape the roadmap.
• Hire and mentor others: You will get to interview people for the team, and to mentor other engineers
Apart from this, you should have great problem solving skills, and confidence in learning and growing your skills in many AWS technologies, including EMR, Kinesis, Lambda, DynamoDB, Spark, EC2, S3 and many others. You love keeping abreast of the latest technology and use it to help you innovate. You should also have strong leadership qualities, great judgment, clear communication skills, and a track record of delivering resilient production ready products.

If this sounds great and you think you have the chops, we'd love to have you on board. Alexa is changing the world and specifically the human machine interaction. This is a great opportunity to be a part of this exciting business.


• 2+ years of non-internship professional software development experience
• Programming experience with at least one modern language such as Java, C++, or C# including object-oriented design
• 1+ years of experience contributing to the architecture and design (architecture, design patterns, reliability and scaling) of new and current systems.
• Bachelor's degree in Computer Science or related technical discipline.
• Minimum of 3+ years of experience in designing and building multithreaded and fault tolerant applications in Java or Scala.
• Experience in big data processing.
• Experience with Spark.
• Distributed systems, design and implementation experience.
• Passionate problem solver, self starter and willing to dive deep into unfamiliar and existing code base.
• Troubleshooting and debugging production issues.
• Writing automated unit, integration and regression tests.
• Scripting in either python, bash or ruby on linux.


• MS in Computer Science or related field/PhD
• Experience with Scala, Rust
• Experience in cloud computing and specifically in AWS services
• Strong customer focus, ownership, urgency and drive
• Demonstrated ability to achieve stretch goals in a highly innovative and fast paced environment
• Excellent communication skills and the ability to work well in a team
• Experience in using popular AWS technologies such as Kinesis, Lambda, EC2, S3.
• Experience in designing and building high-volume, high-performance orchestrated and resilient backend processes.
• Experience in using big data platform and tools such as AWS EMR.
• Experience with building data pipelines and applications to stream and process datasets at low latency.
By submitting your application here, you can apply once to be considered for multiple Software Engineer openings across various Amazon teams. If you are successful in passing through the initial application review and assessment, you will be asked to submit your career and personal preferences so that our dedicated recruiters can match you to the right role based on these preferences.

Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit

Job ID: Amazon-1315032