Senior Software Engineer – Big Data
There’s a certain kind of energy here. You can feel it. It’s what happens when your job makes a tangible, positive impact for your clients, and our world. It’s what happens when you work with a team that’s all in, all the time. It’s what happens when you work at Lytx.
We’re looking for hungry, humble and capable people who want to be part of building something special. Ideally, people who want to play a part in shaping the future where your work makes a direct contribution and helps to save lives. Lytx is rapidly scaling and looking for like-minded software engineers currently working with big data.
The Senior Software Engineer – Big Data will be an integral part of the core team designing and implementing the Hadoop ecosystem for Data Science, Data Analytics, Data Warehouse and Business Intelligence by leveraging variety of data types. One of the primary roles will include designing and developing large scale real-time and batch data ingestion and integrations using the Hadoop Platform. If you love building new big data platforms, guiding or assisting software and data engineering teams toward the development of big data solutions, then this is the role for you!
Key Duties & Responsibilities:
- Design and develop end-to-end Hadoop data ingestion process. Sufficiently decompose and transform the NoSQL, Binary, Sequential Flat file or XML data to Data warehouse through MapReduce, Pig, Hive, Avro and etc.
- Design and develop Hadoop data integration in both real-time and scheduled mode leveraging appropriate Hadoop open integration framework (Storm, Flume, Sqoop, Oozie, and etc.)
- Manage Hadoop clusters and guide the operational best practices for Hadoop cluster configuration and job processing.
- Build Libraries, user defined functions (UDF), and frameworks around Hadoop designed to be reusable and scalable for enterprise wide use.
- Manage the performance and allocations in open distributed file system (HDFS) leveraging the Hadoop data lifecycle and retention policy.
- Implement standards based monitoring and tuning practices for the 20+ nodes Hadoop ecosystem.
- Translate the functional and non-functional requirements from business, product and engineering teams to appropriate Hadoop technology solution design.
- Document the big data solution design and clearly communicate the target solution patterns to the technical audiences.
- Minimum 4 years of experience of Java application development experience; strong knowledge of object-oriented-programming (OOP) with J2EE architecture expertise.
- Must have an experience of setting up J2EE application build, deploy and promote configurations experience; Java library/source control management experience preferred.
- Strong knowledge of SQL (Oracle, NoSQL, SQL Server) databases.
- Solid backgrounds in fundamentals of computer science and development process.
- Experience with open source frameworks and tools.
- Familiarly with Hadoop, Map-Reduce, Hive and other big-data based open source projects.
- Understanding of virtualization and cloud computing concepts and technologies.
- Experience with Unix-like operation systems is required.
- The ability to think clearly and quickly while taking ownership of important issues is imperative.
- Strong foresight and the ability to anticipate challenges or obstacles.
- Drives change in ambiguous situations and is a champion for those new approaches.
- Other duties as assigned.
- 1+ years experience of designing and developing Hadoop based solutions with strong knowledge of related framework including MapReduce, YARN, Hive, Sqoop, Flume and Oozie.
- Experience in setting up, optimizing and sustaining Hadoop ecosystem at mid-to-large scale.
- Familiarity with AWS technologies – EC2, S3, VPC.
- Expert level Java coding experience.
- Greenplum/Teradata database development experience for large scale of data warehouse.
- Experience with distributed systems, persistence, caching and concurrent programming.
- Experience in Storm/Kafka will be a huge plus.
Lytx, Inc. is proud to be an equal opportunity/affirmative action employer. We are committed to attracting, retaining and maximizing the performance of a diverse and inclusive workforce. EOE/M/F/Disabled/Vet.
Meet Some of Lytx's Employees
Director Of Advanced Technologies
As the Director of Advanced Technologies, Gabriela and a team of scientists build the algorithms that collect and analyze data from the hundreds of thousands of Lytx event recorders in fleets around the world.
Back to top