Senior Software Engineer - Big Data
Key Duties & Responsibilities:
- Play a key role in cutting edge big data solution implementation at Lytx.
- Member of core team to design and implement Hadoop ecosystem for Data Science, Data Analytics, Data Warehouse, and Business Intelligence by leveraging a variety of data types.
- Design and develop a large scale of real-time and batch data ingestion and integrations using Hadoop Platform.
- Guide and assist software and data engineering teams in designing and developing appropriate big data solutions in Hadoop framework for different data needs and use cases.
- Design and develop end-to-end Hadoop data ingestion process.
- Sufficiently decompose and transform the NoSQL, Binary, Sequential Flat file or XML data to Data warehouse through MapReduce, Pig, Hive, Avro and etc.
- Design and develop Hadoop data integration in both real-time and scheduled mode leveraging appropriate Hadoop open integration framework (Storm, Flume, Sqoop, Oozie, and etc.).
- Responsible for Hadoop clusters and guide the operational best practices for Hadoop cluster configuration and job processing.
- Build Libraries, user defined functions (UDF), and frameworks around Hadoop designed to be reusable and scalable for enterprise wide use.
- Responsible for the performance and allocations in open distributed file system (HDFS) leveraging the Hadoop data lifecycle and retention policy.
- Implement standards based monitoring and tuning practices for the 20+ nodes Hadoop ecosystem.
- Translate the functional and non-functional requirements from business, product and engineering teams to appropriate Hadoop technology solution design.
- Document the big data solution design and clearly communicate the target solution patterns to the technical audiences.
Education, Experience & Qualifications:
- Requires a Bachelor's degree in Computer Engineering, Computer Science, Mathematics, or related field of study plus six (6) years of experience in the job offered or in related software development role.
- Requires a minimum of 4 years of Java application development experience.
- Knowledge of object-oriented-programming (OOP) with J2EE architecture experience.
- Must have experience setting up J2EE application build, deploy, and promote configurations.
- Requires knowledge of SQL (Oracle, MySQL, SQL Server) databases.
- Requires a background in fundamentals of computer science and development process.
- Must have experience with open source frameworks and tools.
- Requires experience with Hadoop, Map-Reduce, Hive and other big-data based open source projects.
- Must possess an understanding of virtualization and cloud computing concepts and technologies.
- Experience with Unix-like operation systems.
Lytx, Inc. is proud to be an equal opportunity/affirmative action employer and maintains a drug-free workplace. We are committed to attracting, retaining and maximizing the performance of a diverse and inclusive workforce. EOE/M/F/Disabled/Vet.
Meet Some of Lytx's Employees
Senior Client Solutions Consultant
May operates as part of the Implementation Team, working to onboard new clients and manage relationships after the sale. She also trains clients on Lytx programs.
Back to top