Arity - Hadoop Systems Engineer

Founded by The Allstate Corporation in 2016, Arity is a data and analytics company focused on improving transportation. We collect and analyze enormous amounts of data, using predictive analytics to build solutions with a single goal in mind: to make transportation smarter, safer and more useful for everyone.

At the heart of that mission are the people that work here-the dreamers, doers and difference-makers that call this place home. As part of that team, your work will showcase both your intelligence and your creativity as you tackle real problems and put your talents towards transforming transportation.

That's because at Arity, we believe work and life shouldn't be at odds with one another. After all, we know that your unique qualities give you a unique perspective. We don't just want you to see yourself here. We want you to be yourself here.
 

Job Summary

As a Hadoop Systems Engineer, you will be responsible for the design, implementation, Security and on-going support of Arity's Hortonworks Hadoop Distribution on AWS. You will partner with Data Scientists as well as product stakeholders to create an advanced analytics framework.

Key Responsibilities

  • Manage large scale multi-tenant Hadoop cluster environments residing on AWS.
  • Handle all Hadoop environment builds, including design, security, capacity planning, cluster setup, performance tuning and ongoing monitoring.
  • Perform high-level, day-to-day operational maintenance, support, and upgrades for the Hadoop Cluster.
  • Research and recommend innovative, and where possible, automated approaches for system administration tasks.
  • Creation of key performance metrics, measuring the utilization, performance and overall health of the cluster.
  • Deploy new/upgraded hardware and software releases and establish proper communication channels.
  • Work with appropriate stakeholders to ensure we have solid capacity planning and can manage our TCO.
  • Ability to collaborate with product managers, lead engineers and data scientists on all facets of the Hadoop Eco-System.
  • Ensure existing data/information assets are secure and adhering to a best in class security model.


Qaulifications

Technical Experiences Required
  • Possess at least 3 years of managing a multi-tenant production Hadoop environment;
  • A deep understanding of Hadoop internals, design principals, cluster connectivity, security and the factors that affect distributed system performance;
  • Proven experience with identifying and resolving hardware and software related issues;
  • Experience with Hortonworks DataPlatform;
  • Knowledge of best practices related to security, performance, and disaster recovery;
  • Expert experience with at least two of the following languages; SQL, Python, Java, Scala, Spark or Bash.


Additional Experiences Preferred
  • Experience managing Cloud Services (IaaS, PaaS). AWS Certification is preferred;
  • Experience with a number of different real-time integration tools such as Apache Flink, Kafka, Storm or Flume. Leveraging Hortonworks DataFlow is preferred;
  • Experience with CloudFormation, Git, Jenkins, Ansible a plus
  • Experience with complex networking infrastructure including firewalls, VLANs, and load balancers.
  • Experience as a DBA or Linux Admin


Skill/Attributes

  • Proven ability to lead multiple high priorities initiative with aggressive timelines leveraging an agile framework.
  • Comfortable performing in a fast paced, dynamic and ambiguous business environment.
  • Ability to concentrate on a wide range of loosely defined complex situations, which require creativity and originality, where guidance and counsel may be unavailable.
  • Demonstrated a product mindset with an ability to set forward thinking and direction.
  • Excellent listening and communication skills.
  • Must be able to communicate technical information clearly, able to 'translate' between diverse groups of technical and non-technical individuals.
  • Ability to synthesize large amounts of complex data into meaningful conclusions and present recommendations to a vast array of individuals.
  • Partner with business units in order to improve the effectiveness of business decisions.
  • Ability to structure documents to effectively communicate with senior leadership and drive alignment and decision making.
  • Strong problem solving skills with the ability to offer why one technical solutions is more preferable than another.
  • Ability to analyze and interpret complex concepts.
  • Ability to lead a group of employees (directly or indirectly).
  • Ability to work in a team environment in a collaborative manner.
  • Proven track record of sound, effective decision making.
  • Bachelor's Degree in Computer Science / Business / Data Sciences required, Master's Degree Preferred.


The candidate(s) offered this position will be required to submit to a background investigation, which includes a drug screen.

That's the day-to-day, now let's talk about the rest of it. As we mentioned, Arity was founded by The Allstate Corporation. But you'll be working for-and at-Arity. It's the best of both worlds. You'll get access to the full suite of Allstate benefits and work in a fast-paced startup culture. That's more than just free breakfasts, brain breaks and ping pong. It's a culture that encourages you to be you.

Sound like a fit? Apply now! We can't wait to meet you.

Arity.com Instagram Twitter LinkedIn

Allstate generally does not sponsor individuals for employment-based visas for this position.

Effective July 1, 2014, under Indiana House Enrolled Act (HEA) 1242, it is against public policy of the State of Indiana and a discriminatory practice for an employer to discriminate against a prospective employee on the basis of status as a veteran by refusing to employ an applicant on the basis that they are a veteran of the armed forces of the United States, a member of the Indiana National Guard or a member of a reserve component.

For jobs in San Francisco, please click "here" for information regarding the San Francisco Fair Chance Ordinance.

For jobs in Los Angeles, please click "here" for information regarding the Los Angeles Fair Chance Initiative for Hiring Ordinance.

It is the policy of Allstate to employ the best qualified individuals available for all jobs without regard to race, color, religion, sex, age, national origin, sexual orientation, gender identity/gender expression, disability, and citizenship status as a veteran with a disability or veteran of the Vietnam Era.


Back to top