Lead Infrastructure Engineer

Overview
Lead Infrastructure Engineer
Bring your ideas. Make history.
BNY Mellon offers an exciting array of future-forward careers at the intersection of business, finance, and technology. We are one of the world's top asset management and banking firms that manages trillions of dollars in assets, custody and/or administration. Known as the "bank of banks" - 97% of the world's top banks work with us as we lead and serve our customers into the new era of digital.

With over 238 years of rich history and industry firsts, BNY Mellon has been built upon our proven ability to evolve, lead, and drive new ideas at every turn. Today, we're approximately 50,000 employees across 35 countries with a culture that empowers you to grow, take risks, experiment and be yourself. This is what #LifeAtBNYMellon is all about.

• should have strong knowledge in Linux
• should have programming fundamentals
• strong knowledge on distributed computing fundamentals
• should have strong knowledge on data lake - HDFS/Amazon s3
• Should have sound knowledge on data ingestion tools
• should have basic knowledge on cloud computing - AWS/Azure/GCP
• should have strong knowledge on scheduling tool - Apache Airflow
• should have strong knowledge on monitoring tool - nagious, cloudera manager,
• Expert advice on spark application, design to resolve scalability and performance issues
• Extensive experience in implementing data analytical algorithm using spark, mapredue and hive
• should have excellent understanding and hands-on experience with hadoop tools like spark, MR, hive,oozie, impala and sqoop
• Experience in integrating, transformation and aggregating large data sets into the hadoop ecosystem from highly -varied and high velocity data collection system
• Should have knowledge in python, java and shell scripting
• should have strong experience in working with rdbms and bigdata databases systems
• should have strong experience in hive,impala, SQL,HQL and spark and its performance optimization
• Expertise in writing hive queries and spark SQL
• Should have good understanding of the performance tuning of bigdata systems
• should have hands-on experience in developing Restful web services/API for bigdata projects
• Should have knowledge on design and develop a dynamic ETL framework for a bigdata projects
• Should have sound knowledge on jenkins
• should have knowledge on configuring Kerberos for the authentication of users and hadoop daemons
• should have hands on experience in Linux admin activities on RHEL &Cent OS Cluster monitoring
• Experience in understanding and managing Hadoop Log Files
• should have knowledge in leveraging and configured the use of Fair and Capacity schedulers
• experience in Monitoring workload, job performance and capacity planning using Cloudera Manager,experience with working CDP version up to 7.1.6
• should have knowledge in applying patches on Cloudera Cluster
• should have knowledge in Service Now ticketing tool
• Levereged and tuned the performance of Hive queries using the use of TEZ execution
• Experience in deploying and managing the multi-node development and production Hadoop cluster with different Hadoop components
• Worked with application teams to install Hadoop updates, patches, version upgrades as required
• HDFS cluster support and maintenance tasks like adding and removing nodes without any effect to running nodes and data.
• enterprise level security using LDAP, Kerberos and Sentry
• Should have knowledge on troubleshooting in Performance Monitoring of CPU, Memory, Disks, Network using various monitoring tools like Nagios, UNIX and Linux systems
• should have knowledge in ansible to maintain the large nodes cluster
• should have knowledge inversion control GIT with ansible
• should have knowledge in Install, Upgrade and maintain the Hadoop Clusters using Cloudera Manager
• should have knowledge in Design and Configure the Cluster with the services required (Hive server2, Kerberos, HDFS, Hue, Hive, Tez,Zookeeper,Spark3, infrasolr, Ranger Atlas
• Should have knowledge in TLS setup on HDFS, Yarn and Hue services
• should have knowledge in Setting up KDC Kerberos trust between multiple Hadoop clusters and tested validated adding peers and replication manager job set up
• should have knowledge in HDFS encryptions on Hadoop cluster using Ranger key trustee and RMS
• should have strong knowledge in cloudera cluster upgrade, CM upgrade and service pack upgrades

• Fortune World's Most Admired Companies & Top 20 for Diversity and Inclusion
• Bloomberg's Gender Equality Index (GEI)
• Human Rights Campaign Foundation, 100% score Corporate Equality Index
• Best Places to Work for Disability Inclusion , Disability: IN - 100% score
• 100 Best Workplaces for Innovators, Fast Company
• CDP's Climate Change 'A List'