Lead Splunk Administrator

Infosys is seeking a Lead Splunk Administrator.
In the role, you will interface with key stakeholders and apply your technical proficiency across different stages of the Software Development Life Cycle in Splunk application. This position's primarily responsibility is to work on Splunk tools including Splunk Cloud, Splunk Enterprise, ITSI and Observability. You will also support knowledge transfer with the objective of providing value-adding consulting solutions that enable our clients to meet the changing needs of the global landscape. You will be part of a learning culture, where teamwork and collaboration are encouraged, excellence is rewarded, and diversity is respected and valued.

Candidate must be located within commuting distance of Austin/Southlake TX or Raleigh NC, Phoenix AZ or be willing to relocate to the area. This position may require travel to project locations.

• Bachelor's degree or foreign equivalent required from an accredited institution. Will also consider three years of progressive experience in the specialty in lieu of every year of education.
• At least 4 years of experience with Information Technologies.
• Experience in AWS, Prometheus, Splunk Workload-management app and Splunk deployment server.

• At least 4 to 10 years of experience in Splunk engineering and having Splunk certifications including Splunk certification of certified Splunk Architect, certified Splunk ITSI engineer and other Splunk certifications.
• Experience in AWS, Prometheus, Splunk Workload-management app and IT observability, grafana, terraform, ansible, chef or puppet, Splunk deployment server, GITHUB and other CICD tools: multitenant Splunk Core, Splunk ITSI, Splunk ES, SPL, REGEX, Kvstore, Splunk's workload management tool, Phantom, CRIBL, SIEM, DMC, Telemetry, Linux, Shell AWS, bitbucket and Python.
• The Splunk patching exercise happens off-business hours hence need to work on different shifts including weekends.
• Experience in weekly patch work of 800 to 1000, Splunk servers and ability to fix the issues in Splunk patching with proper due-diligence and security vulnerability analysis.
• Experience in supporting production Splunk issues, app releases, config deployments and ability to present the higher management on progress and issues on monthly basis.
• Experience in splunk SME from large scale Splunk multi-site cluster environments of 50+ TB per day ingestion with 50K+ UFs.
• Ability to pinpoint the root cause in no-time, perform RCAs and take measures to prevent.
• Ability to administer, patch, upgrade and handle Splunk premium applications such as enterprise security, ITSI, SIEM, UBA, IT observability, phantom, workload management tool.
• Experience in resolving production KVSTORE, bundle replication, bucket corruption, data rebalance, throughput and pipeline utilization, and capacity, proxy authentication, concurrency balance, Splunk infrastructure performance and other complex issues.
• Ability to implement and recommend the team on Splunk best practices and troubleshoot, resolve complex production issues in Splunk multi-tier environment.
• Ability to build complex Splunk dashboards, glasstables, eventtypes, summary indexes, service analyzers and drilldowns to monitor different complex use cases.
• Experience with advanced configuration of Splunk ITSI (including Entity Integrations, service insights, and event analytics), maintaining and administering enterprise Splunk ITSI and custom ITSI Glass Tables.
• Experience in Splunk search head clustering, indexer clustering, license master, deployment server, DMC, SH dispatch, storage, mem, HF & UF administration and other configurations. monitor, develop, maintain centralized logging system that handled critical infrastructure & operations. manage end-user authentication, authorization with SAML single sign-on, LDAP, OAuth authentication protocols and Splunk authentication.
• Experience of Workload management, Content pack integrations, DB connect, smart store, distributed MC and Splunk base Add-on integrations.
• Experience to analyze and track security vulnerabilities for patching. Splunk integration experience with akamai, tanium, qualys and other vulnerability tools.
• Experience with Splunk regular expressions(regex) and lookups for data enrichment. Built out metrics, reports and alerts with Splunk to achieve a stronger monitoring system.
• Experience with Splunk REST APIs, SDK, Python, Bash, Shell for automation requirements. Onboard data from different domains within the organization using Splunk Universal Forwarders, http event collector, TCP/UDP, Add-ons, Apps, scripted inputs etc.
• Experience of syslog daemon configuration principles, ideally in Syslog-NG and RSyslog configurations. Cloud experience (AWS, Azure, etc.) with GIT and bitbucket.
• Experience in syslog integration, props, transforms, anonymizations, data extraction.
• Experience in Design core scripts to automate Splunk maintenance and alerting tasks. Use Splunk REST API, curl statements to export data from Splunk.
• Experience in environment optimization using best practices, performance tunings.
• Experience on all Splunk knowledge objects like data models, lookups, macros, event type, saved searches, tags etc.
• Experience in SRE principals and establishing SRE program parameters.
• Strong knowledge of AWS services such as ECS, EC2, S3, RDS, and VPC. Experience with Infrastructure as Code (IaC) tools, such as Terraform.