Skip to main contentA logo with &quat;the muse&quat; in dark blue text.

Senior Principal Software Engineer - SDLC Evolution

New York, NY

Splunk is here to build a safer and more resilient digital world. The world's leading enterprises use our unified security and observability platform to keep their digital systems secure and reliable. While customers love our technology, it's our people that make Splunk stand out as an amazing career destination and why we've won so many awards as a best place to work. If you become a Splunker, we want your whole, authentic self, what we call your "million data points". So bring your work experience, problem-solving skills and talent, of course, but also bring your joy, your passion and all the things that make you, you.
Must-have Qualifications

  • Extensive experience (typically 10+ years) in software development with a focus on the SDLC for large-scale code bases (1M+ LOC, 200+ concurrent developers)
  • Track record of leadership roles in large tech companies with demonstrated ability to strategically plan and execute both incremental improvements and revolutionary changes.
  • Strong expertise with the developer lifecycles of multiple programming languages and domains (how to develop, debug, package, test, etc.) as we have code in several languages, ranging from Go and C++ to Python and JavaScript.
  • Experience with implementing software development and testing practices for extensive code bases.
  • Familiarity with hybrid delivery models, delivering software to both customer managed and cloud hosted environments.
  • Excellent communication skills and a history of collaborating effectively with cross-functional teams.
Nice-to-have Qualifications

Want more jobs like this?

Get Software Engineering jobs delivered to your inbox every week.

Select a location
By signing up, you agree to our Terms of Service & Privacy Policy.

We've taken special care to separate the must-have qualifications from the nice-to-haves. "Nice-to-have" means just that: Nice. To. Have. So, don't worry if you can't check off every box. We're not hiring a list of bullet points-we're interested in the whole you.
  • Experience crafting and delivering software as a service and working with cloud infrastructure services such as AWS EC2, S3, Kubernetes, etc. is desirable but not required.
  • Experience working with multiple cloud providers (like AWS, GCP, and Azure).
  • Experience working on decade+ legacy codebases.
  • Familiarity with mono-repos is a plus.
  • Ability to work across our multiple languages (polyglot).
Role Summary

Over the past few years, Splunk has been evolving how we develop and deliver our products to our customers. Through building internal platforms, services, and practices, we have been empowering our developers to own product quality, operations, and reliability. While we have made great strides in our journey to shift left, there is much room to grow.

That's where we need you. We're looking for an engineering leader to bring their experience and expertise in the software development lifecycle (SDLC) to evolve how Splunk developers build, test, deliver, and observe our products.

In this role, you will be measured on your impact to the four key DORA metrics: deployment frequency, lead time for changes, failed deployment recovery time, and change failure rate.

This role empowers you to be a catalyst for positive change, driving innovation and improvement across the entire software development process. Your ability to ideate, gain buy-in, and effectively roll out changes will be pivotal in shaping the future of our engineering culture and the experience felt by each and every customer.
Meet the Team

As Splunk is a 20-year old / multi-million LOC product, we have a ~200 person platform engineering organization that is looking for you, an expert in large-scale SDLCs.

The team you are joining is responsible for the end-to-end feedback cycle of code to customer back to developer. And you would be the key architect of that feedback loop.

Your peers will be Engineering Directors that lead our Developer Platform pillars (build, test, delivery, observability) as well as other Senior Principal Engineers who drive key aspects of the developer experience.
What we offer you
  • A constant stream of new things to learn. You'll learn how our whole stack works, from code compilation to log/metric observation in the wild. We're also always expanding into new areas like bringing in open source projects and contributing back, exploring new technologies, and seeking new ways to make our ecosystem more developer-friendly.
  • Impact. We give our engineers an environment in which they can contribute from day one while also providing opportunities for learning and growth.
  • Skilled and dedicated peers, all the way from engineering to product management and customer support. We are an engineering and product-focused company. Our engineers take a leading role in designing, architecting, building and testing our product.
  • Growth and mentorship. We believe in growing engineers through ownership and leadership opportunities. We also believe mentors help both sides of the equation.
  • A stable, collaborative and supportive work environment. We are totally remote friendly. You can choose to work from a Splunk location or you can be in any US time zone and work with the rest of the team around the globe.
  • Work-life balance. We don't expect people to work 12-hour days. We want you to have a successful time outside of work too. We trust our colleagues to be responsible with their time and commitment, and believe that balance helps cultivate an outstanding environment.
What you'll get to do
  • You will bring and share your diverse ideas and lived experiences to raise the tide for what good could look like.
  • You will spearhead the creation of innovative strategies for enhancing the SDLC, introducing new technologies, and optimizing workflows.
  • You'll collaborate closely with development teams to understand their needs, challenges, and aspirations.
  • You will partner with sister teams across the company (from developers to customer support and more) to create a cohesive story of how a product change gets into our customer's hands.
  • You will mentor engineers across the organization, building connections and support for the changes you are driving.
Splunk is an Equal Opportunity Employer

At Splunk, we believe creating a culture of belonging isn't just the right thing to do; it's also the smart thing. We prioritize diversity, equity, inclusion, and belonging to ensure our employees are supported to bring their best, most authentic selves to work where they can thrive. Qualified applicants receive consideration for employment without regard to race, religion, color, national origin, ancestry, sex, gender, gender identity, gender expression, sexual orientation, marital status, age, physical or mental disability or medical condition, genetic information, veteran status, or any other consideration made unlawful by federal, state, or local laws. We consider qualified applicants with criminal histories, consistent with legal requirements.


Base Pay Range

SF Bay Area, Seattle Metro, and New York City Metro Area

Base Pay Range: $242,400.00 - 333,300.00 per year

California (excludes SF Bay Area), Washington (excludes Seattle Metro), Washington DC Metro, and Massachusetts

Base Pay Range: $218,160.00 - 299,970.00 per year

All other cities and states excluding California, Washington, Massachusetts, New York City Metro Area and Washington DC Metro Area.

Base Pay Range: $193,920.00 - 266,640.00 per year

Splunk provides flexibility and choice in the working arrangement for most roles, including remote and/or in-office roles. We have a market-based pay structure which varies by location. Please note that the base pay range is a guideline and for candidates who receive an offer, the base pay will vary based on factors such as work location as set out above, as well as the knowledge, skills and experience of the candidate. In addition to base pay, this role is eligible for incentive compensation and may be eligible for equity or long-term cash awards.

Benefits are an important part of Splunk's Total Rewards package. This role is eligible for a competitive benefits package which includes medical, dental, vision, a 401(k) plan and match, paid time off and much more! Learn more about our comprehensive benefits and wellbeing offering at

Client-provided location(s): New York, NY, USA; Colorado, USA; South Carolina, USA; Maryland, USA; Texas, USA; Pennsylvania, USA; Arizona, USA; California, USA; Florida, USA; Virginia, USA; North Carolina, USA; Oregon, USA
Job ID: splunk-F5ibqj9pOh4Z
Employment Type: Other

Perks and Benefits

  • Health and Wellness

    • Health Insurance
    • Health Reimbursement Account
    • Dental Insurance
    • Vision Insurance
    • FSA
    • FSA With Employer Contribution
    • HSA
    • Life Insurance
    • Fitness Subsidies
    • On-Site Gym
    • Short-Term Disability
    • Long-Term Disability
    • Mental Health Benefits
    • Virtual Fitness Classes
  • Parental Benefits

    • Birth Parent or Maternity Leave
    • Non-Birth Parent or Paternity Leave
    • Fertility Benefits
    • Adoption Assistance Program
    • Family Support Resources
  • Work Flexibility

    • Flexible Work Hours
    • Remote Work Opportunities
    • Hybrid Work Opportunities
    • Work-From-Home Stipend
  • Office Life and Perks

    • Commuter Benefits Program
    • Casual Dress
    • Pet-friendly Office
    • Happy Hours
    • Snacks
    • Some Meals Provided
    • Company Outings
    • Holiday Events
  • Vacation and Time Off

    • Paid Vacation
    • Paid Holidays
    • Personal/Sick Days
    • Leave of Absence
    • Volunteer Time Off
  • Financial and Retirement

    • 401(K) With Company Matching
    • Company Equity
    • Stock Purchase Program
    • Performance Bonus
    • Financial Counseling
  • Professional Development

    • Tuition Reimbursement
    • Leadership Training Program
    • Learning and Development Stipend
    • Promote From Within
    • Mentor Program
    • Shadowing Opportunities
    • Access to Online Courses
    • Lunch and Learns
  • Diversity and Inclusion

    • Diversity, Equity, and Inclusion Program
    • Employee Resource Groups (ERG)