[Hiring] Senior Linux Systems Engineer @Nationwide IT Services

Role Description

Nationwide IT Services, NIS, is seeking a hands-on Linux Systems Engineer to support mission-critical scientific systems and infrastructure for the U.S. Geological Survey (USGS).

We are looking for someone who enjoys figuring out why complex systems are not working. This position is centered on Linux systems administration and technical operations, but the environment crosses:

• Applications
• Databases
• Networking
• Automation
• Monitoring
• Security
• Messaging systems

The right person does not need to be an expert in every technology in the environment. They do need to be comfortable taking ownership of a technical problem, investigating it across multiple layers, identifying the root cause, and either resolving the issue or providing enough technical information to get it to the right person for resolution.

This is not a traditional CI/CD-focused DevOps position. While DevOps tools and practices are part of the environment, success in this role depends more heavily on:

• Strong Linux fundamentals
• Technical curiosity
• Troubleshooting ability
• Understanding how systems interact

What You Will Do

• Administer, maintain, troubleshoot, and support Linux-based production, development, test, and supporting infrastructure.
• Investigate system and application problems through logs, configuration, monitoring data, network behavior, application code, database activity, and other available evidence.
• Perform root-cause analysis of technical issues and drive problems toward resolution.
• Determine when an issue can be resolved directly and when it should be escalated or routed to software developers, network engineers, security personnel, or other technical specialists.
• Perform Linux system maintenance, patching, configuration, software installation, service management, performance troubleshooting, and operational support.
• Support system security and hardening activities, including vulnerability remediation, access controls, host firewalls, security configuration, and compliance-related changes.
• Use and maintain automation and configuration-management capabilities supporting the environment.
• Work with monitoring systems to understand system health, investigate alerts, determine what should be monitored, and create or modify monitoring checks as needed.
• Support relational databases at an operational level, including installation and configuration, connectivity, access, basic SQL queries, and troubleshooting.
• Support messaging and event-driven infrastructure used to move information between systems and applications.
• Read application and automation code as part of troubleshooting. Trace program logic, identify relevant functions or processes, test components where appropriate, and communicate findings to developers.
• Develop or modify scripts and lightweight automation to support administration, troubleshooting, monitoring, and operational activities.
• Troubleshoot connectivity and system communication issues using a practical understanding of networking fundamentals.
• Work with technical staff across systems, software development, networking, security, and scientific operations to resolve issues that cross traditional technology boundaries.
• Document technical findings, configurations, troubleshooting steps, and operational procedures.

Qualifications

• Strong hands-on Linux systems administration experience, preferably in enterprise or production environments.
• Demonstrated ability to independently troubleshoot complex technical problems and perform root-cause analysis.
• Strong understanding of Linux services, processes, logs, filesystems, permissions, packages, configuration, and general system operations.
• Experience performing system maintenance, patching, configuration, troubleshooting, and operational support.
• Experience using monitoring and logging information to investigate system or application behavior.
• Hands-on scripting or automation experience.
• Ability to read and follow application or automation code sufficiently to support troubleshooting, identify relevant logic, and communicate effectively with software developers.
• Working knowledge of networking fundamentals, including TCP/IP, DNS, ports and protocols, routing concepts, firewalls, and connectivity troubleshooting.
• Practical familiarity with relational databases, including database installation/configuration, connectivity, basic administration, and SQL queries.
• Familiarity with system security concepts such as patching, access controls, host firewalls, vulnerability remediation, and secure system configuration.
• Ability and willingness to investigate unfamiliar technologies rather than stopping troubleshooting when an issue crosses into another technical domain.
• Ability to communicate technical findings clearly and work collaboratively with specialists in other technical areas.
• High school diploma or equivalent.
• Must reside and perform work within the United States.
• Must be able to complete Government Public Trust suitability requirements and obtain and maintain required Government credentials, including a PIV card.
• Must be able to maintain a work schedule beginning between 8:00 a.m. and 9:00 a.m. Pacific Time, regardless of U.S. location. 100% Remote within the United States.

Requirements

• Approximately 5 or more years of relevant hands-on experience is preferred in Linux systems administration, infrastructure operations, systems engineering, production support, technical operations, or a related technical role.
• We will also consider candidates with less traditional experience who can demonstrate equivalent technical depth, strong troubleshooting ability, and the ability to work independently in complex technical environments.

Highly Desirable Experience

• Red Hat Enterprise Linux (RHEL), Rocky Linux, or similar enterprise Linux distributions.
• Python, Bash, or other scripting/programming languages.
• PostgreSQL administration and troubleshooting.
• Ansible or comparable configuration-management and automation technologies.
• Nagios, including creating or modifying monitoring checks or plugins.
• Prometheus and Grafana or comparable metrics and visualization platforms.
• NATS or ActiveMQ.
• Other message brokers or event/messaging platforms such as RabbitMQ, Kafka, or similar technologies.
• SELinux and Linux host firewall configuration.
• DISA STIGs, FISMA, vulnerability remediation, or experience operating systems within Federal or other regulated environments.
• AWS or other cloud infrastructure.
• CI/CD pipelines and automated deployment processes.
• Git/GitLab and source-control-based operational workflows.
• Experience supporting distributed systems where applications, databases, messaging, networks, and infrastructure interact.
• Additional networking experience beyond basic systems-administration networking knowledge.

The Kind of Engineer Who Will Do Well Here

This role is a good fit for someone who sees an unfamiliar technical problem and wants to understand what is actually happening and why. You may not immediately know the answer. That is expected. What matters is knowing how to investigate:

• Reproduce the problem
• Collect evidence
• Examine logs and monitoring data
• Understand dependencies
• Test assumptions
• Isolate components
• Follow application or system behavior
• Continue narrowing the problem until the root cause is understood

You should be comfortable saying: “I don't know yet, but I know how to start figuring it out.” You do not need to be a software developer, database engineer, or network engineer. You should, however, understand enough about those areas to recognize when they are relevant to a problem, investigate across those boundaries, and communicate effectively with the specialists who own them.

If your experience has given you broad technical fundamentals, strong Linux skills, and a habit of digging into difficult problems until they make sense, we would like to hear from you.

Benefits

• Medical, dental, and vision insurance
• Life and disability insurance
• 401(k) plan with employer match
• Paid holidays
• PTO (sick/vacation)
• Commuter benefits
• Employee assistance program (EAP)
• Educational reimbursement
• Pet Insurance

Salary

Salary range $100,000 to $140,000 annually. The salary range is a general guideline. We consider several factors when determining base salary offers, including the position's scope and responsibilities, the candidate's experience, education, and skills, and current market conditions.