Federal Principal Product Support Engineer needed for a
Direct-Hire opportunity with SOC’s client to work hybrid onsite in
Oklahoma City, OK, Dallas or Houston, TX or Montgomery, AL.
** Active Secret Clearance is required for this position**
Job Description:
Role nature: This is a senior field-engineering position, not a traditional support or escalation-management role. You will work hands-on across the full infrastructure stack — from bare metal and hypervisors to networking, containerized workloads, and application-layer issues. Technical depth across multiple layers of the stack is a core requirement; you will be expected to access nodes directly, review system logs, and diagnose live production environments.
How escalations work: You serve as the primary engineering escalation point for on-site support teams. When a field team surfaces a vague symptom “this service isn't responding" — you take ownership: filter the noise, reproduce and isolate the root cause, and deliver a structured defect report with clear reproduction steps that product engineers can act on immediately. You will drive permanent fixes, contribute to product improvements, and keep customers' Private Cloud Enterprise environments operational and secure.
US Citizenship required
What You’ll Actually Do
A typical week in this role involves:
- Receiving escalations from on-site Level 1–3 teams, triaging root cause hands-on, and owning the technical path to resolution across the full infrastructure stack — spanning bare metal, hypervisor, networking, and application layers — using direct access to production environments, log analysis, and service-level debugging
- Engaging directly with engineers: writing structured defect reports with reproduction steps, attending engineering triage calls, and validating patches in customer environments before broader rollout
- Translating field-observed symptoms into actionable technical requirements for product and engineering teams — filtering noise from vague customer complaints into precise, reproducible problem statements
- Training and enabling a managed services team field engineers who are new to the PCE platform — producing runbooks, leading technical walkthroughs, and building their operational confidence
- Monitoring customer environments; triaging alerts and contributing to dashboard and observability improvements
- Conducting on-site work at customer locations
- Hardening environments to DISA STIG requirements and supporting audit readiness activities
- Producing customer-facing incident summaries and internal knowledge base articles after each major resolution
Core Responsibilities
1. Engineering Liaison & Escalation Management
- Act as the primary conduit between on-site support teams and BU engineering — translating field observations into well-structured defect reports, feature requests, and technical requirements
- Own escalations that have exceeded Level 3 resolution capacity; lead hands-on technical investigation and drive closure
- Participate in engineering standups, triage calls, and sprint reviews to represent field and customer priorities
- Validate engineering fixes, patches, and product updates in customer environments before production rollout
2. Deep Technical Troubleshooting (Hands-On)
- Lead in-depth, hands-on diagnostics directly on the full infrastructure stack using log analysis, and service-level debugging:
- Virtualization: VMware ESXi, vCenter, vSAN, NSX — and KVM/libvirt (like Morpheus VM Essentials)
- Containers & Orchestration: Kubernetes (pod networking, persistent storage, ingress, cluster lifecycle)
- Operating Systems: Linux (kernel, systemd, storage subsystems, networking — primary platform) and Windows Server (AD, DNS, Group Policy)
- Networking: TCP/IP, VLANs, routing protocols, overlay networks (VXLAN), load balancers, firewalls, east-west datacenter traffic
- Cloud/Hybrid: AWS, Azure, GCP integration with on-premises Private Cloud environments
- Identify root causes for recurring issues; architect permanent resolutions and drive systemic fixes through engineering
3. Observability, Monitoring & Operational Tooling
- Work within platform monitoring and alerting; manage and triage incidents routed through ITOM/ITSM
- Identify gaps in dashboards, alerting thresholds, and operational visibility; write Jira tickets and requirements to drive improvements through the engineering team
- Contribute to the continuous improvement of observability tooling so day-two field teams can detect and respond to incidents faster and with greater independence
4. Field Team Enablement & Training
- Train and enable a managed services team of field engineers — producing runbooks, delivering technical walkthroughs, and building their operational readiness
- Develop platform onboarding materials, standard operating procedures, and troubleshooting guides for field consumption
- Be a force multiplier: success means the field team escalates less over time, not more
5. Federal Compliance & Security
- Lead DISA STIG compliance efforts — implementation, verification, and documentation
- Support system hardening, audit preparation, and continuous compliance monitoring for federal customer environments
- Apply security best practices across all supported platforms and ensure configurations meet federal standards
6. Technical Requirements & Product Improvement
- Document and communicate product gaps, field-observed defects, and enhancement opportunities to product management and BU teams
- Develop automation and tooling using DevOps principles, CI/CD pipelines, and Infrastructure as Code where applicable
- Author detailed troubleshooting guides, runbooks, and technical documentation for internal and external use
7. Customer Engagement & Communication
- Serve as the trusted technical advisor to federal stakeholders — provide clear, accurate communication during incidents and escalations
- Deliver executive-readable incident summaries alongside deeply technical root cause analyses
- Build and maintain long-term relationships with federal client technical and program teams
8. Mentorship & Knowledge Transfer
- Guide and mentor Level 1–3 support engineers; share expertise and ensure best practices are consistently applied
- Lead knowledge transfer sessions following major resolutions to prevent recurrence
Required Qualifications
- U.S. Citizenship (required without exception)
- Active Secret clearance, or demonstrated ability to obtain one (U.S. citizen with clean background)
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or equivalent hands-on experience
- 10+ years of hands-on experience in infrastructure engineering, systems administration, or technical support — with demonstrated depth in at least three of the following domains:
- VMware (ESXi, vCenter, vSAN, NSX) and/or KVM/libvirt — both are relevant; KVM experience is highly desired
- Kubernetes and containerized workloads (hands-on cluster operations, not solely architectural review)
- Linux systems administration and live troubleshooting (Red Hat, Ubuntu, or similar)
- Enterprise networking (routing, switching, VLANs, overlay networks, load balancing, firewalls)
- Windows Server in hybrid/AD environments
- Hybrid or on-premises cloud infrastructure
- Proven, hands-on troubleshooting experience across the full infrastructure stack — direct access to production environments, log analysis, and live system debugging spanning bare metal, hypervisor, networking, and application layers are daily requirements of this role.
- Proven experience interfacing between field/support teams and engineering or product organizations, including writing structured defect reports
- Strong written communication skills: able to produce engineering-quality defect reports and executive-readable incident summaries
- Experience with enterprise monitoring/observability platforms (OpsRamp, Prometheus, Grafana, or equivalent) and ITSM ticketing systems (ServiceNow or equivalent)
Preferred Qualifications
- KVM/libvirt hands-on experience, or demonstrated ability to rapidly transition between hypervisor platforms
- VMware certifications: VCP-DCV, VCAP, or higher
- Certified Kubernetes Administrator (CKA) or Certified Kubernetes Application Developer (CKAD)
- Red Hat certifications: RHCSA or RHCE
- CompTIA Security+ or higher (CASP+, CISSP)
- Cloud certifications (AWS Solutions Architect, Azure Administrator, GCP Associate) — valuable for hybrid environment context
- Experience with DISA STIGs, NIST 800-53, or other federal security frameworks in an operational (not solely advisory) capacity
- Experience supporting federal or DoD programs in a customer-facing technical role
- Experience training or enabling large field teams on complex platforms — producing runbooks, SOPs, or leading technical onboarding programs
- Familiarity with Private Cloud Enterprise, GreenLake, or Synergy platforms
- Experience with Infrastructure as Code tooling (Terraform, Ansible) and CI/CD pipelines
- Data management experience: relational and NoSQL databases, storage architecture
SOC, a Day & Zimmermann company, is an Equal Opportunity Employer, EOE AA M/F/Vet/Disability.
Note: Any pay ranges displayed are estimations, which may have been provided by job boards. Actual pay is determined by an applicant’s experience, technical expertise, and other qualifications as listed in the job description. All qualified applicants are welcome to apply.
#INDSOC
#DZFED
Estimated Min Rate: $0.00
Estimated Max Rate: $0.00
