Senior Principal Software Engineer (Go)
On-siteColumbia, Maryland, United States
Job Summary
Develop Go-based Crossplane Composition Functions to replace static YAML with runtime logic for provisioning isolated Kubernetes clusters across AWS, Azure, and on-premises. Design and version Kubernetes Custom Resource Definitions (CRDs) that abstract infrastructure configurations into declarative interfaces. Implement advanced sequencing, dependency mapping, and health-validation loops using Crossplane's Usages and Validation APIs to eliminate provisioning race conditions. Maintain and customize Cluster API and Crossplane providers, contributing patches upstream where applicable. Establish best practices for custom controllers, manage the Kubernetes Controller Runtime, and mentor engineers on software-first infrastructure approaches. Contribute to and execute Architecture Decision Records regarding API specifications and controller frameworks.
Required Qualifications
- Advanced Go (Golang) Proficiency: 3+ years of professional experience writing production-grade Go code, with a deep understanding of concurrency patterns (goroutines, channels) and profiling
- Kubernetes Internals & Operator Pattern: Strong hands-on experience using the controller-runtime library, client-go, Operator SDK, or Kubebuilder to build custom controllers and operators
- Cloud-Native Infrastructure Automation: Experience building and managing infrastructure declaratively, ideally with control-plane-based provisioning tools like Crossplane
- GitOps & Continuous Delivery: Proficiency with GitOps workflows and continuous delivery practices, using tools like Flux, Helm, and Kustomize to manage infrastructure lifecycles
- Systems Thinker: Ability to view physical hardware and hyperscale clouds not as static targets, but as endpoints controlled by software APIs
- Collaboration & Communication: Strong capability to bridge the gap between pure application developers and traditional infrastructure engineers, speaking comfortably to both
- Ruthless Solver of Race Conditions: An investigative mindset capable of debugging complex, asynchronous distributed systems where resources depend on sequential, timed events
- Extreme Ownership: A proven track record of acting as a technical steward for a product or subsystem, where you care as much about the operational metrics, edge-case failures, and technical debt of the system as you do about delivering new features
- Defensive Engineering Mindset: An engineering approach that assumes distributed systems will fail asynchronously. You naturally build deep observability, robust error handling, self-healing reconcile loops, and fallback mechanisms into your Go controllers
- Traceability and Documentation: A commitment to maintaining clear architectural records (ADRs), comprehensive unit/integration test suites, and clear API documentation, ensuring that your subsystem can be safely maintained by other engineers
Desired Qualifications
- Live-Production Incident Response: Hands-on experience managing incident response lifecycles and resolving critical failures within live, high-scale production systems
- Telemetry-Driven Debugging: Advanced capability in utilizing telemetry, distributed tracing, and metrics to perform root-cause analysis and resolve complex issues in live production environments
- API Lifecycle & Maintenance: Proven track record in managing long-term API maintenance, including defining upgrade processes and implementing n-1 compatibility testing to ensure seamless version transitions
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.