Kafka Administrator
HybridBengaluru, Karnataka, India
Job Summary
Design and optimize end-to-end data pipelines serving AI and analytics initiatives using Kafka, batch ETL, and cloud data lake patterns. Manage Apache Kafka clusters, including offset management, lag monitoring, and cluster maintenance, while driving migration to Kubernetes. Operate multiple database technologies like Redis, PostgreSQL, and MongoDB, performing routine maintenance, capacity planning, and security hardening. Develop automation scripts and CI/CD pipelines to reduce operational toil, manage AWS infrastructure, and maintain observability tooling. Participate in on-call incident response and support cloud migration tasks between on-prem and cloud environments.
Required Qualifications
- 4+ years of overall engineering experience, with demonstrated work across data pipelines, database operations, and infrastructure—not limited to a single technology.
- Hands-on experience with Kafka or another distributed streaming platform (Kafka, Kinesis, Pulsar) in a production environment.
- Operational experience with at least two of the following database technologies: Redis, PostgreSQL, MySQL, MongoDB, Redshift or Snowflake.
- Proven data pipeline experience: building or maintaining ETL/ELT processes, batch jobs, or streaming pipelines that serve analytics or operational use cases.
- Strong scripting and automation skills (Python and/or Bash) with a track record of reducing manual toil.
- CI/CD pipeline development and Infrastructure-as-Code experience (Terraform preferred).
- Solid Linux fundamentals: networking basics, log analysis, system troubleshooting, process/memory/disk management.
- Cloud engineering experience, preferably AWS (EC2, VPC, IAM, MSK/ElastiCache, CloudWatch).
- Familiarity with observability practices—metrics, logs, tracing—and incident response workflows.
- Understanding of distributed systems trade-offs: availability, consistency, partitions, backpressure.
- Strong communication skills including the ability to present platform status and technical decisions to non-technical stakeholders.
- Flexibility for on-call rotation and occasional weekend support.
Desired Qualifications
- Experience with stream processing frameworks: Kafka Streams, Apache Flink, or similar.
- Familiarity with data lake table formats: Apache Iceberg, Delta Lake, or Apache Hudi.
- Hands-on experience with AWS data services: Athena, Glue, Redshift, or Lake Formation.
- Working knowledge of Snowflake—data loading, query optimization, or integration patterns.
- Understanding of data quality, data lineage, or schema registry concepts.
- Container/Kubernetes depth: deployments, StatefulSets, storage classes, Helm.
- On-prem infrastructure experience: VMware/KVM, SAN/NAS storage, network configuration.
- Security and compliance experience, particularly PCI-DSS (Payment Card Industry Data Security Standard).
- Secrets management tooling: HashiCorp Vault, AWS Secrets Manager, or similar.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.