Distinguished Engineer - Data Architect
On-sitePune, Maharashtra, India
Job Summary
Design and evolve a unified blueprint for streaming and batch data processing to ingest, normalize, and store petabyte-scale security telemetry. Partner with the Lead AI Engineer to architect storage, retrieval systems, and feature stores for low-latency AI inference and training. Enforce strict multi-tenant isolation, data privacy guardrails, and compliance controls into the data fabric. Drive infrastructure cost-efficiency and performance tuning across processing engines and cloud layers while writing clean, foundational code for critical components. Lead the ultimate technical authority on data scale, streaming architecture, and multi-tenant security within a lean three-person team.
Required Qualifications
- 12+ years of data engineering and architecture experience
- Proven track record of designing production systems handling massive, high-throughput streaming datasets
- Deep, authoritative knowledge of distributed processing and streaming engines (e.g., Apache Flink, Spark, Kafka, Pulsar)
- Knowledge of modern lakehouse architectures (e.g., Iceberg, Delta Lake)
- Mastery of polyglot persistence
- Knowledge of traditional relational databases
- Knowledge of NoSQL
- Knowledge of Columnar engines (e.g., ClickHouse, DuckDB)
- Knowledge of Vector databases (e.g., Milvus, Qdrant, Pinecone)
- Strong fundamentals in secure data architectures
- Knowledge of network topography
- Knowledge of encryption-at-rest/in-transit
- Knowledge of containerization (Kubernetes)
- Knowledge of cloud-native infrastructure (AWS/GCP/Azure)
- Rare combination of visionary architecture and hands-on execution
- Ability to thrive in ultra-lean teams
- Enjoyment of writing core code
- Pragmatism to build for immediate needs
- Ability to plan for massive scale
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.