Software Engineer - 2 (Python+AI)
On-siteBengaluru, Karnataka, India
Job Summary
Build and maintain LLM/speech eval frameworks for the voicebot, measuring task success, hallucination, and latency across model and prompt changes. Run fine-tuning experiments on open-weight models to determine when fine-tuning beats prompting, and benchmark engines on cost, latency, and quality to make data-driven recommendations. Diagnose and fix real production conversation failures using logs and traces, then ship changes into the live pipeline with instrumentation and alerting built in from day one. Take full ownership across the SDLC for your changes, including design, deployment, and monitoring, without needing to be unblocked daily.
Required Qualifications
- Solid grounding in ANNs and transformer architecture — attention, tokenization, decoding strategies — enough to reason about why a model behaves a certain way, not just call an API.
- Hands-on experience with LLM evals: building or running eval harnesses, LLM-as-judge setups, regression suites for prompt/model changes.
- Hands-on experience with fine-tuning, including PEFT/LoRA/QLoRA — on at least one open-weight model, for a real task (not just a tutorial).
- Working knowledge of speech/ASR-TTS evaluation — WER, latency, diarization, common failure modes in real (noisy, accented, multilingual) audio.
- Strong Python; comfortable reading/writing production code, not just notebooks.
- 2-4 years of software/ML engineering experience, with at least some of it in a production system (not purely research/academic).
- A track record of shipping and owning your own changes in production — you've been on the hook for something live.
- Strong analytical rigor — you instinctively ask 'how do we measure this' before shipping a change.
Desired Qualifications
- Experience with real-time audio/streaming systems and streaming vs. batch tradeoffs.
- Experience with agentic orchestration and tool-calling patterns for LLMs.
- Exposure to RAG patterns — embeddings, vector stores, retrieval strategies.
- Familiarity with self-hosting/serving open-weight models.
- Familiarity with observability for AI workloads — cost tracking, quality dashboards.
- Experience with multi-tenant SaaS constraints (per-tenant config, isolation).
- Prior experience specifically in voice AI / IVR / contact-center domains.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.