Prompt Engineering Specialist
On-sitePune, Maharashtra, India
Job Summary
Design, write, and iteratively refine prompt templates for internal LLM agents covering code generation, spec interpretation, and enterprise domain reasoning tasks. Build and maintain a structured, versioned prompt library organized by task type and domain context. Develop evaluation frameworks to define success metrics, construct test sets, run A/B comparisons, and document performance regressions. Collaborate with Spec Engineering and AI Toolchain teams to ensure prompts are correctly parameterized and outputs meet downstream code quality standards. Investigate and mitigate failure modes such as hallucination and instruction drift. Stay current with advances in prompt engineering techniques and adapt internal practices accordingly. Document design decisions and empirical results in a shared knowledge base.
Required Qualifications
- 3+ years of hands-on experience in prompt engineering, NLP engineering, or applied LLM development, with a demonstrable portfolio of production prompt systems
- Deep practical knowledge of major LLM APIs (OpenAI GPT-4/o series, Anthropic Claude, Google Gemini) including token economics, context window management, and structured output techniques
- Experience building prompt evaluation pipelines: automated test harnesses, LLM-as-judge patterns, or human evaluation workflows
- Strong Python skills for scripting prompt experiments, parsing LLM outputs, and integrating with LangChain, LlamaIndex, or equivalent orchestration frameworks
- Ability to reason clearly about enterprise domain complexity and encode that domain knowledge into reliable, reusable prompt structures
- B.E. / B.Tech in Computer Science, AI/ML, or Linguistics; or equivalent practical experience with a strong portfolio
Desired Qualifications
- Experience with retrieval-augmented generation (RAG) architectures and vector databases (Pinecone, Weaviate, pgvector) in enterprise contexts
- Familiarity with the data engineering, BI, or analytics domain — ability to write prompts that reason correctly about SQL, data pipelines, ML model outputs, or business KPIs
- Prior work on multi-agent LLM systems where prompt design affects agent-to-agent communication and task decomposition
- Contributions to open-source prompt engineering toolkits, evaluation frameworks, or published benchmarks
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.