Applied Data Scientist, Finance AI Evaluation & Datasets
$150,000–$175,000 year
RemoteInn, Oberösterreich, Republic of Austria
Job Summary
Design and validate datasets for financial AI evaluation, focusing on unstructured and multimodal data like PDFs, scanned documents, and call transcripts. Translate customer goals into concrete dataset specifications, taxonomies, and acceptance criteria for training, fine-tuning, and monitoring LLMs, vision-language models, and AI agents. Build statistical tooling for bias analysis, leakage detection, and distribution shift checks to support model risk management and auditability. Define annotation guidelines, sampling strategies, and adjudication workflows with finance SMEs to ensure domain validity and compliance. Partner with research engineers to instrument datasets into evaluation pipelines and instrument rubric-grounded LLM-as-judge prompts. Own data quality end-to-end, from PII handling to provenance tracking, while supporting customer discovery and proposals. Stay current on regulatory developments and emerging evaluation methodologies for high-stakes financial workflows.
Required Qualifications
- 5+ years of data science experience
- at least 2+ years in financial services, fintech, banking, or a comparable regulated data environment
- Real working knowledge of financial data and workflows: financial statements, SEC filings, transaction data, and other common financial-services document types
- Hands-on experience with unstructured and multimodal financial data — some combination of PDFs, scanned documents, spreadsheets, charts, or call transcripts
- Hands-on experience designing datasets for ML — not just consuming them
- Familiarity with LLM-based and multimodal financial AI workflows: prompt design, rubric-based evaluation, RAG, LLM-as-judge methods, and the limitations of automated evaluation in high-stakes contexts
- Strong Python and SQL
- comfort with pandas, scikit-learn, or equivalent
- working familiarity with Hugging Face, PyTorch, or model APIs
- Statistical literacy: sampling design, inter-annotator agreement metrics (e.g., Cohen's kappa), confidence intervals, and the ability to push back when a number is being over-interpreted
- Solid grasp of financial services privacy, compliance, and governance: PII handling, GLBA or equivalent privacy regimes, MNPI sensitivity, and documentation fit for regulated AI programs
- Excellent collaboration skills — upstream with a Technical Solutions Architect, sideways with research scientists and engineers, and downstream with SME annotators and quality teams
- Degree in a relevant field — statistics, data science, economics, finance, or a related quantitative field, or equivalent demonstrated experience
- Experience designing evaluations for LLMs, VLMs, or multimodal models in financial reasoning, filings analysis, or fraud/AML contexts
- Experience with document AI, OCR/post-OCR quality, or table and chart extraction for complex financial documents
Desired Qualifications
- Familiarity with financial standards or protocols such as XBRL, ISO 20022, or GAAP/IFRS reporting concepts, etc.
- Formal finance credentials such as CFA, FRM, or MBA backgrounds, etc.
- Experience with agentic evaluation, AI observability, experiment tracking, or tools such as Weights & Biases or LangFuse
- Familiarity with model risk management frameworks, validation documentation, fairness/bias auditing, or consumer protection analysis
- Experience with multilingual or cross-border financial data, or published/open-source work in financial AI or model governance
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.