AI SDET

Hiring: AI SDET / Evaluation Engineer

Location: New York City – Hybrid

Employment Type: Full-Time

We are looking for an AI SDET / Evaluation Engineer to help build, test, and evaluate our AI-powered applications and systems.

This is an exciting opportunity for software testing professionals who are interested in working at the intersection of Software Quality Engineering, Generative AI, and AI-powered automation.

What makes this role different?

You will leverage AI coding agents to perform and accelerate traditional SDET activities. You will also set up and use AI coding agents to test our company's AI systems, ensuring the applications are reliable, accurate, secure, and production ready.

Key Responsibilities

• Build and maintain automated evaluation tests and scripts.
• Develop and manage evaluation datasets and test scenarios.
• Evaluate AI outputs for accuracy, completeness, citations, and instruction following.
• Test AI workflows, APIs, document ingestion, integrations, and structured outputs.
• Analyze failures, investigate root causes, and document defects.
• Compare performance across different models, prompts, and application versions.
• Help calibrate AI graders against human evaluations.
• Maintain regression suites and support CI/CD integration.

Required Skills & Qualifications

• 5+ years of software development experience using Python, Java, or C#.
• Foundational understanding of LLMs, Generative AI, and common AI failure modes.
• Hands-on ability to work effectively with AI coding agents.
• Knowledge of APIs, Git, automated testing, and SDLC practices.
• Familiarity with CI/CD tools such as Jenkins and Jira.
• Understanding of statistics, sampling, precision, and recall.
• Strong analytical, troubleshooting, and documentation skills.

Preferred

• Experience with AI application testing, LLM evaluations, or AI quality assurance. Academic, internship, and project experience in AI evaluation is also welcome.