QA Analyst

About The Role

We are seeking a QA Analyst to own quality assurance for LOIS for Word, Filevine’s AI-powered legal drafting product. This role is critical to building and leading the QA function, encompassing test strategy, process development, and automation implementation. The successful candidate will work at the intersection of legal AI and product quality, assessing everything from AI output accuracy to user interface behavior. A key focus is on embedding quality upstream in the development lifecycle, partnering closely with product and engineering teams to ensure test plans are established before feature releases. The ideal candidate has over two years of QA experience, a keen eye for both functional and AI-related quality issues, and thrives in ambiguous, fast-paced environments. They must be proactive in identifying potential issues and committed to delivering a robust, reliable product.

Qualifications

• Minimum of 2 years of QA experience, including ownership of test strategy for a product or significant feature area.
• Demonstrated ability to build QA processes and automation from the ground up in a dynamic, fast-moving environment.
• Strong analytical and problem-solving skills with attention to both functional correctness and AI output quality.
• Comfortable working in high ambiguity with rapid feature deployment and quick adaptation of test methodologies.
• Ability to reason about non-deterministic AI systems and define criteria for correctness where no single answer exists.
• Excellent written communication skills, capable of producing clear, concise, and evidence-backed bug reports.
• Experience with bug tracking tools and test management workflows.
• Preferred qualifications include experience testing AI or Large Language Model (LLM)-based products, CI/CD pipeline experience, UX/design sensibility, prior legal technology exposure, and programming skills beyond test automation.

Responsibilities

• Define and own the comprehensive test strategy, QA processes, and playbook for LOIS for Word.
• Collaborate early and continuously with product and engineering teams to integrate QA into the development lifecycle.
• Evaluate AI output quality across multiple dimensions such as accuracy, citation reliability, and reasoning scope.
• Design and maintain evaluation datasets that account for non-deterministic AI outputs and detect regressions over time.
• Test AI-generated document edits for accuracy and formatting fidelity across diverse document structures.
• Develop and maintain automated guardrails integrated into CI/CD pipelines to cover critical workflows.
• Produce structured, reproducible, and actionable bug reports supported by evidence.
• Partner with legal domain experts to understand varied customer use cases and translate them into effective test coverage.
• Adopt a product and design-oriented mindset to identify opportunities for improvement beyond functional correctness.
• Compare behavior across different product surfaces and document any discrepancies.

Benefits

• Competitive and fair compensation package ranging from $105,000 to $150,000, based on location, qualifications, and experience.
• Flexible work location options including remote work or hybrid/in-office presence in San Francisco, New York, Chicago, or Salt Lake City.
• Comprehensive medical, dental, and vision insurance for full-time employees.
• Maternity and paternity leave benefits for full-time employees.
• Short-term and long-term disability coverage.
• Opportunities for professional growth and learning from a dedicated leadership team.
• Access to premium company swag.
• Dynamic and rapidly growing company culture focused on innovation and employee success.

Equal Opportunity

Filevine is an equal opportunity employer. We are committed to creating an inclusive environment for all employees and do not discriminate based on race, color, religion, gender, sexual orientation, national origin, age, disability, or any other legally protected status. We value diversity and encourage candidates from all backgrounds to apply.