On AI Evals, TAR vs. LLMs and What Will be Different Next Year - Aron Ahmadia (VP Applied Science, Relativity)

In advance of this year’s RelFest in Chicago, Aron Ahmadia, Vice President of Applied Science at Relativity, joins the podcast to discuss AI evaluation, AI benchmarking and the transition from traditional Technology Assisted Review (TAR) to generative AI in legal document review and eDiscovery. The conversation covers the daily operations of a global AI team, the concept of "evaluation as a product," and the methodology behind testing large language models (LLMs) for legal tasks. Aron also discusses a recent study that compared the accuracy and efficiency of TAR and human document review against Relativity's generative AI tool, aiR for Review.

Aron talks about:

• A day in the life of the VP of Applied Science at Relativity.

• Navigating the rapid pace of AI and predicting what remains constant in legal tech.

• Aron's background: From early computer networking in a law firm to data science at DARPA and Capital One.

• Defining "evaluation as a product" and the importance of maintaining private benchmarks for LLMs.

• The trade-offs between human evaluations and automated AI evaluations.

• TAR (Predictive Coding) versus Generative AI: How prompt-based reasoning differs from label-based classification.

Episode Credits

Editing and Production: Grant Blackstock

Theme Music: Home Base (Instrumental Version) by TA2MI

Episode Credits

Editing and Production: Grant Blackstock

Theme Music: Home Base (Instrumental Version) by TA2MI