Will Generative AI awaken the need for serious focus legal research education?

The introduction of Generative AI to the practice of law has been anything but smooth. First there was the unfortunate case of Mr. Schwartz who used Chat GPT-3 to write a brief complete with hallucinated cases which he submitted to a federal  court in New York. Judge castell of the Southern District of New York noted that the attorneys had “abandoned their responsibilities.” More recently there have been the controversies related to a Stanford Human-Centered Artificial Intelligence (HAI) team study criticizing the quality of the Lexis and WL generative AI products. The study was so roundly criticized that it was revised and reissued. The HAI study’s conclusions regarding the Westlaw Precision AI and Lexis+  AI products requires a nuanced understanding of the HAI benchmarking definitions.  The HAI studies flag a wide range of issues including some which appear to be subjective. Problems noted range from a “true hallucination” to a factual error e.g. name of a judge, to the length of responses. Everyone agrees that legal generative AI products require serious benchmarking studies, but Stanford fumbled the ball.

Why are law firms are holding off generative AI adoption for legal research?

Read the full post on Legal TechHub

  • The trustworthiness of current technology solutions 86%
  • The quality of current technology solutions 75%
  • Hallucinated/invented content concerns 74%

Hallucinated cases have created a false hysteria. It is a genuine Issue but there is a real solution.  There are a few factors that have the heightened the genuine risks posed by hallucinated cases.

  • Does generative AI pose truly unique risks for legal research? In my  opinion, there is no risk that could not be completely mitigated by the use of traditional legal research skills? The only real risk is lawyers losing the ability to read, comprehend and synthesize information from primary sources.

Are law firm GC offices and Ethics committees over-reacting to Generative AI after ignoring the risk latent in prior generations of legal research technology? In my opinion yes. They are only reacting to the hyperbole surrounding Generative AI. The risks of GAI can be better understood in the overall context of the long evolution of legal research products and technologies.

 Law firm GCs and ethics officer have largely ignored the general decline in lawyers ability to employ proper legal research techniques.  This is a complaint I have heard from partners for decades. Bar exams don’t test legal research competencies. Most law schools don’t require advanced legal research training. Most law firms don’t have mandatory legal research training requirements. I wish I had a dollar for every associate to excused him/herself from  their new associate legal research training class because they had already been given a legal research assignment…. Before they had been introduced to the best research resources and firm’s research best practices!

Lawyers have been outsourcing pieces of their legal analysis to “technologies” and editors for over 100 years.  Traditional examples of research shortcuts include headnotes, syllabi, citators, citation flags, treatises, practical guidance tools, brief checkers.

Lawyers have been using products that they didn’t completely understand for decades. I don’t recall GC offices and ethics committees trying to holdback the adoption of traditional online legal research products. Lexis and Westlaw were not born as complete repositories of all U.S. legal materials. They grew over time. I remember Westlaw when it only included headnotes, I remember Lexis when it only had Ohio caselaw. Boolean searching and the old research command languages certainly resulted in incomplete research results. There was a time when a lawyer could retrieve a case with “on point” language without realizing  that the text appeared in a dissent. In “the old days” a lawyer would have to print out the full case and read it to discover that the text was in the dissent.

How many lawyers have understood the prior generation of natural language outputs? Lexis, Westlaw and newer rivals Bloomberg Law and Fastcase/vLex offer “non generative AI” products which provide algorithm based natural language research results. Each system displays results ranked by their proprietary  algorithm. The results can be dramatically different. I have never met a technology executive who could  explain the “black box” results of a natural language query.

Is the generative AI hysteria warranted?

Every generation of research tools has contained some level of risk.  Over the years, I have discovered mistakes in such revered tools as Westlaw headnotes and Shepard’s citators. The solution has always been the same – a lawyer must read the underlying authorities and draw their own conclusions.

 The Provenance of Documents is a risk in legal research – Should lawyers be using Google for legal research? I would say “no.” Even though Google doesn’t hallucinate legal documents, it also does not verify the authenticity or provenance of legal documents it serves up in its search results. I do not recommend using public open-source legal documents, even if they are located using a traditional Google search rather than an LLM. Open-source legal documents, even if not hallucinated, could be fakes or contain errors. Law firms subscribe to premium legal research platform (Lexis, Westlaw, Bloomberg Law,vLex, Wolters Kluwer  et.al.) to assure that lawyers have access to authentic and editorially enhanced legal documents.

  Do Generative AI products from Lexis and Westlaw, and vLex  have the same risks of “hallucinated cases as  GAI products using open source data? No. Lexis, Westlaw and vLex have private curated databases which include only authoritative content. They have built controls including Retrieval Augmented Generation (RAG) into their systems which prevent the LLMs from generating “hallucinated” cases, but as of June 2024 can not guarantee that responses such as summarizations are 100% accurate.

 Trust but verify. Lexis + AI, Westlaw Precision AI, vLex Vincent AI will speed up the research process – but in the end these are “trust but verify” technologies. Lawyers will still need to read and cite check any case that is recommended using a commercial Generative AI product designed for the legal market.

Back in the 1980’s there was a big push in the law librarian community to promote legal research training. Bob Berring, Dean and Director of the Library at the University of California Berkley, released his swashbuckling Commando Legal Research video series which contained some commandments of legal research including  the “no longer obvious to associate’s”:  “ read any case you are citing, “ and “cross check and validate your research. “

Maybe GAI can only succeed if we go back to the future?

Law librarians are the GAI training and risk mitigation teams which already exist in many law firms. However legal research training doesn’t necessarily get the same level of endorsement and inclusion as other skillsets. Legal Research education needs to be put on a par with all of the other core competencies included in “Associate Academies.”  if firms want to mitigate the risks of all  legal research technologies including Generative AI.