News You Can Use

Links have updated their AI Benchmark to now include Open AI's o1 and Google's Gemini 2.0 models. This is a test to decide whether LLMs can answer substantive legal questions. The test consists of 50 questions across 10 practice areas and is marked by Linklater's lawyers. They conclude that whilst these new models are better, they are not to be relied upon without human expert supervision.

This is a great look into LLMs answering legal questions and a dive into the concept of whether they are getting to a position to be relied upon for legal advice. i.e. how far away are we from replacing lawyers? This is a different test to what we are doing as we are focused on review and extraction at the moment, but interesting nonetheless in showing that o1 is getting much closer to a lawyer level answer to some of these queries. We may only be one or two versions away from sufficiently good AI to give good answers to legal queries.

OpenAI are aiming to simplify their model offerings, planning to drop 'o3' and focus on GPT-5 as the next release, building in reasoning capabilities, deep research, multi-modal skills and task abilities. This is partly to stop the confusion around so many models all doing different things and also to show the flexibility of the new developments. This will also see an imminent release of GPT 4.5 to boost the quality of the models (possibly in response to Deepseek) before a later release of GPT 5.

Merging all the abilities and skills into one overarching model will be very powerful but may require a rethink as to how tools like AGPT work, with a change in the way we prompt and send data to the model. It also builds in image and voice into one place, meaning that if we wanted to look at doing this with AGPT we could rely solely on GPT-5 to build this out. General progress in this space is super quick at the moment, with the new Gemini model being capable of 2 million input tokens also being released.

King College London held a hackathon with their law students, Ashurst, Charles Russell Speechlys, Linklaters, and PWC. Students were given use cases and tasked with solving a problem using a LLM. They needed to show they understood the problem, apply the LLM method, and prove their solution worked. The winning solution automated the creation of deal bibles. The KCL team developed a solution trained on final and draft documents, using example and validation prompting to identify final documents and automate the creation of a finalised bible with a pre-intake form.

This is a problem we are also trying to solve. The solution from the KCL team looks like a "deal bible" management platform, where the lifecycle of a deal bible can be tracked and automated, risks identified, and key dashboards developed. Despite limited resources, KCL achieved good results in training the LLM to identify and distinguish final documents. Their main issue was a lack of training data, which would not be as much of an issue for us at AG, as we can work with our groups and knowledge lawyers to access completed bibles and final and draft documents. There is also potential to train on in-house legal templates, making this part of our LTC/TDS/MLS offering. The KCL team faced challenges linking software to organise documents into folders, an area that could be explored using Bryter or Microsoft Power platforms. Currently, training on AGPT is limited due to the lack of multi-document review functionality and memory capabilities. The issues highlighted by KCL involved back-end programming, particularly in creating a final folder (bible), suggesting that a platform like DC may be suitable. Considerations: AG's bibles are typically linked to matters, which would likely require integration with iManage or SharePoint. This project would be best initially suited to a "testing" group, such as real estate, and involve collaboration with ILT, KLs, deal leaders, and fee-earning juniors. Testing would be most effectively conducted by ILT and potentially junior champions who usually complete the bibles.