Ten Things to Think About when using AI for Legal Work

AI tools have come an incredibly long way from when Inceptiv first started looking at general LLMs and legal specific AI tools back in 2023.  Beginning in February 2024, Inceptiv started formally reviewing and demoing different AI legal tools, comparing usability, features and pricing.  As a firm we continue to demo different AI tools each year, and have seen marked improvements in features and functionality over time.  

Like many other law firms, Inceptiv has rolled out AI legal tools to all of its attorneys, and we hold regular internal training sessions to discuss how each of us is using such tools to better understand how such tools can make each of us more efficient and effective.  

We also acknowledge that many of our clients are also using AI for their legal work.  As a result of our client’s usage of AI, we understand that the nature of the work that Inceptiv performs, and our relationship with clients, will need to evolve over time.  For many use cases, our client’s use of AI makes a lot of sense, and we have no qualms encouraging our clients to use AI in those situations. 

However, based on our continuing (and constantly evolving) experience with various AI tools, we thought it would be helpful to provide some initial thoughts and advice to our clients regarding their use of AI, so that our clients can leverage AI most effectively.

1. Check the settings on a general LLM to be sure that training is toggled off by default

One of the easiest mistakes to make with a general purpose LLM is assuming that your data inputs are private by default. In many consumer-facing tools, the opposite is true – prompts and uploaded documents may be used to train future versions of the model, which means sensitive client information, draft agreements, or internal communications could end up influencing outputs delivered to other users. 

Inadvertently publicly sharing internal confidential information can have significant negative consequences, from potentially waiving attorney-client privilege with respect to attorney client communications and work product, to losing trade secret or intellectual property rights with respect to such disclosures.  

Before using any general LLM for legal work, we strongly recommend that you take a few minutes to dig into the settings and confirm that sharing of, and training on, your data is turned off.  We also recommend periodically rechecking these settings, particularly after major product updates, as AI providers may occasionally change their default privacy settings or introduce new features (such as memory or chat history sharing) that may re-enable data usage you previously opted out of. A quick audit every few months is a small investment relative to the risk of inadvertently disclosing confidential information.

2. Consider having your organization enter into a team or enterprise level LLM license, so you are able to enter into stronger data handling agreements (eg Data Protection Agreements (“DPAs”), or Business Associate Agreements (“BAAs”)) to protect your data

Individual or consumer-tier subscriptions to LLMs typically come with standard, take-it-or-leave-it terms of service that offer limited protections around data handling, confidentiality, and indemnification. Team and enterprise licenses, on the other hand, *may* allow your organization to negotiate (or at least ask for and sign onto) more robust data and information security agreements which provide for more details in how your data may be used, stored and secured, and what happens in the event of a data security incident or breach.  Such additional agreements may include a separate standalone DPA or information security agreement, or where health information is involved, a BAA.  

For any organization using AI on more than an occasional basis, especially where such use is across a team of sales, finance or business development members, the incremental cost of an enterprise license may be worth the added legal protection and administrative control. For those organizations with centralized IT operations, an enterprise plan may also provide great visibility and control across the organization, by offering centralized administration, audit logs, and the ability to enforce settings (such as disabling training) across the entire organization, rather than relying on each individual user to configure their own account correctly.

3. Consider using a legal-specific AI platform

General purpose LLMs have come a long way, and have become remarkably capable.  However, they are still designed for a broad audience and are not optimized for the particular workflows, terminology, and risk profile applicable to legal work. Legal-specific AI platforms (such as Harvey, Spellbook, GC.AI, Legora, Co-Counsel and others) are purpose-built with legal use cases in mind, and generally incorporate contract-specific review features, citation checking, clause libraries, and tighter controls around confidentiality and data handling.  

That said, legal-specific platforms can be expensive, often charging high per user monthly or annual seat licenses (especially in comparison with a general purpose LLM subscription), and are most effective if used with or by an experienced attorney, who understands the legal issues and nuances that the legal specific AI tool is designed to address.  Moreover, general purpose LLMs, such as Claude, have invested more heavily into supporting legal workflows, and may be more than sufficient for the typical use cases involving legal contracts and legal questions that typically arise with a non-attorney user.  

Ultimately, the right choice depends heavily on the type of work you do and how the tool fits into your existing systems. We generally recommend that clients evaluate a few options, run them against representative documents, and consider whether the legal-specific features justify the additional cost compared to a well-configured general LLM.

4. Try to start from a known template (whether from your attorney or using a standard form in the industry), rather than using an LLM to generate templates/agreements from scratch.  

When asked to generate an agreement from scratch, LLMs will often produce something that looks polished and complete – but on closer review, the document may be missing key provisions, include language that is inconsistent with market practice, or borrow from templates that aren’t well suited to your particular deal or jurisdiction. Even when the output looks reasonable, it can be difficult to know what’s missing without comparing it against a trusted reference.

For instance, we are often asked to do a “quick review” of a standard contract or agreement that our client generated from the ground up using an LLM.  The issue in such case is that we, as attorneys, must then review the entire document, word for word, rather than simply reviewing redlines against a known template, to review and confirm changes the LLM had made. 

Starting from a known template – whether one provided by your attorney, a standard industry form (such as National Venture Capital Association or “NVCA” documents for venture financings, Y-Combinator or “YC” SAFE forms for SAFE financings, YC forms of SaaS agreements for commercial software licensing, Loan Syndications and Trading Association or “LSTA” forms for credit agreements, or California Association of Realtors or “CAR” real estate or lease forms), or a prior agreement your organization has used successfully – gives you a reliable baseline. Other sources of standard templates include Practical Law (subject to a subscription) or Common Paper (free and subscription tiers).  

Ultimately it is much safer and more efficient to use an LLM to help tailor the template to your specific facts vs. asking an LLM to build an entire document from scratch.

5. Unless you provide specific guidance to an LLM, the LLM will generally be more aggressive in reviewing and issue spotting an agreement

Out of the box, most LLMs are tuned to be thorough – they will flag a wide range of potential issues when reviewing an agreement. While this can be helpful as a first pass, it can also generate a long list of comments that don’t reflect your actual priorities, risk tolerance, or the commercial context of the deal. Similar to an inexperienced junior attorney’s instinct to flag every theoretical issue, an LLM will often do the same, which can quickly bog down a negotiation by delving into too many unnecessary or unimportant issues.

To get more useful output, always try to be explicit with the LLM about what you care about – the type of agreement, your role (buyer vs. seller, licensor vs. licensee, etc.), the size and significance of the deal, and any specific provisions you want it to focus on (or ignore). The more context you provide upfront, the more the LLM’s review will resemble the kind of targeted, judgment-driven analysis you’d expect from an experienced attorney.

6. LLMs don’t always flag or pick up on general business issues or business practices that are considered “custom” in an industry or type of agreement 

LLMs are trained on a broad corpus of publicly available documents.  They tend to perform best on issues that are well represented in that training data. However, LLMs are generally less reliable at identifying industry-specific norms, customary practices, or deal-specific business points that aren’t captured in standard contract language. 

For example, an LLM may miss that a particular indemnity cap is unusually low for your industry, or fail to flag that a customary MFN clause is missing from a SAFE.  Or it may accept a set of reps that, while customary and standard, should not apply, or would in particular be detrimental, to a particular client.  In some instances, the LLM may not realize that autorenewals are an issue w/r/t certain types of vendors or services.  Pricing or standard SLA terms may also vary from industry to industry and may need scrutiny in ways that an LLM may not flag or appreciate.  

This is one of the areas where experienced legal counsel continues to add the most value. We recommend treating an LLM review as a useful complement to – but not a substitute for – an attorney who knows your industry and the typical deal terms for the type of agreement you’re negotiating. If you’re using an LLM as a first-pass review, it’s worth flagging to your attorney which issues the LLM raised (and which it didn’t), so the attorney can focus their attention efficiently.

7. Tone matters in comments; always review and modify comments to ensure they reflect the tenor of the negotiation and relationship of the parties

LLMs are quite capable of generating substantive comments and proposed edits, but they generally don’t have a feel for the tone of a particular negotiation or the nature of the relationship between the parties. A comment that reads as appropriately firm in a contested deal between adverse parties may come across as needlessly aggressive in a friendly negotiation between long-standing business partners – or vice versa.

We have experienced countless situations where it appears that an LLM was used to generate comments that are often off-putting or needlessly technical.  In other situations, we’ve seen agreements where it was obvious that both a human and an LLM added comments – and where in such cases the LLM comments stick out like a sore thumb. Human nature being what it is, comments that are obviously AI generated are often discounted and/or disregarded – as they are seen as auto generated rather than being genuine – which may inadvertently degrade your negotiating leverage by including them.  

Before sending any AI-generated comments to the other side, take the time to read through such comments with the relationship in mind, and always be willing to adjust, edit or delete such comments accordingly. Soften language where collaboration matters, sharpen it where it’s warranted, and make sure the overall package reflects how you want to be perceived across the negotiating table. A well-calibrated tone can make the difference between an efficient round of redlines and needlessly antagonistic negotiations going forward.  

8. Think about creating a set of rules or guidelines for contracts (or certain types of contracts) that the LLM should follow in reviewing an agreement

One of the most effective ways to get consistent, useful output from an LLM is to develop a set of standing instructions or guidelines that the LLM should apply when reviewing particular types of agreements. These might include your standard positions on key provisions (limitation of liability caps, indemnity baskets, governing law preferences, etc.), formatting conventions, or specific issues you always want flagged.

Over time, these rules become a form of institutional knowledge that improves the consistency of AI-assisted reviews across your organization and reduces the amount of rework required on each new agreement. Many legal-specific AI platforms have built-in support for this kind of playbook functionality; for general LLMs, you can often achieve a similar result by maintaining a well-crafted set of system prompts or a reusable instruction set that you provide at the start of each review.

9. Ask your attorney if he/she has already integrated AI into their practice, and what tools they use; they may already have fine-tuned instructions that may speed up the review/drafting process.

As noted above, many law firms (including Inceptiv) have already invested significant time evaluating, deploying and refining AI tools for legal work.  As a result, your attorney may already have access to legal-specific AI platforms, enterprise-grade general LLMs, or carefully tuned playbooks and prompt libraries that have been developed and tested across many matters.  Leveraging their diligence and work – rather than duplicating it on your own – can save time, reduce cost, and produce a more reliable result.

We encourage clients to have an open conversation with their attorneys about how each side is using AI, what tools are in play, and where the handoffs make the most sense.  In some cases, your attorney may be able to share their playbook or prompt set with you (or run an initial AI review on their end before you ever see the document); in other cases, it may make sense for you to do a first-pass AI review and then have your attorney focus on the issues that the AI flagged (or missed).  

Aligning on AI usage upfront helps avoid duplicated effort, inconsistent positions, and the awkward situation of two different AI tools generating conflicting comments on the same agreement.

10. LLMs can still make mistakes; you should always review their work, whether on your own or having an attorney review

As far as AI tools have come in the last few years, even the best AI tools make mistakes – sometimes obvious ones, and sometimes subtle errors that are easy to miss without careful review. LLMs can misread defined terms, miscalculate dates or thresholds, hallucinate citations or statutory references, and confidently produce language that sounds authoritative but is simply wrong. The risk is particularly high when the LLM is being asked to do something complex or to apply nuanced legal judgment.

In some instances we have noticed that LLMs still have issues calculating numbers, share counts, etc. especially where the edits require exact math calculations.  

In other instances, we have seen LLMs generate comments or reference sections or language in a contract that simply don’t exist (eg the dreaded “hallucinations” often discussed and referred to in discussing AI usage in the law).  

The practical takeaway is that AI-generated work product should always be reviewed before it is relied upon or sent to a third party. For routine matters, your own careful review may be sufficient; for anything material, we always strongly recommend that you still have an attorney review the output as well.