A review of AI based patent searching – Are we there yet?

Article Overview – 

These days when setting off to conduct a patent search or evaluate patent search firms, you will inevitably come across the phrase “Artificial Intelligence” or “AI”.  The phrase is used by almost all patent search firms and patent search resources to convey the presence of cutting-edge strategies, capabilities, and technologies.  Unfortunately, it is not always clear as to what the AI is actually doing and how it differs from a traditional human/expert conducted search.  Therefore, in this review we aim to define AI as it applies to patent and prior art searching and evaluate its utility.  We do this by defining common terms and buzzwords used when discussing AI and patent searching, then outlining the pros and cons of the most common AI techniques.  Finally, we provide a summary of an evaluation of AI based patent searches and how such search techniques can integrate and complement traditional human/expert search methodologies, and draw conclusions on the current state of AI in patent searching.

What is Artificial Intelligence (AI) and how does it work?

The term “AI” can often be compared to marketing terms like “superfood,” which convey special health benefits, but not a specific benefit, so the meaning is a bit fuzzy.  Superfood isn’t a term generally used by experts in health and nutrition for this reason – it just doesn’t communicate much.  In the same way, AI conveys some kind of high-tech benefit, but not always a specific benefit, so this label is also a bit fuzzy.

Researchers and engineers in various fields that develop or implement AI don’t always refer to their tools or methods as AI, instead they generally prefer to use more descriptive labels.  Like most marketing buzz words, the term evokes a feeling, but isn’t especially informative.  It’s used more like a bucket or label for a class of things that share at least one characteristic.

For the purposes of this article, we’re going to use the following working definition of AI as a class of tools:

Anytime a computer algorithm does something that we think requires or emulates human intelligence.

Common examples

The reader will likely be familiar with AI as a description or feature of the following:

• • • • Facial detection/recognition
      • Text editors/autocorrect
      • Search and recommendation algorithms
      • Chatbots
      • Digital assistants (Siri, Alexa)
      • Self-driving cars
      • Robotics/manufacturing automation
      • Finance/trading and auto-advisors
      • Home automation

Each of the above involves a computer algorithm making decisions, providing recommendations, or taking actions based on some input, in a way that we might previously have thought required human involvement, or with an outcome similar to a human performing the same task. How the algorithm is doing this can vary considerably.

Algorithm Defined

An algorithm is:

A step-by-step procedure for solving a problem or accomplishing some end (Webster – algorithm).

For the rest of this article, we will often compare algorithms to recipes, which provide instructions for taking ingredients (inputs), and turning those into a meal or dish (output). This is essentially what all computer programs are; the computer is a perfect cook, flawlessly following instructions written in their algorithm or recipe book, faster and more precisely than any human.

This doesn’t mean their meals are perfect – the dishes depend on things like the quality of the recipe, and of the ingredients. Remember: this is the world’s most precise cook, and they will follow their recipe to-the-letter, every time, no matter how poorly written the recipe is or how terrible the ingredients are that they’re given. In the same way, the products of AI algorithms depend on the sophistication of the instructions, and the quality of the inputs, and the computer will only execute exactly what it is told to do in the algorithm.

Over time, researchers and engineers have made incredible progress improving the quality and variety of recipes and ingredients available, so our perfect cook can make better and better meals of different types. Breaking from the analogy, this means more sophisticated instructions for the computer, better methods of collecting and using input, and many new applications for these algorithms. These very sophisticated and often layered algorithms (recipes on top of recipes) can give the impression of intelligence and sometimes even agency (take a look at DALL-E2 and GPT-3, for example), but that’s probably not what’s going on.

Algorithms, Not Consciousness

We’re getting somewhat philosophical here, but just briefly, and only to the extent that it relates to the topic of AI in patent searching.

Tools we categorize as AI can sometimes seem so intelligent that we may even think that they are conscious agents. When we say conscious, we usually mean:

Awareness of internal and external (Webster – Consciousness)

Another way of putting it is to say having identity or sense of self, knowing “what is me, and what isn’t.”

There is a LOT of controversy among experts in philosophy and neuroscience over what consciousness actually is. There is also some disagreement amongst both philosophers and computer scientists about whether or not computer algorithms can one day achieve some version of consciousness. Most philosophers and computer scientists agree that while it may be a possibility in the future, no computer algorithm has yet produced consciousness.

One popular model advanced in a paper by Dehaene et al (Dehaene) divides what we call consciousness into the following three categories:

1. C0 – Autopilot; this is unconscious calculations and operations that we don’t pay attention to, things like facial recognition and speech recognition.
2. C1 – Trains of thoughts, pools of info; this is conscious decision-making that we execute following thought and external information processing.
3. C2 – Metacognition; this is thinking about one’s own thoughts, reflecting on one’s own process, which can push us to correct, or to adapt, or to seek to enrich our experience.

Dehaene argues that computer algorithms can perform C0 tasks quite well, but nothing higher. Attempts have been made to allow algorithmic systems to approximate C1 and C2, but to date, no AI appears to fully satisfy the criteria of either type. This shortcoming prevents even the most sophisticated AI from being able to abstract, deal with novel input or make connections in ways that most humans can do naturally. It is true that very sophisticated tools like DALL-E2 and GPT-3 are incredible accomplishments that seem to exhibit creative thought or responsiveness that might pass a Turing test, but they’re doing this by means of complex layering of C0 processes. They are successful at giving the impression of higher-level “thinking” within certain parameters, but aren’t able to generalize outside of those parameters successfully, like most humans can.

Types/Methods of AI in Patent Searching

It can be confusing to group or categorize AI tools, as there is a variety of tasks one can apply these tools to, and also a variety of strategies that AI can employ to accomplish those tasks, and much overlap between those two groupings. We’ll first look at the AI strategy, then at the tasks they can accomplish relating to patent and prior art searching.

Natural Language Processing (NLP)

Natural Language Processing (NLP) – AI computer science of understanding spoken words and natural language text (Wiki – NLP). This might involve text segmentation (dividing text into meaningful segments), normalization (transforming text into standardized segments), lemmatization (grouping together word inflections), stemming (identifying root words), identifying co-occurrences (words that tend to occur together or nearby each other) and vector space modeling (an algebraic model for representing text documents as vector identifiers).

Semantic Technologies

Semantic Technologies – leverages AI to simulate how people understand language and process information (Wiki – Semantic technology). This might involve NLP (hah, see? I told you this would get confusing), lexico-semantic knowledge (the study of word meanings, how they act in grammar and composition, and relationships between the different uses of words), and statistical topic modeling like Latent Dirichlet Allocation (LDA – documents are represented as a mixture of topics, and a topic is a grouping of words that suggest a similar theme).

Machine Learning (ML)

Machine Learning (ML) – the use and development of computer systems that are able to learn and adapt without following explicit instructions, by using algorithms and statistical models to analyze and draw inferences from patterns in data (Lexico – Machine Learning).

Machine Learning is the dominant AI technique disclosed in patents and is included in more than one-third of all identified inventions in this space, over 134,000 patents as of this WIPO Technology Trends report for 2019 (WIPO 2019).

Important to note: this is a point of contention – many experts categorize ML as distinct and separate from other AI tools, whereas some experts in patent searching use AI synonymously and interchangeably with ML. We’re going to keep them grouped together and refer to ML as a subset of AI, because like other tools mentioned, ML still meets our working definition of AI from earlier – a computer algorithm doing something we think requires or emulates human intelligence.

ML is sort of like teaching a chef the basic skills of cooking, and then letting them create their own recipe to make dishes; we don’t write the recipe that makes the dish, we prepare the cook to (hopefully) figure that out themselves. There are two general types of machine learning:

Supervised Machine Learning – machine learning using labeled data. The algorithm (sometimes called a “classifier”) is presented with labeled inputs for training purposes, and uses those to create its own process for splitting inputs into two or more groups.

Think of the chef given a variety of training ingredients to review, and each ingredient is labeled “include me” or “don’t include me.” During this review, the chef tries to create a recipe that includes or excludes all the appropriate training ingredients based on their labels – that’s called a “learning” or “training” phase – and afterwards the AI/ML chef is set loose on a world of unlabeled ingredients to apply the recipe they came up with. If all goes as planned, and if the chef prepared a good recipe based on the training input, they can quickly cover a huge collection, correctly classifying (hopefully) new inputs that they didn’t see during training.

Unsupervised Machine Learning – machine learning using unlabeled data, the algorithm identifies connections and probability densities.

In this case, the chef is presented with a bunch of ingredients but isn’t told what to do with them. Instead, the chef looks for combinations that have something in common, or can be grouped in a way they think might “taste good” (be meaningful).

Patent and Prior Art Searching Tasks that AI Can Perform

There are many opportunities for AI to assist in patent searches, and with varying success. In an upcoming section we’ll take a look at these in greater detail, but here is a summary of some of the major patent searching tasks that AI can perform or enhance:

• Extracting features of interest from patent applications or invention disclosures
• Search query expansion (offering lexically related terms i.e., synonyms, hyponyms, hypernyms, and meronyms; also offering terms which are semantically related, alternative spellings, and even term and phrase autocomplete suggestions)
• Document classification
• Document clustering
• Ranking of results
• Topic modeling and highlighting

Already in Use at the USPTO

As one application example, the USPTO is incorporating AI tools into patent searching and classification to aid the patent examination process. These tools attempt to identify relevant documents and provide suggestions for additional search areas. Beta versions of the tools were implemented with select examiners in March of 2020, and early data looks promising.

Also being tested and utilized is an auto-classification tool for CPC classification. After initial analysis of performance, this system was implemented in December of 2020 and has resulted in reductions in expenditures for acquiring CPC data (AI tools at the USPTO).

Some further examples are:

• Chemical Abstract Services (CAS) completed a collaboration with Brazil’s National Institute of Industrial Property (INPI) to streamline searching with, among other interventions, AI tools – resulting in considerable reduction in examination time (CAS – AI and Patent Offices).
• This same resource indicates that WIPO has over 70 AI-related projects to test or implement various AI tools.

Advantages and Disadvantages of AI-based Patent Searching

Advantages of an AI Patent Search

Returning to the analogy of the computer being the perfect cook, some advantages are obvious.

First, our perfect cook is perfect. They make no mistakes in following their instructions; they execute their recipe exactly the same way, every time. Given the same ingredients and recipe, they’re going to give you the same dish, always. In contrast, any two expert patent researchers given the same search request will produce slightly different result sets – they may leverage different resources or use different key terms and classifications, and they may assess documents differently. It’s even possible for the same researcher to produce slightly different results across time if they repeat the same project. If you don’t change their recipe, the perfect cook won’t do this, and that consistency can be appealing.

Next, because they’re a computer, the cook is inexhaustible and always available. The limit of their availability is the amount you’re willing to pay to keep the tool running. If you keep the AI cook’s kitchen stocked and the lights on, they will keep pumping out dishes. Also, the perfect cook doesn’t care if you ask for one dish or one thousand. Leaving the analogy, a professional human patent searcher is going to have to deal with things like stopping to sleep and eat, and other human obligations that get in the way of digesting huge datasets. Some patent searches can take a human searcher not just hours, but days and even weeks to cover an appropriate search scope. AI can repeatedly power through its algorithm, iterating over the same or a much larger scope in a tiny fraction of that time. This advantage becomes increasingly valuable as the sheer volume of patents increases over time.

Expanding from that, the last advantage we’ll cover is being able to leverage speed to cover an even broader scope than a human researcher might. One of the most important aspects of designing an effective patent search is deciding what scope is likely to contain relevant hits. While an expert researcher can establish a scope that is very likely to contain most relevant documents, there is always the risk that something relevant could exist outside that scope, and would therefore not be found. Since AI has the speed advantage, computer algorithms can cover a much larger scope, and decrease the likelihood of overlooking those outlying important references.

Disadvantages of an AI Patent Search

As for the disadvantages, they’re going to differ according to exactly what type of AI-based patent search tool is being used.

Some of the AI tools currently available require a great deal of up-front work and resources to allow them to work properly. There is proprietary software, or firms that employ computer scientists that can set up a unique AI patent search solution, and this can be prohibitively expensive when compared to a traditional patent search conducted by a human search expert. In the cook analogy, this is a little like our computer algorithm cook having to set up a new restaurant each and every time they start a new search project. There may be cases in which this makes sense, but in most cases, it is exorbitantly wasteful. This can cost orders of magnitude more than a more traditional patent search executed by an expert human searcher.

GIGO (Garbage In, Garbage Out) is another stumbling point for the AI cook – again, the cook pays no mind to what recipe they are handed or what ingredients they are given. The ingredients can be inputs like search strings, key words, classifications, documents thought to be of interest, and so on. An expert human searcher that is familiar with the technology can adapt their strategy based on their prior knowledge, additional research into the subject matter, client input and on the quality of results they see while searching. They can correct for corrupted inputs, building from an understanding of the project objective, and can find useful results in spite of poor inputs. The AI cook will proceed with the recipe with no concern for rotten ingredients and no regard for the quality of the final dish.

Depending on what AI patent search tool is being used, a very common challenge is the number of irrelevant hits. If you were to ask the AI cook to make you any dish that contains a certain ingredient – you can quickly be overwhelmed by the number of dishes produced if that is a common ingredient. One of the advantages of the AI cook works against us in this instance. Imagine asking our perfectionist cook to prepare all dishes that contain all-purpose flour, for example. The tireless cook would cover enormous territory, and come back having prepared hundreds of thousands of unique dishes.

An increasingly common concern (possibly the #1 most common concern among patent search requestors) is the “black box” nature of AI patent search tools. In just about all cases, there is little to no transparency about how the algorithms are operating. The chef’s recipe is often a mystery. Ordinarily, if a search requestor has a question about the search strategy, or about a given document and why it did or did not appear in a result set, an expert human searcher can provide a concrete answer and if necessary, adjust the strategy to capture other references and increase confidence in the search results. In fact, testing and correcting a strategy is a routine step of all expert human searchers. This isn’t always possible in the case of AI patent search tools, and in some cases, no one actually knows how results are identified and collected as the chef wrote the recipe himself, inside that black box.

The final major category of disadvantage covered here relates to the ability to abstract, make new connections, and deal with new input. Like we covered earlier, the AI cook is essentially on autopilot – they don’t deviate from the recipe. If the recipe isn’t sophisticated enough to deal with a new technology, it won’t be successful. If the recipe isn’t sophisticated enough to allow the cook to identify a relevant reference, it will overlook that reference. When discussing patent searching, every professional searcher has personal anecdotes of making connections based on a core understanding of the search objective and the references being reviewed that allowed them to identify key references that otherwise didn’t explicitly match the initial search request parameters. Human experts can consistently generalize and abstract their thinking, and deal with novel situations and technologies in ways that AI isn’t able to do yet.

Should I Commission an AI Patent Search?

When evaluating this question, we need to address issues like “How is the quality of a patent search assessed?” and “What do experts say about the quality of AI patent searches?”

Assessing AI Patent Searches

This won’t be referenced often in this article, but in most discussions about AI based patent search tools, the following terminology shows up frequently, so let’s become acquainted with them.

Industry standard in evaluating IR (Information Retrieval) systems are precision and recall. Precision is the ratio of relevant to irrelevant documents retrieved. Recall is an assessment of the relevant documents overlooked.

To calculate precision (P), divide the number of relevant documents retrieved by the total number of documents retrieved. For example, if the IR system returns 10 documents, and 5 of those are relevant, P = 5/10 = 0.5, or 50%.

To calculate recall (R) divide the number of relevant documents retrieved by the total number of relevant documents in the whole collection. For example, if there are 10 relevant documents in a collection and the IR returns 8 of them, R = 8/10 = 0.8 or 80%.

These calculations can get a bit fuzzy, as there may be disagreement among professionals about what qualifies as “relevant,” and part of the issue at hand is the total number of potentially relevant documents in a collection being an unknown, but when the assessment of a number of professionals converge, we can develop “consensus” answers to these questions for which we can have some confidence, at least for evaluation purposes.

IPO Investigation into AI Patent Searching

In April of 2020, the Intellectual Property Office of the UK (IPO) published a research paper produced by Cardiff University, describing their assessment of AI-assisted patent prior art searching (Cardiff – IPO).

The goal was to assess how various AI based patent search tools could benefit the IPO during prior art searches for patent applications. Researchers at Cardiff University and IPO patent examiners assessed all the AI methods mentioned in this blog, across all the applications given. They were assessing quantitative aspects like precision and recall, and also qualitative aspects like ease of use. Here are some excerpts from the executive summary of that report:

“The research finds clear evidence that none of the available AI algorithms on their own can support every aspect of the prior art search process (classification, forming a search query, retrieval, ranking, identifying similarities, and topic visualization).”

“As a result, this (our recommended*) human-in-the-loop approach aims to maximize performance by combining the AI and human intervention and is designed to supplement, not substitute, human expertise and judgement.”
*text added for clarity

“The user keeps the role of key decision maker, whereas the AI provides intelligent decision support.”

Ultimately, this conclusion was reached because no AI patent search tool alone or in combination was as successful as an expert human searcher at both constructing effective search queries and identifying relevant documents. This emphasis is captured in the following excerpts:

“The success of a prior art search relies upon the selection of relevant search queries (Bashir and Rauber, 2010).”

“Typically, terms for prior art queries are extracted from the claim fields of query patents. However, selecting relevant terms for queries is a difficult task due to the complex technical structure of patents and the presence of mismatched and vague terms; this often involves further research into the domain of the application.”

“Furthermore, patents are complex legal documents, even less accessible than the scientific literature.”

“Patent drafters intentionally try to use entirely different word combinations, not only synonyms but also paraphrasing (Atkinson, 2008). Patentees typically use their own lexicon in describing their inventive details or use abstract or generic terms to maximize the protective scope. Patents often include different data types – typically drawings, mathematical formulas, bio-sequence listings or chemical structures that require specific techniques for effective search and analysis.”

In the model generated by the IPO and Cardiff University researchers, an expert human searcher still reads an application, defines search statements and queries, curates the hits, and performs all of the decision-tasks, but they recommend using various AI patent search tools to recommend document classification, extract suggested keywords, suggest query expansions, group and rank hits, and apply highlighting to aid in document review.

Conclusions on AI and Patent Searching

By our assessment, the vast majority of cases, modern AI based patent search tools are not yet in a position to replace expert human searchers – though there are many opportunities for these tools to complement a searcher, streamline processes and help to capture more relevant art.

1. “While AI-based tools are economical and quick, manual searching is more reliable and relevant. Using a combination of both can ensure truly superior results.” (IPWatchDog – AI Versus Manual Patent Searching)
2. “Patent software companies are now relying on AI to produce advanced patent tools using the hybrid approach of incorporating the features of conventional patent tools with AI capabilities. (Lexology – Why AI is crucial for patent searching and mining)

We find the IPO report compelling, and consider their conclusions to be both justified and consistent with our experience. While we’re aware of some firms that claim to conduct strict AI patent searches with no human expert searcher input, in most cases based on client feedback and our own experiment, they appear to produce underwhelming results. Referring again to the IPO report in discussion of AI tools retrieving and ranking documents, they write:

“The experimental results for precision varies between 30% and 50%, which means that the first 10 search results contain between 3 and 5 relevant documents. Patent examiners involved in this feasibility study agree that this was a higher ‘hit rate’ than they achieve with their current search tools.”

While it is true that 3-5 out of the first 10 references to be reviewed (a P value of 0.3-0.5) is an improvement over previous standard searching practices, this is an early stage of a typical TPR patent search, when the human expert searcher begins reviewing hits and identifying actual documents of interest, eliminating the “false drops” or “noise.” As the IPO concludes, without the input and decision-making of the expert searcher, the search is incomplete. These AI based patent search tools can help make searching more efficient, and draw the searcher’s attention to documents of potential interest earlier, but they lack the sophistication to generate a finished product similar to that of what an expert searcher will produce.

We don’t want to dismiss one point: some Machine Learning (ML) patent search solutions can be quite useful in ongoing and repeated searches covering roughly the same scope over time in a technology where relevant documents are well defined and many examples are available for AI training. Though the process of training the ML AI can be very expensive, that cost can be offset by the speed and efficiency of using that tool on future iterations. Over enough iterations, you’ll eventually see an impressive ROI. If you are assessing options for long-term repeated FTO or landscape searches on the same technology, the technology being well known and understood, and for which you have many examples of art of interest (the number required may vary, but some services state no less than 50 positive examples), and you also have the resources to set up an ML patent search solution, that might be an excellent option for you. But if any of those aspects are missing (i.e. the search is a one-off, is new or not well-developed technology, you have few or no examples of art of interest, or you don’t have the resources to set up an ML patent search solution), then a purely ML AI search is not likely a good option for you. A more robust ML solution will still likely include an expert searcher initially anyway, in order to identify the 50+ positive examples needed to train the ML classifier properly.

TPR continues to keep its eye on this field of research and development, as these tools improve all the time. For now, we have adopted the “human-in-the-loop” approach as the IPO recommends. Most commercial databases and even some open-source resources used by our searchers have implemented AI-driven functionality. This allows us to incorporate those AI based patent search tools into our workflow to expand queries, classify and rank documents, apply highlighting and clustering for visualization, and so on. As more and newer tools become available, we will continue to assess those as well.

If you’d like to discuss any of these points, or have more questions on this topic, please contact us.

© Copyright 2022, Technology & Patent Research International, Inc.

 

References
Webster Algorithm https://www.merriam-webster.com/dictionary/algorithm
Webster Consciousness https://www.merriam-webster.com/dictionary/consciousness
Dehaene https://www.science.org/doi/full/10.1126/science.aan8871
Wiki NLP – https://en.wikipedia.org/wiki/Natural_language_processing
Wiki Semantic Technology – https://en.wikipedia.org/wiki/Semantic_technology
Lexico – Machine Learning – https://www.lexico.com/en/definition/machine_learning
WIPO 2019 Technology Trends – https://www.wipo.int/edocs/pubdocs/en/wipo_pub_1055.pdf
AI tools at USPTO https://www.uspto.gov/blog/director/entry/artificial-intelligence-tools-at-the
CAS – AI and Patent Offices – https://www.cas.org/resources/blog/ai-patent-offices
Cardiff IPO Report – https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/887907/aI-assisted-patent-prior-art-searching-feasibility-study.pdf
IPWatchDog – AI and Versus Manual Patent Searching – https://www.ipwatchdog.com/2021/10/02/ai-manual-patent-searching-pros-cons-hybrid-approach/id=138204/
Lexology – Why AI is crucial for patent searching and mining – https://www.lexology.com/library/detail.aspx?g=ece7e6a5-7a3c-499d-9c00-4cb37f031e78