Although Technology Assisted Review (“TAR”) has been discussed at length over the last 14 months, many folks are still just getting started with the technology. As a computational linguist with an ediscovery background, I thought I would share my top five tips on getting the most out of technology-assisted review.
Tip #1. Know the Technology
Having access to sophisticated review features won’t do you any good unless you understand how to use them. This was true for the previous generation of ediscovery tools, and it’s even more critical with TAR. TAR software relies on behind-the-scenes statistical algorithms to evaluate and classify documents. Don’t worry, you don’t need to know these algorithms to use the software, but you do need a detailed understanding of the software’s options as well as the consequences of choosing those options. If you don’t have the time to develop this expertise yourself, it’s important to work with service
providers who already have it.
Tip #2. Identify Good Training Documents
Before a TAR system can start classifying documents as responsive or non-responsive, it needs to be trained with human-selected sample documents. Choosing the right training documents increases the efficiency of the TAR process, so it’s worth some extra effort at the beginning of a project.
A good training document should fall within the proper date range, and should be clearly responsive on its face. If its responsiveness only comes from a broader context outside of the document, the training won’t work as well – computers haven’t yet mastered the art of implication. Try to find sample documents for each request or issue, but don’t be worried if you don’t have complete coverage at the start. TAR is an iterative process (as explained in Tip #3), so additional issues and sub-issues will be fleshed out over the
course of review.
Tip #3. Take Advantage of the Knowledge Feedback-Loop
The iterative nature of TAR is like a continuing dialog between you and the computer, with each exchange adding to the accumulation of knowledge. The TAR system uses your initial training documents to find a batch of statistically significant documents for you to review. Your responsiveness choices on this batch will allow the system to find even more significant documents for you to review. This back-and-forth dialog continues until the system has classified the entire data set at an accuracy level acceptable to you. At each iteration you learn more about the documents, and this knowledge gets passed back to the system, rippling throughout the data set. The more you know, the better the system performs. The better the system performs, the more you know. It’s a beautiful thing.
This knowledge feedback-loop is one of TAR’s major advantages over manual review. Manual review is a linear process: reviewers work their way through batches of documents, learning more as they go, so by the end of review they have a fairly complete knowledge of the case. But this knowledge can’t be applied back to the documents they’ve already reviewed without a re-review of the entire data set, something prohibitively expensive in most circumstances. In contrast, by the end of the TAR process, both you and the system have a detailed knowledge of the documents, and it’s this detailed knowledge that gets applied automatically across the entire set of documents.
Tip #4. Log Everything
Two other big advantages TAR has over traditional review methods are repeatability and transparency. A TAR system is deterministic – if you feed it the same data, you get back the same output every time. Humans, on the other hand, are notoriously non-deterministic (which can be magical in some arenas, but not in document review). With TAR, every step of the process can be explained, defended, and repeated if necessary.
To take full advantage of TAR’s repeatability and transparency, you must ensure that a record of every step is recorded. Any good TAR system will have some sort of logging or audit functionality, but as I mentioned in Tip #1, this functionality won’t add value unless you know what it does and how to use it.
Tip #5. Post-Review Evaluation
A successful technology-assisted review depends upon three things: technology, human expertise, and a well-planned workflow. Unless you’re a software developer, the technology is outside of your control, so that means you’ll need to focus your efforts on improving expertise and workflow. Post-review evaluations are one of the best ways to do this.
After a review project is complete and you’ve had a few days to celebrate (and catch up on your sleep), it’s important to analyze the project while the details are still fresh in your mind. You’ll want to look for the things that went well and the things that didn’t. Was there an unexpected choke-point in the workflow? Do some of your team members need additional training? Answering these kinds of questions after every project leads to
continual improvement, maximizing the effectiveness and efficiency of TAR.
I’d love to hear your TAR stories. What would you tell new users to look out for before beginning a TAR project? Email me at email@example.com to discuss.