For most companies, business takes the shape of emails, contracts, support tickets, meeting notes, and survey responses. Last year, an estimated 347 billion emails were sent around the world.

Classifying those documents has always been a valuable initiative and a herculean task. Say we wanted to understand those 347 billion emails; hard-to-parse send fields, footers, and all. Normally, we’d have two options: ask a data science team, or ask an LLM.

Data Science-Led Text Classification

The tried and tested (and tedious) method of complex text classification involves handing over an email corpus to an in-house or outsourced data science team. Natural language processing (NLP) capabilities makes automated document classification somewhat possible. The data team constructs a project in Python, imports the emails, and architects an arduous process of text cleaning, vectorization, and model creation.

The result is a shortlist of labels that the emails map to–in theory. In practice, data science-led labeling can fall short of the targeted clarity. Without a business expert in the loop who can add their contextual understanding–how granular the labels should be, what segment of the email chains should direct the classification–the resulting taxonomy can miss the mark.

LLM-Led Text Classification

The efficiencies of LLMs afford a second option. LLMs are trained on vast amounts of data. They can manage a multi-turn analysis and make complex classification decisions with some level of accuracy. Analytics plugins released in the latest wave of GPTs make it easy to drag and drop a file into a ChatGPT-like interface and write a prompt to extract key topics, assess frequency, or perform basic classification. 

But one dimensional prompt interfaces come with limitations. Namely, the labeling and analysis happens without a human in the loop, which prohibits the alignment of human understanding with the LLM’s predictive decision-making. The classification project quickly becomes impossible to audit. Even different models disagree on simple definitions; if business experts can’t inspect and correct the model’s understanding, trusting the resulting taxonomy becomes impossible.

Prompt and Data Engineering for Automated Document Classification

This workflow offers teams a third option. Combining the efficiency of LLM-led document classification with the fidelity of inspectable data engineering methods, and removing the barrier of code, teams can quickly arrive at an automated classification model they can inspect, adjust, and trust.

After following this how-to guide, you’ll have:

Annotated data you can use for dashboards and visualizations

Organized datasets you can analyze further

An operative taxonomy you can use to categorize new documents as they’re created

Download the full guide + video walkthrough to learn more.