Articles

The crux of NLU is managing the long-tail of data.

GREGORY WHITESIDE
December 25, 2020
.
2 MIN READ

> Until now, building and maintaining the training data needed to support a long-tail of intents has been a difficult problem — yet this is what is needed to go from a Gregory Whiteside.

Until now, building and maintaining the training data needed to support a long-tail of intents has been a difficult problem — yet this is what is needed to go from a “dumb” conversational AI to an experience that improves over time and leaves your users amazed. Training and deploying an NLU model is no longer the bottleneck; successfully scaling and managing the underlying training data has become the crux of the problem.

HumanFirst provides a data pipeline & workflow to label and manage training data at scale, with APIs to easily deploy NLU-powered features and analytics within your products, and integrations with 3rd party vendors to use this data at any time within your existing projects (including Rasa, DialogFlow, Watson, Luis, BotPress and others).

HumanFirst is future-proof and accelerates development in a few key areas — let’s explore them!

A scalable pipeline for unlabeled and NLU training data

HumanFirst’s data pipeline ingests unlabeled data (conversation logs, utterances, emails) and / or NLU training data (i.e: existing NLU/chatbot projects) and prepares it for processing; HumanFirst’s UX provides a powerful bottom-up workflow to label and organize this data, with machine-learning and active-learning assistance throughout.

Workflow with intents and training data as first class citizens

Much like code, data has the tendency to become “spaghetti” as projects evolve, and refactoring is necessary to be able to maintain quality (and sanity).

HumanFirst provides as much control over intents and training data you’d get from working with raw files (YAML, JSON, Excel etc), but with machine-learning powered workflows that make refactoring and maintaining this data 10x more efficient.

HumanFirst’s workflow supports

  • Organizing NLU training data in cascading hierarchies
  • Drag & Drop refactoring of intent structure as long-tail grows
  • Splitting intents (and automatically moving training data) into more specific sub-intents
  • Merging ambiguous intents and their training phrases together

Combined together, this unprecedented level of control allows teams to easily:

  • Improve the breadth and coverage of their NLU
  • Improve quality and reduce ambiguity in training data
The ability to organize labeled data in flexible hierarchies that can easily evolve over time also provides an elegant path to creating and managing catalogs of intents, across domains and verticals, that can be re-used across projects. Modularize and re-use! Just like code.

Multi-tenant ready

HumanFirst was built with multi-tenant scenarios in mind, to facilitate the re-usability of data across multiple projects.

Its data pipeline provides powerful abstractions to manage, share and control access to data across multiple workspaces and namespaces:

  • Workspaces are where data for any given project is managed. You can create any number of workspaces, and assign different access controls across them to your team. Workspaces can be cloned and conversational data can be shared across them.
  • Namespaces allow an organization to manage different teams and segregate access to data (a bit like Github Teams)
Note: Namespaces are particularly useful for teams building and managing projects for multiple clients (i.e. agencies, consultants, etc.)

Now what?

Getting started with HumanFirst is quick, easy and free. Simply signup at www.humanfirst.ai to create your account and invite your team.

Subscribe to HumanFirst Blog

Get the latest posts delivered right to your inbox