Personal and professional history might not repeat, but it certainly rhymes. I’m thrilled to join the team at HumanFirst, and reconnect with a team of founders I not only trust, but deeply admire.
The skill of Prompt Engineering has been touted as the ultimate skill of the future. But, will prompt engineering be around in the near future? In this article I attempt to decompose how the future LLM interface might look like…considering it will be conversational.
Personal and professional history might not repeat, but it certainly rhymes. I’m thrilled to join the team at HumanFirst, and reconnect with a team of founders I not only trust, but deeply admire.
eHealth is one of the largest online marketplaces for health insurance in the United States.
NLU and conversational AI are part of their strategic vision: their machine-learning team are working on cutting-edge NLU models trained from transcribed call center voice conversations; among other applications, their AI automatically annotates and summarizes conversations to surface key information that’s useful to the sales and support team, and powers personalized coaching tools for new agents.
To power their roadmap, successfully training and managing the long-tail of intents and conversation data is needed — eHealth’s NLU classifier is continuously growing and is now composed of over 1000 intents containing 100+ training phrases each.
Before HumanFirst
eHealth’s first version of their classifier was created using unsupervised clustering techniques: the initial set of unsupervised utterance clusters contained 10,000+ ambiguous, overlapping and unstructured Level 1 clusters. Additionally, some contextual cluster embeddings were performed to help generate Level 2 clusters.
eHealth initially took on the task of refactoring this initial list of Level 2 clusters to a two-level hierarchy of ~1000 clean intents, with a workflow dependent on excel, .txt files and human annotation.
Prior process
Unmanageable data at scale
Separating the intent hierarchy (in excel) from the respective training utterances (in .txt files) made it extremely difficult for eHealth to visualize the structure and content of their data. An example of an intent catalog managed in excel:
Public data-set, not eHealth
Understanding and modifying a hierarchy is almost impossible at scale.
Scattered workflow
Working with excel and .txt files required careful coordination between team members to ensure the right versions of the files were worked on, and that a given model’s training data was correctly versioned and trackable. This became unmanageable as the classifier evolved, and more contributors joined the project.
Lack of visibility
Stakeholders couldn’t easily take decisions on what parts of the training data needed improvement or modification, or easily track & communicate progress: the machine-learning team even built their own web-based UI to expose the state of the intent and training data — until they found HumanFirst.
With HumanFirst
“We would never go back to doing this without HumanFirst” — Zhe Jia, Senior NLP Data Scientist @ eHealth
Current process
Easy to visualize
With HumanFirst, eHealth has been able to organize their NLU data in well structured hierarchies to facilitate navigation and visualization of their data. This has helped their team become more efficient when exploring and making changes to their classifier.
Below is an example of data visualized in HumanFirst :
Public data-set, not eHealth
Centralized and collaborative
The team collaborates around their centralized NLU data in HumanFirst: annotators fix classification issues and add training examples to intents, data engineers refactor the NLU hierarchy by merging and splitting intents and their training data, while product and operations have instant access to the latest state of the model’s training data, and can more easily map the output of the model to their analytics and automation features.
With changes made on an ongoing basis, eHealth found it easy to create versions and snapshots of their classifier using workspaces in HumanFirst:
Machine-learning assisted workflows
HumanFirst’s data engineering workflows have made annotators and data engineers more confident and efficient when making changes. It allowed eHealth to hire less annotators that were previously budgeted, and to successfully apply their large-scale NLU classifier to improve their operations during the highest-volume period of the year.
Moving misclassified utterances to the correct intent is easy with HumanFirst’s real-time recommendations
“Having more than 1000 intents with 100+ utterances on average, we were finding it impossible to manage via Excel & .txt files. HumanFirst helped efficiently maintain and organize our intents at scale as well as made it easier for our data annotator to edit them, data scientists to build models on them and provide transparency to our business stakeholders.” — Joel Prince Varghese, Staff Data Scientist @ eHealth
Just the beginning
The effort eHealth is putting into building and continuously improving their NLU classifier is a long term investment that will continue powering their analytics, augmentation and automation roadmap — and their CX!
We’ve just scratched the surface of what we can do with NLU. We’re excited to see how much HumanFirst allows us to do today, but are also really excited about the upcoming features in their short-term roadmap, (like Tuning/Evaluation tools and Analytics API) that can help take our NLU to the next level. — Jing Zhou, VP Data Science & Data Engineering @ eHealth
About eHealth eHealth, Inc. (NASDAQ: EHTH) operates a leading health insurance marketplace at eHealth.com and eHealthMedicare.com with technology that provides consumers with health insurance enrolment solutions. Since 1997, eHealth has connected more than 8 million members with quality, affordable health insurance, Medicare options, and ancillary plans. The company’s proprietary marketplace offers Medicare Advantage, Medicare Supplement, Medicare Part D prescription drug, individual, family, small business and other plans from over 180 health insurance carriers across fifty states and the District of Columbia. About HumanFirst HumanFirst is the developer-first labelling and data engineering platform for long tail conversational search and analytics. With a powerful data pipeline, ML assisted workflows and API - teams use HumanFirst to continuously optimize their NLU data at scale and build NLU-powered search and analytics.
Subscribe to HumanFirst Blog
Get the latest posts delivered right to your inbox
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
.svg)
.svg)
%20(42%20x%2024%20in)%20(1).png)
.png)
.png)
%20(280%20x%20208%20px)%20(4032%20x%202304%20px)%20(1).png)
.png)
.png)
.png)
.png)