Lessons learned from LLM based Chatbot study and assessment…#3
Continuing on in the series of lessons learned from the chatbot study, next let’s discuss use case discovery.
This was always a tricky topic, as from the beginning without a particular methodology for discovery or generation of hypotheticals, you will quickly run into a bit of a chick and egg problem. Trying to talk to potential end users about a technology that might as well be magic to them, quickly leads to wild overestimation of it doing everything, or the apprehension to even try it in the first place. Neither is helpful, and leaves you back where you started struggling to find something specifically valuable to the customer, and technically feasible for the technology to accomplish.
After stumbling through these types of conversations a number of times to ultimately pry out something feasible to experiment with, it was clear a better path was needed to at least start the initial conversations with end users to then ultimately drive selection of target use cases.
What was developed is a methodology based on data analysis from the FLASK research done out of South Korea (https://lnkd.in/gdpsVAM4), to generate six primary functions that encapsulated the “sweet spot” of all LLM functions. All LLMs can accomplish these core functions, some better than others obviously, but targeting use cases pertaining to these functions guarantee the best possibilities of potential success.
From there it was a matter of using career field specific tasks and duties data to create use case descriptions, references, and suggested source material. This was a nontrivial matter to both construct and refine to a stable and high quality state, but ultimately it was achieved, and produced over 600+ examples across 60+ career fields.
This provided us both baseline metrics for a whole range of value insights to LLM applications across the force, but also plenty of potential specific examples tied directly to individual career field data to facilitate those initial conversations with end users.
