Cover of the book with images of ChatGPT use cases.

UX for Enterprise ChatGPT Solutions
by Richard H Miller, Ph.D.

Companion References

I hope you are here because you are enjoying my book. I found it a usability issue to include URLs in a physical book. Kind of hard to click a link in a book... So, I thought one way to help was to create this web page. So, it is a single source of truth for all the book recommendations and online references. One-stop shopping; I hope you find it helpful!

Need to order the book? Order via Amazon, happy to sign it if you are in the San Francisco Bay Area or at any presentation or seminar!

Thanks for visiting, Richard Miller richard@uxdforai.com

Part 1: UX Foundation for Enterprise ChatGPT

Every good story has a beginning, a middle, and an end. In this part, we’ll start our journey by exploring how traditional user experience methods and best practices can be applied to creating world-class solutions powered by ChatGPT. We will then explore essential user research methods and provide tips and secrets of the trade that work when creating conversational designs. This will lead us to explore how to define and pick the use cases that large language models (LLMs) are best suited to solve. A user experience approach is taught to prioritize use cases. When combined with a method to include development costs into the mix, this powerful method from Agile allows for prioritizing the most valuable solutions to build. Although we’ll consider how these use cases play out with ChatGPT, almost all the learnings can apply to any LLM model. We’ll look at LLM-powered applications, chat-only experiences, and robust chat-powered graphical user interfaces, and we’ll even explain how to work with ChatGPT when there is no UI. This part has the following chapters:


Chapter 1: Recognizing the Power of Design in ChatGPT

  1. GitHub: Repository for Book Material. Richard Miller
  2. Website: OpenAI ChatGPT Experience. OpenAI
  3. Website: Quickstart guide for developers. OpenAI.
  4. Article: Computing Machinery and Intelligence (1950). Alan Turning in Mind.
  5. Wikipedia: Turing Test.
  6. Wikipedia: ELIZA details.
  7. Demo: ELIZA demo. Michael Wallace and enhanced by George Dunlop.
  8. Wikipedia: History of Chatbots.
  9. Video: Steve Martin teaching his kid for the first day of school. Steve Martin.
  10. Wikipedia: The History of VoiceXML.
  11. Article: The origins of OpenAI. Karl Montevirgen
  12. Article: Chatbot Failures. Cem Dilmegani
  13. Wikipedia: Survey of LLMs.
  14. Article: Google's LaMDA. Google
  15. Article: Meta's LlaMa. Meta
  16. Article: Anthropic's Claude. Anthropic
  17. Article: OpenAI's GPT Models. OpenAI
  18. Article: Hick-Hyman Law.
  19. Wikipedia: Explanation of working memory.
  20. Books Collection of useful design books.
  21. Article: ChatGPT AI for Slack. Slack
  22. Article: AI in Salesforce. Salesforce
  23. Article: Feeding Sensitive Data to ChatGPT. Robert Lemos.
  24. Figure 1.3: Getting an account with OpenAI
  25. Figure 1.3: The OpenAI Playground.
  26. GitHub: The Full_Thesis.pdf file. Richard Miller
  27. Article: File Search from OpenAIOpen AI.


Chapter 2: User Research

  1. Figure 2.1: The broad landscape of user research methods https://www.nngroup.com/articles/which-ux-research-methods. Christian Rohrer
  2. Website (bonus link): Nielsen Norman Website Survey of research methods. Christian Rohrer
  3. Book: A deeper dive into user research methods The User Experience Team of One: A Research and Design Survival Guide (2nd Edition)_. Leah Buley
  4. Website: Using comic strips to storyboard UX Comic Strip for Storyboarding. Chris Spalton
  5. Book: Writing Effective Use Cases Writing Effective Use Cases. Alistair Cockburn
  6. Book: Survey Methods Questionnaire Design: How to Plan, Structure and Write Survey Material for Effective Market Research. Brace and Bolton (5th Edition, 2022)
  7. Article: Primer on Likert Scales What is a Likert scale?. Surveymonkey
  8. Article: Creating an Interview Scripts Example User Interview Script. Surveymonkey
  9. Website: Discount Usability Methods Discount Usability Methods. Jakob Nielsen
  10. Website: User Interviews User Interviews. Jakob Nielsen
  11. GitHub: Spreadsheet for Conversational Analysis Examples to Review. Richard Miller and Surlina Yin
  12. GitHub: Spreadsheet template for Conversational Analysis Template for Analysis. Richard Miller and Surlina Yin


Chapter 3: Identifying Optimal Use Cases for ChatGPT

  1. Book: Recommendation on Use Cases Writing Effective Use Cases. Alistair Cockburn
  2. Book: PDF of Alistair's Book Online PDF of Book. Alistair Cockburn
  3. Article: Amazon Searches Top 100 Amazon Searches. Source
  4. Article: The Trolly Problem The Trolly Problem. Source


Chapter 4: Scoring Stories

  1. Article: Weighted Shortest Job First WSJF Discussion. Scaled Agile, Inc.
  2. Book: Software development process The Principles of Product Development Flow. Donald Reinertsen
  3. Article: Comparison of UI evaluation techniques User Interface Evaluation in the Real World: Comparison of Four Techniques. Robin Jeffries, James R. Miller, Cathleen Wharton, and Kathy M. Uyeda. ACM (1991)
  4. Article: Agile Estimating Poker Estimating Poker. Scaled Agile, Inc.
  5. Wikipedia: Accessibility Learning about Accessibility. Wikipedia
  6. Wikipedia: Language Support NLS Support. Wikipedia
  7. Article: Explaining story points Why do we use Story Points for Estimating? Derek Davidson, Scrum.org.
  8. GitHub: Example File for Scoring Stories Scoring Stories Samples. Richard Miller
  9. GitHub: Worksheet Scoring Stories Scoring Stories Worksheet. Richard Miller
  10. Article: Explaining story points Explaining Story Points. Source
  11. Article: More on Story Points Don’t Equate Story Points to Hours. Mike Cohn, Mountain Goat Software.
  12. Website: SAFe Scaled Agile Framework. Source
  13. Article: Using WSJF and the use of Fibonacci Prioritize your backlog: Use Weighted Shortest Job First (WSJF) for improved ROI. Matthew Heusser
  14. GitHub: Deep dive on severity (PDF) More details on setting up a Severity Rubric. Richard Miller


Chapter 5: Defining the Desired Experience

  1. Article: The Role of Micro-interactions in Modern UX Micro-interactions. Mads Soegaard.
  2. Book: (Online Bonus) Recommendation Microinteractions: Full Color Edition: Designing with Details. Dan Saffer and Don Norman
  3. Article: The Essence of Effective Rich Internet Applications RIA introduction. Kevin Mullet, Macromedia Experience Design Team.
  4. Video: Resturant ordering via voice Resturant ordering via voice. Soundhound.
  5. Video: A generative AI co-pilot feature in a GUI Demo of Oracle Virtual Assistant. Jurgen Kress, Oracle.
  6. Article: Current Desktop addressable sizes Current browser usage and statistics. Statcounter.
  7. Books: The work of Edward Tufte The collective works of Edward Tufte. Edward Tufte.
  8. Article: Recognition over recall Recognition over recall. Raluca Budiu, Nielsen Normal Group
  9. Article: Web guidelines for accessibility Web Content Accessibility Guidelines (WCAG) 2. World Wide Web Consortium.
  10. Article: US Federal Standards for software Section 508 Standards. US Government.
  11. Article: Accessible Rich Internet Applications suite of web standards WAI-ARIA Guidelines. World Wide Web Consortium.
  12. Article: ADA website American with Disabilities Act. US Government.
  13. Article: AI Guidance . Algorithms, Artificial Intelligence, and Disability Discrimination in Hiring.
  14. Article: European Standards European Standards. Publisher.
  15. Article: Human Factors standards for video (conferencing) Human Factors in Videotelephony. European Telecommunications Standards Institute (ETSI).
  16. Article: ITU-T F.922 Guidelines ITU-T F.922 Guidelines. International Telecommunication Union (ITU).
  17. Website: Reference for a counterpoint to the concept of cheap, fast, or good. Choose two. Cheap, Fast or Good, Choose Two. Benek Lisefski
  18. Book: Recommendation on writing and punctuation Eats, Shoots & Leaves: The Zero Tolerance Approach to Punctuation. Lynne Truss
  19. Web link: Google Sheets for Testing Translation Length Testing Translation Length. Richard Miller (feel free to clone your own sheet!)
  20. Website: Reference on spelling out numbers When to spell out numbers. MasterClass
  21. Website: Reference on pluralization Introduction to pluralization. Lingo Hub
  22. Website: Reference on plural rules Plural Rules for Countries. Unicode
  23. Article: The Science of Word Recognition The Science of Word Recognition. Microsoft.
  24. Website: Reference on word recognition The Science of Word Recognition (ALL CAPS discussion). Microsoft Team
  25. Website: Reference on cultural design Primer on Cultural Design. Google
  26. Website: Reference on bi-directionality BI-directional design. Goggle


Part 2: Designing LLMS

This part focuses on the most technical part of LLM design, the work done in the LLM, or when using other tools and techniques to make an end-to-end LLM solution. We will start by introducing practices such as Retrieval Augmented Generation (RAG) to integrate enterprise datasets with the generative ability of an LLM. You’ll then learn about the fundamentals of prompt engineering for enterprise applications. The examples follow a non-technical route, so you can get the most out of learning about the steps involved without coding. We’ll then explore fine-tuning to make models think and act based on examples provided that will follow the style and tone needed for any business or enterprise. We’ll also explore a few case studies and give some hands-on experiences to get a feel for the process. This part includes the following chapters:


Chapter 6: Gathering Data: Content is King

  1. Article: Open AI Knowledge Retrieval Documentation. OpenAI
  2. (Updated) Documentation: Supported File Formats. OpenAI
  3. Article: Amazon’s RAG Explanation. Amazon
  4. Article: Leveraging LLMs on your domain-specific knowledge base. Michiel De Koninck
  5. Video: Accelerate your Generative AI journey. Databricks, Hanlin Tang, Ina Kolea, and Akhil Aggrawal
  6. Article: A survey of RAG for LLMs. Yunfan Gaoa, Yun Xiong, et. al.
  7. GitHub: FAQ Collection for Testing. Richard Miller
  8. Demo: OpenAI Playground.
  9. Website: The freight forwarding use case Wove.com.
  10. Article: Rate Sheet Terms and Introduction. Publisher.
  11. Documentation: ChatGPT documentation on function calling. OpenAI
  12. Website: Cohere, an AI platform. Cohere
  13. GitHub: FAQ Sample Document. Richard Miller
  14. GitHub: Zip of FAQS as unique PDFs. Richard Miller
  15. GitHub: Zip of 18 FAQ Files (each with 25 or so FAQs). Richard Miller
  16. GitHub: Single PDF with all 441 FAQs. Richard Miller
  17. Video: A Survey of Techniques for Maximizing LLM Performance. OpenAI, Colin Jarvis and John Allard
  18. GitHub: Transcripts of FAQ Test. Richard Miller.
  19. Article: Prolego tips for RAG development. Prolego


Chapter 7: Prompt Engineering

  1. Documentation: Prompt Examples Prompt Documentation. OpenAI.
  2. Video: Techniques for improving LLM Quality (45 minutes) A Survey of Techniques for Maximizing LLM Performance. OpenAI.
  3. Video: Prompt Eng, RAG, and Fine Tuning (15 minutes) Prompt Engineering, RAG, and Fine-tuning: Benefits and When to Use. Mark Hennings of Entry Point AI
  4. Article: Getting started with LLM prompt engineering Getting started with LLM prompt engineering. Shane Peckham, Jeff Day, and Diana Hbr for Microsoft.
  5. Article: Prompt Techniques Best Prompt Techniques for Best LLM Responses. Jules Damji.
  6. GitHub: Basic prompting from the CO-STAR framework https://colab.research.google.com/github/dmatrix/genai-cookbook/blob/main/llm-prompts/1_how_to_use_basic_prompt.ipynbBasic prompting from the CO-START framework
  7. Resource: Training from Miha Fine Tuning Training from Mihael Cacic
  8. Video: Optimize Instruction tuned Conversational AI/LLM OpenAI Playground: Optimize Instruction tuned Conversational AI /LLM. code_your_own_AI
  9. Article: Self-consistency improves chain of thought reasoning in language models Self-consistency improves chain of thought reasoning in language models. Wang, et. al..
  10. Article: Writing clear instructions Strategy: Write clear instructions. Open AI.
  11. Article: Chaining complex prompts for stronger performance Chaining complex prompts for stronger performance. Publisher.
  12. Video: Andrew NG Agentic Presentation Video: Andrew NG Agentic Presentation. Publisher.
  13. Article: SELF-REFINE: Iterative Refinement with Self-Feedback SELF-REFINE: Iterative Refinement with Self-Feedback. Madaan et al.
  14. Article: Reflexion: Language Agents with Verbal Reinforcement Learning Reflexion: Language Agents with Verbal Reinforcement Learning. Shinn et al.
  15. Article: Gorilla: Large Language Model Connected with Massive APIs Gorilla: Large Language Model Connected with Massive APIs. Patil et al.
  16. Article: MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action. Yung et al.
  17. Article: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al.
  18. Article: HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. Shen et al.
  19. Article: Communicative Agents for Software Development Communicative Agents for Software Development. Qian et al.
  20. Article: AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. Wu et al.
  21. Article: A guide to prompt techniques A guide to prompt techniques. Tensor Ops.
  22. Article: Improving LLMs with emotional prompts Large Language Models Understand and Can Be Enhanced by Emotional Stimuli. Cheng Li et al..
  23. Article: The Good, The Bad, and Why? Unveiling Emotions in Generative AI The Good, The Bad, and Why? Unveiling Emotions in Generative AI. Publisher.
  24. Demo: Playground for learning about Temperature and Top P Playground for learning about Temperature and Top P. Open AI.
  25. Article: Prompt Engineering Guide Prompt Engineering Guide. Publisher.
  26. Article: Google explains multi-modal prompting for Gemini Google explains multi-modal prompting for Gemini. Google.
  27. Article: Language Is Not All You Need: Aligning Perception with Language Models Language Is Not All You Need: Aligning Perception with Language Models. Shaohan Huang et. al. Microsoft.
  28. Documentation: Ingredients of a Prompt Template Ingredients of a Prompt Template. Salesforce.
  29. Documentation: Guide to the prompt builder Guide to the prompt builder. Salesforce.
  30. Article: Lost in the Middle: How Language Models Use Long Contexts Lost in the Middle: How Language Models Use Long Contexts. Nelson F. Liu et. al. Stanford


Chapter 8: Fine-Tuning

  1. Demo: Tokenizer Tokenizer. Open AI.
  2. Article: When to use fine-tuning (same link as above) When to use Fine Tuning. Open AI.
  3. Demo: OpenAI Fine Tuning OpenAI Fine Tuning. Open AI.
  4. GitHub: Training Data with ten examples 10 Fine Tuning Examples. Richard Miller.
  5. GitHub: 20 validation examples 20 Fine Tuning Validation Examples. Richard Miller.
  6. GitHub: Training Data with 30 examples 30 Fine Tuning Examples. Richard Miller.
  7. GitHub: 49 training examples 49 Fine Tuning Examples. Richard Miller.
  8. GitHub: (Online bonus) Complete file with all training examples All 78 training examples. Richard Miller.
  9. Training: Miha's training website where I got Fine Tuning training https://miha.academy/LLM Training Academu. Mihael Cacic.
  10. Article: Training vs. Validation Loss Training vs. Validation Loss. Publisher.
  11. Documentation: Calling functions with chat models Calling functions with chat models. Open AI.
  12. Article: Learning Rate What is Learning Rate. ChatGPT Guide.
  13. Documentation: ChatGPT Fine Tuning Documentation Fine Tuning Intergrations. Open AI.
  14. Article: Getting Started with OpenAI Evals Evals. Open AI.

Part 3: Care and Feeding

In the last phase of your journey, we’ll explore the tools and resources needed to understand how the LLM is performing and what to look for to fix it if it isn't up to expectations. You’ll first learn how to apply existing guidelines and heuristics that can be used in ways you didn’t expect to craft LLM solutions. We’ll even give you the skills to adapt web and traditional design thinking to conversational AI. Then, you’ll dive into understanding how to monitor and evaluate solutions based on industry AI and user experience metrics, covering objective and subjective performance measures in the care and feeding process. The lifecycle of an LLM solution demands good business processes to be successful. Best practices that can work at the most prominent enterprises are shared to help create effective methods for creating world-class LLM solutions. This part includes the following chapters:


Chapter 9: Guidelines and Heuristics

  1. Article: 944 Guidelines for Designing User Interface Software Smith and Mosier’s Guidelines for Designing User Interface Software. Publisher.
  2. Article: First Principles of Interaction Design First Principles of Interaction Design. Bruce Tognazzini
  3. Article: Intro to Heuristic Evaluation (HE) What is Heuristic Evaluation (HE)?. Interaction Design Foundation.
  4. Article: How to perform a heuristic evaluation How to perform a heuristic evaluation. Nielsen Normal Group.
  5. Article: Memory Recognition and Recall in User Interfaces https://www.nngroup.com/articles/recognition-and-recall/. Nielsen Normal Group.
  6. Article: Proposal to include Accessibility as the 11th Heuristic Proposal to include Accessibility as the 11th Heuristic. Jim Ekanem.
  7. GitHub: Guidelines typical of a GUI and supporting conversational experiences Heuristic Checklist example. Richard Miller.
  8. Website: Writing standards in the Chicago Manual of Style Chicago Manual of Style. Publisher.
  9. Article (Online Bonus): The 10 Usability Heuristics Reimagined The 10 Usability Heuristics Reimagined. Jakob Nielsen


Chapter 10: Monitoring and Evaluation

  1. Documentation: Introduction to RAGAs Introduction to RAGAs. Ragas.
  2. Tutorial: Installing RAGAs Installing RAGAs. Ragas.
  3. Article: Evaluating the performance of RAG solutions Evaluating the performance of RAG solutions. Ragas.
  4. Article: LLM Leaderboard Hughes Hallucination Evaluation Model (HHEM) Leaderboard. Hughes.
  5. Discussion: Text Embedding Issues Discussion: Text Embedding Issues.
  6. Article: Semantic Answer Similarity: The Smarter Metric to Score Question Answering Predictions Semantic Answer Similarity: The Smarter Metric to Score Question Answering Predictions. Isabelle Nguyen.
  7. Article: Semantic Answer Similarity for Evaluating Question Answering Models Semantic Answer Similarity for Evaluating Question Answering Models. Risch et al..
  8. Article: Metrics for evaluating LLMs Metrics for evaluating LLMs. Publisher.
  9. Article: Evaluating RAG Applications with RAGAs Evaluating RAG Applications with RAGAs. Publisher.
  10. Article: Automating Hallucination Detection Automating Hallucination Detection. Vectara.
  11. Article: An evaluation on large language model outputs: Discourse and memorization An evaluation on large language model outputs: Discourse and memorization. Wynter et al., 2023.
  12. Article: Taxonomy of Hallucinations in LLMs Taxonomy of Hallucinations in LLMs. Deval Shah, Lakera.
  13. Article: A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. Huang et al., 2023.
  14. Video: Techniques for maximizing LLM performance Techniques for maximizing LLM performance. Open AI.
  15. Documentation: OpenAI's view on testing strategy OpenAI's view on testing strategy. Open AI.
  16. Article: Out of Domain Detection Out of Domain Detection. Open AI.
  17. Video: Tutorial for improving retrieval Tutorial for improving retrieval. Open AI.
  18. Table 10.7 Article: How to Use Rouge 2.0 How to Use Rouge 2.0. Open AI.
  19. Table 10.7 Article: Perplexity and Burstiness Perplexity and Burstiness. Open AI.
  20. Table 10.7 Product: Originality AI Originality AI. Open AI.
  21. Table 10.7 Aiesera: LLM Evaluation: Metrics and Benchmarking Performance Aiesera: LLM Evaluation: Metrics and Benchmarking Performance.
  22. Article: How to Evaluate LLMs: A Complete Metric Framework How to Evaluate LLMs: A Complete Metric Framework.
  23. Article: Patterns of Trustworthy Experimentation: During-Experiment Stage Patterns of Trustworthy Experimentation: During-Experiment Stage.
  24. Article: How to modify the NPS Survey question to meet your needs How to modify the NPS Survey question to meet your needs.
  25. Table 10.9 Article: NPS scores for the table, Nice Source Nice NPS scores.
  26. Table 10.9 Article: NPS scores for the table, CustomerGauge Source CustomerGuage NPS scores.
  27. Table 10.9 Article: Typical NPS for a product Simplesat NPS scores.
  28. Article: Net Promoter Score Net Promoter Score
  29. Article: How to SUS out usability scores How to SUS out usability scores.
  30. Article: Is the SUS Too Antiquated Is the SUS Too Antiquated.
  31. Article: 5 Ways to Interpret a SUS Score 5 Ways to Interpret a SUS Score.
  32. Article: Software Usability Scale Software Usability Scale.


Chapter 11: Working Design Processes into a Development Lifecycle

  1. Website: AI marketplace for Jira and Confluence Atlassian Marketplace.
  2. Video: Introduction to Agile Introduction to Agile. Agile Alliance.
  3. Article: State of Agile Report (linked to latest report) State of Agile Report. Digital AI..
  4. Article: The Agile Manifesto The Agile Manifesto.
  5. Data: Ultrachat 200,000-row data set Ultrachat 200,000-row data set.


Chapter 12: Summary


Books Recommendations: A summary of all books referenced and recommended in this book.

  1. A primer for learning interaction designer (Chapter 1) The Design Of Everyday Things Paperback. Don Norman (Revised & Expanded Edition, 2013)
  2. A reference book for design principles (Chapter 1) Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design. Lidwell, Holden, and Butler (2010)
  3. The psychology of design (Chapter 1) Designing with the Mind in Mind: Simple Guide to Understanding User Interface Design Guidelines. Jeff Johnson (3rd Edition, 2020)
  4. Introduction to design (Chapter 1) Don't Make Me Think, Revisited: A Common Sense Approach to Web Usability. Steve Krug (3rd Edition, 2013)
  5. The human factors underpinning of UX design (Chapter 1) Engineering Psychology and Human Performance. Christopher D. Wickens, Helton, Hollands, and Banbury (5th Edition, 2021)
  6. Research methods (Chapter 2) The User Experience Team of One: A Research and Design Survival Guide. Leah Buley
  7. Writing use cases (Chapters 2 and 3) Writing Effective Use Cases. Alistair Cockburn (2000)
  8. Survey Method (Chapter 2) Survey Methods. Brace and Bolton (5th Edition, 2022)
  9. Product development principles (Chapter 4) The Principles of Product Development Flow. Donald Reinertsen (2009)
  10. Microinteraction (Online Bonus) (Chapter 5) Microinteractions: Full Color Edition: Designing with Details. Dan Saffer and Don Norman
  11. The books of Edward Tufte
  12. On the use of punctuation and writing quality (Chapter 5) Eats, Shoots & Leaves: The Zero Tolerance Approach to Punctuation. Lynne Truss (2006)
  13. Classic reference on interface design (Chapter 9) Tog on Interface. Bruce Tognazzini (1992)
  14. How to understand the goal of a business (Chapter 12) The Goal: A Process of Ongoing Improvement. Eliyahu M Goldratt, Jeff Cox, and David Whitford (North River Press).















































And yes, I do hope to earn a few bucks with any book purchases for my reference material. Writing a book is a labor of love, so please use my links. It is at no cost to you!

© Packt Publishing and Richard H. Miller, All Rights Reserved.