At Braid Technologies, our mission is to power up AI teams to deliver faster. We upskill and organise new teams to deliver with a world class blend of AI technologies, industry knowledge, and AI ways of working, including AI-enabled tools and patterns.
This blog goes through the architecture of one of our tools – ‘Boxer,’ an interactive AI learning assistant.
The Knowledge Base
The first thing we need to establish is a knowledge base – the set of foundational knowledge that will underpin how our client’s team build systems with embedded AI. This consists of a curated set of Videos, Blogs, and GitHub repositories, mostly sourced from Stanford University, the Silicon Valley start-up crew, and the big commercial players in AI.
To turn this into something an AI system can work with, we need to ingest it and categorise the embedded knowledge. We do this by:
1. Chunking. Large documents are divided up into manageable sections. Small fragments of text, such as the transcript lines of a video, are merged into larger ones. In either case, we ‘overlap’ the chunks in case a key concept has been split into two parts by our subdivision.
2. Embedding. Each chunk is loaded into a language model, which gives back an ‘embedding’ - a vector representation of the document – a summary of the neural network’s state when the document is ingested. This is something we can use later - similar documents will ‘point’ in the same direction in the vector space of the embedding.
3. Summarisation. We ask the language model to produce a summary of the document, which can be used by our teams to preview the document and decide if they want to invest the time to watch or read the entire thing.
At the end of this process, we have a database of document chunks, with a summary and an embedding for each.
Querying the Knowledge Base
When our teams have questions about how to develop AI-components for their applications, we provide a ChatGPT style front end to capture the question. We then do some basic ChatGPT actions, and combine them with some more sophisticated ones for our specific learning environment.
1. Answer the question. We relay the question to the LLM, and provide the direct answer. Like ChatGPT.
2. Enrichment. To go beyond this, we use a key feature of LLMs – their ability to ‘imagine.’ In this case, given a question, we ask the LLM to imagine a document that provides a great answer. A great document would of course answer the question, but would also cover underlying concepts needed to provide the answer, and might also cover other related concepts to those included in the question or the answer. We ask the LLM to provide a summary of this great document, as though it exists. We then generate an embedding for this great answer, and we look for the ‘closest’ documents in our repository. We also give these in response to the question. We are not just answering the question, but are providing links that help the learner go deeper.
3. Follow up. When our learner interacts positively with part of the answer (by following the link, or clicking a ‘thumbs up’ icon, we do a final round trip with the LLM. We ask it to generate a question that a developer building AI-enabled systems might have after reading the article. We surface this to the learner in the user interface, and they can ask it with a click if they wish.
In this part of the process, we have gone beyond basic Q&A, to provide:
How do we evaluate all this?
The most common answer these days is to use an LLM. Sometimes you might want to use a different one to the main one in your production system, as a sort of cross check.
The workflow in this case is:
1. Generate questions. We ask the LLM to adopt the persona of an application developer seeking to build AI-enabled technologies into their application, and we ask it to generate a batch of questions.
2. Measure coverage. We create embeddings for the questions, and check that we can answer them (i.e. we can find an answer that meets a minimum relevance threshold). This allows us to find gaps in the knowledge base.
3. Evaluate usefulness. Where we have matches, we look at the ‘relevance’ scores for our answers. If we dip below a good relevance threshold, we need better quality material in the knowledge base on these topics.
4. Evaluate relevance of our suggested questions. We check our questions against the knowledge base, as a miss will imply the question is not relevant, or that we have a gap in the knowledge base. We also ask the LLM, in its persona as a developer, if this is a relevant question about AI. If it says ‘no,’ we have some improvements to make.
Where we have gaps or quality issues, we can add content, tune the prompts, or tune the thresholds for relevance. And of course, we need to combine these metrics with empathetic thinking and observation of the learners, to ensure we are using engineering measures that correlate with learning outcomes.
Summary
In the words of our assistant:
“Braid Technologies' AI learning assistant, 'Boxer,' is designed to help clients build high-performing AI teams. It uses a curated knowledge base of videos, blogs, and GitHub repositories, which are ingested, chunked, embedded, and summarised. When users query the system, it provides direct answers and enriched responses by imagining and linking relevant documents. Follow-up interactions generate further questions to deepen learning. The system is tested by generating developer-like questions, measuring coverage, and evaluating answer relevance. This ensures comprehensive, adaptive learning and continuous improvement of the knowledge base. Metrics and empathetic observation are combined to enhance learning outcomes.”
(c) Braid Technologies Ltd, 2024. All rights reserved.
We hope that this has been informative. If you would like to work with us, please get in touch. Find us at: https://www.linkedin.com/company/braid-technologies-ltd.
