Handling Full Context Windows

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Handling Full Context Windows

One of the errors you handled with a specific case when building the app in the first lesson was for exceeding the maximum context window length. The context window is the maximum number of tokens that the model can consider simultaneously. When you changed the app in the last lesson to use a single LanguageModelSession, all the prompts and responses formed a single context window. This context window is how the model references the fruits from one response in a later prompt in the last lesson. Anything not in the context window must be embedded in the model or added using other methods, such as Retrieval Augmented Generation (RAG), which is not directly supported in the initial version of Foundation Models.

Exceeding the maximum context window length.
Ojfoubipz bcu xokicuh qirlatm fixfuj guyxwp.

func summarizeChat() {
}

summarizeChat()

// 1
var allText = ""
// 2
for entry in session.transcript {
  // 3
  switch entry {
  case .prompt(let prompt):
    allText += prompt.description + "\n"
  case .response(let response):
    allText += response.description + "\n"
  default:
    allText += "\n"
  }
}
// 4
addMessage("Context windows exceeded. Summarizing Chat", isFromUser: false)

let summarySession = LanguageModelSession(instructions: "Summarize all text presented to the model.")
let summarizedText = try? await summarySession.respond(to: allText).content

// 1
if let summarizedText = summarizedText {
  // 2
  resetChatHistory()
  addMessage(summarizedText, isFromUser: false)
  // 3
  session = LanguageModelSession(instructions: promptInstructions)
  // 4
  let response = try? await session.respond(to: summarizedText)
  addMessage(response?.content ?? "", isFromUser: false)
} else {
  // 5
  resetChatHistory()
}

Produce a brief summary of the following text.
The summarized text
Rfa bevsufuxek muqc
See forum comments
Download course materials from Github
Previous: Foundation Model Options & Tuning Next: Conclusion