TinkeReReader

Book Exploration with LLM’s

// @daneroo

Choice of Weapons

• Locally running LLM’s
  • Ollama (llama2 7b / mistral 7b)
• LangChain (.js)
• Sources
  • Choice ePub ebooks
  • Thesis
  • Synthesized text (Thanks GTP4)

Summarization - 1st attempt

Using LangChain(.js)

const chain = loadSummarizationChain(model, { type: "refine" });

summary = summarize(chunk1)
summary = summarize(chunk2, summary)
summary = summarize(chunk3, summary)
...

When this is performed on a large number of chunks (>30), the running summary becomes very forgetful.

Summarization - 2nd attempt

Repeatedly split, summarize, concat

level0Chunks = split(OriginalText)
level0Summaries = [...level0Chunks].map(summarize)
level1Txt = concat(level0Summaries)

level1Chunks = split(level1Txt)
level1Summaries = [level1Chunks].map(summarize)
level2Txt = concat(level1Summaries)

...

Until levelNText is small enough.

This turns out to be a very effective approach, and produces a very good summary.

Example Result

Hero of Ages - Summary ↗️

~10:1 reduction per level

Character Extraction - 1

• extract the characters from a novel
• aggregate their descriptions

Same as with summarization

• langChain’s refine is not suited for long text.
• The refine chain is too lossy, or forgetful

Character Extraction - 2.1

• Extract characters from each chunk (LLM)
  • Constrain to JSON (with a schema)

chunk1:[
  { "name": "Dr. Yamada", "description": "A scientist" },
  { "name": "Kaito", "description": "A hacker" }
]

chunk2:[
  { "name": "Kaito", "description": "invaluable to Dr. Yamada" }
]
...chunkN:

Then …

Character Extraction - 2.2

• Aggregate the descriptions (JavaScript)
• Synthesize a description (LLM)

{
  "Kaito": ["A hacker", "invaluable to Dr. Yamada"],
  "Dr. Yamada": ["A scientist"]
}

“Kaito is a hacker, invaluable to Dr. Yamada”

Character Extraction - Example

Neon - Characters - llama2 ↗️

• Kaito
  • A young street-smart hacker with a …
  • He joins the trio stop The Architect …

Reformulated

Kaito is a young street-smart individual with a reputation within the underground networks, … Kaito joins the trio on their quest to stop The Architect.

Hero Of Ages - Characters - llama2 ↗️

Future Work

• Lot’s of refactoring ;-)
• JSON OutputParser can be made more robust
• Disk caching (refinements)
• Extend to other aspects of long-form text
  • Locations
  • Events
• Combine with RAGs
  • By indexing the summaries and aggregations

Takeaways

• A concrete project is invaluable to learning LLM’s
• LangChain(.js) is a great tool to start
• Local LLM’s are truly a feasible option
• Map/Reduce is a powerful pattern

Thank You

// @daneroo

\[ \text{Daniel Lauzon} = \sum_{meetups} \text{people}_{met} + \epsilon \]

The End.

Map / Reduce (ctd)

Math Reduce

\[ \text{length}_\text{tot} = \sum_{\text{doc} \in \text{docs}} \text{length}(\text{doc}) \]

MathJax

A thin wrapper for MathJax

\[P(E) = {n \choose k} p^k (1-p)^{ n-k} \]

\[ \frac{1}{\Bigl(\sqrt{\phi \sqrt{5}}-\phi\Bigr) e^{\frac25 \pi}} = 1+\frac{e^{-2\pi}} {1+\frac{e^{-4\pi}} {1+\frac{e^{-6\pi}} {1+\frac{e^{-8\pi}} {1+\ldots} } } } \]

\[ {}_{\text{Ottawa}}^{\,\quad\text{AI}} \text{Tinkerer}_{\text{Hackathon}}^{2023} \]