Ragwalla builds a knowledge graph from your files: it reads each file, pulls out the people, companies, projects and products it mentions, and links them to each other and to the files they came from. The model that does the reading is the graph's extraction_model, and until this week it had to be a chat model writing JSON.
Cloudflare's new extraction model, Clairic (@cf/cloudflare/clairic), looked like a good fit for that job. It takes a document and a set of fields, fills each field with text taken straight from the document, and costs $0.10 per million input tokens with free output. So we added a Clairic mode to Ragwalla's knowledge graphs and tested it against gpt-6-luna, a small chat model at $0.10 / $0.50 per million tokens.
The short answer: Clairic reads long files without trouble, but the graph it builds is much noisier. We are keeping gpt-6-luna for knowledge graphs.
How Clairic mode works
Clairic fills fields, and a list field holds plain strings. So in Clairic mode a graph gets one list field per entity type (the schema's types, or a default set), and each name Clairic lists becomes an entity of that type. Clairic cannot write relationships, so each entity is tied to its files only through the segments it was read from. A chat model writes entities and relationships together, as JSON.
Ragwalla refuses Clairic as extraction_model when the graph's schema asks for something it cannot fill, such as document-level fields or entity properties beyond a name and type.
Test 1: does a long file fit?
Clairic reads at most 64K tokens, and its docs say fields are split into 8,192-token passes. Ragwalla sends up to two 32,768-character segments at a time, so we checked that a long file is read whole. We used a 39,952-character meeting transcript from the EnterpriseRAG benchmark corpus. Clairic found an entity at character 39,767, so the whole file was read. Nothing was cut off.
Test 2: ten files, same schema
We picked 10 files from the corpus: one from each of its nine apps (Confluence, Fireflies, GitHub, Gmail, Google Drive, HubSpot, Jira, Linear, Slack), plus one more Gmail file, all 2K to 30K characters, chosen with a fixed random seed. We built two graphs from them with the same six-type schema (person, organization, customer, project, product, team), one with Clairic and one with gpt-6-luna.
| Clairic | gpt-6-luna | |
|---|---|---|
| Entities | 1,319 | 553 |
| Distinct names | 599 | 544 |
| Short fragments (4 chars or fewer, not a word in the text) | 124 | 0 |
| Names filed under 2 or more types | 360 | 9 |
| Names filed under 3 or more types | 180 | 0 |
| Relationships | 0 | 689 |
| Real people found | about 24 | about 25 |
Three things went wrong with Clairic:
- Fragments. Clairic listed single letters and cut-off words as names: "Q", "L", "S", "K", "T" and "R" each appear five or six times, and one file name became "LedgerBank_KMS_Quest". gpt-6-luna listed none.
- The same name under every type. Clairic filled each type's list on its own, so a prominent name lands in most of them. "LedgerBank", a customer in the corpus, is filed as a customer, an organization, a person, a product and a team. LedgerBank, Silverpine and Helios each appear under all six types once their variants are counted. gpt-6-luna files LedgerBank as a customer and LedgerBank Security as a team.
- No relationships. This is by design, since Clairic's lists hold only strings, but it is most of what makes a graph useful. gpt-6-luna wrote 689 relationships from the same files.
Of Clairic's 599 distinct names, 60% were filed under more than one type. Of gpt-6-luna's 544, 98% were filed under exactly one.
Both models found about the same real people, and both missed the people named only in a file's metadata (authors and reviewers). That text probably never reaches extraction, which we will look into separately.
The schema matters for chat models too
In an earlier two-file run without a schema, gpt-6-luna invented 99 different entity types for 363 entities. With the six-type schema, every entity got one of the six. If you use a chat model for extraction, give the graph a schema with a fixed list of entity types.
What it cost
Both 10-file graphs cost a few cents to build. Clairic was cheaper, but at these prices the difference does not matter next to the quality gap.
Would a graph help retrieval?
We also asked whether a knowledge graph would help on the EnterpriseRAG benchmark itself. Our best search (hybrid search, then the Jev reranker over 90 candidates) misses 81 of the 364 gold documents behind the benchmark's dev questions. A graph adds one new way to search: take the names a question mentions and return every file tied to them. So we checked, over all 511,962 documents, how often that lookup could reach the missed gold. We counted any document containing a name word for word, which is generous to the graph.
| What we found | Missed gold documents |
|---|---|
| The question names no entity at all | 22 |
| The gold document never mentions the name the question uses | 35 |
| The gold mentions some of the question's names, not all | 8 |
| The gold is in the lookup, but the lookup is over 100 documents | 12 |
| The gold is in a lookup small enough to rerank | 4 |
Two things stop it. Most of the benchmark's questions (147 of 236) describe things instead of naming them: "the call-summarization customer", not a company name. And when a question does name a customer, the answer often lives in a general runbook or policy that never mentions that customer. Two questions ask about Northbridge Bank's log retention, and none of their four gold documents contains the word "Northbridge". A graph ties a customer to its own case files, not to the runbook that answers the question.
So a knowledge graph is useful for questions about the entities themselves, such as who works on what or which customers use which product. It is not a fix for this benchmark's retrieval misses.
What we're doing
We are keeping gpt-6-luna, with a schema, as the extraction model for knowledge graphs. We are not building one over the benchmark corpus. Clairic mode stays in Ragwalla for anyone who wants a quick list of names from a document, but it is not the right tool for building a graph.
The measurement script, the entity lists and the chart code are in the ragwalla-taubench repository under results/blog-kg-extraction/, and the retrieval check under results/kg-entity-probe/.