What does a good record look like?↗
By Mark Tebeau and Claude Fable 5 (Anthropic)
A note on authorship: ARCADE is built through AI-inflected practice. The post was written by Mark Tebeau and Claude Fable 5. Effective use of AI announces its provenance and use.
Two weeks ago I could not answer a basic question about our own work: what does a good record look like? I could recognize one. I could not define one, and neither, candidly, can the field — the literature on AI-generated metadata reaches for rubrics and gold standards, and both assume the thing we lacked, a settled account of quality. So we stopped trying to define good and started trying to recognize it, on the oldest finding in the judgment literature: people choose between two things far more reliably than they score one.
The test collection was the Hermit Herald, 150 issues of a satirical newsletter that Peter Bundy, writing as “Piero Bundini,” sent to family and friends from Florida through three years of the pandemic, and contributed to our COVID-19 archive, A Journal of the Plague Year. We had two Claude models each describe the same ten issues. Then three of us — Katy Kole de Peralta, Erin Craft, and I — judged the pairs blind, labels shuffled, key sealed, one reason per choice.
Neither model won. My choices split four to four; Erin’s five to five; Katy leaned six to four. Agreement between any two of us ran at or below a coin flip. The models, it turned out, were not the interesting variable. We were. Each of us judged coherently and by a different standard: I judged whether the description captured the newsletter’s humor, Katy rewarded context and care, Erin rewarded structure and usable vocabularies. A rubric with scores would have averaged that away. Ten pairs and thirty sentences of reasoning surfaced it in an afternoon.
The judging also forced a harder question. Our records carried fields addressed to a reviewing curator — uncertainty notes, verification flags, instructions for checking. For whom were those fields written? The collection we aspire to describe at scale runs to two million photographs. No curator reviews two million records, so a field written for the curator is written for no one. We rebuilt the record for the public reader, entirely — unknowns stated inline the way catalogers have always stated them, a rationale that says what the description emphasizes and why, evidence notes that say where consequential claims come from. Archival description’s own standard agrees; DACS opens its principles by stating that users are the fundamental reason description exists. And on contested claims the record now follows the tradition rather than the fact-checking reflex: say whose claim it is, once, and leave truth to the reader. Attribute, don’t adjudicate.
Then we made the design earn its keep. We wrote the collection’s facts and decisions into a Collection Charter, and ran the same ten issues through four models — Claude Sonnet and Opus, and two open-source models, Qwen3-235B and GPT-OSS-120B, on Arizona State University’s research computing. Every record from every model held the new form. The Charter, not the model, carries the structure. The frontier models produced the richer, more discursive records — descriptions that quote Bundy’s jokes rather than report that jokes occur — and stand nearly ready for collection-scale description. The open models performed well: rich, discursive, acceptable, and more prone to small confident errors, a garbled name, a misdated issue, that a person skimming forty records would likely miss. Our evaluation caught them because it reads every record; error-checking now lives inside the process, for every model.
The honest caveats are real. This is one collection, ten issues, and judges who built the system they judged; the open models read extracted text while the frontier models read the original pages, which may explain part of the gap; and criteria applied by an AI agent need humans spot-checking the agent. Those limits are why the full 150-issue runs are executing as this post is written.
But the question that started this has an answer now. What does a good record look like? Not like a score. It looks like a choice a knowledgeable person can make between two real records and defend in a sentence — and after enough choices, the sentences become standards, the standards become a charter, and the charter teaches any model, frontier or open, what this collection needs. Quality was never in the model. It was in the judgment, and judgment can be written down.
So: here is what we think a good record looks like
This week. For this collection. Our current answer, not our last one — below is issue 1 as Claude Sonnet described it under the Collection Charter — read the record against the original issue in the archive — printed whole, with our pencil in the margins: the blue circles mark where the weekend’s judging changed what records say, and each note beneath explains the mark.
ARCADE Metadata Record
Record identifier: Hermit Herald, issue 1 (provisional). The archive item ID, covid-19archive.org URL, and corpus filename are in the corpus manifest.
Object or item type: Newsletter issue (text only); satirical community newsletter, first issue.
Title or supplied title: The Hermit Herald, Vol. 1, Issue 1
Creator: Peter Bundy, writing as “Piero Bundini.” The issue is signed “Your faithful scribe, Piero Bundini.”
Date or date range: March 19, 2020 (from the masthead)
Place: Florida, written from there. The exact place is unknown. The Peanut Island reference probably points to the Palm Beach area.
Collection or provenance context: The Hermit Herald, a collection of 150 weekly issues (March 19, 2020 to March 19, 2023) in A Journal of the Plague Year COVID-19 Archive (covid-19archive.org), Arizona State University. Issue 1 belongs to the earlier group of issues (1 through about 24), which Bundy sent by email and Erin Craft made into PDFs. Later issues came through the archive’s web form.
Language: English
Format or medium: PDF, 4 pages. Bold serif body text under a display-lettered masthead. No illustrations.
Description: The first issue of The Hermit Herald, a four-page newsletter dated March 19, 2020, under the masthead “Hermit Herald, Vol 1 Issue 1.”
It opens in mock grandeur. The writer, living a “monastic existence” and able to read “the great works of the world” only so many hours a day, decides to place “upon their shoulders the angst of my ennui” by writing to “several circles of friends.” He promises “some true and helpful tidbits, some thoughtful musings and a good sprinkling of innocuous drivel,” and calls this “the acclaimed pre publication issue” that “may or may never hit the news stands again.” The paper’s origin story is twenty-five minutes spent staring at his feet, from 9:03 to 9:28 one morning.
Five short items follow, each a take on the week’s news.
- Hydroxychloroquine. “BREAKING: first good news on Covid-19” reports the old malaria drug as promising, “probably wishful thinking,” but exciting to “a bunch of medical people.”
- Tokyo Olympics. The writer objects (“You must be kidding”) that the IOC is “doubling down” on holding the July 2020 games although its deputy head has the virus.
- Naming the virus. The item headed “Corona virus” argues that “Chinese Virus” is a fair name, comparing it to “the Spanish virus” and to Lyme disease, and blames a long tradition of consuming animal organs.
- Incarceration. “Real jails, real prisoners” says lockdown gives a “personal sense” of total incarceration, and recommends the Nordic countries, especially Sweden, as models of humane treatment.
- Social distancing. “Secret escapes” describes a spotless, nearly empty Florida restaurant, compares it to the Kennedy-era bunker on Peanut Island, and offers to reveal its location to the first ten subscribers for $10 and a self-addressed envelope.
Even at the start, the pattern of the series is visible. The voice is a deadpan lockdown persona joking at a world suddenly at home. The week’s news (treatments, sports, the language of the disease, prisons, distancing) serves as occasion for opinion. The jokes carry the commentary, and the commentary turns blunt where the jokes stop.
Subject terms: COVID-19 pandemic; Social commentary; Satire; Humor; Newsletters; Lockdown; Social distancing; Hydroxychloroquine; Tokyo Olympics; Disease names; Prisons; Restaurants; Florida
Extracted keywords: Hermit Herald; monastic existence; ennui; innocuous drivel; Covid-19; Hydroxychloroquine; Malaria; Olympics; Tokyo; IOC; corona virus; Chinese Virus; Spanish virus; Lyme Disease; incarceration; Nordic countries; Sweden; Social Distancing; Peanut Island; Kennedy; Florida; restaurant; subscribers; Piero Bundini
Imputed keywords: early pandemic; isolation; boredom; self-mockery; mock pomposity; pen name; family and friends; chain newsletter; first issue; stigma; incarceration and confinement; public health; sports; dining out; deadpan; sardonic humor
Rights and access: Open access; public on covid-19archive.org. CC BY-SA 3.0; see covid-19archive.org.
Ethics and bias note: A satirical community newsletter; everything in it is opinion. The item headed “Corona virus” defends the name “Chinese Virus” and ends “thanks for your disease.” These are the author’s views, and the term was publicly contested as stigmatizing when the issue was written.
Rationale: The description gives the opening conceit and the five items nearly in full. The voice, mock-grand and self-deprecating, is part of what the object is, and the items show how the series turns the week’s news into opinion. The early date matters: the issue was written in the first days of the lockdowns, with the Olympics still scheduled and hydroxychloroquine still “promising.” The “Chinese Virus” item is described plainly, because researchers on the language of the pandemic will look for it. Where it is quoted, the quotation is the author’s. The pen name is kept as the signature. Bundy is named as creator because the collection documentation attributes the pen name to him.
Evidence note: The description is drawn from the text and page images of the four-page PDF. The masthead gives the date, March 19, 2020, and the volume and issue number. This is established. The signature “Piero Bundini” appears on page 4. The attribution of that pen name to Peter Bundy comes from the collection’s documentation, not from the issue, which carries only the pen name. This is established. That Bundy wrote from Florida is well supported by the collection documentation and by the issue’s reference to a restaurant in “nearby Florida.” The Palm Beach area is proposed, from the mention of Peanut Island. The email-era provenance and Erin Craft’s making of the PDF come from the collection’s documentation. This is well supported. The rights statement is the collection-wide Creative Commons license.
Read it the way we now read all of them: not asking whether it is correct, but whether you would choose it over another. And if you find yourself wanting to circle something we didn’t — that instinct is the method. What would you circle?
Sources
Society of American Archivists. Describing Archives: A Content Standard, “Statement of Principles,” 2019 revision.
Thurstone, L. L. “A Law of Comparative Judgment.” Psychological Review 34, no. 4 (1927): 273–286.
The Hermit Herald collection. A Journal of the Plague Year: An Archive of COVID-19. Arizona State University. covid-19archive.org.