---
title: "August 2026 notes on stubs"
content_id: "garden/draft/2026-08-stubs"
date: "2026-08-03"
slug: "2026-08-stubs"
summary: "Longer focus posts I want to write or find"
original_url: "https://dataleverageshortposts.leaflet.pub/3ms72ezdgnysp"
captured_from: "https://nickmvincent.github.io/short-focus-posts/2026-08-stubs.html"
source_revision: "cbe747fa1c1f30dd85d9cf552f4db279c7be2d7d"
source_sha256: "a3c39ebca9b77101cd72517c24ccfb0b1701cb6a28c39c6a458557744ff8adb3"
status: "draft"
visibility: "public"
---

These are some one-liners that I want to expand into ~600-word focus posts to link to from other posts (or just find a good canonical reference that makes the point).

- eval data is what you eval on...
	- Generally, there is no special token within a document, property within a JSON object, or column within a tabular dataset that makes a certain data record "eval data."
	- Amongst the universe of all data objects, some are more useful for performing "evaluation operations".
	- In traditional supervised ML, eval data is just holdout data.
	- In the modern LLM world, eval data stands in stark contrast to pretraining data because it has lots of explicit structure. The contrast between eval and post-training data is less stark, making dual use easier.
- there's always a model in prod...
	- all models are wrong, some models are useful, and the powers that be, by definition, always have a "model" running in prod (from a systems/control perspective -- there's always something making decisions, even if the decision is to go with the default, baseline, or dummy option)
- scaling laws for evals...
- pretraining detente and clearer public/private split
- centrist take on distillation
- we should freak out about Terms of Service from organizations trying to build "AGI"
- it's a worthy target for advocacy to just get key actors in the data debates to state earnestly the IP regime and data rules they'd like to see