Discussion about this post

User's avatar
Flows and Textures TLC's avatar

The invocation of John 15:2 is a kind of confession that 1) salvation lies in surrender to the way ecologies (say, vine/fruit systems) work, and in which we invest ourselves productively by faith, and not by works; and therefore 2) ONLY ecological complex adaptive systems methods will be able to anticipate alignment; but therefore 3) Bayesian predictions for alignment may or may not succeed because our societies, segmented and 17-c mechanistically constituted (focused on, say, private property and free will as steam-engine-like forces it does not harm us not to understand) as we are, decline to operate using such methods; and 4) how willing would you be to hand over your children’s care to even a modestly inscrutable caretaker? so 5) the references to how we practice, learn, remember, and use neurons feel vulnerable to the John 15:2 reference’s wisdom.

Let’s prune carefully, allowing plenty of feedback and guardrails from an ecosystem of users, because there’s much more at stake than grapes.

Robin Reese's avatar

Dean, your Brakhage opening captures both the promise and the hazard of Cammarata’s reframe. An eye freed from familiar categories may discover structure we’ve missed, or project a more elegant story. His circuit and steering evidence is promising; the test is whether those explanations reliably predict new cases at scale.

Even if they do, understanding a network, predicting its behavior, controlling it, and judging the wisdom of its deployment are four distinct achievements. Deception may emerge from interacting capacities rather than a discrete “cheating part” to switch off. Structure isn’t goodness.

Then there’s the American business model. You write that “we” have perfect information about a network’s internals. Which we? Cammarata’s language-model demonstrations rely substantially on Qwen3-32B, an open-weight model. Independent researchers have no equivalent access to the internals of proprietary American frontier models.

You’ve championed open-weight research and identify concentration of power as your chief concern. Yet “iterative deployment” is both a way of learning and a commercial strategy. If interpretability removes a barrier to acceleration without expanding independent scrutiny, might it actually deepen the concentration of power you fear?

If networks are already clean, who gets to certify that judgment—and who gets to check?

Ready for more?