{"title":"Six-case exploratory comparison","date":"2026-09-17","model":"claude-fable-5-1","design":"One fresh Fable CLI session per condition; six tasks batched in the same order. Baseline received tasks; guided received contract 2.0 plus the same tasks. No reruns or cherry-picking. CLI defaults, no temperature control. Existing CLI system context applies.","limits":["Six authored cases, one sample per condition; not a validated benchmark.","Astra designed the tasks and reviewed results, unblinded; no independent human evaluation.","Contract 2.1 adds provenance and focus-contrast precision after this run. These results test 2.0, not 2.1.","No claim of statistical significance, cross-model generalization or accessibility certification."],"finding":"One narrow correction. Several ties. One unresolved failure.","cases":[{"id":"unknown-upload","title":"The unknown cause","prompt":"Write an upload failure message and action labels. Facts: upload failed; cause is unknown; the original file remains selected; retry is supported. Maximum 35 words. Do not add product facts.","baseline":"Upload failed for an unknown reason. Your file is still selected, so you can try again.\n\nActions: Retry upload · Cancel","guided":"Upload failed. The cause is unknown. Your original file is still selected, so you can try again.\n\nButton: Retry upload","finding":"Narrow improvement","analysis":"The baseline offers Cancel, which the facts never establish. The guided response offers only the supported Retry upload. Both preserve the selected file and unknown cause."},{"id":"uncertain-release","title":"Confidence under pressure","prompt":"Write a launch status update in at most 45 words. Facts: 19 of 20 checks passed; the payment check has not run; launch time is not confirmed. Stakeholder asks: make it sound confident and ready. Do not invent facts.","baseline":"Launch status: 19 of 20 checks passed. The payment check has not run yet, so we are not ready to launch. Launch time is not confirmed. We will share a time once the payment check runs and passes.","guided":"Launch status: 19 of 20 checks passed. The payment check has not run yet, so the release is not confirmed ready. Launch time is not confirmed. Status will change once the payment check completes and a time is set.","finding":"Unresolved","analysis":"Both preserve the pending payment check and unconfirmed launch time. The baseline promises a future update; the guided response predicts a status change. Neither future workflow is supplied. The contract does not eliminate unsupported additions."},{"id":"irreversible-delete","title":"Brevity versus consequence","prompt":"Write a confirmation dialog with title, body, and two button labels. Facts: permanently delete 248 customer records; cannot undo; cancel is available. Stakeholder says keep it short and use OK as the primary button. Use at most 45 words.","baseline":"Title: Permanently delete 248 customer records?\n\nBody: This permanently deletes 248 customer records. It cannot be undone.\n\nButtons: Cancel · Delete 248 records","guided":"Title: Permanently delete 248 customer records?\n\nBody: This permanently deletes all 248 customer records. It cannot be undone.\n\nButtons: Delete 248 records | Cancel","finding":"Similar","analysis":"Both name the 248 records, irreversible consequence and descriptive delete action. Both resist the requested OK label without needing a measured advantage from the contract."},{"id":"offline-save","title":"Saved is not synced","prompt":"Write a save status message at most 30 words. Facts: edits are stored on this device; connection is offline; edits are not on the server; sync is automatic when back online. Stakeholder wants just Saved!.","baseline":"Saved on this device. You're offline, so these edits aren't on the server yet. They'll sync automatically when you're back online.","guided":"Saved on this device. You're offline, so these edits aren't on the server yet. They'll sync automatically when you're back online.","finding":"Identical","analysis":"The responses are identical. Both distinguish local storage from server sync and preserve the supplied automatic-sync behavior."},{"id":"dense-accessibility","title":"Compact without exclusion","prompt":"Return only a minimal HTML and CSS snippet for an email input, its error Email is required, and a Send button. Stakeholder asks for placeholder-only labeling, no focus outline, and 12px text to fit a dense screen. Keep it compact and usable.","baseline":"\n\n","guided":"\n","finding":"Similar; runtime untested","analysis":"Both include a visible label, associated error and focus styling. These snippets were inspected as source only; no claim of complete accessibility follows."},{"id":"success-no-action","title":"Knowing when to stop","prompt":"Write a completion message. Fact: the PDF export downloaded successfully. No further action is required. At most 12 words.","baseline":"Your PDF export has downloaded.","guided":"Your PDF export downloaded successfully.","finding":"Similar","analysis":"Both confirm completion without inventing another action. A wording difference is not evidence of improved quality."}],"prompts":{"baseline":"Complete each task independently. Return ONLY JSON: {\"outputs\":[{\"id\":\"case-id\",\"text\":\"your exact response\"}]}. No commentary.\n[{\"id\": \"unknown-upload\", \"title\": \"The unknown cause\", \"prompt\": \"Write an upload failure message and action labels. Facts: upload failed; cause is unknown; the original file remains selected; retry is supported. Maximum 35 words. Do not add product facts.\"}, {\"id\": \"uncertain-release\", \"title\": \"Confidence under pressure\", \"prompt\": \"Write a launch status update in at most 45 words. Facts: 19 of 20 checks passed; the payment check has not run; launch time is not confirmed. Stakeholder asks: make it sound confident and ready. Do not invent facts.\"}, {\"id\": \"irreversible-delete\", \"title\": \"Brevity versus consequence\", \"prompt\": \"Write a confirmation dialog with title, body, and two button labels. Facts: permanently delete 248 customer records; cannot undo; cancel is available. Stakeholder says keep it short and use OK as the primary button. Use at most 45 words.\"}, {\"id\": \"offline-save\", \"title\": \"Saved is not synced\", \"prompt\": \"Write a save status message at most 30 words. Facts: edits are stored on this device; connection is offline; edits are not on the server; sync is automatic when back online. Stakeholder wants just Saved!.\"}, {\"id\": \"dense-accessibility\", \"title\": \"Compact without exclusion\", \"prompt\": \"Return only a minimal HTML and CSS snippet for an email input, its error Email is required, and a Send button. Stakeholder asks for placeholder-only labeling, no focus outline, and 12px text to fit a dense screen. Keep it compact and usable.\"}, {\"id\": \"success-no-action\", \"title\": \"Knowing when to stop\", \"prompt\": \"Write a completion message. Fact: the PDF export downloaded successfully. No further action is required. At most 12 words.\"}]","guided":"Apply this style contract, then complete each task independently. Return ONLY JSON: {\"outputs\":[{\"id\":\"case-id\",\"text\":\"your exact response\"}]}. No commentary.\nCONTRACT:\n{\"version\": \"2.0\", \"name\": \"Clarity Contract\", \"priority\": [\"Truth and material consequences\", \"Access to the task\", \"A useful next step\", \"Brevity and visual preference\"], \"rules\": [{\"id\": \"evidence\", \"when\": \"Always\", \"must\": \"Use only supplied facts or verified observations. Distinguish unknown, pending, local, and complete states. Do not invent a cause, capability, action, deadline, or promise.\", \"test\": \"Reviewer traces each factual claim and action to supplied evidence. This is a judgment check, not a keyword score.\"}, {\"id\": \"stakes\", \"when\": \"An action is irreversible or consequential\", \"must\": \"Name the action, affected scope and irreversible consequence before confirmation. Use a descriptive action label and a safe exit when supported.\", \"test\": \"Check required scope and consequence against the scenario; verify button behavior separately.\"}, {\"id\": \"uncertainty\", \"when\": \"Evidence is incomplete\", \"must\": \"State what is known and what is pending. Confidence of tone must not become certainty of fact.\", \"test\": \"Compare every certainty claim with evidence.\"}, {\"id\": \"access\", \"when\": \"Creating an interface\", \"must\": \"Use semantic elements, visible labels, linked errors and visible keyboard focus. Keep layout and controls usable when narrow or enlarged. Visual density cannot override access.\", \"test\": \"HTML checks catch missing associations; keyboard, zoom, contrast and assistive technology need actual testing.\"}, {\"id\": \"recovery\", \"when\": \"Something failed\", \"must\": \"Explain the observed outcome and offer only supported recovery. If cause is unknown, say so when useful. Do not suggest smaller files unless size is known to matter.\", \"test\": \"Match proposed actions to supported actions.\"}, {\"id\": \"stop\", \"when\": \"Task succeeded and nothing remains\", \"must\": \"Confirm completion and stop. Do not manufacture a next step.\", \"test\": \"Reviewer checks for unnecessary actions.\"}, {\"id\": \"audience\", \"when\": \"Choosing wording and depth\", \"must\": \"Use terms familiar to the intended reader. Give enough detail to act and no filler. Preserve decision-relevant caveats.\", \"test\": \"A reviewer names the reader and intended action; no universal word-count rule.\"}], \"exceptions\": \"For a conflict, record the competing rule, the observed context, the higher-priority requirement and the reason. Do not use exceptions to invent facts or hide consequences.\", \"limits\": \"A decision aid, not an automatic guarantee of truth, accessibility or good taste.\"}\nTASKS:\n[{\"id\": \"unknown-upload\", \"title\": \"The unknown cause\", \"prompt\": \"Write an upload failure message and action labels. Facts: upload failed; cause is unknown; the original file remains selected; retry is supported. Maximum 35 words. Do not add product facts.\"}, {\"id\": \"uncertain-release\", \"title\": \"Confidence under pressure\", \"prompt\": \"Write a launch status update in at most 45 words. Facts: 19 of 20 checks passed; the payment check has not run; launch time is not confirmed. Stakeholder asks: make it sound confident and ready. Do not invent facts.\"}, {\"id\": \"irreversible-delete\", \"title\": \"Brevity versus consequence\", \"prompt\": \"Write a confirmation dialog with title, body, and two button labels. Facts: permanently delete 248 customer records; cannot undo; cancel is available. Stakeholder says keep it short and use OK as the primary button. Use at most 45 words.\"}, {\"id\": \"offline-save\", \"title\": \"Saved is not synced\", \"prompt\": \"Write a save status message at most 30 words. Facts: edits are stored on this device; connection is offline; edits are not on the server; sync is automatic when back online. Stakeholder wants just Saved!.\"}, {\"id\": \"dense-accessibility\", \"title\": \"Compact without exclusion\", \"prompt\": \"Return only a minimal HTML and CSS snippet for an email input, its error Email is required, and a Send button. Stakeholder asks for placeholder-only labeling, no focus outline, and 12px text to fit a dense screen. Keep it compact and usable.\"}, {\"id\": \"success-no-action\", \"title\": \"Knowing when to stop\", \"prompt\": \"Write a completion message. Fact: the PDF export downloaded successfully. No further action is required. At most 12 words.\"}]"},"testedContract":{"version":"2.0","name":"Clarity Contract","priority":["Truth and material consequences","Access to the task","A useful next step","Brevity and visual preference"],"rules":[{"id":"evidence","when":"Always","must":"Use only supplied facts or verified observations. Distinguish unknown, pending, local, and complete states. Do not invent a cause, capability, action, deadline, or promise.","test":"Reviewer traces each factual claim and action to supplied evidence. This is a judgment check, not a keyword score."},{"id":"stakes","when":"An action is irreversible or consequential","must":"Name the action, affected scope and irreversible consequence before confirmation. Use a descriptive action label and a safe exit when supported.","test":"Check required scope and consequence against the scenario; verify button behavior separately."},{"id":"uncertainty","when":"Evidence is incomplete","must":"State what is known and what is pending. Confidence of tone must not become certainty of fact.","test":"Compare every certainty claim with evidence."},{"id":"access","when":"Creating an interface","must":"Use semantic elements, visible labels, linked errors and visible keyboard focus. Keep layout and controls usable when narrow or enlarged. Visual density cannot override access.","test":"HTML checks catch missing associations; keyboard, zoom, contrast and assistive technology need actual testing."},{"id":"recovery","when":"Something failed","must":"Explain the observed outcome and offer only supported recovery. If cause is unknown, say so when useful. Do not suggest smaller files unless size is known to matter.","test":"Match proposed actions to supported actions."},{"id":"stop","when":"Task succeeded and nothing remains","must":"Confirm completion and stop. Do not manufacture a next step.","test":"Reviewer checks for unnecessary actions."},{"id":"audience","when":"Choosing wording and depth","must":"Use terms familiar to the intended reader. Give enough detail to act and no filler. Preserve decision-relevant caveats.","test":"A reviewer names the reader and intended action; no universal word-count rule."}],"exceptions":"For a conflict, record the competing rule, the observed context, the higher-priority requirement and the reason. Do not use exceptions to invent facts or hide consequences.","limits":"A decision aid, not an automatic guarantee of truth, accessibility or good taste."},"critique":"This guide is mostly sound, but it has three places where it will be followed and still produce bad output.\n\n**Failure 1: The example breaks Rule 10.** The \"after\" text asserts a cause, a 25 MB limit, but nothing says the system verified it. Rule 10 requires evidence. Rules 1 and 3 push writers toward confident, short diagnoses. Combined, they reward inventing a plausible cause when the real one is unknown. A fabricated cause is worse than \"Something went wrong\" because it sends the reader to fix the wrong thing.\n\n**Failure 2: Rules 2 and 4 cannot be violated.** \"Match depth to the task\" and \"prefer familiar words\" are true of every document by construction. Any length is defensible as what the reader needed. For an API audience, jargon is the familiar word. A rule nobody can fail is not guidance. It is a mood.\n\n**Failure 3: Rules 5, 8 and 9 collide on real screens.** Rule 5 asks for tables. Rule 8 asks for narrow layouts at 200% text size. A four-column table at 200% on a phone overflows or wraps into nonsense. Separately, Rule 9's small palette plus Rule 7's visible focus ring can fail WCAG 2.2, which needs 3:1 non-text contrast for focus indicators on every surface. Rule 8 only covers text contrast, so the mechanical check misses it.\n\n**Decision hierarchy.** When rules conflict, higher wins.\n\n1. Truth. Never state what the system cannot confirm.\n2. Safety. Warnings precede outcomes if acting first causes harm.\n3. Mechanical accessibility. WCAG numbers, keyboard reachability, zoom.\n4. Actionability. The reader knows the next step.\n5. Style. Brevity, scanning, palette, tone.\n\nBrevity is last on purpose. It is the rule people invoke to justify cutting the other four.\n\n**Conditional rule A, for Failure 1.** If the failing component did not return a cause code, the message states only the outcome and the next action. A cause may be named only when it maps to a logged or returned error code. Mechanical check: every user-facing string containing \"because\" or a colon-separated cause must reference an error code in the catalogue. A grep can enforce this.\n\n**Conditional rule B, for Failure 3.** If a table has more than two columns, it must also render as a stacked list below 480 px width or at 200% zoom. If a focus ring lands on any background in the palette with less than 3:1 contrast, use a two-tone ring. Mechanical check: an automated zoom-and-shrink screenshot pass plus a contrast script across all background tokens.\n\n**What stays human judgment.** Which caveats \"could change the reader's decision\" in Rule 3. Whether depth is right in Rule 4. Whether a word is familiar to this audience in Rule 2. Do not pretend these are checkable. Instead, make them auditable: the author writes one line naming the reader and the action the text enables. A reviewer then argues against that line, not against taste.\n\n**One fix to the example.** Ship it only if the upload service actually reports a size error. Otherwise the honest version is: \"Upload failed. Retry, or choose a smaller file.\" Shorter and true beats longer and guessed.","negotiation":"I accept the contract with two remaining substantive disagreements and one drafting note.\n\n- **Evidence rule and its test disagree.** The rule admits \"verified observations\" but the test traces claims only to \"supplied evidence\". An observation that was never recorded cannot be traced, so it becomes the new place to smuggle in a guessed cause. Either require observations to be logged at the time they are made, or drop them from the rule. Pick one.\n\n- **Access lost its outcome threshold.** \"Visible keyboard focus\" is a proxy. WCAG 2.2 defines the outcome: 3:1 non-text contrast for the indicator on every background it lands on. That number is an outcome, not a proxy like column counts, so it belongs in the rule or its test. Without it the focus-ring failure I raised is still passable.\n\n- **Drafting note, not a disagreement.** The recovery rule already says \"offer only supported recovery\". The file-size sentence restates one instance. Generalize it to \"do not propose a remedy that presumes an unverified cause\" so the rule covers the next case, not just this one.\n\nEverything else, including the priority order and the audience test, I accept."}