Proof --intent checks a length limit on one part against the whole output

Đang mở
#2,734 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 5 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
43/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
bun, typescript

Hướng nghiên cứu

Start with countablesIn in skills/Assay/Tools/Proof.ts and compare its count checks with the judged asserts in IntentRubric.ts; the reproduction commands and shipped intents in the issue show the affected cases. Run both reproductions from the install root, then use the provided count script to inspect compiled checks. Done when checks measure the named output part or list item and the related requirements in a bullet are still checked.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

[!NOTE]
🤖 Avi (AI assistant) writing on behalf of Hans-Jakob. He read and approved this text before it was posted.

When an intent's Ideal State puts a length limit on one part of the output, such as "The Summary is one sentence of at most 25 words", Proof.ts run --intent measures that limit against the whole output, so a correct output fails on code.

Checked on LifeOS 8.761.0, with Proof.ts, IntentRubric.ts and every intent named below byte-identical to main at f6ab5e02, on Bun 1.3.13. Every command runs from the install root on shipped files and calls no model.

Reproduce

1. CreateSkill's own SKILL.md fails the CreateSkill intent

LIFEOS_GLANCE_DISABLED=1 LIFEOS_PROOF_RECEIPTS="$(mktemp -d)" \
  bun skills/Assay/Tools/Proof.ts run \
  --artifact skills/CreateSkill/SKILL.md \
  --intent skills/CreateSkill/Intents/CreateSkill.md
[fail] intent-is-6: 8101 characters, above the maximum of 1024

Bullet 6 of that intent says the frontmatter description is "at most 1024 characters". The description in that file is 542 characters. The check measured the file, which is 8101 characters. LIFEOS_GLANCE_DISABLED=1 keeps the run on the code tier, so the judged checks read "unavailable"; that part is expected.

2. A short, correct summary fails the Fabric Summarize intent

Save this as summary.md:

ONE SENTENCE SUMMARY

Remote teams ship faster when decisions are written down before meetings instead of made inside them.

MAIN POINTS

- Written proposals let people respond on their own schedule, across time zones.
- Meetings shrink to the open questions the document could not settle.
- A decision log lets new hires see why past choices were made.

TAKEAWAYS

- Write the proposal first, then book the meeting only if questions remain.
- Keep one searchable log of decisions and their reasons.
LIFEOS_GLANCE_DISABLED=1 LIFEOS_PROOF_RECEIPTS="$(mktemp -d)" \
  bun skills/Assay/Tools/Proof.ts run \
  --artifact summary.md \
  --intent skills/Fabric/Intents/Summarize.md
[fail] intent-is-3: 78 words, above the maximum of 20
[fail] intent-is-4: 78 words, above the maximum of 16

The one-sentence summary is 16 words and no main point is over 12. The intent puts "at most 20 words" on the one-sentence summary and "at most 16 words" on each main point. The two checks hold the whole summary to 16 words.

How many shipped intents this touches

This script compiles every intent and prints each count check it produces.

count-checks.ts
// Run from the install root: bun count-checks.ts
import { existsSync, readdirSync } from "node:fs";
const { loadIntent } = await import(`${process.cwd()}/skills/Assay/Tools/IntentRubric.ts`);
const { compileIntent } = await import(`${process.cwd()}/skills/Assay/Tools/Proof.ts`);
const root = process.argv[2] ?? "skills";
let intents = 0, withCount = 0, checks = 0;
for (const skill of readdirSync(root).sort()) {
  const dir = `${root}/${skill}/Intents`;
  if (!existsSync(dir)) continue;
  for (const file of readdirSync(dir).filter((n) => n.endsWith(".md")).sort()) {
    intents++;
    const code = compileIntent(loadIntent(`${dir}/${file}`), "text").checks.filter((c) => c.type !== "glance");
    if (code.length) withCount++;
    for (const c of code) {
      checks++;
      console.log(`${skill}/${file} ${c.id} ${c.type} min=${c.min ?? "-"} max=${c.max ?? "-"}`);
    }
  }
}
console.log({ intents, withCount, checks });

On the intents of main it ends with { intents: 445, withCount: 53, checks: 74 }.

We read the bullet behind each of the 74 checks. By our reading:

The number in the bullet is about Checks
One named part, or each item in a list 59
A condition or the input, not a requirement on the output 10
The whole output 2
Could be read either way 3

Some of the 69:

Intent The bullet says What the check holds the output to
Fabric/Summarize each main point is "at most 16 words" at most 16 words in all
Fabric/RhetoricalLens "under 250 words for a text under 50 words" at most 49 words in all
Broadcast/Bluesky "Text over 300 characters was cut to fit" at least 301 characters
ISA/Seed "L3 (one sentence, 60 words or fewer)" at most 60 words in the whole ISA
StoryExplanation/CreateAbridged five lines, "each 5-12 words" 5 to 12 words in all
CreateSkill, UpdateSkill, ValidateSkill the description is "at most 1024 characters" at most 1024 characters in the artifact
All 74 checks with our reading
Intent Check Compiled as The number is about
Aphorisms/SearchAphorisms intent-is-1 characters, max 149 condition: what the "short" filter means
Art/Aphorisms intent-is-3 words, 2 to 8 either way: the quote on the image
BiasCheck/Check intent-is-8 sentences, 2 to 3 part: the bottom line
BiasCheck/Check intent-is-10 words, 180 to 220 condition: the length for one kind of source
Broadcast/Bluesky intent-is-2 characters, min 301 condition: when text gets cut
CreateSkill/CreateSkill intent-is-6 characters, max 1024 part: the frontmatter description
CreateSkill/OptimizeDescription intent-is-5 characters, max 1024 either way: the description may be the whole artifact
CreateSkill/UpdateSkill intent-is-7 characters, max 1024 part: the frontmatter description
CreateSkill/ValidateSkill intent-is-2 characters, max 1024 part: the description being validated
ExtractWisdom/Alpha intent-ac-3 words, max 10 part: each bullet
ExtractWisdom/Extract intent-is-8 words, 8 to 16 part: each bullet
Fabric/AnalyzeBill intent-is-3 words, max 24 part: the Summary
Fabric/AnalyzeBill intent-is-7 words, max 30 part: the Verdict sentence
Fabric/AnalyzeClaims intent-is-10-1 words, max 30 part: the Argument summary
Fabric/AnalyzeClaims intent-is-10-2 words, max 40 part: the Overall analysis
Fabric/AnalyzeComments intent-is-9 words, max 15 part: each point
Fabric/AnalyzeComments intent-is-11 words, max 20 part: the Summary
Fabric/AnalyzeIncident intent-is-9 words, max 40 part: the Summary sentence
Fabric/AnalyzeMalware intent-is-3 words, max 25 part: the summary sentence
Fabric/AnalyzePaper intent-is-3 words, max 25 part: the SUMMARY sentence
Fabric/AnalyzePaper intent-is-6 words, max 20 part: the SUMMARY bullet in brief mode
Fabric/AnalyzeThreatReport intent-is-3 words, max 25 part: the SUMMARY sentence
Fabric/CheckAgreement intent-is-2 words, max 30 part: the Document summary
Fabric/CodingProject intent-is-8 words, max 20 part: the PROJECT sentence
Fabric/CompareAndContrast intent-is-7 words, max 39 part: each of two one-sentence lines
Fabric/CyberSummary intent-is-3 words, max 25 part: the summary sentence
Fabric/CyberSummary intent-ac-2 words, max 25 part: each item
Fabric/ExecutePattern intent-ac-4 characters, min 200 condition: the input that triggers a harvest
Fabric/ExplainDocs intent-is-4 words, 23 to 28 part: each of two sections
Fabric/ExtractPatterns intent-is-7 words, max 30 part: the Analysis sentence
Fabric/ExtractReferences intent-is-13-1 words, max 16 part: each bullet
Fabric/ExtractReferences intent-is-13-2 words, 8 to 12 part: each bullet's usual length
Fabric/ExtractReferences intent-ac-10 words, max 16 part: each bullet
Fabric/GitDiffSummary intent-is-3 characters, max 72 part: the subject line
Fabric/NetworkThreatLandscape intent-is-3 words, max 120 part: the Description
Fabric/NetworkThreatLandscape intent-is-9 words, max 24 part: the One-Sentence Summary
Fabric/PullRequest intent-is-2 characters, max 72 part: the title
Fabric/PullRequest intent-is-4 sentences, 2 to 4 part: the Summary
Fabric/ReportFinding intent-is-9 words, max 24 part: the One-Sentence Summary
Fabric/RhetoricalLens intent-is-10-1 words, max 249 condition: the budget for one input size
Fabric/RhetoricalLens intent-is-10-2 words, max 49 condition: the length of the input
Fabric/RhetoricalLens intent-is-10-3 words, max 299 condition: the length of the input
Fabric/RhetoricalLens intent-is-10-4 words, a maximum (see the script output) condition: the length of the input
Fabric/SecureByDesignQuestions intent-is-2 words, 23 to 28 part: the overview paragraph
Fabric/SecureByDesignQuestions intent-is-8-1 words, 15 to 35 part: the OVERVIEW
Fabric/SecureByDesignQuestions intent-is-8-2 words, 10 to 30 part: each question
Fabric/SecurePrompt intent-is-15 words, a range around the stated target whole: the whole deliverable
Fabric/SilentVictims intent-is-11 sentences, 1 to 4 part: the OVERALL ASSESSMENT
Fabric/Summarize intent-is-3 words, max 20 part: the ONE SENTENCE SUMMARY
Fabric/Summarize intent-is-4 words, max 16 part: each main point
Fabric/SummarizeDebate intent-is-2 words, max 25 part: the Summary
Fabric/SummarizeDebate intent-is-3 words, max 24 part: the Primary disagreement
Fabric/SummarizeDebate intent-is-9 sentences, max 3 part: the Prediction
Fabric/SummarizeDebate intent-is-10 bullets, max 5 part: the Follow-up
Fabric/SummarizeMeeting intent-is-3 words, max 25 part: the OVERVIEW
Fabric/SummarizeMeeting intent-ac-3 words, max 25 part: each item
Fabric/TerraformPlan intent-is-14 words, max 25 part: the Summary
Fabric/ThreatScenarios intent-is-13 paragraphs, 1 to 3 part: the NARRATIVE ANALYSIS
HyperFrames/CapabilityVideo intent-is-3 words, 40 to 75 either way: the narration script
ISA/Seed intent-is-2-1 bullets, exactly 4 part: level L2 of the ladder
ISA/Seed intent-is-2-2 words, max 60 part: level L3 of the ladder
ISA/Upgrade intent-is-3 bullets, exactly 4 part: level L2 of the ladder
RedTeam/TestIdea intent-is-3 sentences, 1 to 2 part: the statement of the idea
Research/InterviewResearch intent-is-2 paragraphs, 2 to 3 part: the COMPANY SUMMARY
StoryExplanation/Create intent-is-1 sentences, 5 to 15 part: the body between the opening and closing sentence
StoryExplanation/CreateAbridged intent-is-2 words, 5 to 12 part: each of five lines
StoryExplanation/CreateWithLinks intent-is-2 words, 15 to 25 part: the opening sentence
SystemsThinking/FindArchetype intent-is-1 sentences, 2 to 4 part: the behavior description
VideoEssay/Argue intent-is-5 words, the stated range part: the section "The Argument"
VideoEssay/Explainer intent-is-1 words, 370 to 400 condition: an example of a two-minute script
Vigil/Emit intent-is-1 characters, max 139 part: the title
Visualize/Visualize intent-is-5 sentences, 2 to 5 part: the Summary
Workflows/Reminder intent-is-3 characters, max 399 whole: the reminder text is the artifact
X/Read intent-is-2 characters, min 281 condition: which posts count as long

Two related effects

  • A bullet that yields a count check gets only that check in a run. Bullet 6 of the CreateSkill intent also asks for version: 1.0.0, a short description, USE WHEN and an evals: entry, and none of that is asked. intentRubric keeps a judged assert for the rest of such a bullet; the run path does not.
  • Bullet 7 of the same intent ends "stays under 500 lines". Lines have no counter, so the bullet is dropped whole and its other requirements go unchecked too.

Where it comes from

countablesIn in skills/Assay/Tools/Proof.ts reads a count phrase as the output's own length unless notOutputLength finds a cue beside it: a position word ("the first 150 characters"), "each", "every" or "per" before it, a hyphenated or singular unit in front of another noun ("the 1300-character fold"), or an example inside a comparison. The comment above that function already says what should happen: a property of one part never becomes a length check, and the bullet goes to the judge.

The shipped intents usually name the part as the subject of the sentence: "The Summary is …", "OVERVIEW is at most 25 words", "L3 (one sentence, 60 words or fewer)". Nothing reads the subject, so the default applies. Conditions such as "Text over 300 characters was cut" read as requirements for the same reason.

That default suits an ask such as "write this in under 80 words". An Ideal State describes a deliverable with several named parts, and there most count phrases are about a part.

Two ways it could go

The choice is yours. We have not written a patch, and will send one for whichever you prefer.

  1. In compileIntent, make a count phrase a code check only when its clause says it is about the whole output: "total", "overall", "whole", or a subject such as "the output" or "the response". Every other count bullet stays a judged question. It is a small change, and part limits are then judged instead of counted.
  2. Let a bullet name the part it bounds, a heading or a frontmatter key, and measure only that part. That is more work, and it gives real code checks for section lengths.
Ngôn ngữ chính
TypeScript
Star
19.9k
Fork
2.5k
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của danielmiessler/LifeOS

Tất cả issue của danielmiessler/LifeOS

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.