AI suggestions describe punctuated Chinese, Japanese and Hindi prose as "commands, code or queries" and give it a 32-token budget

Open Beginner friendly
#726 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
macos, swift
Domain
ai, testing

Research direction

Start in Sources/UttrflowPredict/Register.swift around lines 222–240, then read RegisterTests.swift and RegisterPropertyTests.swift. Run the supplied reproduction or the focused tests first, and use make verify with the documented DEVELOPER_DIR. Done means Chinese, Japanese, and Hindi prose receive prose classification and sentence hints, while English and shell behavior remains covered by the existing tests.

Written by the indexing model from the issue text.

Description

area:ai-suggestions bug good first issue P2

What happens

Register.infer describes where a line is being written, and the description goes straight into the suggestion model's prompt (register.hints, Sources/UttrflowLocalModel/PromptBuilder.swift:74) and sets its token budget (register.maxTokens, AppleCandidateGenerator.swift:35, MLXCandidateScorer.swift:316). Two of its measures assume English text:

  1. Symbol share (Sources/UttrflowPredict/Register.swift:223-231) counts every visible character that is not a letter or digit as a symbol, and above 0.10 (:33) the text is declared "commands, code or queries rather than prose" (:80, :130-133) with a 32-token budget (:111). Chinese and Japanese pack a word into one or two characters, so ordinary sentence punctuation (, , ) is a much larger share of the characters; Hindi's is counted the same way.
  2. Sentence case (:234-240) requires the first character to be upper-case and the last to be one of .!?. Scripts without case never pass, and , , and are not accepted, so the person is described as writing "casually, without sentence punctuation" (:137).

Measured with a throwaway test on three short, fully punctuated lines per language:

Language symbolShare Expected kind / hint Actual kind / hint
English 0.068 line; "writes in full sentences" line; "writes in full sentences"
Chinese 0.143 line "command, query or line of code"; "commands, code or queries rather than prose"
Japanese 0.103 line same as Chinese
Hindi 0.111 line same as Chinese

(Once symbol share is fixed, the sentence-case hint would read "writes casually, without sentence punctuation" for all three, because sentenceCaseShare is 0.0.)

Why it matters

For anyone typing prose in these languages the model is told the field holds code, is given 32 tokens, and, after that is fixed, is told the writer does not punctuate. Suggestions come out shaped like commands or unpunctuated, or are cut short.

How to reproduce

In Tests/UttrflowPredictTests/RegisterTests.swift:

let own = ["我们明天下午三点开会,请准时参加。", "谢谢,到时候见。", "你能把笔记发给我吗?"]
#expect(Register.symbolShare(of: own) <= Register.symbolicShare)   // 0.143 today
#expect(Register.sentenceCaseShare(of: own) >= 0.5)               // 0.0 today

Acceptance criteria

  • Sentence punctuation (Unicode general category punctuation used to end or separate clauses, including ,。?!、।) does not count as a symbol, or the share is weighed per word rather than per character; shell and code lines keep their current classification (the existing RegisterPropertyTests still pass).
  • A line in an uncased script counts as a sentence when it ends in sentence punctuation of any script; cased scripts keep the capital check.
  • Tests cover English, Chinese, Japanese and Hindi lines, plus a shell line.

Where to start

  • Sources/UttrflowPredict/Register.swift:222-240.
  • Tests: Tests/UttrflowPredictTests/RegisterTests.swift, RegisterPropertyTests.swift.
  • Run make verify (export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer first) and read CONTRIBUTING.md.

Size: S, an hour or two.

Dominant language
Swift
Stars
4
Forks
17
Avg merge
18h 15m
Merged PRs (30d)
442

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from uttrflow/uttrflow-swift

All issues in uttrflow/uttrflow-swift

Similar issues

More Swift issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.