`FMComposedPrompt` still leaked by `stream_response()` and `SystemLanguageModel.token_count()` (follow-up to #17 / #18)

Aperta
#23 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Read src/apple_fm_sdk/session.py at _stream_response_basic(), src/apple_fm_sdk/core.py at SystemLanguageModel.token_count(), and src/apple_fm_sdk/prompt.py at _composed_prompt_from_prompt(). Then inspect tests/test_composed_prompt_cleanup.py and its fake native bindings. Done means the composed prompt is released on success, error, cancellation, and rejected components, and invalid streaming prompts raise to the caller.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Description

#18 releases the native composed prompt in the three respond() paths. Three other places that
build one with _composed_prompt_from_prompt() still never release it:

  1. LanguageModelSession._stream_response_basic(), which builds the prompt inside the stream thread:
    https://github.com/apple/python-apple-fm-sdk/blob/0918b60/src/apple_fm_sdk/session.py#L833
  2. SystemLanguageModel.token_count(), in the prompt branch:
    https://github.com/apple/python-apple-fm-sdk/blob/0918b60/src/apple_fm_sdk/core.py#L440
  3. _composed_prompt_from_prompt() itself, when a component is rejected (PromptError,
    ImagePromptError): the pointer is created first and never returned to the caller:
    https://github.com/apple/python-apple-fm-sdk/blob/0918b60/src/apple_fm_sdk/prompt.py#L300-L319

Each call leaks one ComposedPrompt. With an ImageAttachment it also keeps one file descriptor
open, which is the symptom from #17.

A related effect of (1): because the prompt is built in the stream thread, an invalid prompt
raises there and the exception is lost. stream_response() then waits indefinitely and
asyncio.wait_for() cannot interrupt it:

Exception in thread Thread-1 (_start_stream):
apple_fm_sdk.prompt.PromptError: Unsupported prompt component type <class 'int'>, ...
(still waiting after 10 s, despite asyncio.wait_for(..., 3))

Reproduction

import asyncio, gc, os, tempfile
from pathlib import Path
from PIL import Image
import apple_fm_sdk as fm

def open_fds(): return len(os.listdir("/dev/fd"))

async def main():
    img = Path(tempfile.mkdtemp()) / "leak.png"
    Image.new("RGB", (32, 32), "white").save(img)
    for _ in range(2):   # warm-up, then measure
        before = open_fds()
        for _ in range(15):
            async for _ in fm.LanguageModelSession().stream_response(["Reply with OK.", fm.ImageAttachment(img)]):
                pass
        gc.collect()
    print(f"stream_response: open fds {before} -> {open_fds()}")

asyncio.run(main())

Results with 15 calls, after warm-up (macOS 27.2.1, M1, Python 3.14.6):

main @ 0918b60 with the fix below
respond() with image (fixed in #18) fds 7 → 7 fds 7 → 7
stream_response() with image +15 fds per 15 calls (the snippet prints 22 -> 37; the warm-up round already leaked 15) +0 (7 -> 7)
token_count() with a 200 KB prompt, 60 calls RSS +12.7 MB RSS +1.6 MB
stream_response(123) hangs raises PromptError immediately

Proposed fix

Same approach as #18:

  • _stream_response_basic(): build the composed prompt before starting the stream thread, and
    release it in the existing finally block, next to the stream pointer. PromptError then
    reaches the caller.
  • token_count(): release the composed prompt in a finally around _token_count().
  • _composed_prompt_from_prompt(): release the pointer before re-raising when a component is
    rejected.

FMLanguageModelSessionStreamResponse and FMSystemLanguageModelTokenCountForPrompt read
.promptRepresentation synchronously, so releasing afterwards is safe.

The patch extends tests/test_composed_prompt_cleanup.py in the same style as #18 (faked native
bindings, no model needed). It adds 8 tests: success, error and early exit for streaming, the
invalid-prompt case, success, error and cancellation for token_count, and the rejected
component. All 8 fail on main (0918b60) and pass with the fix. The existing 12 tests are
unchanged and still pass. With the on-device model, test_streaming.py, test_token_count.py,
test_prompts.py, test_image_prompts.py, test_session.py and test_respond_options.py also pass
on the branch.

Branch with the fix and tests:
https://github.com/apple/python-apple-fm-sdk/compare/main...luiacuaniello:python-apple-fm-sdk:fix/composed-prompt-leak-stream-token-count

Note: PyPI still serves 0.2.1, which predates #18, so respond() leaks too for anyone who
installs from PyPI.

Lingua principale
Python
Stelle
1.4k
Fork
95
Merge medio
1m
PR unite (30g)
1

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apple/python-apple-fm-sdk

Tutte le issue di apple/python-apple-fm-sdk

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.