feat(agent): add session cleanup lifecycle for built-in and custom agents
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go
- Domain
- api, backend, documentation, testing
Research direction
Start with agent.Agent, SessionResumer, and defaultEvaluator.executeCaseOnce to trace session creation, retries, judging, artifact collection, and runtime closing. Then inspect the Custom Engine local and HTTP transport contracts; done means attributable sessions are cleaned exactly once across all listed outcomes, cleanup errors do not mask primary results, and concurrency and lifecycle tests pass.
Written by the indexing model from the issue text.
Description
Problem / motivation
Agent sessions created during an evaluation have no agent-level cleanup lifecycle today.
The current flow closes the per-case runtime, but agent.Agent has no session cleanup capability. This is a different resource boundary: a built-in CLI may persist a conversation under its home/session store, while a stateful Custom Engine may allocate a server-side session and return its opaque session_id.
For stateful custom engines, skill-up propagates SessionResult.session_id between turns and then drops it when the case ends. There is no callback or declarative custom transport operation that lets the engine release the session. The same gap applies to sessions created by built-in agents. It is especially visible with parallel cases, retries, failures, and timeouts, where evaluation runs can leave many sessions behind.
Relevant current boundaries:
agent.Agentprovides install, check, and run operations, but no cleanup operation.SessionResumerprovidesRunTurnonly.defaultEvaluator.executeCaseOncedefersruntime.Close, which cleans the runtime/workspace boundary but does not express agent-session cleanup.- stateful Custom Engines receive and return
session_id, but the Custom Engine contract has no terminal lifecycle notification.
Proposed solution
Add an explicit, best-effort agent-session cleanup lifecycle that works for both built-in and custom agents.
One possible shape is an optional capability so existing third-party/test agents remain source-compatible:
type SessionCleaner interface {
CleanupSession(ctx context.Context, rt Runtime, opts ExecOptions, sessionID string) error
}
The evaluator should track the latest non-empty session ID created by each agent invocation and call cleanup exactly once after all required transcript/artifact collection and judging for that session, but before closing its runtime.
The lifecycle must cover:
- single-turn and multi-turn sessions;
- normal completion, FAIL, ERROR, cancellation, and timeout;
- every retry attempt, not only the final attempt;
- main runner sessions and separate
agent_judgesessions; - session IDs that rotate between turns, cleaning the final live session according to the adapter contract.
For Custom Engines, define a declarative cleanup operation for both transports:
- local: render and invoke a cleanup command with the final
${session_id}and the same safe environment/secret handling as run; - HTTP: call a configured cleanup endpoint/request with the final session ID and the same bounded timeout, response-size, redirect, and credential rules as the existing HTTP transport.
Built-in adapters should implement cleanup where their CLI/session store exposes a safe, attributable deletion mechanism. An adapter that cannot safely delete a session should leave the optional capability unsupported rather than guessing paths or deleting unrelated state.
Cleanup should use a short context detached from the already-canceled case context, similar to post-run session artifact collection, so timeout/cancellation does not make cleanup impossible. Cleanup errors should be observable and reported without replacing the primary case result/error.
Design questions
- Should cleanup be enabled by default for sessions created by skill-up, with an opt-out such as
keep_sessions, or remain opt-in initially? - Should failed cases retain sessions for debugging by default?
- For rotating custom session IDs, does the backend guarantee that cleaning the latest ID releases the whole conversation, or should the evaluator retain and clean every observed ID?
- Should the cleanup contract receive a terminal reason such as completed, failed, timed_out, canceled, or retrying?
Alternatives considered
Rely on runtime.Close
This only works when the session storage is fully contained in an ephemeral runtime. It cannot clean server-side Custom Engine sessions and does not model the agent resource explicitly.
Let each run command clean up internally
A run command cannot reliably know whether another turn will resume the session, whether judging/artifact extraction still needs it, or whether the evaluator will retry after an error.
Delete known built-in session files directly in the evaluator
This couples orchestration to adapter-specific storage layouts and risks deleting the wrong session. Cleanup ownership should stay with the agent adapter.
Acceptance criteria
- An agent can expose a session cleanup capability without forcing all existing Agent implementations to add a no-op method.
- The evaluator invokes cleanup once per created session on success, failure, timeout, cancellation, and retry paths before
runtime.Close. - Cleanup runs only for a non-empty, attributable session ID and cannot delete unrelated sessions.
- Main runner and
agent_judgesessions are both covered. - Stateful Custom Engine local and HTTP transports can configure a cleanup operation using the final session ID.
- Cleanup uses a bounded context that still runs when the case context has already been canceled.
- Cleanup failures are logged/reported without masking the primary evaluation result.
- Concurrency tests prove session IDs do not cross cases.
- Tests cover success, failure, timeout, cancellation, retries, rotating IDs, and cleanup transport errors.
- User and Custom Engine contract documentation explains cleanup timing, defaults, and retention/debugging behavior.
Additional context
This follows the stateful Custom Engine work in #249 / #250. That work established session creation and resumption; this issue covers the missing terminal lifecycle.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remotePossibly taken @bababoom721 claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day