context length and trajectory problems

Inizia tracciando come viene creato e utilizzato task.md durante il rollout, come vengono assemblati i prompt durante la riflessione e dove viene generato trace raw.txt. Confronta i percorsi Claude Code Exec e skillopt sleep per determinare se vengono conservati il contesto, i sample falliti e le traiettorie intermedie degli strumenti; il lavoro sarà completato quando il contesto e i dati di traiettoria richiesti raggiungeranno la riflessione senza troncamento.

Context size issue: For example, when training on the SearchQA dataset using Claude Code Exec as the backend, the context and question are written into a task.md file, which Claude reads to generate answers. This avoids the problem of input text being too long during the rollout phase, making it impossible for Claude to fully process the content. However, during the reflection phase, Skillopt currently does not support full Claude Code Exec as the backend. Instead, relevant information—including the context needed to answer the question—must be included directly in the prompt, then calling claude once. This leads to potential context truncation issues.

Alse skillopt sleep：During the rollout phase, skillopt sleep did not use full Claude execution but instead treated it as a chat endpoint, so context was likely limited. In the reflection phase, reference materials for context probably weren't fed into the reflector either, and each failed sample's question, answer, and failure reason were truncated.

Do you have any plans to optimize the above two context-related scenarios in the future?

The issue regarding the intermediate process trajectory: Taking the training data of the Searchqa dataset as an example, when using the Claude code execution backend, the intermediate execution trajectory of Claude code (such as the intermediate thinking process, tool calls, etc.) was not parsed and saved. Although I noticed that a trace raw.txt file was generated in the code, the content was just a very simple summary. Additionally, the trajectory process was not sent to the reflection stage.

Does the official have any optimization plans for this issue? If this situation can be supported, then the dataset only needs to provide the questions and answers. The agent will provide the intermediate trajectory and the final result, and all of them will be sent to the reflect stage. The reflector can simultaneously analyze the agent's output and the intermediate process, and propose more targeted skill optimization suggestions.