Rubric design may be too strict for open-ended workspace tasks

Bắt đầu bằng việc đọc task_lite_clean_cn/152/metadata.json và task_lite_clean_cn/158/metadata.json từ dataset Hugging Face được liên kết, rồi so sánh mô tả của từng task với rubric tương ứng. Xem xét cách các tiêu chí đánh giá hiện tại xử lý tên tệp thay thế và các quyết định lập kế hoạch có lý do chính đáng. Công việc được xem là hoàn tất khi các task mở có thể đánh giá các đầu ra bắt buộc, mức độ bao quát, tính đúng đắn về ngữ nghĩa và chất lượng mà không chỉ dựa vào các câu trả lời chính xác được định nghĩa trước.

Hi, thanks for building and releasing Workspace-Bench Lite. I found the benchmark very useful for evaluating agent behavior in realistic workspace environments.

I would like to raise one concern about some open-ended tasks: the task descriptions are relatively broad and allow multiple reasonable solutions, but the rubrics sometimes enforce very specific output details as if there were only one correct answer. This may cause valid task completions to be marked as failures.

Example 1: Task 152
https://huggingface.co/datasets/Workspace-Bench/Workspace-Bench-Lite/blob/main/task_lite_clean_cn/152/metadata.json

The task asks the agent to organize several scientific drawing icons into a new folder and rename them so that their contents can be clearly identified.

This is an open-ended file organization task. A user would likely accept multiple reasonable names, such as:

• 对话气泡-问号.png
• 问号-气泡.png
• 灯泡创意-问号.png
• 绿色数据表格.png

However, the rubric requires exact filenames such as 问号-气泡.png, 问号-人.png, 表格-绿色.png, etc. This makes the evaluation closer to exact-answer matching rather than checking whether the agent correctly understood and renamed the icons.
Also, the task explicitly asks to create a new folder, but the rubric focuses heavily on filenames and image details, while the folder-organization requirement seems less directly evaluated.

Example 2: Task 158
https://huggingface.co/datasets/Workspace-Bench/Workspace-Bench-Lite/blob/main/task_lite_clean_cn/158/metadata.json

The task asks the agent to generate an operational work plan based on several user communication records and team responsibilities.

This is also an open-ended planning task. The agent needs to extract user needs, prioritize them, assign responsibilities, and produce an actionable plan. However, the rubric requires very specific conclusions, such as fixed priority levels, fixed department assignments, fixed follow-up mechanisms, and exact demand counts.

Some of these requirements may be reasonable, but they are not always uniquely implied by the task description. For example, different priority assignments may be valid if the agent provides a clear rationale. A good answer should be judged by coverage, reasoning quality, structure, and actionability, not only by whether it matches a predefined plan.

Suggestion

For open-ended workspace tasks, it may be better to split rubrics into different levels:

Hard constraints: required output file exists, correct format, no missing input files, images remain valid, etc.
Content coverage: all key input information is extracted or processed.
Semantic correctness: filenames, summaries, or plans are reasonable and clearly reflect the input.
Quality criteria: consistency, actionability, prioritization rationale, and absence of obvious mismatches.

For tasks like 152, the rubric could allow semantically equivalent filenames instead of requiring exact names.

For tasks like 158, the rubric could evaluate whether the priority assignment is justified and complete, rather than requiring one fixed priority mapping.

This adjustment may make the benchmark better reflect real-world agent performance in open-ended environments, while still keeping the evaluation reliable and diagnosable.