DQL still learning at evaluation time
还没有人认领这个 Issue。
- #177 来自 @lxcxjxhx —— 已关闭，未合并
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 38/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 技术栈
- jupyter-notebook, python
调研方向
从 cyberbattle/agents/baseline/learner.py 中的 learner.epsilon_greedy_search 和 cyberbattle/agents/baseline/agent_dql.py 中的 learner.on_step 开始，然后检查 notebook_benchmark-chain.ipynb 和 notebook_benchmark-toyctf.ipynb 中的评估流程。确认评估不会继续进行优化，并将报告的 chain 和 ToyCTF 基准测试结果与所描述的数据进行比较。
由索引模型根据 Issue 内容生成。
描述
Issue forked from #87 by @kvas7andy
learner.epsilon_greedy_search(...) is often used for training agents with different algorithms, including DQL in the dql_run. However dql_exploit_run with input network dql_run as policy-agent and eval_episode_count parameter for the number of episodes, gives an impression that runs are used for evaluation of the trained DQN. The only distinguishable difference between 2 runs is epsilon queal to 0, which leads to exploitation mode of training, but does not exclude training, because during run with learner.epsilon_greedy_search the optimizer.step() is executed on each step of training in the file agent_dql.py, function call learner.on_step(...).
- Solution: I will include in Pull request the code I used for better evaluation (based on learner.epsilon_greedy_search(...) and generate pictures below.
- Screenshots: Figure 1 & 2 and figure 3 & 4 , shows result of chain network evaluation using corresponding new cell in notebook_benchmark-chain.ipynb. As you can see on figure 1 training on the initial 50 episodes is not enough for owning 100% of the network (AttackerGoal), whereas original run
dql_exploit_runinternally usinglearner.on_step(...)figure 2 leads to much better results, due to optimization process, which still process ongoing experience of agent. We can overcome this inaccurate evaluation and still reach the goal in 100% of times figure 3, while training on 200 episodes with commentedlearner.on_step(). It fixes trained network and stops optimizing during evaluation, but leads to the ownership of all the network with larger amount of learning episodes. This means with 200 episodes it is feasible to learn optimal path of agent attacks inside chain network configuration.
Lastly, figure 4 we can compare those runs with correct evaluation runs on 20 episodes reach 6000+ and 120+ cumulative reward for for 200 and 50 training episodes correspondently.
Figure 1: (after PR) no optimizer during evaluation, 20 trained episodes, 20 evaluation episodes
Figure 2: (before & after PR) dql_exploit_run with optimizer during evaluation, 20 trained episodes, 5 evaluation episodes
Figure3: (after PR) no optimizer during evaluation, 200 trained episodes, 20 evaluation episodes
Figure 4: (after PR) comparison of evaluation for network trained on 200 and 20 episodes, chain network configuration
ToyCTF benchmark is inaccurate, because with correct evaluation procedure, like with chain network configuration, agent does not reqch goal of 6 owned nodes after 200 training episodes.
- 主要语言
- Jupyter Notebook
- 星标
- 1.8k
- 派生
- 283
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
microsoft/CyberBattleSim 的其他 Issue
-
难度 5/5 一周以上 新手友好度 25/100
microsoft/CyberBattleSim#156 ·
-
难度 4/5 3-5 天 新手友好度 30/100
microsoft/CyberBattleSim#152 ·
-
question
难度 5/5 一周以上 新手友好度 25/100
microsoft/CyberBattleSim#117 · 1 条评论 ·
-
enhancement
难度 5/5 一周以上 新手友好度 20/100
microsoft/CyberBattleSim#96 · 3 条评论 ·
-
needsclarification question
难度 3/5 1-2 天 新手友好度 35/100
microsoft/CyberBattleSim#90 · 1 条评论 ·
查看 microsoft/CyberBattleSim 的全部 Issue
相似的 Issue
-
`ingest` fails with pynndescent 0.7可能已有人在做 @eroell 今天认领。 未关闭Triage 🩺
难度 2/5 1-3 小时 新手友好度 68/100
维护者通常 1 天内回复
-
[VLM] granite_vision preset produces empty output and runs ~3× slower with Granite Vision 3.3-2B than with Granite Vision 4.1-4B可能已有人在做 @prassu017 今天认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 74/100
docling-project/docling#5068 · 4 条评论 ·
维护者通常 1 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 66/100
Azure/azureml-examples#4139 ·
维护者通常 4 天内回复
-
bug triage
难度 2/5 1-3 小时 新手友好度 74/100
guaardvark/guaardvark#349 ·
维护者通常 1 天内回复
-
Bug in Web UI: Unknown model arch 'sd15' when selecting SD 1.5可能已有人在做 @ngo-binh0108 今天认领。 未关闭
难度 1/5 1 小时以内 新手友好度 86/100
ostris/ai-toolkit#1092 ·
维护者通常 1 天内回复