Inference Gym: adding and/or updating ground truth expectations
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 25/100
- Issue 类型
- 功能
- 描述清晰度
- 需要澄清
- 活跃度
- 停滞
- 技术栈
- jupyter-notebook, python
调研方向
首先，使用 Stan 和 CmdStanPy 审查现有的 inference-gym ground-truth 流程，以及所引用的 VectorModel 和 BrownianMotionUnknownScalesMissingMiddleObservations 模型。将运行时间更长的 Blackjax NUTS 结果与当前预期进行比较，然后确定应如何处理二阶矩期望值，以及如何处理可能的 Blackjax 或 TFP NUTS 工作流。如果结果得到确认，并且已经达成一致的方法且更新了 ground truths，则视为完成。
由索引模型根据 Issue 内容生成。
描述
(I've divided this into three subsections, and can split into separate issues if preferable)
Length of ground truth runs
As I understand it, the current ground truth estimates are obtained from Stan with 150000 samples and 10 chains.
For certain models, such as `gym.targets.VectorModel(gym.targets.BrownianMotionUnknownScalesMissingMiddleObservations(), flatten_sample_transformations=True,), I have produced my own ground truths via longer runs of Blackjax's NUTS (10 million steps, 4 chains), and found results that differ enough to matter for my use cases (namely, estimating efficiency of different samplers)
from blackjax run: [ 0.11525708 0.09256472 0.05635736 -0.03410918 -0.05100336 -0.18196875
-0.18945307 -0.25923407 -0.25987643 -0.32402724 -0.22958763 -0.28165078
-0.3362609 -0.38868254 -0.44175696 -0.4945148 -0.5447676 -0.6013282
-0.6559048 -0.7087315 -0.75866866 -0.8134075 -0.8074223 -0.7784713
-0.82167107 -0.7737639 -0.743899 -0.7613981 -0.6401507 -0.6669518
-0.64461184 0.11305185]
from gym: [ 0.11984811 0.10274264 0.06093274 -0.03870019 -0.04362268 -0.19021639
-0.1856622 -0.26851514 -0.26010785 -0.3334386 -0.21788554 -0.2735482
-0.33083084 -0.38252977 -0.43280044 -0.49400684 -0.54860604 -0.60449123
-0.65569454 -0.7083658 -0.76391494 -0.8189823 -0.8105346 -0.7771473
-0.8268097 -0.7768991 -0.7374106 -0.7740582 -0.6294383 -0.670295
-0.6432216 0.10105278]
(See e.g. the hierarchical params, in particular, the second and final elements of the array).
If my results are actually more accurate (of course it's possible there's a mistake on my end), then would it be possible to switch to the results of a longer run (either of Stan or Blackjax, but see the final section below) in inference-gym?
Adding ground truth expectations of second moment
I would also like to add ground truth estimates of the second moment, i.e. $\mathcal{E}[x^2]$. Would it be possible for me to add these to certain inference-gym models?
Blackjax vs Stan
Currently, Stan is used by inference-gym to produce samples for ground truth estimates, run via CmdStanPy. How open would inference-gym be to switching to Blackjax's NUTS implementation instead, to obtain a fully Python setup? (Or even the TFP NUTS implementation)
- 主要语言
- Jupyter Notebook
- 星标
- 4.4k
- 派生
- 1.1k
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
tensorflow/probability 的其他 Issue
-
KL divergence between two `TransformedDistribution`s with a shared non-injective bijector (`AbsoluteValue`) returns the base KL可能已有人在做 @Andreas-Hum 于 8 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
tensorflow/probability#2036 ·
-
难度 3/5 1-2 天 新手友好度 68/100
tensorflow/probability#2034 ·
-
难度 5/5 一周以上 新手友好度 35/100
tensorflow/probability#2021 ·
-
难度 4/5 3-5 天 新手友好度 30/100
tensorflow/probability#2015 ·
-
难度 3/5 1-2 天 新手友好度 48/100
tensorflow/probability#2013 ·
查看 tensorflow/probability 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 62/100
vllm-project/vllm-ascend#21392 · 1 条评论 ·
维护者通常 1 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 68/100
维护者通常 2 天内回复
-
难度 1/5 1 小时以内 新手友好度 85/100
维护者通常 1 天内回复
-
type:epic-wave
难度 2/5 1-3 小时 新手友好度 75/100
jejjohnson/filterax#204 ·
维护者通常 1 天内回复
-
documentation good first issue
难度 2/5 1-3 小时 新手友好度 88/100
selimfirat/pysad#644 ·
维护者通常 1 天内回复