Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Efficient AD for the simulation of gradients

オープン
  • 何か問題がありますか？
#504 コメント 6 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
説明が足りない
活発さ
停滞
技術スタック
julia
領域
machine-learning, performance

調査の方向性

まず chainrules.jl と、issue でリンクされている複数出力の入力空間設計を読んでください。現在の autodiff パスを、提案されている等方的カーネルおよび定常カーネルのケースと比較し、そのうえで、範囲を限定した一次導関数のアプローチとその検証基準を定義してください。issue では chainrules.jl 以外に具体的なテストやファイルは指定されていません。

索引モデルが issue の本文から書いたものです。

説明

Introduction: How would you simulate gradients

If you want to simulate the gradient of a random function $Z$, it turns out that you simply need to take derivatives of the covariance funcion, as (w.l.o.g. $Z$ is centered)

\text{Cov}(\nabla Z(x), Z(y)) = \mathbb{E}[\nabla Z(x) Z(y)] = \nabla_x \mathbb{E}[Z(x)Z(y)] = \nabla_x C(x,y)

And similarly

\text{Cov}(\nabla Z(x), \nabla Z(y)) = \nabla_x\nabla_y C(x,y)

For stationary covariance functions $C(x,y) = C(x-y)$ this simplifies to

\begin{aligned}
\nabla_x C(x,y)  &= \nabla C(x-y)\\
\nabla_x\nabla_y C(x,y) &= -\nabla^2 C(x-y)
\end{aligned}

So if we consider the multivariate random function $T_1(x) = (Z(x), \nabla Z(x))$, then its covariance kernel is given by

\begin{aligned}
C_{T_1}(x,y) 
&= \begin{pmatrix}
     C(x,y) & \nabla_y C(x,y)^T\\
     \nabla_x C(x,y) & \nabla_x \nabla_y C(x,y)
\end{pmatrix}\\
&\overset{\mathllap{\text{stationary}}}= 
\begin{pmatrix}
     C(x-y) & -\nabla C(x-y)^T\\
     \nabla C(x-y) & -\nabla^2C(x-y)
\end{pmatrix}\\
&\overset{\mathllap{\text{isotrope}}}=
 \begin{pmatrix}
     C(d) & -f'\bigl(\frac{\|d\|^2}{2}\bigr)d^T\\
     f'\bigl(\frac{\|d\|^2}{2}\bigr) d & -\Bigl[f''\bigl(\frac{\|d\|^2}2\bigr)dd^T + f'\bigl(\frac{\|d\|^2}2\bigr) \mathbb{I}\Bigr]
\end{pmatrix}
\end{aligned}

where we use $d=x-y$ and $C(d) = f\bigl(\frac{|d|^2}2\bigr)$ in the last equation.

Performance Considerations

In principle you could just directly apply Autodiff (AD) to any kernel $C$ to obtain $C_{T_1}$, but since I heard that the expense of autodiff scales with the number of input arguments, this would be really wasteful when we only really need to differentiate a one dimensional function in the isotropic case.

Unfortunately the way that KernelFunctions implement length scales results in general kernel functions, so I am not completely sure how to tell the compiler, that "these derivatives are much simpler than they look".

One possibility might be, to add the Abstract types IsotropicKernel and StationaryKernel and carry these types over when transformations do not violate them. Scaling would not, more general affine transformations would violate isotropy but not stationarity, etc. This could probably be done with type parameters.

But even once that is implemented, how do you tell autodiff what to differentiate? I have seen the file chainrules.jl in this repository, so I thought I would ask if someone already knows how to implement this.

Kernels for multiple outputs considerations

Since you implemented kernels for multiple outputs as an extension of the input space, the reuse of the derivative $f'(\frac{|d|}{2})$ also becomes more complicated.

Maybe this is all premature optimization, as the evaluation of the kernel is complexity wise in the shadow of the cholesky decomposition.

Extension: Simulate $n$-order derivatives

In principle you could similarly simulate $T_n(x) = (Z(x), Z'(x), \dots, Z^{(n)}(x))$. But AD is already underdeveloped for hessians, so I don't know how to get this to work. It might be possible to write custom logic for certain isotropic random functions, like with the squared exponential one.

What do you think? I handcrafted something for first order derivatives in a personal project, but for KernelFunctions.jl a more general approach is probably needed.

主要言語
Julia
スター
276
フォーク
42
平均マージ
19時間
マージ済み PR（30日）
2

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

JuliaGaussianProcesses/KernelFunctions.jl のほかの issue

JuliaGaussianProcesses/KernelFunctions.jl の issue をすべて見る

似ている issue

Julia の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。