Then, how do SBMs approximate distributions on manifold?

문제 제기: 어떻게 score based model(SBMS)은 manifold에 분포에 근사 할지

Fig. 1 summarises our main finding. The left two panels are two rotated views of the example data density q0 (blue-orange) on a 2D manifold in a 3D ambient space. The density implies a conserved vector field. The right two panels show the vector field of a score model g which approximates the score of q0 but is not guaranteed to be conservative.

문제 제기: Fig 1에 왼쪽 두 패널들은 3D ambient space 안에 있는 2D manifold에 data density q0(blue-orange) 의 2가지 관점임. density는 conserved vector filed를 암시함. 오른쪽 두 패널들은 q0의 score를 근사한 score model g의 vector field인데 consertative를 보장하지 않음.

We find that, in this score model, the vector field is non-conservative only within the manifold; whereas the field in directions normal to the manifold remains close to the conservative score field of the noisy data distribution, constraining the samples to stay around the data manifold. Further, the local features of g span the same local subspace of an effective density function that is consistent with g in the sense we clarify soon.

score model에 vector field는 manifold에 non-conservative함. 반면 noisy data 분포의 score field는 conservative함. 또한, g의 local features는 g와 일치하는 effective density function의 동일한 로컬 하위 공간에 걸쳐 있음.

1. Local orthogonal features of approximate score functions

A local approximation of the score gσ(x) ≈ gσ(x0) + ∇xgσ(x)|x0 (x − x0) around x0 involves the score Jacobian, so we use its singular value decomposition (SVD) to analyse gσ locally:

SVD를 이용해 g를 분석함. ui,vi그리고 si 각각 i번째 left sigular vector, right sifular vector 그리고 g의 singular vector임. 만약 g가 socre density function이면 g는 conservative하고 델타 g는 symmetric 그리고 u=+_vi임.

To build more intuitions about score Jacobians, consider the multivariate Gaussian N (x; µ, Σ) which has a constant score Jacobian $−Σ^{−1}$ . Each pair of its singular vectors have opposite signs. In particular, the singular value of rank i is equal to the inverse variance along the i’th singular vector.

score Jacobians에 대해 더 intuitions하기위해, multivariate Gaussian을 고려함. singular vectors의 각 쌍은 opposite signs를 가짐. 특히, rank i의 singular value는 invercse variance의 i번째 singular vector와 동일함.

After adding a small additive Gaussian noise on R d , the SVD of the score Jacobian shows interpretable properties of the data distribution: large singular values or small variances appear along directions with abrupt changes in the score, reflecting steep curvatures along the off-manifold directions. Conversely, small singular values or large variances are associated with on-manifold directions along which the data density varies smoothly. This pattern generalises to curved manifolds as long as the noise is small compared to the local curvature. In practice, the Jacobian of a learned score estimator ∇xgσ(x) may not be symmetric as that of the Gaussian, but we can compare it to an effective conservative (energy-based) score field.

$R^d$에 작은 가우시안 노이즈를 추가한 후에, score Jacobian의 SVD는 data distribution의 interpretable properties일때 보여주는 현상은 : 큰 singular values 또는 small variances는 score에 갑작스러운 변화를 보여줌. off-manifold 방향으로 터무니 없게 curvatures함.

거꾸로 작은 singular values나 large varicance는 data desity에 다양한 smoothly한 방향으로 on-manifold에 따름. 이 패턴은 노이즈가 local curvature에 비해 작은 한 curved manifolds에 일반화됨.

This means that, given a sample path, g˜σ is a valid score of an (unknown) density function equivalent to gσ in terms of xt’s likelihood.

의미는 주어진 sample path, g˜σ 는 xt’s likelihood의 terms 안 gσ 에 동일한 (unknown) density function의 valid score임.

Similar to but unlike the SVD of ∇xgσ(x), the eigendecomposition of the symmetric ∇xg˜σ(x) reveals the local features of the equivalent density: following the intuition of the Gaussian distributions, we see that the eigenvectors with negatively large eigenvalues correspond to off-manifold directions (−Σ −1 is negative semi-definite); eigenvectors with small-inmagnitude eigenvalues indicate on-manifold directions. Positively large eigenvalues indicate positive curvature, and we find that they exhibit on-manifold features as shown in our experiments.

∇xgσ(x)의 SVD와 비슷하지만 다르게, 대칭 ∇xg˜σ(x)의 고유값분해(EVD)는 동일한 density의 local features를 드러냄. 가우시안 분포의 intuition을 따라, off-manifold directions에 맞는 negatively large eigenvalues와 함께 eigenvectores를 볼 수 있음. small-inmagnitude eigenvalues에 eigenvectors는 on-manifold 방향임, positively large eigenvalues는 positive curvature과 on-manifold임.