We attribute the reason to the high curvature of the learned generative trajectories. The curvature is intriguing since it is directly related to the truncation error of a numerical solver. Intuitively, zero curvature means that generative ODEs can be accurately solved with only one function evaluation.
We find that the rectified flow perspective offers an interesting insight into the relationship between the forward process and the curvature. Based on our observation, we propose an efficient method of training the forward process to reduce curvature.
recent ODE-based models do not require ODE simulations during training and therefore are more scalable. At a high level, they define a forward coupling q(x, z) between data distribution p(x) and prior distribution p(z) and subsequently an interpolation xt(x, z)
Here, a neural network xθ(xt, t) is trained to reconstruct the data x from the corrupted observation xt. In the following, we briefly review two popular instances of such models: the denoising dif-fusion model and rectified flow
Rectified flows
However, the choice of Eq. (2) seems unnatural from a rectified flow perspective as it unnecessarily increases the curvature of generative trajectories.
In rectified flow (Liu et al., 2022), the intermediate sample xt is rather defined as a linear interpolation
it has a constant velocity across t for given x and z.
The effectiveness of this sampler in reducing the sampling costs has been previously investigated in Karras et al. (2022) under the variance-exploding context. Also, Eqs. (3) and (4) are a special case of flow matching (Lipman et al., 2022).
<aside> 💡 A.1 Rectified flows, Flow Matching은 논문 참고.
</aside>
For a generative process Z = {zt(z)} with the initial value z1(z) = z , we informally define curvature as the extent to which the trajectory deviates from a straight path:
