Given a set of correct keypoint matches and a fundamental matrix, to optimize the coordinates of these key points such that they satisfy the epipolar constraint.

A point (x,y) on the left image (pose: [I|0]) and (x’,y’) on the right image (pose: [R|t]). These points are undistorted and in normalized image coordinates. Having known the pose already, we can get to the fundamental matrix. F= [t]_x * R. See Hartley-Zisserman.

Now we want to get another points (u,v) which is near to (x,y). Also (u’,v’) which is near to (x’,y’) such that these new points (u,v) and (u’,v’) satisfy the epipolar constraint, ie. (u,v,1) F (u’;v’;1).

This can be set up as an optimization problem. Detailed derivation in my note, here. Since this optimization problem involves an equality constraint, we can use the Lagrange-multiplier method to arrive at a solution. Since the Lagrangian problem is a quadratic and convex cost function, 1 step of Newton’s method is sufficient to arrive at an optimum.