Hasta La Motion!

I need to rebuild a scene in 3D from various pictures that were shot on-set and by using project mapping retexture the whole thing – the aim is to create a more interesting camera move than the one that we got on the shoot.. Now my question is how do I go about it correctly..?

My first thought was to undistort the plates, then import them to the 3D package, match the cameras position and remodel the whole scene. Now I found that matching camera is really tricky especially for wider shots. If I use undistorted plate it seems that I won’t be able to match them perfectly because the 3D camera will introduce its own curvature..

Any tips and/or suggestions would be highly appreciated on this topic..

Best Regards