Rig Removal In Fusion
Posted in: Digital Fusion / DFX+I was having a coffee this weekend with a good friend (aka Neybs). We fired Furnace for Fusion up and started poking some of the tools, one of which was the Rig removal tool which was so cool I had to share.
To my shame I never looked at Furnace for Fusion when I was with eyeon, so I thought I would give them a poke and see what they did. There is some very cool technology with Furnace (motion estimation/optical flow, Inference matting etc etc), but when I asked some local Fusion users if they had used Furnace for Fusion, most had no idea what they did.
Anyway, please don’t flame me…(I am looking at you Eric..Theo… 🙂 ) I just wanted to share a bit of joy we had when we ran the Furnace tools in Fusion, and hoped you would get the same. You can pop me a mail if you want to try for yourself, you do need Fusion 5.3, as this is an OFX plug-in.
Below is how I did it…Please take note of the professional Roto job I did further down. Thank You Neybs for helping me get the grabs from Fusion over the weekend, I owe you more Instant coffee, which I know is your fav.
M
First look at the Furnace Rig Removal Tool in Fusion
One of the first steps I take with any new tool is to load it, and try to work out what it does, (and then later when things are not going to plan, read the manual). Actually I do that with everything…
A lot of the tools I have tried over the years have not fallen well into the poke it first, read the manual later category, but the Furnace rig-removal tool for Fusion does, and it was such a jaw dropping few clicks, I thought I had better share this with as many Fusion users as I could.
When I dragged the rig tool onto the Fusion flow, I genuinely expected the same tools that we have all seen many times before, key frame a marker around a wire, advance, rinse and repeat. What I got though was something quite astonishing, so here is the result of my first click through.
The example file was a scooter, whizzing through the streets of London, passing behind a tree, several posts, and in front of windows. I really thought I had opened the wrong example, so checked and it was correct.
The Fusion view screen grab below is a still of the clip, and you can see the little scooter is passing behind all of these objects, the camera was hand held and shaky as hell, and it was PAL footage.
Grab One. Furnace Rig Removal Test Shot. Scooter, trees, posts.
The real eye opener was the view of the rig removal tool, no roto work, no key, but on this frame, the scooter was removed completely and the background was replaced perfectly. Take a look below.
Grab 2. Loader connected to source input of Furnace Rig Removal tool
I followed the few steps in the tutorial, and the end result of my first pass was a 98% work done scooter removal. I have since tried the rig removal tool on several shots, and in most situations it works perfectly. I did have a few shots that did not get results as clean as this one without a few tweaks and there are going to be some shots and situations where there isn’t enough data to get a result like this (or at all), but for those shots where you do!
The rig removal works by looking at the background motion between frames, ignoring the foreground object, and then using motion information to look forwards and backwards in the sequence in order to find the correct piece of background to fill in the missing region. Here is a run through of using the tool in Fusion. Add a loader, and then pick the F_RigRemoval tool from the Furnace menu.
Grab 3. Pick: Furnace F_RigRemoval
Grab 4. View the Rig tool, and see the red default fail marker displayed.
There are a few ways of telling the Furnace rig tool what to look for (a bounding box, source alpha and Luminance), and you can also specify regions with a mid grey mask for it to ignore, but my first poke was using the default bounding box, you can see this in red above when you view the Furnace tool, attached to the scooter clip in Fusion.
Grab 5. Scale bounding box over removal area
Next step was to drag the bounding box over the region you want to remove. The bounding box is colored red to let you now that you need to tweak the settings as the removal has failed. You do this by adjusting the number of frames the tool looks forwards and backwards to analyze and find enough data in the image to repair the sequence. All you do here is adjust the number of frames slider, until the red goes away.
This allows the rig removal tool to gather information from this number of frames before and after (if they are there) from the current point. I slid the slider up to 30 frames. (I later found that 5 frames worked fairly well on this shot, but I had not read the manual at this point).
The scooter (and the red fail marker) disappeared. Zooming into this first attempt you can see that the background has been replaced perfectly, and the scooter has been removed. If you do not get a good solution, and the number of frames becomes high (this will take much longer to calculate) you can skip frames with the frame spacing setting.
Grab 6. Adjust number of frames until the fail marker disappears.
I experimented with masking the image later with the mid grey alpha, which also worked extremely well.
Grab 7. This is what blew me away. It took less than a second for this frame to be processed.
The next step was to key frame the bounding box across the shot, I created 4 or 5 keys, and only got a red fail marker at the very end of the clip. This could be adjusted out I am sure (after I read the manual), or perhaps 2 frames of paint would be required. Step through the clip, and animate the Rig region Bottom left (BL) and Top right (TR) parameters below.
Grab 8. Animate the bounding box, and key frame the Rig Region BL and TR parameters
Hitting go at this point, the Furnace rig removal tool had a think about the problem I gave it, the progress marker moved across pretty quickly, and then when it had what it needed it processed the remaining shot in less than a second per frame.
Grab 9. Finished rig removal showing the section behind the tree.
I had tried the bounding box default on a couple of other shots, and on some of these I did need to provide a mask to the rig removal tool. The ability to segment the mask into useable areas, and ignore areas (with the mid grey alpha) made some of the harder shots I threw at this tool work real well. If you find that additional motion (like people crossing over each other) are causing problems, use the mid grey feature !
Grab 10: Add a Bezier shape to the loader and create a shabby Bezier mask.
My next test I created a (shabby) Bezier spline around the scooter, and key framed it across the shot as before with the bounding box. I fed the alpha output of the loader, to the Rig Mask input of the rig removal tool. I hit go again, and the rig removal did an 80% job this time. If you look at the road below the scooter (in the grab below), you can see some errors with the background around the bottom right of the Bezier region, which flashed when the tool played back. This was caused by cutting the spline in too far under the scooter, and the Rig tool not having enough data around the removal to do a great job.
Grab 11. Mask problem on bottom right (just above the bottom of the shabby roto).
Adjusting the Bezier in the Loader view gave me an almost instant update in the rig removal tool, so flattening out the bottom of the mask showed an immediate fix to error introduced by my terrible Bezier mask.
Grab 12. Fixed the Bezier, and viewed the rig removal result.
The final step for the Bezier mask version of the rig removal was to flatten the Bezier spline across the original key frames. Using the Fusion Spline view I stepped through the original key frames and flattened the bottom of the Bezier at each key.
Grab 13. Jump to each key and flatten the Bezier mask at the bottom of the scooter.
Running the Furnace rig removal again with the fixed Bezier mask took the same amount of time (and gave the same result) as with the default bounding box. There are options to use the alpha and inverted alpha from the source image, as well as luminance and inverted luminance. The final result from the Bezier mask version is below, and again it provided a 98% solution, with just one or 2 frames at the end that required adjusting. In this shot there was also a glass covered yellow sign in the front of the church, which reflected the scooter as it went by, however this would be extremely fast to fix, and overall the few seconds it took to set up and analyze this shot, was magnitudes faster than other methods to remove this scooter.
I also made a before and after grab for you to take a look at here.
Finally from the Furnace manual, below are some settings, however in the test shots I tried I only needed to adjust the number of frames, and everything else was left at default with both the default bounding box and the Bezier mask version. This is one of 33 tools Furnace contains and I will do some other run through examples (depending on the flame level !).
Other settings in the Furnace Rig Removal tool.
Max Motion – defines the width of the border around the rig region in other frames.
Luminance Correct – Corrects for luminance changes from information taken from other frames. This is particularly important if the lighting changes throughout the sequence.
Blend Overlap – The repair is built up using slices of information from other frames in the sequence. These slices can be overlapped and blended to give a more natural looking repair. This parameter controls how much the regions overlap. Increasing this parameter too much will degrade image sharpness.
Filtering – sets the filtering quality.
Low – low quality but quick to render.
Medium – uses a bilinear filter. This gives good results and is quicker to render than high filtering.
High – uses a sinc filter to interpolate pixels giving a sharper repair.
Perspective – switch this on to correct for minor perspective changes.
Direction – sets whether to search forwards, backwards or in both directions to find missing data.
Both – searches before and after the current frame.
Forward – searches frames after the current frame.
Backward – searches frames before the current frame.
Fail Marker Alpha – Transparency of the red pixels used to show where the repair has failed.
Box – controls the size and shape of the rectangular area used in the repair.
