Wednesday, 15 December 2010

Three Pass Keying Tutorial

Before and After
After enjoying using KeyLight in Nuke to consolidate an edge matte, a core matte and a garbage matte with one node I decided to see if the same could be done in After Effects.  Unsurprisingly it is not quite as simple in AE but after reading the Foundry's user guide and a blog post from Mark Christiansen as well as watching a light wrap video tutorial by Jerzy Drozda Jr, I have found a workflow which meets my needs.

The footage I am using was captured directly from the EX1 to ProRes4444 using an AJA break out box for Rachel McLean's Scottish Arts Council project.  I now realise that 422 chroma sampling is all we were getting out of the camera but I'm sticking with 444 to avoid a rerender.

After Effects Flow Chart

This tutorial covers the whole three-pass keying process but doesn't discuss the principals of keying or KeyLight in any depth.  For a KeyLight tutorial, check out Andrew Kramer's video here.  A still frame of the footage is available here for anyone wishing to follow along.


Core Matte: The core matte is used to make sure there is no transparency in the core of the element.

Pull the footage into a new composition and apply KeyLight.  In KeyLight sample the 'Screen Colour' from your footage and use the Screen Matte controls to create an eroded matte while in the Screen Matte view.  Make sure that the core matte does not extend to the edges of the element and that the background has a consistent value of 0 and the foreground has a consistent 100% white value (I later increased 'Screen Softness' to a value of 6 to allow for better blending with the edge matte).  Finally set the view to Intermediate Result to allow the core matte to be passed through to the edge matte.

Generate Core Matte
Set Core Matte to Intermediate Result

Edge Matte: The edge matte is used to create a fine detail edge for the element which retains transparency.

Create another instance of KeyLight on the same footage, then go to Inside Mask and select 'Add to Inside Mask' under 'Source Alpha'.  This adds the core matte to the edge matte.  Then choose the same 'Screen Colour' as for the core matte and use the Screen Matte controls to create a fine detail edge matte without pushing the 'Clip Black' and 'Clip White' values too hard.  Some transparency values in the foreground an opacity values in the background are acceptable in this pass.  Finally set the view to Final Result.

Set Core Matte as Inside Mask of Edge Matte

Garbage Matte: The garbage matte is used to remove unwanted elements such as lights from the shot as well as avoiding the need for aggressive keying in the fine detail passes.

Pull another instance of the footage into the composition above the first instance and apply KeyLight.  Create a matte similar to the core matte by using the 'Clip Black' and 'Clip White' values aggressively but this time grow the matte beyond the edges of the element. Now draw a rough mask round the footage to exclude any unwanted elements and animate if required.  Finally set the view to final result and change the blending mode to 'Stencil Alpha'.

Create Garbage Matte

Set Blending Mode to Stencil Alpha

Spot Fix: Often pulling a matte from an unevenly lit greenscreen needs to be done in multiple passes to avoid resorting to an overly aggressive edge matte.  In this case the area at the elements head was significantly darker, requiring a spot fix.

First create a new composition with another instance of the footage.  Next add an adjustment layer and mask the area of the fix, next add a Color Key effect to the adjustment layer and sample the key colour of the offending area.  When refining the key, make sure not to remove any of the edge detail of the element.  Finally pull the spot fix comp into the original comp and apply a 'Stencil Alpha' blend mode above the keyed footage.

Spot Fix in New Composition

Set Blending Mode to Stencil Alpha

Matte Footage:  KeyLight is a terrific keyer but it applies operations to the RGB values of the footage as well as creating an alpha.  Although these operations can help compositing it is often preferable to apply things such as despill and light wrap separately.  To avoid any unintentional RGB operations the keyed footage can be used to matte another instance of the footage.

Create a new composition and pull in the original keyed element composition.  To reduce fringing add a 'Minimax' effect, set the operation to Minimum, Radius to 1 and Channel to Alpha.  Pull in another instance of the footage an place it below the keyed element composition.  Then set the 'TrkMat' (track matte) mode of the footage to Alpha Matte.

Apply Minimax to Keyed Element

Matte Footage with Keyed Element's Alpha

Background: The generic background for this example was generated with a radial 'Ramp' effect applied to a background.  This must be placed in it's own composition for the light wrap effect to recognise the ramp.

Radial Ramp Background Composition

Composite:  Once the element is successfully matted, operations must be made to the RGB of the element to match it to the background.  If premultiplication artifacts such as black haloing occur, the RGB operations should be applied to the footage before it is matted.
Place the matted element composition over the background composition, then apply a Spill Suppress effect setting the screen colour as the colour to suppress.  Next apply a Levels effect to match the Input Black, White and Gamma to the background.  If possible use the RGB read out in the info panel to accurately match elements, in this cases such as this it must be done by eye.

Composite Matted Element over Background

Suppress Colour Spill and Match Luminance

Light Wrap: Light wrap is required in most cases to simulate the light from the background hitting the edges of the element.  This recipe was taken from Jerzy Drozda Jr's video tutorial.

Add another instance of the matted element composition over the original.  Apply a 'Set Channels' effect and set the source RGB values to the background composition.  Apply a 'Channel Blur', blur the RGB channels by 10 and blur the alpha channel by around 200.  Invert the alpha channel by applying a 'Levels' effect and swapping the 'Output Black' and 'Output White' values.  Matte the result with the original alpha by applying 'CC Composite' and setting 'Composite Original' to Stencil Alpha.  Next duplicate the light wrap composition and change the Alpha Blurriness to a value of about 30 for a sharper light wrap.  Finally set the blend modes of the light wrap compositions to Add and reduce the opacity to taste.

Add Soft Light Wrap

Add Sharp Light Wrap

Match Grain:  The final step in matching the elements of the composition is matching the grain.

Add another instance of the footage below all the compositions then apply the 'Match Grain' effect to the background, set the 'Viewing Mode' to Final Output and the 'Noise Source Layer' to the footage layer below.

Match Grain


Final Result