There are few Photoshop tools more powerful and confounding than layer masks. They’re powerful, of course, because they allow for non-destructive editing in layered image files. But they can be confounding, too, when trying to create a detailed mask of a highly complex image element. It’s tiny little edge details that make masks difficult. Flyaway hair, for instance, or foliage and tree branches. These finely detailed subjects can be especially tricky to accurately select and mask. There are countless tools that help refine the edges of selections and masks, but even they can come up short on occasion. There’s one approach, however, that is both a shortcut and a better method for making detailed selections. Instead of hand-painting a mask’s edge, pixel by pixel, this technique allows you to make a fast selection and refine it with a slider. What is this wonder tool? It’s the combination of channels and levels. Here’s how they come together to make easy and accurate selections and masks of even the most complex, finely detailed subjects.
To get started, we need to select just one channel and use the tonal values in it to create the selection. Open the Channels palette and click on any individual channel—red, green or blue. The image will turn grayscale (because you’re just looking at the luminance values for the individual color of the channel you’ve selected). On the blue channel, dark gray and black tones represent deep blue colors, while light grays and whites are very bright and pale blues. Simply click from one channel to another, looking for the channel that shows the highest contrast between the subject you’d like to select and the background. Are you starting to get the idea about how this works? By isolating a single channel, you can make it easier for Photoshop to more accurately identify even the finest edge details of the subject.
Once you’ve found the ideal channel, click and drag that channel to the New Channel icon at the bottom of the Channels palette to duplicate the channel. This way you can work on it without ruining the colors in the original image.
Next, use the Levels adjustment controls to increase the contrast even further, ensuring bright background elements become lighter and closer to pure white, while dark subject elements become darker, closer to black. (It doesn’t so much matter which is white and which is black at this point, so long as the contrast between what you want to select and what you don’t is high.) The middle gray point button on the Levels adjustment slider helps with this too, but you’re primarily going to be dragging the right side slider to bring down the white point and the left side slider to bring up the black point—effectively making a high-contrast black and white image that becomes even easier to identify edge transitions and make for an accurate, highly detailed selection. Ideally, the closer you get to pure black and pure white tones dominating the image, the better. But simply dragging either slider to the extreme will end up growing the selection too much and ultimately defeating the purpose and rendering unnecessarily jagged edges rather than more refined selections. Instead, consider going lighter with the contrast and then refining it after the next step by using a lasso tool to fine tune what areas are included in the selection.
With a now high-contrast black and white image in front of you, you may notice it reminds you of something already: a layer mask. But to actually turn it into a layer mask you’ll want to control-click (or command-click on a Mac) on the channel icon in the channels palette to make a selection that includes every white pixel in the image. (If that’s backwards for you, simply invert the channel prior to making the selection, or inverse the selection after making it.) Now you can turn off that channel’s visibility, then go back and click on the RGB composite channel to show the image once again in full color. Now click the Add Layer Mask icon to create a mask based on the selection—or even drag the layer to the New Layer icon in the layer’s palette to generate a new layer with the selected area masked, all in one click.
Another way to use levels to refine masks is to create the mask how you normally would—say, using the Select Subject control. You can then refine the edges as best you can with the Select and Mask or Refine Edge tools. Then you can click on the mask icon in the layers palette and use the same levels adjustments to expand or contract the black and white areas of the mask. As you do this you’ll see the edge expand and contract in the image as you drag the shadow and highlight sliders.
Because masks themselves are black and white—black showing opacity on the mask, white allowing the details behind to show through—you can understand how the levels control effectively expands and contracts the edges of the mask, allowing it to grow and incorporate more details or shrink to eliminate some of the fine halos or fog that can occur with wispy edges like feathers, hair and foliage. All told, it’s a much simpler approach than trying to select and paint the edges of a complex mask, and it can be done in a what-you-see-is-what-you-get manner as well. It’s a simple technique that’s also incredibly effective.