Subtract from selection or add to selection

When I am trying to lock a bunch of items down within a frame but leave post-it notes unlocked, it would be nice if I could hold down a key and get a selection subtraction to remove items. Similar to Photoshop subtract/add from mask.