Optimize runs without Default_Ignorable's

Now that we have a buffer-wide scratch flags facility, use it to
optimize away a few passes.