Currently, if "Allow pages to choose their own fonts, instead of your selections above" is disabled, all site fonts are overridden. This is of course, intended behavior, however, some site's don't play well with it, such as Google Docs and Office Online. It would be nice if there was some sort of list users could make which allows that site to load it's fonts.
A alternative solution might be to keep a list of sites to disable it for and automatically disable the override if it requires it.
I've done a quick mockup of my first idea which you can see as a attachment.