When addressbar was redesigned, the main annoyance was (and still is) that hitting Esc no matter how many times only restored original address, but didn't "unfocus" the address bar. The only way to close it now is to find some empty space on the page (not even toolbar) to click or press F6 again.
While it's good that I can restore original address and close it from keyboard on F6 (not really intuitive), Esc remains the button for cancelling actions and closing modals/popups. It feels wrong, when it doesn't quite work like that on address bar.
If it helps - F6 only cycles through "panes" or frames. Alt+D (on Windows) is the definitive address bar shortcut (same as Chrome/Edge).
But yep I agree that hitting Esc should return you back to your browsing/focused section.
Eh, clicking on the webpage is more intuitive to me, but whatever.