I can't speak to all our future plans, but at this point, we will at least be supporting Windows 7 and Windows 8 to at least the 3Q of next year via the Firefox ESR.
So although everything says private browsing in the UI, it is not private browsing. If you close and reopen, you'll see things in the history.The fact that you see any private browsing UI at all is an issue, though, I'll get a bug opened for that.