When modifying the toolbar in Firefox you are not allowed to get rid of the forward and back buttons. When you drag them out, they jump back. This seems like quite an arbitrary decision, since you are allowed to remove and shuffle around most other elements of that toolbar.
I only use the forward and backward keys on my mouse, so the back/forward buttons on the toolbar serve no purpose. The only thing they do is that I occasionally accidentally click on them when I'm in a hurry and trying to click a (pinned) tab, to great annoyance. I wanted to solve that problem by removing them altogether and found out that you are not allowed to.