Yes, before you can set up the QoS to be in control of every single packet that is leaving the firewall, you will need to get rid of the router and replace it with a modem.
Otherwise you can only limit the bandwidth so that you are only allowing IPFire to pass 90-ish % of traffic and hope for the best that no other traffic is coming from anywhere else.