If it is the speed from your IP, you’re going to have problems as the router is going to maintain some bandwidth in reserve for any additional connections that may come on. This can be changed on higher end or consumer friendly routers where they give you plenty of expert options to turn off unneeded connections