You may accidentally add an "extra" player you didn't want when you pressed a button on your controller or mouse. 

To remove this extra player, hold the Back / Give up button until the extra player's cursor disappears.

If the unwanted device is connected as Player 1, this will bring you back to the game's main menu. When you see Start displayed, make sure to use the device you want so it gets set as Player 1.