Did you remove the metal retainer and also the signal arm. You will also have to loosen up the metal cover plate further down the column to pull the wires out with the switch.