1. Can you put your phone on vibrate - is correct. The other two request the listener to make their phone begin vibrating, right now.
2. Can you mute your phone - You would say this if the phone was making noise. You'd probably do like 'vibrate' 'could you set it on silent'. Mute is a word that is used to make a microphone not work. You would mute a call that is in progress, to make the microphone not work - so the caller could not hear you.