It's anything distasteful which makes you react by putting the edges of of your teeth together. Like this! http://mychinaconnection.com/wp-content/uploads/2011/01/set-ones-teeth-on-edge-150x150.jpg

Something that irritates you can be said to "set your teeth on edge". It is a strong statement and indicates distaste.