" What is this ? " most likely, you are pointing or staring at something and ask about it. " What is it ? " is often part of question, where "it" refer to something intangible Ex. What is it that makes you happy ?

As Ahmed said - " What is this?" you are looking at something and asking about it. 'What is it' could be part of a question, or can just be a question on its own, in which case it is similar to 'What is the matter?', 'What's wrong?' or 'What's up?', or it could be exactly the same as 'What is this?'. You need context to know what someone means by it. As a reply, either is OK, but 'It is...' would usually be more natural. If someone stressed the 'this' in the question, in an accusatory way, you could also use 'this' in the reply, and stress it as well, to mock their tone of voice.