'On average' is a phrase meaning 'generally' or 'in general'. Young people are generally taller now. In the second example, 'average' is just an adjective describing GPAs. 'In' doesn't connect to 'average'. It connects to 'increase'. There is an increase in GPAs, an increase in unemployment, an increase in global temperatures, etc.

You should use on average. On average means that something is generally at about that level.

Use "increase IN average" in situations where you want to describe an increase in the value of the average itself. For instance, you would say "assigning lots of homework resulted in an increase IN the average test scores for the class". In your example, something else is going on: "on average" is an adverbial clause that modifies the verb "increased". Its purpose is to specify the method whereby the "increase" is computed. It informs the reader that the number, 1 1/2 inches, signifies the average value of increase. However, you would use "in average" if you changed around the sentence to read "There was an increase in average height of 1 1/2 inches". Do they mean the same thing? Yes.

The others' answers are correct, but I still think another phrasing might help. In the examples above, we have "on average" used as a phase and "increase in average GPAs". Here, "on average," as a phrase means something like, "when considering averages of the numbers." In the second example, "increase in average...", "in average" is not a phrase and should not be considered meaningful when taken out of context. You should consider this phrase as [noun] + [preposition] + [noun]: "increase" + "in" + "average GPAs".