In my opinion they would often be used in different contexts.
To "reach new heights", tends to have a more grand and poetic meaning, often having to do with people and their hopes. "She always wanted to reach new heights." this could describe an ambitious person.
To "reach a new high" is less poetic and more straightforward/literal. Perhaps a number on a graph has reached a new high. "The number of people supporting this idea has reached a new high".
I'm sure there is some crossover, but in general this is how I think of it.
Hope this helps!