The main difference you should focus on is that "to" always has a weak accent, whereas "two" and "too" have strong accents. So yes, we can hear the difference but it's a matter of stress; not the actual pronunciation of to/two/too, which is the same.