Hailey, you have asked a very good question.
I have never heard of a rule for the use of these 2 phrases.
It's better in this way. IT'S MUCH MORE COMMON TO SAY "IT'S BETTER THIS WAY".
You're going about it in the right way = YOU'RE GOING ABOUT IT THE RIGHT WAY.
Why do I feel this way? CORRECT, AND WE WOULD NOT SAY "WHY DO I FEEL IN THIS WAY".
I think English speakers say "this way" more than 95% of the time.
Sometimes, but not very often, we will use "in this way" to describe a method or technique or practice (a habit or how we do something), for example:
- We should work harder. In this way (means by following this practice) we will be more productive.
- I should exercise more often. In this way (means 'by exercising') I would improve my health.
- You should have an anti-virus program on your computer. In this way (means 'by following this advice) you will help to keep your computer safe.
Now that I've written my 3 examples, I notice that "in this way" starts the second sentence and the word "way" refers back to the topic or subject of the first sentence.
I hope I haven't confused you :)