Yes, the "adverb of time" usually comes at the end of the phrase. This is where it naturally sits in English.

Be careful of putting your first-language rules (ie. Japanese grammar) on top of English grammar. This is one of the main reasons learners make mistakes in their target language.

However, yes you can put the adverb of time at the beginning of the phrase. Traditionally, you would add a comma after the adverb to show you've cut it from the end of the sentence and moved it to the front. However, if the meaning is clear then there's no real need for a comma.  The meaning of the sentence stays the same.

In your first sentence, it would be OK and feel equally natural to say: 

"Yesterday, I went to a park."

In the second you could also say: 

"The weather report says that, tomorrow morning, there will be rain / it will rain."

So you could keep your habits from Japanese in many cases.  

Thanks  a lot, Michel and Peachey.

I got it!

I went to the park yesterday. 

Yesterday, I went to the park.

Both are fine.

More examples:

I've been working hard lately/ Lately, I've been working hard.  I'm going swimming next week/ Next week, I'm going swimming.

I've always got up late in the mornings/ In the mornings, I've always got up late.   I eat pizza on Fridays/ On Fridays, I eat pizza.

In poetry they sometimes put it in the middle. "The tree, this summer, will blossom!" So the time phrase can go in the middle - but it usually sounds weird unless you have a very good reason for putting it there. Mind you, I might say, "I really am going, this summer, to diet!"  Or, "I really am going to, this summer, diet!" But I'm saying it with lots of emphasis, and it sounds weird unless you know all about my attempts to diet, my failures, and my frustration. And you need to hear my voice. BUT I WOULDN'T PUT IT IN THE MIDDLE IN AN EXAM.

This is on the DISCUSSION page, so I'm discussing it! Is there anybody else out there who gets poetic sometimes?