I believe this is not an idiom but an actual track, depending on the context of the story. It might be a race track for horses (where people bet on the horse winning the race), or maybe a track for people to run on. So, his father literally took him to some kind of track to do something there. Hope that makes sense regarding the text you're reading.

It definitely means they went to a horse racing track to watch the horses race. It's a common expression just for that type of activity.

I agree with Marileen. It's an actual track, for races of some kind. The father took his child out of school and spent the day at the track, perhaps watching horse-racing or car or bike racing. Or they may even have taken part. Whatever kind of track it was, the word is meant literally rather than idiomatically.