A coast is anywhere the land meets a large body of water. The entire western and eastern sides of the United States can be called coasts. A bay is part of a major body of water that is set off from the main body and mostly surrounded by land. The Bay of Fundy is a very good example: http://encarta.msn.com/map_701510917/Bay_of_Fundy.html Note that in a bay, the line where the land meets the water would still be called a coast.

We can also refer to the COASTLINE as a series of BAYS and headlands. The headland is the part that sticks out into the sea because it is made of hard rock and thus harder to erode. The bay is formed where the rock was softer and the sea eroded it away to form a (normally) curved area with a beach (sand/stone). This area is the bay.