"by" implies and stresses that two places in question are side-by-side or next to each other (so I guess that also makes them near each other). So, you could say: the house is by the sea OR the house is near the sea. To reiterate, using "by" would stress that the house is probably directly facing the sea whilst using "near" does not. hope that helps!