Could is grammatically proper for a request, although the preferred formal way would be "May I look at the book?" either one is proper.
Can is used to express ability to do something. Technically, asking "can I look at the book?" is asking someone if you are physically able to look at it.
However, please note that only English / grammar teachers would be likely to say anything about using "can" in this case, and that most native speakers will use can in this situation. Could or may would be more appropriate in formal situations, as they sound more respectful, and a little old-fashioned.
But the short answer is, you could use any of those in any case, realistically (unless you're writing University essays, or business papers where proper grammar is very important, then using could / may would be preferred).