Sure, you can use either phrase. In terms of a library, they basically mean the same thing. "Checking out" is really just describing the actual process of showing your library card and putting your name in the library records for borrowing the book.

That makes perfect sense :) I personally usually talk about borrowing books from the library, rather than checking them out.

To me, "check out a book" sounds like bad English, but maybe it is acceptable in American English? As far as I know, we usually "borrow books".

As far as I've used it, it's actually preferable to say "borrow a borrow a book from the library". Maybe there are other speakers who use "check out a book".

Yes, you can. It still makes sense.