It sounds best to use "would have been," because you're talking about why you didn't do something in the past.
In the present, it's probably still hard to go there without a car, but since you're not trying to go there now, the fact that it's hard now is irrelevant. The relevant time is in the past. "It would have been hard" explains your past action. If you say "it would be hard," it sounds like you're talking about a possible future action.