Yes, there is a risk that some persons would consider that offensive. A lot depends on context, like whether the person in question is your friend and you know he is not offended by it. But walking up to a Latino stranger, while in an English-speaking area, and calling him “amigo” might be regarded as offensive either because (i) it places inappropriate emphasis on his ethnicity, or (ii) it assumes, without evidence other than his appearance, that he is a Spanish speaker.