Hello Addison,
Yeh the difference is exactly what you mentioned:
熱い means hot .When referring to physical heat like hot water or hot weather it means that the thing referred to has higher than desirable temperature, giving off heat or causing a sensation of heat and burning.
暖かい means warm .When the climate ,the body or water is warm it means it has or produces a comfortable and agreeable degree of heat.