In RP - the standard, textbook or 'BBC' pronunciation of the south of England - the lips are relaxed for the /a:/ sound. It is a very 'open' sound, with no tension in the lips.
/a:/ is an open, back, unrounded vowel. 'Open' tells you that the mouth is open and the tongue is low, 'back' tells you that the tongue is a long way back in your mouth, and 'unrounded' tells you that the lips are relaxed.
The rounded version is /ɒ/ . If you round your lips, it will sound more like the /ɔ:/ sound in 'core' rather than 'car'.
[British English speaker/ RP]