"With that being said" (also, "that being said", "that said", "saying that", "having said that") is used when you are about to say something that contradicts or is an exception to something you've just said. For example, "I love the UK and have been there many times. That being said, there are a few parts that aren't very pleasant".