Both are possible, but the meanings would be slightly different. The sentence with the infinitive would be best translated as "nodoby whom to talk to". It has the same meaning and it does not change the infinitive of the auxiliary verb "poder".

The easiest answer is that it's used the same way in English if you break it down. nadie con quien poder hablar = "No one (with whom) to be able to speak"

En tu frase "poder" es la palabra correcta porque se encuentra en su forma base, si tú dijeras "pude" estarías hablando en tiempo pasado.