I had to watch this passage several times - the way the sentence is split, the fact it is written and the lack of punctuation does not help.
Here is the whole sentence with two commas I've added to clear the confusion:
'I will later blame you for not being who I imagined, without checking, you were'.
Or to put it differently and further clarify the message (as I understand it): 'I will later blame you for not being who I imagined you were, even though I didn't check it with you first/without checking it with your first.'
I hope this helps