You're correct in stating that the का/के/की agree with the possessed thing. And you're correct in the fact that हाथ is masculine. Normally, it would have been भाईसाहब का हाथ (assuming it is only one hand). But because you have the word में after हाथ, the का turns to के. This rule is called 'Oblique Case' in Hindi. In Oblique Case, a sentence unit, if male, singular and ending in the '--a' sound, when used before a postposition, changes its sound to '--e'. For example, 1. भाईसाहब का हाथ (Brother's hand) changes to भाईसाहब के हाथ में (in the brother's hand). Here, में is the postposition. 2. पिछला हफ़्ता (Last week) changes to पिछले हफ़्ते में (in the last week). Here, में is the again the postposition; the word 'Pichhla' changes to 'Pichhle' because of the Oblique Case. 3. मेरा कुत्ता (My dog) changes to मेरे कुत्ते का (my dog's). Here, का is the postposition; the word 'Mera' changes to 'Mere' and 'Kutta' changes to 'Kutte'. 4. हिंदी भाषा (Hindi language) DOES NOT change with a postposition. For instance, हिंदी भाषा में (in the Hindi language). This is because even though 'Bhaasha' is placed before a postposition, even though it ends in '--a' sound, and even though it's singular, it is NOT masculine. Therefore, it does not follow the Oblique Case.

You got it right about का / के / की. They agree with the possessed like here with the hand. के is usually used to signify the plural form where as का and की are used for singular form for masculine and feminine respectively. To get a clear idea about these three and their usages, have a look at these links: https://en.wikibooks.org/wiki/Hindi_Lessons/Lesson_7 https://www.rocketlanguages.com/forum/hindi-grammar/ka-ki-or-ke/ Hope this helps!