Yes, you are right, different fruits have different 'things' inside them, depending on the size.
If it's large and hard, we can call it a stone (GB) or a pit (US). You'd use these words for peaches, plums, cherries, dates and so on. If it's small, you'd call it a seed (mainly US) or a pip (mainly GB) . Apples, for example, have seeds/ pips.