You use "Inte jag heller." when you want to say "Me neither.". You can't use "mig varken" as it is almost only used in sentences like "Det säger mig varken bu eller bä." meaning "That doesn't tell me anything (I want to know)". It's more usual to say "Varken eller." meaning "(I don't want) either of those."