I mean, I will absolutely forget. but then I'll ask my opponent, and if we both forget we either count the cards (difficult) or shrug and say "yeah it was probably you/probably not you"
(also it's only hard to remember game one as long as everyone chooses to play, which does help the memory...