It just grows more distant as time goes on and you grow apart. Even when you have other YFs, his memory will still bring up emotions.

At least assuming your feelings are similar to my own.