If anyone asks or gets suspicious I would just ask why?
Why can't an adult have a meaningful relationship with a kid?
Why can't we just be friends?
It doesn't mean that you have to be creepy or mean, just put up a tough face I'm sure most of us have learned to put on.

You know I have a relative I'm close too, like - we're related, he's not my YF or anyone I've ever even though about having sexual encounters with, and his uncle asked me what I meant when I sent a text telling him "I love you".

I could have said:
1) He needs to hear it
2) his parents suck, he needs to hear it
3) he lives in a terrible home with multiple bad influences, INCLUDING YOU, he needs to hear it.

The fact is, it's quite a silly question. If your neighbors like you at all, or have any common sense, unless they see you smack your YF's butt in public or here screaming and crying from your house/apartment whatever, they should just leave it be.

It doesn't make since to dial 911 every time a boy is with a man. Went off and digressed a little, but just be honest. He's a friend, a close friend who used to live where you used to live. It's not good for a boy to lose an adult figure; whether that's backed up by science or not (which it is).
He's your buddy, your pal. That's what it is. The fact that you find him attractive, or beautiful or whatever are private mental thoughts, not your cities'

Best of luck : )