What's "threatening" (though I am not TOO scared) in your viewpoint is that there's a scientific principle that a causal theory should feature identical causes yielding identical effects.

A theory that is claimed only to be true to explain one instance is something that formally can't be tested.

Therefore it isn't science.

Therefore, it's either a matter of faith, or just a notion. Notion sounds like a bad word, but art and literature tend to be quite notional.

So your idea about your father-son relationship could yield a good bit of autobiography or a play or a novel, but it can't yield any valid psychological science.

Such theories don't have to explain everyone who has a particular sexual orientation, since there may be multiple paths to developing a sexual orientation, but biology is not so complicated that there can be a separate path for each individual.

On the other hand, we all have unique childhood experiences with our parents. So if we rate the things they did as causal, we may well feel that we are the only one caused to be as we are.

Overall, this will tend to be true, but I don't think it will apply to the relatively simple process of the development of sexual orientation.

Sexual orientation really only has three variables: 1) whether we get an erection for M, F or both, 2) whether we erotically most treasure our partner's vulnerability or our own, and 3) what the best sexual symbol of vulnerability/genuineness might be for us (just being naked and aroused alone, or beyond that, being a member of a 'weaker' sex [a disappearing notion but once the mainstay of 'normal' sex life], being young, being very young, being in pain, being humiliated, being about to wet oneself, being urinated on, being tickled, exposing the bare feet, etc., etc.). [Oh yes, and the opposite side of the coin for variable number 3, erotic symbols of invulnerability/protectiveness, such as toughness, muscularity, commanding personality, wealth, fame, hirsuteness, uniform, boots and other heavy costume items, inflicting pain or humiliation, punitive or territorial urination, etc., plus the traditional notion of belonging to a more commanding, potentially violent sex.]

You have a gender or two, a power status or two, and some symbols of the latter and that basically wraps the erection business up.

There's a lot of leeway in the symbols but this situation isn't going to generate a different causal process, per se, for sexual orientation in each human individual.

In sum, you haven't found a theory that works for you, but rather, a story that works for you. When that's clear, the 'threat' is gone.