Testosterone, estrogen, and other hormones are produced at various sites throughout the body (not just the gonads), circulate throughout the bloodstream, are received by cells throughout the body, and ultimately get degraded or reprocessed into other molecules.
Cells throughout the body are continuously looking for hormonal signals. Everything from the brain, muscles, bone, fat, breasts, reproductive organs, and even blood vessels constantly receive hormonal signals. These signals influence how tissues grow, change in response to stimuli, and maintain homeostasis. Hormone signaling even has feedback loops, and can change how hormones are produced or received by cells over time. Incorrect hormonal signals can cause psychiatric symptoms, cardiovascular problems, brittle bones, and other health issues throughout the body.
Cells watch for hormones by producing a variety of receptor proteins. Some of these receptors are highly specific for one hormone, while others are promiscuous. You can guess, based on their names, that the Androgen Receptor (AR, a.k.a. gene NR3C4) preferentially detects androgens like testosterone, that Estrogen Receptors Alpha and Beta (ERɑ and ERβ, a.k.a. genes NR3A1 and NR3A2) prefer estrogens, and that the Progesterone Receptor (PR, a.k.a. gene NR3C3) prefers progesterone. Other receptors for these hormones include GPER, GPRC6A, ER-X, OXER1, ZIP9, and PAQRs.
Cell types across the body differ in the types of hormone receptors they express. This is true even when comparing the main Estrogen Receptors ERɑ and ERβ. Both ERɑ and ERβ are expressed in the central nervous system, the cardiovascular system, the ovaries, and adipose tissue. However, ERɑ is more prominent in mammary glands, the uterus, and bone, whereas ERβ plays a larger role in the bladder, the colon, and the immune system (Paterni et al, Steroids, 2014).
The differences between tissues in how they express hormone receptors, and how they interpret signals, help explain why hormones have such complex and far-ranging effects. Some parts of the body can grow while others shrink, and some seemingly-unrelated physiological processes may change, when HRT is applied. At the end of the day, some HRT drugs will be more appropriate (in terms of both effectiveness and safety) to use in a given situation than others. And, as with many things in medicine, it is genuinely possible for doctors to be unsure as to which specific treatment would be optimal for you at first.