TThis phase II trial tests whether relugolix and radiation therapy works to shrink tumors in patients with prostate cancer that has spread in a limited way to 1 to 5 other parts of the body (oligometastatic). Testosterone can cause the growth of prostate cancer cells. Relugolix lowers the amount of testosterone made by the body. This may help stop the growth of tumor cells that need testosterone to grow. Giving relugolix with radiation therapy may help lower the chance of prostate cancer growing or spreading.