The aim of this study was to examine how androgen deprivation therapy (ADT) may influence the perceptions of 14 men with prostate cancer regarding their quality of life and physical activity post-diagnosis and to the potential benefits and risks associated with increasing physical activity levels. To address these questions, two focus groups were conducted, with these separated into ADT (n = 6) and non-ADT (n = 8) groups. The probe questions used assessed the link between quality of life and physical activity as well as the benefits and risks associated with increasing levels of physical activity. Data was transcribed verbatim and themes identified using a general inductive thematic approach. The primary themes identified were sexual health, “plumbing” and non-urogenital side-effects, return to and increased levels of physical activity post-diagnosis, physical health and psychological benefits of physical activity as well as over-doing it and age-related risks of excessive physical activity. However, not all themes were present in both the ADT and non-ADT groups. These results further highlight the link between physical activity and quality of life in prostate cancer survivors and how each group may use physical activity as a part of their survivorship process. Of particular interest was how several men on ADT used resistance training to counteract some of the ADT-related side-effects affecting their perceived maleness and potency. As the benefits of physical activity for men with prostate cancer are becoming better understood, cancer clinicians and service providers may need to focus more of their counseling on these issues, especially for those men on ADT.