This randomized phase II trial studies how well veliparib or pembrolizumab work with combination chemotherapy and radiation therapy in treating patients with rectal cancer that has spread from where it started to nearby tissue or lymph nodes (locally advanced). Veliparib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Monoclonal antibodies, such as pembrolizumab, may interfere with the ability of tumor cells to grow and spread. Drugs used in chemotherapy, such as modified (m)FOLFOX6 regimen, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Radiation therapy uses high-energy x-rays to kill tumor cells and shrink tumors. Giving veliparib or pembrolizumab with combination chemotherapy and radiation therapy may kill more tumor cells, make the tumor smaller, and reduce the amount of normal tissue that needs to be removed.