Inhibition of return (IOR) refers to the slowing of response times (RTs) for stimuli repeated at previously inspected locations, as compared with novel ones. However, the exact processing stage(s) at which IOR occurs, and its nature across different response modalities, remain debated. We tested predictions on these issues originating from the FORTIOR model (fronto-parietal organization of response times in IOR; Seidel Malkinson & Bartolomeo, 2018), and from accumulator models such as LATER (linear approach to threshold with ergodic rate; Carpenter & Williams, 1995). We reanalysed RT data from a target-target IOR paradigm (Bourgeois et al., 2013a, 2013b) by using a Linear Ballistic Accumulator model, to test the predictions of FORTIOR, and specifically whether IOR could occur at early sensory/attentional stages of processing, or at later stages of decision and action selection. We considered the following conditions: manual or saccadic response modality, before or after TMS perturbation over four cortical regions. Results showed that the slope parameter best explained both manual and saccadic IOR, suggesting that in both response modalities IOR may result from slower accumulation of evidence for repeated locations. Additionally, TMS stimulation over the right intra-parietal sulcus (IPS) affected the model's goodness of fit, and the slope parameter best explained the perturbed manual IOR, suggesting that the right IPS might be involved in the accumulation of evidence towards a decision. TMS perturbation over regions that did not affect IOR, nonetheless significantly decreased the goodness of fit of manual IOR models in a nonspecific manner, suggesting that TMS increases intra-individual RT variability. These results support the hypothesis that IOR is a predominantly sensory/attentional phenomenon, and is modulated by activity in fronto-parietal networks.