The recent Spectre attacks exploit speculative execution, a pervasively used feature of modern microprocessors, to allow the exfiltration of sensitive data across protection boundaries. In this paper, we introduce a new Spectre-class attack that we call SpectreRSB. In particular, rather than exploiting the branch predictor unit, SpectreRSB exploits the return stack buffer (RSB), a common predictor structure in modern CPUs used to predict return addresses. We show that both local attacks (within the same process such as Spectre 1) and attacks on SGX are possible by constructing proof of concept attacks. We also analyze additional types of the attack on the kernel or across address spaces and show that under some practical and widely used conditions they are possible. Importantly, none of the known defenses including Retpoline and Intel’s microcode patches stop all SpectreRSB attacks. We believe that future system developers should be aware of this vulnerability and consider it in developing defenses against speculation attacks. In particular, on Core-i7 Skylake and newer processors (but not on Intel’s Xeon processor line), a patch called RSB refilling is used to address a vulnerability when the RSB underfills; this defense interferes with SpectreRSB’s ability to launch attacks that switch into the kernel. We recommend that this patch should be used on all machines to protect against SpectreRSB.