Groups, Rings, and Fields Without the Pain
I used to skip this chapter. Groups, rings, fields. The unsexy stuff. Then RSA showed up and I was lost. Same with elliptic curve crypto. Computational Number Theory and Modern Cryptography by Song Y. Yan (ISBN 978-1-118-18858-3) starts Chapter 2 with this exact trap.