Coffee consumption seems generally safe within usual levels of intake, with summary estimates indicating largest risk reduction for various health outcomes at three to four cups a day, and more likely to benefit health than harm. Robust randomised controlled trials are needed to understand whether the observed associations are causal. Importantly, outside of pregnancy, existing evidence suggests that coffee could be tested as an intervention without significant risk of causing harm.
Upshot: the risks associated with coffee have, in general, been misunderstood and over-estimated.
Your particular situation may warrant more caution, as mine does, but, for most people 3-4 cups does more good than harm.