Nothing to do with atheism, it’s a “southern” saying that just means getting really angry, akin to “losing one’s rag”, “being at the end of one’s tether”…
I arranged this in E minor, forgotten what the original key is, to be honest…Am, is it? There are some lovely “misheard” lyrics in this song, but given that Michael Stipe from the earliest days of REM was reluctant to enunciate and often ad libbed random words that’s perhaps not a surprise, although this song is perhaps the most lucid in terms of hearing the words albeit least understood in terms of the meaning…not that I claim to know. He’d probably tell you it was about the Exxon Valdez disaster, which he “discussed” at length on the Green tour (last time I saw them live). Anyway, waffled out now…