Ray J suffered a setback in his legal battle with his ex, Kim Kardashian, and her mom, Kris Jenner ... the judge just shut down his attempt to get their defamation case against him tossed out of court ... TMZ has learned. According to court docs,…