Searching for books tagged as “challenging”, I’m seeing it used both of those ways (and the former has two distinct subcategories, “experimental” and “dense and complex”, though of course some books are both). Which is kind of an inherent problem with user-applied tags, I think: individual users are going to apply them according to their understanding of the meaning, and there might not be as much consensus on a narrow and specific definition as you would hope there would be when using them to search.
So it’s not a fully avoidable problem, IMO, but if we were to brainstorm a pool of mood tags to use on IFDB, I would certainly support having, say, “experimental”, “complex”, and “discomforting” in there separately.