Weird, the first time loaded the char set and set that mode it worked as I expected, but then didn't afterwards.
I guess it's a bit unintuitive for me, setting MCM mode in my brain would disaplay all chars in multicolor mode. There is no color information in a character set so that just seems weird to me. I get it though because of how the c64 works with how color memory is set. It'll just take some getting used to.