Why does my Palette use the wrong color on some layers?

For some reason my Palette 3 Pro is selecting the wrong color when its prints it's middle layers. It doesn't appear to be from a miss calculated measurement (meaning the distance from the Palette to the nozzle.) The color changes seem to happen at the correct times but it isn't the right color. The first layer goes on correctly and the last layer does, but the middle seems to have random other colors for the print, for both the wall and infill layers. Am I missing a setting? Or has anyone else experienced this issue? I wouldn't mind if it was only the infill because it wouldn't necessarily be seen, but since it's the walls also, it doesn't look good. I have it connected to an Elegoo Neptune 2S.