Hide only second atrtibute and set always visible the first attribute

We have two attributes for variations: Size and Colour.
If I select “hide out of stock atributte” we have a problem: It is not possible to set Size visible and hide just Colours.
Example:
If a client select “Small and then Red colour” and we dont have Large size in Red colour that user cant go back to “large” and see what colours are avalible in that size because Large was hidden. That user has to change the colour to whetever is avalible in Large size and then he could select again “Large” size and see what are the avalible colours.
That user has to guess what colour has Large size to see what are Large variations avalible.
Is it possible just to hide “colours” and keep “size” always visible?
I mean hide when “Colours are out of stock” and keep size always visible so our client could navigate more fluid