I can see that. If it helps, I think taking a piece of paper and starting from scratch by putting the cards you truly want in helps a lot. It helped me shed a lot of filler and my cube is far more typical than yours.
I think it can be done but it will be difficult at first since, like you say...