If you read up on the history of business class, it started out as a marginal improvement to economy class, to cater to business travelers flying with full-fare economy class tickets. First class was meant as a luxury, whereas business class was meant to give business travelers more comfort.
However, as time went on, business class started getting fancier, that the seats surpassed the first class seats way back when. But, the level of service hasn't. And, a lot of carriers have jettisoned the first class service or limited it to only certain routes.
So, pretty much, we're back to a 2 class standard. With premium economy, if it gets too fancy, business travelers would end up flying that, abandoning business class. And, then the result would be a new cycle starting again.
So in a nutshell, Y has to remain lousy and inferior so that business travelers will avoid it and stay in business class.