During the early 2000s, I saw some document prepared by Air France comparing the operating costs of the 777 against the 747-400 on a per seat basis. The difference was HUGE in favour of the 777.
My guess is that the GE90 class of engines, despite being older than GE-nx still outperform economically because you only need 2 engines. If they were to grow the GE-90s to allow a 747 to be twin engined, this could change the economics quite a bit.
With regards to market for VLA:
While a number of US carriers have long ago shunned the 747, one needs to consider that none of those carriers exist in their original form. Delta has merged with Northwest which has trans-pac routes where 747 are still viable. United has merged with Continental, and UA has routes where a 747 is valid. AA and US may not have an affinity for the 747 but...
With merged airlines becoming behemoths, one needs to consider if main routes between large cities might benefit from a VLA when you combine traffic from two merged airlines. This is especially true for cities where concessions in landing slots had to be made to allow the merger.
During the 787 propaganda era, Boeing was all about point to point, smaller planes for long thin routes. But the actual trend says otherwise: planes are growing, not shrinking.
The 787-8 is bigger than the 767 it replaces. Yet, most orders went for the 787-9 which is even bigger. Airlines look at the per seat cost, and math is quite harsh for this: stretch an aircraft and per seat cost goes down. In fact. Airbus scored a lot of 350 sales because its first variant was bigger than Boeing first 787 variant.
And meanwhile, the A330 has basically cannabalised remaining 767 sales, again, as a bigger aircraft.
Bombardier long ago stopped the 50 seat CRJ. Stretched CRJs still produced and Bombardier now after the 100-130 seat market.
Same with 777. Airlines no longer going for the -200, they want the bigger ones. And Boeing is stretching the 777 again.
The old argument was that people valued price of flight over comfort and food. Airlines did away with all nice stuff on flights. Heck, we even have to pay for luggage now.
Then, airlines stopped (or reduced) aircraft waste by reducing parked time at a terminal (which not only increases aircraft usage, but also reduces gate costs because you need fewer gates). May not look as good in CRS when the connection time is longer, but airlines save a bundle. So airlines did away with nicer looking schedules on paper to save a huge amount of money.
What is next ? The next step is to grow aircraft and remove unnecessary flght duplications and reduce frequencies to a reasonable level. And that means larger aircraft.
The 777X is already beyond size limits and needs folding wings.
So, in the future, perhaps the need for VLA will become more apparent. Whether Boeing recycles the 747 with a new 747-9, or whether it starts from scratch, I don't know. But I think the 747-8 is the last of teh 4 engined 747 and if the 747 is to survive beyond the -8, it will be as a twin engined aircraft.
And I am pretty sure Airbus would also love to be able to make a twin engine A380.