In general there's a few disadvantages to using an eSIM. The primary one isn't related to eSIM as such, just the way the business model works for "travel" eSIMs.

In the majority of "travel" eSIMs are not based in the country that you're using them in, but instead from another country and then roam into the country you're in. For example, I was recently using an Airalo eSIM in Japan, however the eSIM itself was from Singtel in Singapore, and was roaming onto the Softbank network in Japan. This works fine, but the nature of mobile phone "roaming" is that data is almost always backhauled to the country the SIM is from - so all of my traffic in Japan was backhauled to Singapore before being released onto the internet, which will add to latency. Compare that to using a local SIM where the SIM will be for the local provider, so the traffic goes out locally. For Japan->Singapore that's likely not a big issue, but if you're in Japan and you're using an eSIM that's from a provider in Europe it's not going to be a great experience.

In some occasions the travel eSIM will be from a local provider in which case this isn't an issue. For example, Airalo's eSIMs in Thailand are for DTAC, a local Thai provider. So in that case you'll get the same experience regardless of whether you're using a DTAC physical SIM, a DTAC eSIM, or an Airalo eSIM.

Another disadvantage is that eSIM's are often data only, so if you need to make phone calls/SMS/etc you can't.

The other disadvantage of eSIMs is the one you've mentioned - you often can't move them between phones, or at least, not easily. This isn't really a technical limitation, but one that travel eSIM companies put in place likely to avoid people sharing eSIMs. How big a disadvantage that is depends on how often you flip between phones, but for a travel SIM likely isn't a big problem.

In general, the advantages of an eSIM will far outweigh those disadvantages - but obviously it depends on the person and the use case!