There is existing passenger-rail track from LA out to Victorville, it is just not "high speed rail" track. PAX can get on the train at LA 's Union Station, from which it will run at 80 mph out to Victorville (on the existing track), then switch onto the high-speed track and run the rest of the way (Victorville to LV) at 185. Even with the slower-speed track from LA to Victorville, the travel time from LA to LV will only be 2 hours or so.

BTW, this is exactly the approach the French took with their TGV -- they started with high speed rail segments out in the country, while retaining the normal speed rail closer in to the cities.