I think you won't be alone but yes, far less people than Machu Picchu. It's like going to a non visited segment of the Great Wall.

Probably the most known way to get there is thru Cachora, a quiet town, and it's a 30 km trekking. No roads, no trains, but that's why that makes this place more misterious and lonely.