Depends if you want to see the tuna auction or not. I heard from others that attended the 1st annual FT Japan Do in March who reported showing up at Tsukiji around 4AM only to find all the spots for the tuna auction already assigned. I opted out of the middle of the night fun and went later, around 8 AM, and walked around the market for an hour or two and enjoyed the experience before taking in the cherry blossoms at the adjacent Hama Rikyu park.

BTW I'd use the forum search using "Tsukiji" for past threads on visiting the market.