I found the neighborhood around the CY to be fine and I travel there frequently. You are in the thick of things. There are plenty of restaurants and shops where Whalley Ave and Dixwell Ave converge. If you are used to staying in the suburbs, then you may find it unsafe or terribly sketchy. I am a woman who travels alone, and I don’t find it sketchy at all. It is located right near many Yale buildings, so it isn’t exactly smack in the middle of public housing. If urban areas make you nervous, then yes, you will find it “unsafe.” If you have basic street smarts, you will be just fine.
New Haven has some great restaurants and some of the best pizza on the planet. I love going there.