"The DIRECTIVE 2004/58/EC which clearly says, that: 5. The right of all Union citizens to move and reside freely within the territory of the Member States should, if it is to be exercised under objective conditions of freedom and dignity, be also granted to their family members, irrespective of nationality. (I don't see any mentioning about visa)."
Moldova is not a territory inside of EU, therefore that directive does not apply on your case.