I've got family in Brazil and travel there about every other year to visit them. Every trip I have brought coffee back. I wouldn't worry about being singled out for any sort of additional questioning or inspection. Neither myself or any family have every had any issues bringing coffee from Brazil. Enjoy your trip!