I wouldn't make a blanket generalization like this. For instance, sour cream is cream that's fermented with specific bacteria. If other types of bacteria start to grow in it, it will go bad, which is why sour cream comes with an expiration date. Similarly, blue cheeses contain specific molds that are safe to eat, but if your cheddar starts growing mold, that is something you shouldn't eat. So some of what grows in your milk may be safe, some of it may not. Unless you know for sure, it's safest to toss it.