The "downside" of course assumes in this case that the stand out card is specifically not a creature (or land I suppose), which they often are anyways.
You can also put a card in the GY that activates other cards in ways that drawing them can't do (putting spell in GY for mizzix's mastery, putting anything in GY to be a target for Eternal Witness, activating delirium, etc.) AKA the self-milling often has real upsides that make the potential 'downside' either really minimal or a symptom of being included in a deck it shouldn't be in.
You can also put a card in the GY that activates other cards in ways that drawing them can't do (putting spell in GY for mizzix's mastery, putting anything in GY to be a target for Eternal Witness, activating delirium, etc.) AKA the self-milling often has real upsides that make the potential 'downside' either really minimal or a symptom of being included in a deck it shouldn't be in.