The Quality of the Observer

In class we discussed the state of the observer as whether or not they are able to decide what is good art. Some of the examples given were the chemically impaired, the very young, and the in-love. I brought up a few examples of chemically impaired artists whom we consider great, and the hypothetical of a young person who knows a lot about art. This was countered by saying that while there are anecdotal instances of these exceptions to the rules, generally people who fall into these categories are not qualified to determine good art.
Is this fair? Where do we draw the line between someone who is too intoxicated to appreciate art? If we are basing on definition of art on what is popularly accepted aren't we committing a logical fallacy? I believe that we are cutting out huge portions of the art/observer community by excluding people deemed too young, too old, too drunk, too high or simply too stressed out to truly appreciate art.
What do you think? Is there really a group of people who are qualified to judge all art?