The concern is that, if we doubt the experts, many people will draw on cherry-picked facts and self-serving anecdotes to furnish their own versions of reality.
This is certainly the case. But, in fixating on this danger, we neglect an important truth: it is simply not feasible to outsource to experts all of our epistemological work — nor would it be desirable. We frequently have no alternative but to choose for ourselves which beliefs to accept. The failure to come to grips with this fact has left us without the kinds of strategies and tools that would enable non-experts to make more effective use of the increasingly opaque theories that explain our world.