Don't use it if the readability is not improved.
Using it has a fixed performance overhead, which is noticeable for operations that run on the order of nanoseconds.
performance is becoming less of an issue. For example https://github.com/golang/go/commit/f8b2314c563be4366f645536e8031a132cfdf3dd
Like any language feature, defer should be used when it's useful: it's the default and idiomatic way to implement the function-scoped RAII pattern in Go, and that property exists independent of the length of the function. And, like any language feature, performance should only become a consideration when profiling reveals it to be a problem.
Avoiding defer where it makes sense in deference to an ominous fixed performance overhead is pre- and micro-optimization. Don't do it.
For typical uses of defer (that is, closing resources), using defer is not an aesthetic decision - if there's a panic, you won't run the cleanup. Sure, defer has some overhead, but like Peter said, you should only worry about it when it's measurably causing problems. To discourage defer in "idiomatic" code is a mistake.
I agree with the points made above. This entry tried to say "use this language feature when it's appropriate; don't use it when it's not", which can be said for any language feature. I'll close this now (so it won't appear on the index page).