Don't use it if the readability is not improved.
Using it has a fixed performance overhead, which is noticeable for operations that run on the order of nanoseconds.
performance is becoming less of an issue. For example https://github.com/golang/go/commit/f8b2314c563be4366f645536e8031a132cfdf3dd
Like any language feature, defer should be used when it's useful: it's the default and idiomatic way to implement the function-scoped RAII pattern in Go, and that property exists independent of the length of the function. And, like any language feature, performance should only become a consideration when profiling reveals it to be a problem.
Avoiding defer where it makes sense in deference to an ominous fixed performance overhead is pre- and micro-optimization. Don't do it.