If one is too austere or too sensuous one cannot control the mind.
Although one is seriously engaging the mind in Kṛṣṇa consciousness, the mind is so flickering that it may suddenly be deviated from its spiritual position. One should then carefully bring the mind back under the control of the self. It is stated in Bhagavad-gītā that if one is too austere or too sensuous one cannot control the mind. Sometimes one may bring the mind under control by allowing the material senses limited satisfaction. For example, although one may eat austerely, from time to time one may accept a reasonable amount of mahā-prasādam, opulent foods offered to the temple Deities, so that the mind will not become disturbed. Similarly, one may occasionally relax with other transcendentalists through joking, swimming and so forth. But if such activities are performed excessively, they will lead to a setback in spiritual life. When the mind desires sinful gratification such as illicit sex or intoxication, one must simply tolerate the mind’s foolishness and by strenuous effort push on with Kṛṣṇa consciousness. Then the waves of illusion will soon subside, and the path of advancement will again open wide.
Source: A.C. Bhaktivedanta Swami Prabhupada (2014 edition), “Srimad Bhagavatam”, Eleventh Canto, Chapter 20 – Text 19.
