The miserable or pleasurable reactions of karma ultimately act upon the false ego.
The material body is composed of earth, water, fire and air, just like bricks, stones and other objects. Our consciousness, falsely absorbed in the body, experiences happiness and distress, and fruitive work (karma) is performed when we falsely consider ourselves to be the enjoyers of the material world. False ego is thus the illusory combination within our minds of the self and the body, which are actually two separate objects. Since karma, or material work, is based on illusory consciousness, these activities are also illusory and have no factual basis in either the body or the soul. When a conditioned soul falsely considers himself to be the body, and consequently the enjoyer of the material world, he tries to find pleasure in illicit connection with women. Such sinful activity is based on his false concept of being the body and thus the enjoyer of women and of the world. Since he is not the body, his activity of enjoying a woman does not actually exist. There is merely the interaction of two machines, namely the two bodies, and the interaction of the illusory consciousness of the man and woman. The sensation of illicit sex occurs within the material body and is falsely assimilated by the false ego as its own experience. Thus the miserable or pleasurable reactions of karma ultimately act upon the false ego and not upon the body, which is composed of dull matter, nor upon the soul, which has nothing to do with matter. False ego is the illusory concoction of the mind; it is specifically this false ego that is suffering happiness and distress. The soul cannot become angry at others, since he is not personally enjoying or suffering. Rather, the false ego is doing this.
Source: A.C. Bhaktivedanta Swami Prabhupada (2014 edition), “Srimad Bhagavatam”, Eleventh Canto, Chapter 23 – Text 54.
