Toddler Playing With Other Nipple While Nursing

Dear Naomi, I'm breastfeeding my two-and-a-half year old son and really enjoying the ongoing connection that we have. We co-sleep, and my husband and I both have a generally peaceful, struggle-free relationship with him. I must work two days a week to help pay our bills (we live very simply; I truly have no alternative), and my husband is the caregiver on those days. The struggle I'm having is that my son is playing with my other nipple while breastfeeding at night and it instantly and fully wakes me up, so that I can't get back to sleep until it stops. He seems not to be able to get back to sleep without doing it; when I've gently but persistently moved his hand to my shoulder or belly, he has woken up fully. The waking up is recent; previously, I could put an arm over his hand and he would be still and go back to sleep. I suspect I is my lack of clarity that is the problem: I know his habit doesn't phase some moms, and I feel guilty that it bothers me. Do you think he will question my love for him or have our attachment harmed in some way if I "wean" his hands? I recognize that he misses me when I work and wonder if this behaviour is some sort of test of my love. Thank you for your thoughts.