Being less than gaga about a grandchild isn’t something most grandparents want to admit out loud. “Though generally not spoken about in social settings,“ says Ben Michaelis, Ph.D., author of Your Next Big Thing, and clinical psychologist,  “I do hear it often in my private practice.”

For Susan Bender Phelps, 62, founder of Odyssey Mentoring and Leadership executive coaching, and grandmother of five, connecting emotionally to her youngest grandchild has been a challenge. “From birth through the toddler years into school age, she only wanted her parents and siblings,” says Phelps. “She just didn’t want any of us to hug her or play with her, and as a result, it was hard to be smitten. We spent most of our time backing off.”

When a grandparent has a grandkid with whom it’s hard to connect—whether from shyness, hyperactivity, a negative temperament, unmet expectations, or hypersensitivity, for example—it can lead to resentment on the grandchild’s part and guilt for the grandparent, neither of which is good for a relationship. The reassuring news: Not feeling intense love for a grandchild doesn’t make you a horrible person. Rather it makes you human, says Nancy Rose, author of Raise the Child You’ve Got –Not the One You Want. But what to do in this situation? Think about the following: 

1. Acknowledge your trigger points.

The first step is asking yourself why you feel as you do. Perhaps, the child is not affectionate and reminds you of your frosty mother. Maybe the child doesn’t meet your expectations in terms of expressing gratitude for all you do. “Try journaling or meditation—or even therapy, if necessary,” says Rose, “because whatever feelings you don’t recognize in yourself will lead to greater disconnection.”

2. Accept your grandchild’s core traits—especially if they don’t align with yours.

Each of us is born with certain wiring which dictates whether we are more optimistic or pessimistic, anxious or laid back, orderly or a slob. Rather than being irritated by a grandchild who is not just like you, appreciate your grandkid’s uniqueness. But also use unpleasant or anti-social behavior as a teaching moment. To the child who never says thank you (and let’s be honest, you’re really mad at your adult child, not your grandchild), it’s okay to say, “I love doing fun activities with you and giving you special gifts. At the same time, it feels bad not hearing a thank you, because everyone likes to be appreciated. Let’s both say thank you to each other and everyone else who does something nice things for us, okay?”

3. Let go of expectations.

We’re not talking about expectations that revolve around kindness and decency. We’re talking externals—being the top of the class or winning a soccer tournament. Letting go (or keeping your mouth shut) is a must if you’re to have a loving, harmonious relationship. “Try to see the pain your expectations cause,“ says Rose. “No one likes to feel they’ve let someone down or that they aren’t likable as is.” Your role: unconditional love, no matter how hard you have to work at it.

4. Be honest.

If you find yourself dreading time together, be honest with the parents. Say, “I love Susie to pieces and I want to spend time with her, but I’m totally exhausted from her high energy. Can you give me tips for dealing with her effectively?” Let the parents help you when you’ve really hit the wall.

5. Share activities.

What kids really want is a connection, says Rose. And for most kids that means sharing an activity that they like to do (rather than forcing an activity you think is more worthwhile). Do something that engages them like playing a computer game or doing an art project. The pay-off: a closer relationship and shared memories.

And finally, practice patience. Kids usually grow out of—or learn to modify—certain behaviors. For Susan Phelps Bender, patience has had a big pay off. “Our youngest grandchild is now six. She is still very shy, but last month she spent the night at our house—first time. Though it took several hours for her to warm up, once she did, we were able to hug, read stories, play, and have conversations,“ says Bender. “It was well worth the six-year wait.”