That is totally normal. My recommendation:

Be chill. The woman (your wife; his mom) played a much bigger role than you did in his early life, so you are pretty much screwed if you expect an even playing field for the first few years.

As a rule, you should pretty much let him be with his mother whenever he wants; that's the best way to build trust.

When you are with him, do a really good job of taking care of him -- feeding him, changing him, putting him to sleep, etc. Get him to trust that you're competent and confident.

When you are with him alone, you should be able to distract him quickly from thinking about mom.

It's normal for children to be more attached to one parent than the other at various times during their development.

At times it will be for one parent, and later, without any rhyme or reason, it can switch to the other parent.

As long as there's not something in your relationship with your son that might be causing the distance (ie, lack of time together, lack of affection, anything that might be effecting a strong bond forming), then don't worry, your time will come!