Am I allergic to milk?

Everytime I drink milk my nose is a little congested and I just don't feel all that great. I don't get an upset stomach but it's like it changes my mood. Could this be possible?