Not me I keep my mat on the outside. I have to anyway, because I use a sleeping quilt and not a sleeping bag, my quilt has no back and no hood. I lie directly on my mattress (A Thermarest Neo Air), OK, I admit to using a fitted sheet!
Not knowing anything about what mat you use, but could it be that its not good enough to stop the cold come from up from the ground, through your mat, through your sleeping bag and through to your skin?
My Thermarest is so good I've never experienced this problem. For me, from ground level, I have my groundsheet, my tent bottom layer, my mattress and then my skin. If its a bit chilly I wear my baselayer, sometimes my sleep leggings and if its really cold I may put on some form of warm jacket to get me comfortably through the night.
As regards me getting cold from the ground itself, sorry never happened to me yet.