It does not matter -- me.A.to B.with 为什么选A?

不是what the matter with you
是What's the matter with you?
It doesn't matter to me （对我来说没什么）里的matter是动词.
What's the matter with you?（你怎么了）里的matter是名词.
matter to sb.和 be the matter with sb./st.是固定搭配.