The more I try to learn physics, the more I need to see the mathematics to understand the ideas. Without the math, I don't believe it is possible to break beyond the surface 'armchair' level of understanding.
I started by building an intuitive understanding of advanced physics before I really dug into the math. It is absolutely possible. That being said, I did have a very good understanding of advanced mathematical concepts to start with, so I could think intuitively about things like high-dimensional spaces and eigenfunctions.