I took Chinese in college and the problem for me was the tones. If you don't grow up hearing them then they are very hard to pickup as an adult. The characters you an learn with effort. But the tone shifts just didn't register.