Are earthly pleasures a hindrance to Spiritual growth?

Following is an excerpt from a samvaad (dialogue) session with Acharya Prashant.

Question: On one hand I want to spiritually grow towards enlightenment, but at the same time, some side of me wants to have earthly freedom, experiences like traveling, even though I know that those things won’t bring me true Joy.