Modi drove to the temple at Seer Govardhan, considered the 'karma bhumi' of the mystic poet-saint, from the Banaras Hindu University.
Due to security reasons, all houses near the temple were taken over by commandos and SPG personnel and shops were shut down.
Sant Niranjan Das, a prominent seer, was stopped by security personnel from going to the premises during Modi's visit. Irked by this, some of his supporters raised slogans like 'Modi go back'.
Later Monday, which is being observed as Ravidas Jayanti, Delhi Chief Minister Arvind Kejriwal will visit the temple.
The Bahujan Samaj Party is also organising an event in Varanasi. Uttar Pradesh's ruling Samajwadi Party is holding a function in Lucknow.
Sant Ravidas was a poet-saint of the Bhakti movement and has a wide following in northern India in particular. (IANS)