The bait store that comes to mind is capitol but they moved from W. 23rd street to the W 36th St.
Get as far south as you can for the bass. Anywhere near the Long Island Sound - the battery / Raritan bay will be fish. Or any of the parks on the Jersey side. Look up the HRFA site for launch ramps and access.
HRFA is Hudson River Fishing Association.
See if one of the Chinese restauraunts in chinatown will order you some eels.
feature will get you exact info on where capitol is.
Good luck this season and welcome aboard.