There are four caverns in the Castleton area: the Blue John Cavern, the Treak Cliff Cavern, the Speedwell Cavern and the Peak Cavern. The first three are all around the Winnats Pass; Peak Cavern is just outside Castleton itself (underneath Peveril Castle, in fact).
Speedwell cavern sticks in the memory as it is accessed by an underground boat trip along an old mine adit. Not something you get to do everyday!
Check the websites listed below for opening times etc.