Of course, the more primitive and the longer the road in, the quieter the campsites will be at night. Justin and I have camped at Colorado Bend SP twice and loved both trips. We had a small group of fishmermen in a site near us on one 2-night trip, and never saw them -- they got up before we did, we were out hiking when they were back at camp snoozing and eating, and when we got back for dinner/evening, they were out for more fishin'.
I think we might even have taken my Toyota Corolla out there once, though the ruts in the road are a hassle if its wet (4x4 recommended, in those instances).
Sadly, both times we didn't do the cave crawling or walking tours... there's a lot to see there, if one looks.