Yes it will make a performance difference.  You need a crank length to suit the bio mechanics of your legs, there are various rough guides but the only real way to get it right is to be assessed by someone who knows what they're doing. http://www.cyclefit.co.uk/what_is_a_cyclefit.html

Having said that loads of people are happy to ride around on the wrong size frame, saddle in the wrong place, bars in the wrong place, cleats wrong etc etc.

I managed to get a pair of 160mm cranks for my small son, certainly makes it easier for him to pedal, and less likely for hitting the ground (24" wheels).

To be honest, I've never really felt much of a difference between 175mm and 170mm cranks.  It's only 10mm difference in total, most spd shoes have more 'fore' and 'aft' adjustment than that!