When you know ahead of time that you will be speaking in public, dress appropriately.
You can dress down if you are speaking to a group of children at a summer camp, but dress more formally if you are making remarks at a business luncheon.