I've thought this before! Sometimes we really shouldn't say something because it will help keep the peace. Sometimes peace is better than saying what's on your mind. Not all the time, of course, but you need to pick your battles.
Some people say, "I'm just being honest." You can be honest and keep your mouth shut too, you don't have to give your opinion on everything, especially if it's not asked for. 'If you can't say anything nice, don't say anything at all.'