Campaigning is not allowed inside polling places, or a certain distance from them. Once you are inside your polling place, no one is allowed to try to influence your vote — and you’re not allowed to try to influence anyone else’s vote inside the polling place either. This is a really strict rule, and it probably means that you won’t be allowed to wear a shirt or button with your candidate’s name/logo on it inside the polls.
The rules against campaigning inside a polling place are really strict in order to protect voters from intimidation. Voting is private, polls are private places, and there are a lot of rules and laws in place to protect them. You also can’t do things like pass out flyers, hold up signs, or answer questions about candidates inside the polls. Generally speaking, anything with a candidate or party’s name visible is likely to be seen as campaigning — including a button or t-shirt that you are wearing. (Or a candidate’s logo, or anything else obviously partisan.)
Since people wear election shirts/buttons as a way to campaign for their candidates, they’re generally not allowed inside the polls. So, if you wear a candidate shirt/button to the polls on election day, you will most likely be asked to take it off or cover it up. If you’re planning to wear a pin, you can put it in your pocket or pin it to the inside of your shirt while you’re inside the polls. If you’re planning to wear a campaign t-shirt, bring something with you to cover it up with. The exact rules vary state-by-state, but it’s best to err on the side of caution.
tl;dr Campaigning is not allowed inside polling places. Wearing a t-shirt with a candidate’s name or party’s name visible is likely to be considered campaigning, just like if you were holding up a sign for your candidate. If you’re planning to wear a campaign t-shirt or button on election day, be prepared to take it off or cover it up while you are actually voting.