As it stands now: If you vote by mail you fill out your ballot and put it into a return envelope provided by the town clerk's office. You then sign your name to that envelope. Because your ballot is in the signed envelope, your ballot is no longer secret.

The change: Include with each ballot a secrecy envelope. When you complete your ballot, put it in the secrecy envelope and seal it. Then put that in the return envelope provided by the town clerk and sign it.  Now your ballot in the secrecy envelope can be placed with the other ballots waiting to be counted and no one will know whose ballot it is.

Part 2 of the voting by mail. The return envelope sent with your ballot has your return address on it.  However that address is always the resident address not the mailing address. If somehow the address to the town clerk's office gets unreadable, the ballot will be returned to the sender's residence. The envelope will then be returned  by the Post Office when it can not be delivered and since the mailing address is unreadable in the first place that ballot will never get to its destination and will not be counted. The solution is to put the voter's mailing address on the return envelope. At least that will give the voter the chance to get it returned to them and redelivered either by correcting the address of the town clerk's office or hand delivering it.