Thanks for the post! Currently, there is not a direct way to restrict specific files from being uploaded to TestRail.
If you are concerned about the uploaded files, you can scan the attachment directory on your TestRail server with a virus scanner or even delete files of a specific extension. Those files would not be downloadable/accessible anymore (please do not remove the attachment links from the database though).
That said, we are happy to review attachment restrictions for a future release of TestRail and I have added your vote to this feature request.