Thanks for your posting. A unique constraint for custom fields is not supported by TestRail as a built-in feature currently. Please do not make manual changes to TestRail’s database schema, as this can cause problems with future database upgrades and we cannot support such changes.
When TestRail sees a database error (including validation errors for unique keys) it displays a separate error page (and the user can no longer change the submission form).
One possible workaround for your question would be to add a UI script that soft-checks the unique constraint via a server-side script when the user clicks the Add/Update button and displays a meaningful error message in case the unique constraint would be violated (and otherwise submits the form as usual). This is not guaranteed to be unique (as two requests with the same value could be made simultaneously in theory), but would this work for you? If so, I’m happy to explain the details.