Rights are additive so if you granted see custom field at any other level then the right will be given to that user/group.
To only allow Everyone to see some custom fields you want to remove wherever you grant the SeeCustomField right at the Global group and queue group level and instead grant the right to everyone at the custom field level