The glue record is only a helper. The A/AAAA record has to pre-exist in your domain as well. Check if there is a trailing point to your ns's fqdn or not.
Does it mean that if I enter host ns1.mydomain.com IP address 3.3.3.3 in the control panel, the Netcup panel's software will first query 3.3.3.3 to check if it serves mydomain.com before allowing the entry to succeed?
If so how can check that the nameserver IP address I've been assigned by the service provider will reply whether or not it has been configured to serve my domain, eg query 3.3.3.3 to see if it provides name service for mydomain.com.
The provider has given 2 IP addresses which I use for ns1.mydomain.com and ns2.mydomain.com in their DNS configuration page and both fqdns have trailing dots. They are automatically added if I miss the trailing dot. So I guess if they are correclty configured there must be a way of querying the IP directly to check if provides name service for the domain.