If the server hosting the site does not support IPv61), then you can disable this protocol by deleting all records of the "AAAA". If there are no such records, then the client connecting via IPv6 will switch to IPv4 and get to your site via IPv4.
By standard system tools DNS domain can only be directed to IP. The port and access directory must be configured directly on the web server running on this IP. Alternatively, you can use an intermediate proxy server that will redirect all requests addressed to the domain to the correct IP and port.