DNS resolve through a coredns pod on a different node timeout

[ygao-armada]

What happened:
We have a EKSA cluster with persisting issue of DNS resolve:

If using coredns pod on the same node, it's fine (command "dig @<coredns IP> kubernetes.default" instantly responds)
If using coredns pod on a different node, command "dig @<coredns IP> kubernetes.default" timeout.
However, ping <coredns IP> always works

What you expected to happen:
command "dig @ kubernetes.default" instantly reponds no matter which coredns in use.

How to reproduce it (as minimally and precisely as possible):

kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools
dnstools# ping <coredns IP on different node>
dnstools# dig @<coredns IP on same node> kubernetes.default
dnstools# dig @<coredns IP on different node> kubernetes.default

Anything else we need to know?:
We have another cluster with the same issue, but now self-healed.

Environment:

• EKS Anywhere Release:
• EKS Distro Release: