kubernetes default gateway not routing to local network -

-

i'm seeing weird issue on kubernetes , i'm not sure how debug it. k8s environment installed kube-up vsphere using 2016-01-08 kube.vmdk

the symptom dns container in pod not working correctly. when logon kube-dns service check settings looks correct. when ping outside local network works should when ping inside local network cannot reach of hosts.

for following host network 10.1.1.x, gateway / dns server 10.1.1.1.

inside kube-dns container:

(i can ping outside network ip , can ping gateway fine. dns isn't working since nameserver unreachable)

kube@kubernetes-master:~$ kubectl --namespace=kube-system exec -ti kube-dns-v20-in2me -- /bin/sh / # cat /etc/resolv.conf nameserver 10.1.1.1 options ndots:5 / # ping google.com ^c / # ping 8.8.8.8 ping 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes 8.8.8.8: seq=0 ttl=54 time=13.542 ms 64 bytes 8.8.8.8: seq=1 ttl=54 time=13.862 ms ^c --- 8.8.8.8 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 13.542/13.702/13.862 ms / # ping 10.1.1.1 ping 10.1.1.1 (10.1.1.1): 56 data bytes ^c --- 10.1.1.1 ping statistics --- 4 packets transmitted, 0 packets received, 100% packet loss / # netstat -r kernel ip routing table destination     gateway         genmask         flags   mss window  irtt iface default         10.244.2.1      0.0.0.0         ug        0 0          0 eth0 10.244.2.0      *               255.255.255.0   u         0 0          0 eth0 / # ping 10.244.2.1 ping 10.244.2.1 (10.244.2.1): 56 data bytes 64 bytes 10.244.2.1: seq=0 ttl=64 time=0.249 ms 64 bytes 10.244.2.1: seq=1 ttl=64 time=0.091 ms ^c --- 10.244.2.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.091/0.170/0.249 ms

on master:

kube@kubernetes-master:~$ netstat -r kernel ip routing table destination     gateway         genmask         flags   mss window  irtt iface default         10.1.1.1        0.0.0.0         ug        0 0          0 eth0 10.1.1.0        *               255.255.255.0   u         0 0          0 eth0 10.244.0.0      kubernetes-mini 255.255.255.0   ug        0 0          0 eth0 10.244.1.0      kubernetes-mini 255.255.255.0   ug        0 0          0 eth0 10.244.2.0      kubernetes-mini 255.255.255.0   ug        0 0          0 eth0 10.244.3.0      kubernetes-mini 255.255.255.0   ug        0 0          0 eth0 10.246.0.0      *               255.255.255.0   u         0 0          0 cbr0 172.17.0.0      *               255.255.0.0     u         0 0          0 docker0 kube@kubernetes-master:~$ ping 10.1.1.1 ping 10.1.1.1 (10.1.1.1) 56(84) bytes of data. 64 bytes 10.1.1.1: icmp_seq=1 ttl=64 time=0.409 ms 64 bytes 10.1.1.1: icmp_seq=2 ttl=64 time=0.481 ms ^c --- 10.1.1.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.409/0.445/0.481/0.036 ms

version:

kube@kubernetes-master:~$ kubectl version client version: version.info{major:"1", minor:"4", gitversion:"v1.4.5", gitcommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", gittreestate:"clean", builddate:"2016-10-29t01:38:40z", goversion:"go1.6.3", compiler:"gc", platform:"linux/amd64"} server version: version.info{major:"1", minor:"4", gitversion:"v1.4.5", gitcommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", gittreestate:"clean", builddate:"2016-10-29t01:32:42z", goversion:"go1.6.3", compiler:"gc", platform:"linux/amd64"}

kubernetes-minion-2 (10.244.2.1):

(per @der's response adding info 10.244.2.1)

kube@kubernetes-minion-2:~$ ip addr show cbr0 5: cbr0: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc htb state group default     link/ether 8a:ef:b5:fc:28:f4 brd ff:ff:ff:ff:ff:ff     inet 10.244.2.1/24 scope global cbr0        valid_lft forever preferred_lft forever     inet6 fe80::38b5:44ff:fe8a:6d79/64 scope link        valid_lft forever preferred_lft forever kube@kubernetes-minion-2:~$ ping google.com ping google.com (216.58.192.14) 56(84) bytes of data. 64 bytes nuq04s29-in-f14.1e100.net (216.58.192.14): icmp_seq=1 ttl=52 time=11.8 ms 64 bytes nuq04s29-in-f14.1e100.net (216.58.192.14): icmp_seq=2 ttl=52 time=11.6 ms 64 bytes nuq04s29-in-f14.1e100.net (216.58.192.14): icmp_seq=3 ttl=52 time=10.4 ms ^c --- google.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 10.477/11.343/11.878/0.624 ms kube@kubernetes-minion-2:~$ ping 10.1.1.1 ping 10.1.1.1 (10.1.1.1) 56(84) bytes of data. 64 bytes 10.1.1.1: icmp_seq=1 ttl=64 time=0.369 ms 64 bytes 10.1.1.1: icmp_seq=2 ttl=64 time=0.456 ms 64 bytes 10.1.1.1: icmp_seq=3 ttl=64 time=0.442 ms ^c --- 10.1.1.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.369/0.422/0.456/0.041 ms kube@kubernetes-minion-2:~$ netstat -r kernel ip routing table destination     gateway         genmask         flags   mss window  irtt iface default         10.1.1.1        0.0.0.0         ug        0 0          0 eth0 10.1.1.0        *               255.255.255.0   u         0 0          0 eth0 10.244.0.0      kubernetes-mini 255.255.255.0   ug        0 0          0 eth0 10.244.1.0      kubernetes-mini 255.255.255.0   ug        0 0          0 eth0 10.244.2.0      *               255.255.255.0   u         0 0          0 cbr0 10.244.3.0      kubernetes-mini 255.255.255.0   ug        0 0          0 eth0 172.17.0.0      *               255.255.0.0     u         0 0          0 docker0 kube@kubernetes-minion-2:~$ routel          target            gateway          source    proto    scope    dev tbl         default           10.1.1.1                                     eth0        10.1.1.0 24                       10.1.1.86   kernel     link   eth0      10.244.0.0 24       10.1.1.88                                     eth0      10.244.1.0 24       10.1.1.87                                     eth0      10.244.2.0 24                      10.244.2.1   kernel     link   cbr0      10.244.3.0 24       10.1.1.85                                     eth0      172.17.0.0 16                      172.17.0.1   kernel     linkdocker0        10.1.1.0          broadcast       10.1.1.86   kernel     link   eth0 local       10.1.1.86              local       10.1.1.86   kernel     host   eth0 local      10.1.1.255          broadcast       10.1.1.86   kernel     link   eth0 local      10.244.2.0          broadcast      10.244.2.1   kernel     link   cbr0 local      10.244.2.1              local      10.244.2.1   kernel     host   cbr0 local    10.244.2.255          broadcast      10.244.2.1   kernel     link   cbr0 local       127.0.0.0          broadcast       127.0.0.1   kernel     link     lo local       127.0.0.0 8            local       127.0.0.1   kernel     host     lo local       127.0.0.1              local       127.0.0.1   kernel     host     lo local 127.255.255.255          broadcast       127.0.0.1   kernel     link     lo local      172.17.0.0          broadcast      172.17.0.1   kernel     linkdocker0 local      172.17.0.1              local      172.17.0.1   kernel     hostdocker0 local  172.17.255.255          broadcast      172.17.0.1   kernel     linkdocker0 local             ::1              local                   kernel              lo          fe80:: 64                                   kernel            eth0          fe80:: 64                                   kernel            cbr0          fe80:: 64                                   kernel         veth6129284         default        unreachable                   kernel              lo unspec             ::1              local                     none              lo local fe80::250:56ff:fe8e:d580              local                     none              lo local fe80::38b5:44ff:fe8a:6d79              local                     none              lo local fe80::88ef:b5ff:fefc:28f4              local                     none              lo local          ff00:: 8                                                      eth0 local          ff00:: 8                                                      cbr0 local          ff00:: 8                                                   veth6129284 local         default        unreachable                   kernel              lo unspec

how can diagnose going on here?

thanks!

first, figure out what's kubernetes-mini. on you've done 2 nodes you've shown us. traffic between 10.1.1.0 , 10.244.2.0 goes through it. it, however, may have bad route 10.1.1.0 net.


Comments

Post a Comment

Popular posts from this blog

php - How to display all orders for a single product showing the most recent first? Woocommerce -

-
i have password protected page on frontend. on page, want display full list of purchases associated single woocommerce product id. option 1: i've looked through woocommerce docs , , found: wc_cli_order , list_( $args, $assoc_args ) function lists orders. presumably, orders can filtered product id [--=] filter orders based on order property. line item fields (numeric array, started index zero): line_items.0.product_id option 2: i found article , based on customer id. i've modified code based on guesses meta_key , meta_value. $customer_orders = get_posts( array( 'numberposts' => -1, 'meta_key' => '_product', 'meta_value' => get_product_id(), 'post_type' => wc_get_order_types(), 'post_status' => array_keys( wc_get_order_statuses() ), ) ); how display orders single product showing recent order first? here solution put based on these articles: http://www.w...
Read more

asp.net - How to correctly use QUERY_STRING in ISAPI rewrite? -

-
i'm having difficult time , hope guys can help. using helicon isapi rewrite version 3.1.0.104 on our iis server. edit http.conf file , have tried , still fail. this trying do: redirect url: https://www.domain.com/switch-by-version?version=2.8.5.2594 to: https://test.domain.com/load/load.aspx?tver=2.8.5.2594 the version number @ end source url change, , need target url have same version number @ end of url example above shows. i tried following not work: rewritecond %{query_string} ^version=(\d\d?)\.(\d\d?)\.(\d\d?)\.(.*)$ [nc] rewriterule ^/switch-by-version(.*)$ https://test.domain.com/load/load.aspx?tver=%1 [r=307,nc,l] any appreciated! thanks. if it's running in equivalent context of .htaccess , rule shouldn't start / . here's simpler version: rewritecond %{query_string} ^version=(\d+\.\d+\.\d+(?:\..+)?)$ [nc] rewriterule ^switch-by-version/?$ https://test.domain.com/load/load.aspx?tver=%1 [r=307,nc,l]
Read more

angularjs - How restrict admin panel using in backend laravel and admin panel on angular? -

-
https://scotch.io/tutorials/token-based-authentication-for-angularjs-and-laravel-apps following tutorials, try write authentication restrict admin panel, didn't understood how can that. this admin panel, admin panel don't want show , request database unauthenticated users. my laravel-routes route::get('/admin',function(){ return view('index'); }); route::get('/api/v1/employees/{id?}', 'employees@index'); route::get('/api/v1/users/{id?}', 'employees@users'); route::post('/api/v1/employees', 'employees@store'); route::post('/api/v1/employees/{id}', 'employees@update'); route::delete('/api/v1/employees/{id}', 'employees@destroy'); route::group(['prefix' => 'api'], function() { route::resource('authenticate', 'authenticatecontroller', ['only' => ['index']]); route::post('authenticate', 'authentic...
Read more