• Two virtual machines
The first one is used as k8s to deploy istio, the second one is used as vm, the system is centos8, centos 7 is very troublesome to upgrade glibc, the second one accesses the pod inside k8s through static routing,
This environment:
vm1:192.168.8.K8s pod cird on 131vm1:10.201.0.0/24vm2:192.168.8.170vm2 add static route: route add-net 10.201.0.0 gw 192.168.8.131 netmask 255.255.255.0
• 1.16 and above k8s•k8s apiserver open service account volume projection
- - - service-account-api-audiences=api,istio-ca ---service-account-issuer=kubernetes.default.svc ---service-account-signing-key-file=/etc/kubernetes/pki/sa.key
• Set the environment variables VM_APP, WORK_DIR, VM_NAMESPACE, and SERVICE_ACCOUNT
# VM_APP:The name of the service that this virtual machine will run#VM_NAMESPACE:Service namespace name#WORK_DIR: working directory#SERVICE_ACCOUNT k8s service account name used for this virtual machine cat vm.envexport VM_APP=testexport VM_NAMESPACE=testexport WORK_DIR=./testexport SERVICE_ACCOUNT=test. vm.env
• Create a working directory
mkdir -p "${WORK_DIR}"
• Install Istio.
istioctl install --set profile=demo --set meshConfig.defaultConfig.proxyMetadata.ISTIO_META_DNS_CAPTURE='\"true\"'
There were some problems during the deployment time. The pilot-agent of the pod could not recognize the ISTIO_META_DNS_CAPTURE normally. As a result, dns resolution was not turned on. It felt that it was a character splicing problem, or it might be a wrong posture.
Manually exportistioctl manifest generate --set profile=demo --set meshConfig.defaultConfig.proxyMetadata.ISTIO_META_DNS_CAPTURE=\'true\'> config.yamlThen modifyISTIO_META_DNS_CAPTURE: "true"
•Deploy east-west gateway
In this example, the virtual machine will be directly connected to the pod IP without deployment
samples/multicluster/gen-eastwest-gateway.sh --single-cluster | istioctl install -y -f -
• Expose the control plane using the provided example configuration
kubectl apply -f samples/multicluster/expose-istiod.yaml
• Create the namespace that will host the virtual machine:
kubectl create namespace "${VM_NAMESPACE}"
• Create a service account for the virtual machine:
kubectl create serviceaccount "${SERVICE_ACCOUNT}"-n "${VM_NAMESPACE}"
• Create WorkloadGroup template for vm
WorkloadGroup is a newly added CRD in 1.8, used to describe a collection of workload instances. It provides a specification that workload instances can be used to bootstrap its agents, including metadata and identity. It is only intended to be used with non-k8s workloads (such as virtual machines) and is intended to mimic the existing Sidecar injection and deployment specification model for Kubernetes workloads to bootstrap Istio agents.
istioctl x workload group create --name "${VM_APP}"--namespace "${VM_NAMESPACE}"--labels app="${VM_APP}"--serviceAccount "${SERVICE_ACCOUNT}"> workloadgroup.yaml
• Use the istioctl x workload entry command to generate the files needed for the virtual machine installation:
istioctl x workload entry configure -f workloadgroup.yaml -o "${WORK_DIR}"#cluster.env: Contains metadata identifying which namespaces, service accounts, network CIDRs, and (optional) inbound ports to capture.#istio-token: The Kubernetes token used to obtain the certificate from the CA.#mesh.yaml: Provides other Istio metadata, including network name, trust domain and other values.#root-cert.pem: Root certificate used for authentication.#hosts: host to which istiod is bound
Run the following command on the virtual machine you want to add to the Istio grid:
• Send the contents of the workdir to the virtual machine. When choosing a way to transfer these files securely, you should consider your information security strategy. For convenience, transfer all necessary files to the "${HOME}" directory in the virtual machine. • Install the root certificate /etc/certs in the following location:
sudo mkdir -p /etc/certssudo cp "${HOME}"/root-cert.pem /etc/certs/root-cert.pem
• Install the token in the following location /var/run/secrets/tokens:
sudo mkdir -p /var/run/secrets/tokenssudo cp "${HOME}"/istio-token /var/run/secrets/tokens/istio-token
• Install the software package that contains the integrated runtime of the Istio virtual machine:
curl -LO https://storage.googleapis.com/istio-release/releases/1.8.0/rpm/istio-sidecar.rpmsudo yum install -y istio-sidecar.rpm
• Install cluster.env in the /var/lib/istio/envoy/ directory
sudo cp "${HOME}"/cluster.env /var/lib/istio/envoy/cluster.env
• Install the grid configuration to /etc/istio/config/mesh:
sudo cp "${HOME}"/mesh.yaml /etc/istio/config/mesh
•Add the istiod host to /etc/hosts:
Here we have no external lb, no need to set
sudo sh -c 'cat $(eval echo ~$SUDO_USER)/hosts >> /etc/hosts'
Check itiod's IP and bind host
10.201.0.14 istiod.istio-system.svc
•Modify permissions:
sudo mkdir -p /etc/istio/proxysudo chown -R istio-proxy /var/lib/istio /etc/certs /etc/istio/proxy /etc/istio/config /var/run/secrets /etc/certs/root-cert.pem
Start the Istio agent:
systemctl start istio
• Check the login /var/log/istio/istio.log. You should see entries similar to the following:
[ root@worker ~]# tail -f /var/log/istio/istio.log2020-11-21T04:27:44.248846Z info sds resource:default pushed key/cert pair to proxy2020-11-21T04:33:30.756067Z info xdsproxy disconnected from XDS server: istiod.istio-system.svc:150122020-11-21T04:33:31.285240Z info xdsproxy Envoy ADS stream established2020-11-21T04:33:31.285453Z info xdsproxy connecting to upstream XDS server: istiod.istio-system.svc:150122020-11-21T04:38:48.670065Z info xdsproxy disconnected from XDS server: istiod.istio-system.svc:150122020-11-21T04:38:49.105717Z info xdsproxy Envoy ADS stream established2020-11-21T04:38:49.105884Z info xdsproxy connecting to upstream XDS server: istiod.istio-system.svc:150122020-11-21T04:54:00.782318Z info xdsproxy disconnected from XDS server: istiod.istio-system.svc:150122020-11-21T04:54:01.037151Z info xdsproxy Envoy ADS stream established2020-11-21T04:54:01.037506Z info xdsproxy connecting to upstream XDS server: istiod.istio-system.svc:15012
• Create a namespace to deploy Pod-based services:
kubectl create namespace samplekubectl label namespace sample istio-injection=enabled
• Deploy HelloWorld service:
kubectl apply -f samples/helloworld/helloworld.yaml -n sample
Send the request from your virtual machine to the service:
[ root@worker ~]# curl helloworld.sample.svc:5000/helloHello version: v2, instance: helloworld-v2-54df5f84b-tts2z[root@worker ~]# curl helloworld.sample.svc:5000/helloHello version: v1, instance: helloworld-v1-776f57d5f6-f72k9
Create workloadentry and service
apiVersion: networking.istio.io/v1beta1kind: WorkloadEntrymetadata: name: test-vm-2 namespace: testspec: serviceAccount: test address:192.168.8.170 labels: app: test instance-id: vm2---apiVersion: v1kind: Servicemetadata: name: test namespace: test labels: app: testspec: ports:- port:80 name: http-vm targetPort:80 selector: app: test
python3 -m http.server 80
kubectl exec -it helloworld-v1-776f57d5f6-nhmbc -n sample -- curl test.test.svc.cluster.localDefaulting container name to helloworld.Use 'kubectl describe pod/helloworld-v1-776f57d5f6-nhmbc -n sample' to see all of the containers inthis pod.<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd"><html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><title>Directory listing for/</title></head><body><h1>Directory listing for/</h1><hr><ul><li><a href="cluster.env">cluster.env</a></li>...</ul><hr></body></html>
From the above we can see that the internal services of k8s can be easily accessed on the vm. First, we enable the ISTIO_META_DNS_CAPTURE function in 1.8 to realize the function of pilot-agent to realize dns resolution. Pilot-agent monitors 15053 udp/tcp port, iptables Redirect 53's DNS request to pilot-agent, the rules are as follows:
- A POSTROUTING -p udp -m udp --dport 15053-j SNAT --to-source 127.0.0.1-A POSTROUTING -p udp -m udp --dport 15053-j SNAT --to-source 127.0.0.1-A POSTROUTING -p udp -m udp --dport 15053-j SNAT --to-source 127.0.0.1-A OUTPUT -p udp -m udp --dport 53-m owner --uid-owner 991-j RETURN-A OUTPUT -p udp -m udp --dport 53-m owner --gid-owner 991-j RETURN-A OUTPUT -p udp -m udp --dport 53-j DNAT --to-destination 127.0.0.1:15053-A OUTPUT -p udp -m udp --dport 53-m owner --uid-owner 991-j RETURN-A OUTPUT -p udp -m udp --dport 53-m owner --gid-owner 991-j RETURN-A OUTPUT -p udp -m udp --dport 53-j DNAT --to-destination 127.0.0.1:15053-A OUTPUT -p udp -m udp --dport 53-m owner --uid-owner 991-j RETURN-A OUTPUT -p udp -m udp --dport 53-m owner --gid-owner 991-j RETURN-A OUTPUT -p udp -m udp --dport 53-j DNAT --to-destination 127.0.0.1:15053-A ISTIO_REDIRECT -p tcp -m tcp --dport 53-j REDIRECT --to-ports 15053
ISTIO_META_DNS_CAPTURE normally only affects pods managed on k8s, but in cluster.env, the pilot-agent function on vm is enabled through ISTIO_META_DNS_CAPTURE=true.
Recommended Posts