12. Network Programming in Python3

The Essence of the Internet####

The communication between two computers is the same as the principle of two people calling.

# 1. First, it must be connected through various physical connection media
# 2. Find the exact computer(Accurate to the software)s position
# 3. Adopt a unified standard(General sub-agreement)Data forwarding

# The physical connection medium, this is what the network engineer considers, and I will briefly talk about it later, we are mainly learning this unified standard.

# English has become the unified standard of communication for all people in the world. If computers are regarded as people distributed all over the world, then the internet connecting two computers is actually a series of unified standards. These standards are called Internet Protocols. The essence is a series of protocols, collectively referred to as "Internet Protocol Suite" (Internet Protocol Suite).

# The function of the Internet protocol: defines how a computer accesses the internet, and the standard for computer communication connected to the internet.

Since the birth of the Internet, almost all programs are now network programs, and there are very few stand-alone programs.
A computer network connects computers together so that the computers in the network can communicate with each other. Network programming is how to realize the communication between two computers in the program.
For example, when you use a browser to visit Sina.com, your computer is connected to a certain Sina server via the Internet. Then, Sina's server transmits the content of the web page as data to your computer via the Internet.
Since your computer may not only be a browser, but also QQ, Skype, Dropbox, mail clients, etc., other computers connected to different programs will also be different, so, more precisely, network communication is on two computers Communication between two processes. For example, the browser process is communicating with a certain web service process on a Sina server, while the QQ process is communicating with a certain process on a Tencent server.

Introduction to TCP/IP####

Although everyone is familiar with the Internet, computer networks appeared much earlier than the Internet.
​ In order to connect a computer, a communication protocol must be specified. In early computer networks, each manufacturer specified a set of protocols. IBM, Apple and Microsoft have their own network protocols, which are incompatible with each other. This is like a group of people. Some people speak English, some speak Chinese, and some speak German. People who speak the same language can communicate, but not between different languages.
​ In order to connect all the different types of computers in the world, a set of universal protocols must be specified. In order to achieve the goal of the Internet, the Internet Protocol Suite (Internet Protocol Suite) is the universal protocol standard. The Internet is a combination of the words inter and net. The original intent is to connect to the "network" network. With the Internet, any private network can connect to the Internet as long as it supports this protocol.
​ Because the Internet protocol contains hundreds of protocol standards, but the two most important protocols are TCP and IP, so everyone refers to the Internet protocol as TCP/IP.
​ When communicating, both parties must know the identity of the other party, just as they must know the other party's email address when sending an email. The unique identifier of each computer on the Internet is the IP address, similar to 123.123.123.123. If a computer is connected to two or more networks at the same time, such as a router, it will have two or more IP addresses. Therefore, the IP address corresponds to the computer's network interface, usually a network card.

​ The IP protocol is responsible for sending data from one computer to another computer through the network. The data is divided into small pieces and then sent out via IP packets. Due to the complex Internet links, there are often multiple lines between two computers. Therefore, the router is responsible for deciding how to forward an IP packet. The feature of IP packets is that they are sent in blocks and routed through multiple routes, but they are not guaranteed to arrive, nor are they guaranteed to arrive in order.

​ The IP address is actually a 32-bit integer (called IPv4). The IP address represented by a string, such as 192.168.0.1, is actually a 32-bit integer that is grouped by 8 bits to make it easier to read.
​ An IPv6 address is actually a 128-bit integer, which is an upgraded version of the currently used IPv4, expressed as a string similar to 2001:0db8:85a3:0042:1000:8a2e:0370:7334.
​ The TCP protocol is based on the IP protocol. The TCP protocol is responsible for establishing a reliable connection between two computers to ensure that data packets arrive in order. The TCP protocol will establish a connection through a handshake, and then number each IP packet to ensure that the other party receives it in order. If the packet is lost, it will be automatically resent.
​ Many commonly used higher-level protocols are based on the TCP protocol, such as the HTTP protocol for browsers, the SMTP protocol for sending mail, etc.
​ In addition to the data to be transmitted, a TCP message also contains the source IP address and destination IP address, source port and destination port.
​ What does the port do? When two computers communicate, it is not enough to only send the IP address, because multiple network programs are running on the same computer. After a TCP message arrives, whether it is delivered to the browser or QQ depends on the port number to distinguish it. Each network program applies to the operating system for a unique port number. In this way, two processes need their own IP addresses and their own port numbers to establish a network connection between two computers.
​ A process may also establish links with multiple computers, so it will apply for many ports.

​ After understanding the basic concepts of the TCP/IP protocol, the concepts of IP addresses and ports, we can start network programming.

Software Development Architecture

CS architecture, BS architecture#####

English name of the client: Client,

The English name of the browser: Browser.

English name of the server: Server.

C/S architecture: Based on the architecture between the client and the client. For example: QQ, WeChat, Youku, Baofengyingyin, etc.

• Advantages: The interface and operation of the C/S architecture are very rich to meet the individual requirements of customers, the safety is easy to guarantee, and the response speed is faster.
• Disadvantages: Need to develop two sets of programs, client and server, high development cost and maintenance cost, poor compatibility, fixed user group, etc.

B/S architecture: a special C/S architecture based on the C/S architecture, the architecture between the browser and the server.

• Advantages: strong distribution, almost no maintenance on the client side, simple development, strong sharing, simple and convenient maintenance.
• Disadvantages: low personalization, security and response speed require huge design costs.

Summary: CS has a fast response speed and strong security. It is generally used in local area networks, but the development and maintenance costs are high; BS can achieve cross-platform, client-side zero maintenance, but it has low personalization ability and slow response speed. Therefore, some units use BS in their daily office work and use the CS structure in actual production.

# C/S architecture
	C:Client
	S:Server

# B/S architecture
	B:Browser
	S:Server
        
# C/S architecture:Need to install to use
# client client:We need to install
# server server

# B/S Architecture: Baidu Blog Park Google Code Cloud
# browser browser
# server server

# b/s and c/s what's the relationship?
# B/S architecture is C/A type of S architecture

# C/Benefits of S architecture
# Can be used offline/More complete functions/Better security

# B/S architecture
# Can be used without installation
# Unified PC user portal

# Mobile terminal:Like C/The S architecture is relatively hot, but it is not the case. WeChat applets and Alipay third-party interfaces are similar to B/S architecture
# The purpose is to unify the interface:Gather user groups

# PC side:BS is more popular
# Nature: B/The S architecture itself is also C/S architecture

Client and server concepts

# Server:Provide services 24 hours a day, I will serve whoever comes
# Client:When you want to experience the service, go to the server to experience the service

What can I do to learn network programming

# Development C/S architecture software

Learn what concurrency, database, front-end can do

# Development B/S architecture software

Origin of Network Programming Technology

# The rise of most advanced technologies basically comes from the military. The technology of network programming comes from the US military in order to realize the remote transmission of data..

The way humans realize remote communication

# Phone
# Big butt computer with network cable
# Laptop with wireless network card

# In summary, we can conclude the first law:The first requirement to realize remote communication is:Physical connection medium
# Let’s think about the communication between people. Chinese people speak Chinese and foreigners speak foreign languages.,
# So if you want to achieve barrier-free communication between people from different countries, is it necessary to stipulate a language that everyone can understand?>>>English
# Come back to the computer field,In order to realize remote communication between computers, in addition to the physical connection medium, should there be a set of public standards??This set of standards is
# >>> OSI seven-layer protocol(Also called OSI seven-layer model)

OSI seven-layer protocol (model)

# Application layer
# Presentation layer
# Session layer
# Transport layer
# Network layer
# data link layer
# Physical connection layer

# Some people group it into five layers
# Application layer
# Transport layer
# Network layer
# data link layer
# Physical connection layer

Physical layer#####

The origin of the physical layer: As mentioned above, in order to play together between isolated computers, they must be connected to the internet. The implication is that the computers must be networked.

Physical layer function: It is mainly based on the characteristics of the circuit to send high and low voltages (electrical signals), high voltage corresponds to number 1, low voltage corresponds to number 0

# Realize the physical connection between computers, the data transmitted are all 010101 binary
# Working principle of electrical signal:Point only high and low

Fiber

Twisted pair

data link layer#####

Origin of the data link layer: Pure electrical signals 0 and 1 have no meaning. It is necessary to specify the number of electrical signals in one group and what each group means.

The function of the data link layer: defines the grouping method of electrical signals

Ethernet protocol

In the early days, each company had its own grouping method, and later formed a unified standard, namely the Ethernet protocol ethernet

ethernet regulations

A group of electrical signals form a data leopard, called a'frame'

Each data frame is divided into two parts: header and data.

# head contains:(Fixed 18 bytes)
 # Sender/source address, 6 bytes
	#Recipient/target address, 6 bytes
	#Data type, 6 bytes
    
# data contains:(The shortest length is 46 bytes, and the longest length is 1500 bytes)
	#The specific content of the data packet
# head length+data length=The shortest length is 64 bytes, the longest length is 1518 bytes. If the limit is exceeded, it will be sent in fragments

mac address

The origin of the source and destination addresses contained in the head: ethernet stipulates that all devices connected to the internet must have a network card. The addresses of the sender and receiver refer to the address of the network card, that is, the mac address
Mac address: Each network card is burned with a unique mac address in the world when it leaves the factory. The length is 48-digit binary, usually represented by 12-digit hexadecimal number (the first six digits are the manufacturer number, and the last six digits are the pipeline number)

broadcast

With the mac address, two hosts in the same network can communicate (one host obtains the mac address of the other host through the arp protocol)
The ethernet uses the most primitive way, broadcast communication, that is, computer communication basically depends on roaring

# 1. Specifies the grouping method of binary data
# 2. It is stipulated that as long as it is a computer connected to the Internet, it must have a network card
# The network card is engraved with a unique number in the world
# Each network card will be burned with a unique mac address in the world when it leaves the factory
# The length is 48-digit binary, usually represented by 12-digit hexadecimal number (the first six digits are the manufacturer number, and the last six digits are the assembly line number)
# The number engraved on our pipe network card is called the computer>>>mac address
# —–>The above two regulations are actually "Ethernet Protocol"!

# Communication based on Ethernet protocol:Communication basically depends on howling!!!Disadvantages:Broadcast storm
# switch: 如果没有switch，你的电脑就变成了马蜂窝，有了switch吼，所有的电脑只需要一个网卡连接switch，即可实现多台电脑之间物理连接

Network layer#####

The origin of the network layer: With the transmission method of ethernet, mac address, and broadcasting, computers in the world can communicate with each other. The problem is that the world-wide Internet
It is composed of small local area networks that are isolated from each other. If all communication uses the broadcast method of Ethernet, then the packets sent by one machine will be received by the world.
This is not just a problem of low efficiency, it will be a disaster

The conclusion of the above figure: It is necessary to find a way to distinguish which computers belong to the same broadcast domain and which are not. If they are, they will be sent by broadcast. If not,
Using routing (distributing data packets to different broadcast domains/subnets), the mac address is indistinguishable, it is only related to the manufacturer
Network layer function: introduce a new set of addresses to distinguish different broadcast domains/subnets, this set of addresses is the network address

***IP protocol: ***

• The protocol that specifies the network address is called the ip protocol, and the address it defines is called the ip address. The widely used version of v4 is ipv4, which specifies that the network address is represented by a 32-bit binary system
• Range 0.0.0.0-255.255.255.255
• An ip address is usually written as a four-segment decimal number, for example: 172.16.10.1

The ip address is divided into two parts

• Network part: Identify the subnet
• Host part: identify the host

Note: The pure ip address segment only identifies the type of ip address, and it is impossible to identify the subnet where an ip is located from the network part or the host part.

Example: 172.16.10.1 and 172.16.10.2 are not sure that they are in the same subnet

Subnet mask

The so-called "subnet mask" is a parameter that indicates the characteristics of the subnet. It is equivalent to the IP address in form and is also a 32-bit binary number. Its network part is all 1 and the host part is all 0. For example, if the IP address is 172.16.10.1, if it is known that the network part is the first 24 digits and the host part is the last 8 digits, then the subnet mask is 11111111.11111111.11111111.00000000, which is 255.255.255.0 in decimal.

Knowing the "subnet mask", we can determine whether any two IP addresses are on the same subnet. The method is to perform an AND operation on the two IP addresses and the subnet mask (both digits are 1, and the result of the operation is 1, otherwise it is 0), and then compare whether the results are the same. If so, it means that they are in the same subnet. In the network, otherwise it is not.

For example, the subnet masks of the known IP addresses 172.16.10.1 and 172.16.10.2 are both 255.255.255.0, are they in the same subnet? Both and the subnet mask are ANDed separately,

172.16.10.1 ：10101100.00010000.00001010.000000001

255255.255.255.0:11111111.11111111.11111111.00000000

The network address result of AND operation: 10101100.00010000.00001010.000000001->172.16.10.0

172.16.10.2 ：10101100.00010000.00001010.000000010

255255.255.255.0:11111111.11111111.11111111.00000000

The network address result of AND operation: 10101100.00010000.00001010.000000001->172.16.10.0

The results are both 172.16.10.0, so they are in the same subnet.

To sum up, there are two main functions of the IP protocol. One is to assign IP addresses to each computer, and the other is to determine which addresses are on the same subnet.

ip packet

The ip data packet is also divided into the head and data parts, there is no need to define a separate field for the ip packet, and it is directly put into the data part of the Ethernet packet

head: 20 to 60 bytes in length

data: The maximum length is 65,515 bytes.

The "data" part of an Ethernet packet is only 1500 bytes long. Therefore, if the IP data packet exceeds 1500 bytes, it needs to be divided into several Ethernet data packets and sent separately.

ARP protocol

The origin of the arp protocol: computer communication basically relies on roaring, that is, broadcasting. All upper-layer packets must be encapsulated with an Ethernet header at the end, and then sent through the Ethernet protocol. When talking about the Ethernet protocol, we learned

Communication is realized based on the mac-based broadcast method. When a computer sends a package, it is easy to obtain its own mac. How to obtain the mac of the target host needs to pass the arp protocol

arp protocol function: send data packets in broadcast mode to obtain the mac address of the target host

Protocol working mode: each host ip is known

For example: host 172.16.10.11/24 visit 172.16.10.11/24

One: First distinguish the subnet you are in by ip address and subnet mask

2: Analyze 172.16.10.11/24 and 172.16.10.11/24 are in the same network (if not the same network, then the target ip in the following table is 172.16.10.1, and the MAC of the gateway is obtained through arp)

Three: This packet will be transmitted in the self-network where the sender is located in a broadcast manner. After all hosts receive it, they will open the packet and find that the target ip is their own. They will respond and return their mac

# ​ It is stipulated that all computers must have an IP address
# IP address characteristics:Dotted decimal
# ​ There are two versions ipv4 and ipv6 in order to be compatible with more computers
# The smallest:0.0.0.0  
# ​ Maximum:255.255.255.255  
# IP protocol can be transmitted across LAN
# The ip address can uniquely identify a unique machine on the Internet!
# [ http://14.215.177.39/](http://14.215.177.39/)

Transport layer (port protocol)

The origin of the transport layer: the ip of the network layer helps us to partition the molecular network, and the mac of the Ethernet layer helps us find the host. Then everyone uses applications. You may open QQ, Baofengyingyin, and other applications on your computer at the same time. program,

Then we found a specific host through ip and mac, how to identify the application on this host, the answer is the port, the port is the number associated with the application and the network card.

Transport layer function: establish port-to-port communication

Supplement: The port range is 0-65535, 0-1023 is the system occupied port

tcp protocol:

Reliable transmission, TCP data packet has no length limit, theoretically it can be unlimited, but in order to ensure the efficiency of the network, usually the length of TCP data packet will not exceed the length of IP data packet to ensure that a single TCP data packet does not need to be split.

tcp message

tcp three handshake and four wave hands

udp protocol:

Unreliable transmission, the "header" part has only 8 bytes in total, and the total length does not exceed 65,535 bytes, which fits exactly into an IP data packet.

# TCP, UDP work based on ports!
# In fact, the communication between computers is actually the communication between the application on the computer and the application
# port(port): Uniquely identifies a certain application based on network communication on a computer
# Port range:0~~65535(Dynamic allocation)  
# Note: 0~~1024 is usually the port number assigned by the operating system
# Under normal circumstances, the software port number we write is recommended to start after 8000
# Flask framework default port 5000
# django framework default port 8000
# ​ The default port of mysql database is 3306
# Redis database default port 6379
# note:A port number can only be occupied by one application at the same time on a computer

# Small summary:  
# IP address: uniquely identifies the unique machine connected to the Internet in the world
# port number:Uniquely identify an application on a computer
# ip+port :Able to uniquely identify an application on a unique computer in the world

# supplement:  
# arp protocol:Resolve mac address according to ip address

Application layer (HTTP protocol, FTP protocol)

The origin of the application layer: all users use applications and all work at the application layer. The Internet is developed, and everyone can develop their own applications. There are many kinds of data, and the organization of the data must be specified.
Application layer function: specify the data format of the application.
Example: TCP protocol can transfer data for various programs, such as Email, WWW, FTP, etc. Then, there must be different protocols that specify the format of email, web pages, and FTP data, and these application protocols constitute the "application layer."

TCP\UDP protocol (streaming protocol, reliable protocol)

# Three handshake and four waves
# Three-way handshake to establish a connection
# Wave four times to break the connection

# Tcp(voice chat/Video chat),Offline cache of high-strength movies\QQ remote control
# Need to establish a connection before you can communicate
# Occupy connection\reliable(Message will not be lost)\High real-time\slow

# UDP(Send message)-Play video online\QQSend message\WeChat message
# You can communicate without establishing a connection
# Do not occupy connection\Unreliable\Messages are lost due to network instability\fast

Network communication realization

To realize network communication, each host needs to have four elements

• IP address of this machine
• Subnet mask
• IP address of the gateway
• DNS IP address

There are two ways to obtain these four elements

1. Static acquisition

Manual configuration

2. Dynamic acquisition

Get through dhcp

(1) The first "Ethernet header", set the MAC address of the sender (this machine) and the MAC address of the receiver (DHCP server). The former is the MAC address of the local network card, the latter does not know at this time, just fill in a broadcast address: FF-FF-FF-FF-FF-FF.

(2) The "IP header" at the back, set the IP address of the sender and the IP address of the receiver. At this time, the machine does not know about both. Therefore, the sender's IP address is set to 0.0.0.0, and the receiver's IP address is set to 255.255.255.255.

(3) In the last "UDP header", set the sender's port and receiver's port. This part is stipulated by the DHCP protocol. The sender is port 68 and the receiver is port 67.

After the data packet is constructed, it can be sent out. Ethernet is a broadcast transmission, and every computer on the same subnet has received this packet. Because the receiver's MAC address is FF-FF-FF-FF-FF-FF, it is not clear who it was sent to, so every computer that receives this packet must also analyze the IP address of the packet to determine it. Not for yourself. When you see that the sender's IP address is 0.0.0.0 and the receiver's is 255.255.255.255, the DHCP server knows that "this packet is sent to me", and other computers can discard this packet.

Next, the DHCP server reads the data content of this packet, assigns an IP address, and sends back a "DHCP response" packet. The structure of this response packet is similar. The MAC address of the Ethernet header is the network card addresses of both parties, and the IP address of the IP header is the IP address of the DHCP server (sender) and 255.255.255.255 (receiver), UDP header The ports are 67 (sender) and 68 (receiver). The IP address assigned to the requester and the specific parameters of the network are included in the Data section.

The newly added computer receives this response packet, so it knows its own IP address, subnet mask, gateway address, DNS server and other parameters

Network communication process

Local acquisition

• The IP address of this machine: 192.168.1.100
• Subnet mask: 255.255.255.0
• The IP address of the gateway: 192.168.1.1
• DNS IP address: 8.8.8.8

4.2 Open the browser and visit

Want to visit Google, enter the URL in the address bar: www.google.com.

dns protocol (based on udp protocol)

13 Taigen dns:

A.root-servers.net198.41.0.4 United States
B.root-servers.net192.228.79.201 United States (IPv6 is also supported)
C.root-servers.net 192.33.4.12 France
D.root-servers.net128.8.10.90 United States
E.root-servers.net192.203.230.10 United States
F.root-servers.net192.5.5.241 United States (IPv6 is also supported)
G.root-servers.net192.112.36.4 United States
H.root-servers.net128.63.2.53 United States (IPv6 is also supported)
I.root-servers.net192.36.148.17 Sweden
J.root-servers.net192.58.128.30 United States
K.root-servers.net193.0.14.129 UK (also supports IPv6)
L.root-servers.net198.32.64.12 United States
M.root-servers.net202.12.27.33 Japan (also supports IPv6)

Domain name definition: http://jingyan.baidu.com/article/1974b289a649daf4b1f774cb.html

Top-level domain names: .com, .net, .org, .cn, etc. are international top-level domain names. According to the current Internet domain name system, international top-level domain names are divided into two categories: category top-level domains (gTLD) and geographic top-level domains (ccTLD) ) Two kinds. Category top-level domain names are domain names ending in "COM", "NET", "ORG", "BIZ", "INFO", etc., all managed by foreign companies. A top-level geographic domain name is a domain name ending with a country or region code, such as "CN" for China and "UK" for the United Kingdom. Geographic top-level domain names are generally managed by each country or region.

Second-level domain name: The second-level domain name is a geographic domain name based on the top-level domain name. It is a metaphor for the second-level domains in China, such as .com.cn, .net.cn, .org.cn, .gd.cn and other subdomains. The subdomain of the parent domain name is analogous to the parent domain name being abc.com, and the subdomain name is www.abc.com or .abc.com.* Generally speaking, the second-level domain name is a record of the domain name, such as alidiedie.com. Domain name, www.alidiedie.com is one of the more commonly used records, and this is generally used by default, but domain names like *.alidiedie.com are all called second-level alidiedie.com

Socket (Socket Programming)

Before looking at the socket, let’s review the five-layer communication process:

But in fact, starting from the transport layer and the following, the operating system helps us to complete the various header encapsulation processes below, should we do them one by one? **NO! **

Socket is also called socket. It is an abstraction layer of middleware for communication between application layer and TCP/IP protocol suite. It is a set of interfaces. In the design mode, Socket is actually a facade mode, which hides the complex TCP/IP protocol family behind the Socket interface. For users, a set of simple interfaces is all, allowing Socket to organize data to conform to the specified protocol. When we use different protocols to communicate, we have to use different interfaces and deal with various details of different protocols. This increases the difficulty of development and the software is not easy to expand (just like we develop a company management system) , Functions such as account reimbursement, meeting reservations, and leave requests do not need to be written to the system separately, but multiple functional interfaces on a system, without knowing how each function is implemented. So UNIX BSD invented the socket, which shields the communication details of each protocol, so that programmers do not need to pay attention to the protocol itself, and directly use the interface provided by the socket to communicate between processes of interconnected different hosts. This is like the operating system provides us with system calls that use the underlying hardware functions. Through the system calls, we can conveniently use disks (file operations) and use memory without having to read and write disks and manage memory ourselves. Socket is actually the same thing, that is, it provides an abstraction of the tcp/ip protocol and provides a set of interfaces to the outside world. With this interface, you can use the functions of the tcp/ip protocol uniformly and conveniently.

In fact, from your perspective, a socket is a module. We establish the connection and communication between the two processes by calling the methods implemented in the module. Some people also refer to socket as ip+port, because ip is used to identify the location of a host on the Internet, and port is used to identify an application on this machine. So we only need to establish the ip and port to find an application, and use the socket module to communicate with it.

Socket is also known as "socket". Applications usually send requests to the network or respond to network requests through "sockets", so that communication between hosts or between processes of a computer is similar to an operating system, which is complicated and ugly The operation of the control computer hardware is packaged into a unified and simple interface, and the user only needs to learn how to operate the operating system to operate the computer hardware simply and quickly

Socket development history and classification

Socket originated from the Unix version of the University of California, Berkeley in the 1970s, which is known as BSD Unix. Therefore, people sometimes refer to sockets as "Berkeley sockets" or "BSD sockets". In the beginning, sockets were designed for communication between multiple applications on the same host. This is also called inter-process communication, or IPC. There are two types of sockets (or called two races), which are file-based and network-based.

Socket family based on file type

The name of the socket family: AF_UNIX

Everything in unix is a file. The file-based socket calls the underlying file system to fetch data. Two socket processes run on the same machine and can communicate indirectly by accessing the same file system.

Socket family based on network type

The name of the socket family: AF_INET

( There are also AF_INET6 used in ipv6, and there are some other address families, but they are either only used on a certain platform, or have been obsolete, or are rarely used, or are not implemented at all, all address families Among them, AF_INET is the most widely used one. Python supports many address families, but since we only care about network programming, we only use AF_INET most of the time)

TCP and UDP comparison#####

TCP (Transmission Control Protocol) is a reliable, connection-oriented protocol (eg: phone call), low transmission efficiency, full-duplex communication (sending buffer & receiving buffer), and byte stream-oriented. Applications using TCP: Web browsers; file transfer programs.

UDP (User Datagram Protocol) is an unreliable and connectionless service with high transmission efficiency (small delay before sending), one-to-one, one-to-many, many-to-one, many-to-many, packet-oriented (data packet), Do your best to serve without congestion control. Applications using UDP: Domain Name System (DNS); video streaming; Voice over IP (VoIP).

Socket under TCP protocol

A scene in life. If you want to call a friend, dial the number first, and the friend will pick up the phone after hearing the ringing. At this time, you and your friend will establish a connection and you can talk. When the exchange is over, hang up the phone to end the conversation. Scenes in life explain how this works.

Let's start with the server side. The server first initializes the Socket, then binds to the port, listens to the port, calls accept to block, and waits for the client to connect. At this time, if a client initializes a Socket, and then connects to the server (connect), if the connection is successful, then the connection between the client and the server is established. The client sends a data request, the server receives the request and processes the request, then sends the response data to the client, the client reads the data, and finally closes the connection, and the interaction ends

The following will introduce the socket() module function usage

In Python, we use the socket() function to create a socket, the syntax format is as follows:

socket.socket([family[,type[,proto]]])import socket
socket.socket(socket_family,socket_type,protocal=0)
 socket_family can be AF_UNIX or AF_INET。socket_type can be SOCK_STREAM or SOCK_DGRAM. protocol is generally not filled,The default value is 0.

 Get tcp/ip socket
tcpSock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

Get udp/ip socket
udpSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

Because there are too many attributes in the socket module. We made an exception here'from module import *'Statement. use'from socket import *',We have brought all the attributes in the socket module to our namespace,This can greatly shorten our code.
E.g. tcpSock=socket(AF_INET, SOCK_STREAM)
Server socket function
s.bind()Bind(Host,The port number)To socket
s.listen()Start TCP listening
s.accept()Passively accept TCP client connections,(Blocking)Waiting for the connection

Client socket function
s.connect()Actively initiate TCP server connection
s.connect_ex()connect()Extended version of the function,Error code is returned when an error occurs,Instead of throwing an exception

Public-use socket functions
s.recv()Receive TCP data
s.send()Send TCP data(send when the amount of data to be sent is greater than the remaining space in the buffer area,data lost,Will not finish)
s.sendall()Send complete TCP data(The essence is to call send in a loop,sendall when the amount of data to be sent is greater than the remaining space of its own buffer,No data loss,Call send in a loop until it is sent)
s.recvfrom()Receive UDP data
s.sendto()Send UDP data
s.getpeername()The address of the remote end connected to the current socket
s.getsockname()The address of the current socket
s.getsockopt()Returns the parameters of the specified socket
s.setsockopt()Set the parameters of the specified socket
s.close()Close socket

Lock-oriented socket method
s.setblocking()Set the blocking and non-blocking mode of the socket
s.settimeout()Set the timeout period for blocking socket operations
s.gettimeout()Get the timeout period for blocking socket operations

Functions for file-oriented sockets
s.fileno()File descriptor of the socket
s.makefile()Create a file related to the socket

parameter

# family:Socket family can make AF_UNIX or FA_INET
# type:The socket type can be based on the connection-oriented or non-connection atmosphere SOCK_STREAM or SOCK_DGRAM.
# protocol:Generally default is 0.

Socket object (built-in) method

Example1

# server.py
import socket
sk = socket.socket()
sk.bind(('127.0.0.1',9000))

# Waiting for others to connect with me
sk.listen()
# Waiting for customers to link me,And there will be a link address
conn,addr = sk.accept()print(conn,"CCCCCCCCCOOO")print(addr,"IPAIP")
conn.send(b'hello')
msg = conn.recv(1024)print(msg)
# Receive up to 1024 bytes of data
conn.recv(1024)
conn.close()
# Turn him off and don’t let others link
sk.close()

# client.py
import socket
sk = socket.socket()
sk.connect(('127.0.0.1',9000))
msg = sk.recv(1024)print(msg)
sk.send(b'byebyte')
sk.close()

Example2

Server

# We use the socket function of the socket module to create a socket object, and the socket object can set up a socket service by calling other functions.
# Now we can pass bind(hostname,port)Function to specify the port of the service(port)
# Next, we call the acoepl method of the socket object, which waits for the client's connection and returns the connection object, indicating that the client is connected.

#! /usr/bin/python3
# File name: server.py

# Import socket and sys modules
import socket
import sys

# Create socket object
serversocket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)

# Get the host name of the machine
host = socket.gethostname()
port =9999

# Binding port number
serversocket.bind((host,port))

# Set the maximum number of connections, queue after exceeding
serversocket.listen(5)while True:
  # Establish client connection
 clientsocket,addr = serversocket.accept()print("Connection address: %s"%str(addr))
 msg='Welcome to YouMenDemo!'+"\r\n"
 clientsocket.send(msg.encode('utf-8'))
 clientsocket.close()

**Client **

# Next we write a simple client instance to connect to the service created above, the port number is 9999
# socket.connect(hostname,port)Method to open a TCP connection to the service provider whose host is hostname and port is port,
# After connecting, we can get data from the server. Remember, you need to close the connection after the operation is complete.

#! /usr/bin/python3
# File name: client.py
import socket
import sys

# Create socket object
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)

# Get local hostname
host = socket.gethostname()

# Set the port number
port =9999

# Connect service, specify host and port
s.connect((host,port))

# Accept data less than 1024 bytes
msg = s.recv(1024)

s.close()print(msg.decode('utf-8'))

Now we open two terminals and execute the server.py file, and the other terminal executes the client.py file, it will appear as follows

# Welcome to YouMenDemo!

# Open the first terminal at the same time, the following information output will appear:
# Connection address:('172.19.0.2',51157)

Example3

TCP and multiple client communication

# Server side
import socket
sk = socket.socket()
sk.bind(('127.0.0.1',9005))
sk.listen()while True:
 conn,addr = sk.accept()print("conn :",conn)while True:
  send_msg =input('>>>')if send_msg.upper()=='Q':break
  conn.send(send_msg.encode('utf-8'))
  msg = conn.recv(1024).decode('utf-8')if msg.upper()=='Q':breakprint(msg)
 conn.close()
sk.close()

# Client side
import socket
sk = socket.socket()
sk.connect(('127.0.0.1',9005))while True:
 msg = sk.recv(1024)
 msg2 = msg.decode('utf-8')if msg2.upper()=='Q':breakprint(msg,msg2)
 send_msg =input('>>>')if send_msg.upper()=='Q':break
 sk.send(send_msg.encode('utf-8'))

sk.close()

Python Internet Module#####

Some important modules of Python network programming are listed below

Example2TCP ServerServer Structure

tcps =socket() #Create server socket
tcps.bind()      #Bind the address to the socket
tcps.listen()      #Monitor link
while True:      #Server infinite loop
 tcpc = tcps.accept() #Accept client link
 while True:         #Communication loop
  tcpc.recv()/tcpc.send() #dialogue(Receive and send)
 tcpc.close()    #Close client socket
tcps.close()        #Close server socket(Optional)

Timestamp server instance

#! /usr/bin/python3
# - *- coding:utf-8-*-from socket import*import time
COD ='utf-8'
HOST ='192.168.164.141' #Host ip
PORT =21566 #Software port number
BUFSIZ =1024
ADDR =(HOST, PORT)
SIZE =10 
tcpS =socket(AF_INET, SOCK_STREAM) #Create socket object
tcpS.setsockopt(SOL_SOCKET,SO_REUSEADDR,1) #Add socket configuration, reuse ip and port
tcpS.bind(ADDR) #Bind ip port number
tcpS.listen(SIZE)  #Set the maximum number of links
while True:print("The server starts and listens for client connections")
 conn, addr = tcpS.accept()print("Linked client", addr)while True:try:
   data = conn.recv(BUFSIZ) #Read messages sent by linked customers
  except Exception:print("Disconnected client", addr)breakprint("What the client sends:",data.decode(COD))if not data:break
  msg = time.strftime("%Y-%m-%d %X") #Get structured event stamp
  msg1 ='[%s]:%s'%(msg, data.decode(COD))
  conn.send(msg1.encode(COD)) #Send a message to the linked client
 conn.close() #Close client link
tcpS.closel()

**TCP client *client structure

tcpc =socket()    #Create client socket
tcpc.connect()    #Try to connect to the server
while True:        #Communication loop
 tcpc.send()/tcpc.recv()    #dialogue(send/receive)
tcpc.close()      #Close client socket

Timestamp client instance

#! /usr/bin/python3
# - *- coding:utf-8-*-from socket import*from time import ctime
HOST ='192.168.164.141' #Server ip
PORT =21566 #Server port number
BUFSIZ =1024
ADDR =(HOST, PORT)
tcpCliSock =socket(AF_INET, SOCK_STREAM) #Create socket object
tcpCliSock.connect(ADDR) #connect to the server
while True:
 data =input('>>').strip()if not data:break
 tcpCliSock.send(data.encode('utf-8')) #send messages
 data = tcpCliSock.recv(BUFSIZ) #Read message
 if not data:breakprint(data.decode('utf-8'))
tcpCliSock.close() #Close client

Sticky package phenomenon#####

Before talking about sticking packages, let's take a look at the socket buffer problem:

After each socket is created, two buffers are allocated, an input buffer and an output buffer.

write()/send()It does not transmit data to the network immediately, but first writes the data into the buffer, and then sends the data from the buffer to the target machine by the TCP protocol. Once the data is written to the buffer, the function can successfully return, regardless of whether they reach the target machine or when they are sent to the network, these are the things that the TCP protocol is responsible for.

TCP protocol is independent of write()/send()Function, the data may be sent to the network as soon as it is written into the buffer, or it may continue to be backlogged in the buffer. The data written multiple times is sent to the network at one time, depending on the current network situation and whether the current thread is idle Many factors are not controlled by the programmer.

read()/recv()The same is true for functions, which read data from the input buffer instead of directly from the network.

These I/O buffer characteristics can be sorted as follows:

1. I/The O buffer exists separately in each TCP socket;
2. I/O buffer is automatically generated when the socket is created;
3. Even if the socket is closed, the data left in the output buffer will continue to be transmitted;
4. Closing the socket will lose the data in the input buffer.

The default size of the input and output buffers is generally 8K, which can be passed through getsockopt()Function acquisition:

1. unsigned optVal;2.int optLen =sizeof(int);3.getsockopt(servSock, SOL_SOCKET, SO_SNDBUF,(char*)&optVal,&optLen);4.printf("Buffer length: %d\n", optVal);

Socket buffer explanation

Detailed explanation of socket buffer:
    
The sending end can send data with one K and one K, and the application at the receiving end can withdraw data in two K and two K. Of course, it is also possible to withdraw 3K or 6K data at a time, or only a few bytes of data at a time. In other words, the data seen by the application is a whole, or a stream, how many bytes in a message is invisible to the application, so the TCP protocol is a stream-oriented protocol, which is also easy The reason for the sticky package problem. UDP is a message-oriented protocol. Each UDP segment is a message. The application must extract data in units of messages, and cannot extract any byte of data at a time. This is very different from TCP. How to define the message? It can be considered that the other party writes at once/The send data is a message. What you need to understand is that when the other party sends a message, no matter how the bottom layer is fragmented, the TCP protocol layer will sort the data segments that make up the entire message before displaying it in the kernel buffer.

For example, a tcp-based socket client uploads a file to the server. When sending, the content of the file is sent in a byte stream segment by segment. When the receiver looks at it, it doesn’t know where the byte stream of the file starts. Where does it end

The so-called sticky packet problem is mainly caused by the fact that the receiver does not know the boundary between messages and how many bytes of data are extracted at one time.

In addition, the sticky packet caused by the sender is caused by the TCP protocol itself. In order to improve the transmission efficiency of TCP, the sender often needs to collect enough data before sending a TCP segment. If the data that needs to be sent several times in a row is very small, usually TCP will combine these data into a TCP segment according to the optimization algorithm and then send it out at once, so that the receiver receives the sticky packet data.

TCP (transport control protocol, transmission control protocol) is connection-oriented, flow-oriented, and provides highly reliable services. Both the receiving and sending ends (client and server) must have a pair of sockets. Therefore, the sender uses an optimization method (Nagle algorithm) in order to send multiple packets to the receiver more effectively to the other party. Combine multiple data with small intervals and small data volume into one large data block, and then packetize. In this way, the receiving end is difficult to distinguish, and a scientific unpacking mechanism must be provided. That is, flow-oriented communication has no message protection boundary.
UDP (user datagram protocol, user datagram protocol) is connectionless, message-oriented, and provides high-efficiency services. Does not use block merge optimization algorithm,,Since UDP supports one-to-many mode, the skbuff on the receiving end(Socket buffer) adopts a chain structure to record every UDP packet that arrives. In each UDP packet, there is a message header (message source address, port and other information). In this way, for the receiving end, It is easy to distinguish and deal with. That is, message-oriented communication has message protection boundaries.
TCP is based on data flow, so the messages sent and received cannot be empty. This requires adding an empty message processing mechanism on both the client and server to prevent the program from getting stuck, while udp is based on datagrams, even if you enter It is empty content (Enter directly), it is not empty message, udp protocol will help you encapsulate the message header, the experiment is omitted
The recvfrom of udp is blocking, a recvfrom(x)Sendinto(y),After receiving x bytes of data, it is complete,If y>x data is lost, which means that udp will not stick to the packet at all, but it will lose data, which is unreliable

TCP protocol data will not be lost. If the packet is not received, the next time it is received, it will continue to receive the last time. The own end will always clear the buffer content when it receives an ack. The data is reliable, but it will stick to the package.

Only TCP has the phenomenon of sticking packets, UDP will never stick to packets

# Only appears in the TCP protocol, because there is no boundary between multiple messages in the TCP protocol, and there are a lot of optimization algorithms.

# Two cases will stick to the bag:
# Sender:The receiver did not receive the packets in the buffer in time, causing multiple packets to be accepted(The client sent a piece of data, and the server only received part of it,When the server receives the data next time, it still takes the data left over from the buffer to generate sticky packets)

# Receiving end:The sender needs to wait for the buffer to be full before sending out, causing sticky packets(The time interval of sending data is very short, and the data is also very small.)

UDP protocol#####

Datagram protocol

# 1. udp protocol client allows to send empty
# 2. udp protocol will not stick to the package
# 3. If the udp protocol server does not exist, the client will still not report an error

Example1

# Socket_Server_UDP.py
import socket

sk = socket.socket(type=socket.SOCK_DGRAM)

server =('127.0.0.1',8089)while True:
 msg =input('>>>')if msg.upper()=='Q':break
 sk.sendto(b'ip,port,server',server)
 msg = sk.recv(1024)if msg.upper()=='Q':breakprint(msg)
# Socket_Client_UDP.py
import socket

sk = socket.socket(type=socket.SOCK_DGRAM)
sk.bind(('127.0.0.1',8089))while True:
 msg,addr = sk.recvfrom(1024)print(msg)
 msg=input(">>>")
 sk.sendto(msg.encode('utf-8'),addr)

Example2

#! /usr/bin/env python3
# - *- coding:utf-8-*-

# Server
import socket

server = socket.socket(type=socket.SOCK_DGRAM)
server.bind(('127.0.0.1',8000))

msg,addr=server.recvfrom(1024)print(msg.decode('utf-8'))
server.send(b'hello',addr)

server.close()

#! /usr/bin/env python3
#- *- coding:utf-8-*-

# Client
import socket

client=socket.socket(type=socket.SOCK_DGRAM)
server_addr=('127.0.0.1',8000)

client.sendto(b'hello server baby!',server_addr)
msg,addr=client.recvfrom(1024)print(msg,addr)

"""
# udp features>>>No link,It’s similar to sending a text message, just send it and the other party loves it, it doesn’t matter
# When the server is turned off, the client can still send data when it starts up. Because there is no need to consider whether the server can receive
"""
# Verify whether there is a sticky packet problem in the udp protocol
import socket
server = socket.socket(type=socket.SOCK_DGRAM)
server.bind(('127.0.0.1',8080))print(server.recvfrom(1024))print(server.recvfrom(1024))print(server.recvfrom(1024))import socket
client = socket.socket(type=socket.SOCK_DGRAM)
server_addr =('127.0.0.1',8080)
client.sendto(b'hello',server_addr)
client.sendto(b'hello',server_addr)
client.sendto(b'hello',server_addr)

Give UDP a simplified version of QQ

#! /usr/bin/env python3
# - *- coding:utf-8-*-

# Server
import socket

server = socket.socket(type=socket.SOCK_DGRAM)
server.bind(('127.0.0.1',8000))while True:
 msg,addr=server.recvfrom(1024)print(addr)print(msg.decode('utf-8'))
 info=input('>>>:').encode('utf-8')
 server.sendto(info,addr)

server.close()

#! /usr/bin/env python3
#- *- coding:utf-8-*-

# Client
import socket

client=socket.socket(type=socket.SOCK_DGRAM)
server_addr=('127.0.0.1',8000)while True:
 info=input('>>>:')
 info=('Message from client 1:%s'%info).encode('utf-8')

 client.sendto(info,server_addr)
 msg,addr=client.recvfrom(1024)print(msg.decode('utf-8'),addr)

client.close()

to sum up:

TCP protocol is similar to calling
UDP protocol is similar to sending SMS

TCP complete file upload

server.py

import socket
import subprocess
import json
import struct
phone = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

phone.bind(('127.0.0.1',8001))

phone.listen(5)
file_positon = r'd:\Upload Download'

conn, client_addr = phone.accept()

# # 1 , Receiving fixed 4 bytes
ret = conn.recv(4)
#
# 2, Use struct module to reverse ret to head_dic_The total number of bytes in bytes.
head_dic_bytes_size = struct.unpack('i',ret)[0]
#
# 3, Receive head_dic_bytes data.
head_dic_bytes = conn.recv(head_dic_bytes_size)

# 4, Head_dic_Bytes are decoded into json string format.
head_dic_json = head_dic_bytes.decode('utf-8')

# 5, Restore the json string to dictionary mode.
head_dic = json.loads(head_dic_json)

file_path = os.path.join(file_positon,head_dic['file_name'])withopen(file_path,mode='wb')as f1:
 data_size =0while data_size < head_dic['file_size']:
  data = conn.recv(1024)
  f1.write(data)
  data_size +=len(data)
    

conn.close()
phone.close()

client.py

import socket
import struct
import json
import os
phone = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  #Buy phone

phone.connect(('127.0.0.1',8001))  #Establish a connection with the client, dial

# 1 Make file_info
file_info ={'file_path': r'D:\lnh.python\pyproject\PythonReview\network programming\08 File upload and download\low version\aaa.mp4','file_name':'aaa.mp4','file_size': None,}
# 2 Get and set file size
file_info['file_size']= os.path.getsize(file_info['file_path'])

# 2 , Use json to change the head_dic into a string
head_dic_json = json.dumps(file_info)

# 3, Head_dic_json into bytes
head_dic_bytes = head_dic_json.encode('utf-8')

# 4 , The head_dic_The size of bytes is converted into a fixed 4 bytes.
ret = struct.pack('i',len(head_dic_bytes))  #Fixed four bytes

# 5, Send fixed four bytes
phone.send(ret)

# 6 Send head_dic_bytes
phone.send(head_dic_bytes)

# Send File:
withopen(file_info['file_path'],mode='rb')as f1:
    
 data_size =0while data_size < file_info['file_size']:
 # f1.read()You can’t read them all out, and you can’t send them all, so if the send is too large, there will be problems. To be on the safe side, send at most each time(1024 bytes)
  every_data = f1.read(1024)
  data_size +=len(every_data)
  phone.send(every_data)
        
phone.close()

Verify the legality of documents

server.py

import os
import socket
import hashlib

secret_key = b'alex_sb'
sk = socket.socket()
sk.bind(('127.0.0.1',9001))
sk.listen()

conn,addr = sk.accept()
# Create a random string
rand = os.urandom(32)
# Send random string
conn.send(rand)

# According to the string sent+secrete key for summary
sha = hashlib.sha1(secret_key)
sha.update(rand)
res = sha.hexdigest()

# Waiting to receive summary results from the client
res_client = conn.recv(1024).decode('utf-8')
# Compare
if res_client == res:print('Is a legitimate client')
 # If consistent,It is a legitimate client
 # And can continue
 conn.send(b'hello')else:
 conn.close()
 # If inconsistent,The connection should be closed immediately

client.py

import socket
import hashlib

secret_key = b'alex_sb979'
sk = socket.socket()
sk.connect(('127.0.0.1',9001))

# Receive a random string sent by the client
rand = sk.recv(32)
# According to the string sent+secret key for summary
sha = hashlib.sha1(secret_key)
sha.update(rand)
res = sha.hexdigest()
# The summary result is sent back to the server
sk.send(res.encode('utf-8'))
# Continue to communicate with the server
msg = sk.recv(1024)print(msg)

Socketserver realizes concurrency####

Why use socketserver? Can the socket of the tcp protocol we wrote before can only communicate with one client at a time? If you use socketserver, you can communicate with multiple clients. It is a layer of encapsulation based on the socket, which means that the bottom layer is still the called socket. It is called SocketServer in py2.7, which means two S in uppercase, and lowercase in py3. The FTP job we will write later needs to be used to achieve concurrency, that is, it can communicate with multiple clients at the same time, and multiple people can upload and download at the same time.
So let's look at how to use socketserver first, and then analyze, first look at the following code

import socketserver  #Import module

classMyServer(socketserver.BaseRequestHandler):   #The class name is defined arbitrarily, but it must inherit socketserver.BaseRequestHandler class
    
 def handle(self):  #Write a handle method, fixed name
  while1:
   # self.request is equivalent to conn pipeline
   from_client_data = self.request.recv(1024).decode('utf-8')print(from_client_data)
   to_client_data =input('The server returns information:').strip()
   self.request.send(to_client_data)if __name__ =='__main__':
 ip_port =('127.0.0.1',8080)
 # socketserver.TCPServer.allow_reuse_address = True  #Allow port reuse
 server = socketserver.ThreadingTCPServer(ip_port,MyServer)
 # To socketserver.The ThreadingTCPServer class instantiates an object, passes in the ip address, port number and class name defined by itself, and returns an object
 server.serve_forever() #Object execute serve_forever method, open the server

Specific process analysis:

# In the socketserver module, there are actually two things done: 1. One is the part of cyclically establishing links, each client link can connect successfully. 2. The part of a communication cycle, that is, after each client is successfully connected, To communicate with the client cyclically.
# Look at the code: server=socketserver.ThreadingTCPServer(('127.0.0.1',8090),MyServer)

# Remember object-oriented inheritance? Come, everyone try to see the source code:

# The order of finding attributes: ThreadingTCPServer->ThreadingMixIn->TCPServer->BaseServer

# Instantiate to get the server, first look for the ThreadMinxIn__init__Method, found that there is no init method, and then find the class ThreadingTCPServer__init__,Found in TCPServer, created socket object inside, and then execute server_bind (equivalent to bind),server_active (click to see if listen is executed)

# Find serve under server_forever,Found in BaseServer, and then execute self._handle_request_noblock(), The method is also in BaseServer

# Execute self._handle_request_noblock()And then execute the request, client_address = self.get_request()(It is self in TCPServer.socket.accept()），然后Execute self.process_request(request, client_address)

# Find process in ThreadingMixIn_request, open multi-thread to deal with concurrency, and then execute process_request_thread, execute self.finish_request(request, client_address)

# The above four parts complete the link cycle. This part begins to process the communication part, and find the finish in BaseServer_request,Trigger the instantiation of our own defined class, go find__init__Method, and the class defined by ourselves does not have this method, then go to its parent class, BaseRequestHandler, to find....

# Source code analysis summary:

# Based on the tcp socketserver in the class defined by ourselves

　　self.server is the socket object
　　self.request is a link
　　self.client_address is the client address
Based on udp socketserver in our own class defined

　　self.request is a tuple (the first element is the data sent by the client, and the second part is the udp socket object of the server), such as(b'adsf',<socket.socket fd=200, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('127.0.0.1',8080)>)
　　self.client_address is the client address
```![]( https://img2020.cnblogs.com/blog/1871335/202005/1871335-20200517124446986-1187051722.png)