python learning route

reptile

1. Basic knowledge: basic principles of website, html, python, multi-process/multi-thread/coroutine, etc. (must learn)

2. HTML foundation, network request module: requests (must learn), urllib (you can understand)

3. Need to understand some common anti-climbing strategies and corresponding solutions: common ones include IP frequency restrictions, User-Agent, Referer, Origen verification, cookie restrictions, dynamic loading and verification codes, etc.,

Corresponding processing methods include IP proxy pool, forged header, cookie storage and processing (basic advanced)

4. Web page analysis extraction: Beautifulsoup&Xpath (choose one), regular expression (must learn)

5. Dynamic execution of JS, js encryption and Selenium, OCR recognition or coding platform (optional)

6. Data storage (file read and write, database, Excel/CSV module, etc.) (required)

7. Network packet capture analysis (optional)

8. Crawler framework: Scrapy (optional), pyspider (optional)

9. Distributed crawler (optional)

Data analysis and processing

1. Basic knowledge: python (functions, modules, object-oriented), regular expressions, JSON (must learn)

2. Related to the above crawler:

·Basic knowledge: basic principles of website, html, python, multi-process/multi-thread/coroutine, etc. (required)

·HTML basis, network request module: requests (must learn), urllib (you can understand)

·Need to understand some common anti-climbing strategies and corresponding solutions: common IP frequency restrictions, User-Agent, Referer, Origen verification, Cookie restrictions, dynamic loading and verification codes, etc.,

Corresponding processing methods include IP proxy pool, forged header, cookie storage and processing (basic advanced)

·Web page analysis and extraction: Beautifulsoup&Xpath (choose one), regular expression (must learn)

·Dynamic execution of JS, js encryption and Selenium, OCR recognition or coding platform (optional)

·Data storage (file read and write, database, Excel/CSV module, etc.) (required)

3. Data analysis related libraries: Pandas, Numpy, Scipy, stutter analysis, etc. (required)

4. Chart drawing and visualization: Matplotlip, word cloud (must learn)

Big data (data mining, machine learning)

1. Basic knowledge: python (basic + advanced) (must learn)

2. Finance, statistics, econometrics, investment (required)

3. Data storage (file read and write, database, Excel/CSV module, etc.) (required)

4. Data analysis related libraries: Pandas, Numpy, Scipy, stammering word segmentation (must learn)

5. Chart drawing and visualization: Matplotlip, etc. (required)

6. Machine learning related model knowledge: Naive Bayes, decision tree, Logistic regression, linear regression, KNN algorithm, SVM,

Boosting, clustering, recommendation system, pLSA, LDA, GDBT, Regularization, anomaly detection, EM algorithm, Apriori,

FP Growth, etc. (required)

7. Machine learning related libraries: sklearn (required), keras, statsmodels, tensorflow (optional)