Centos7 build python3.8.5+scrapy+gerapy
Environment: CentOS Linux release 7.6.1810 (Core) mini
Update Centos7 yum source
yum -y install epel-release
Install the compilation environment
yum groupinstall -y "Development tools"
Update bash and openssl vulnerabilities and basic software
yum install -y bash openssl* ntp vim wget telnet nscd
Update server time
ntpdate ntp1.aliyun.com
Install python3 may use dependencies
yum install -y openssl-devel bzip2-devel expat-devel gdbm-devel readline-devel sqlite-devel zlib zlib-devel libffi-devel
yum install -y readline-devel.x86_64 (You must install this package, otherwise you will find that deleting the backspace key will not work after the upgrade is complete)
Download the latest version of python
cd /opt/
wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz
Compile and install python3.8.5
tar zxvf Python-3.8.5.tgz -C /usr/src/
cd /usr/src/Python-3.8.5
Note: --with-ssl must be specified, otherwise when using the pip3 installation package, when accessing the ssl link, a TLS/SSL error will be reported!
. /configure --with-ssl --prefix=/usr/local/python38
make && make install
The following message appears, indicating successful installation
Installing collected packages: setuptools, pip
WARNING: The script easy_install-3.8 is installed in '/usr/local/python38/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts pip3 and pip3.8 are installed in '/usr/local/python38/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-20.1.1 setuptools-47.1.0
Add python38 to the system environment variables
vim /etc/profile
Add in the last line
export PATH=$PATH:/usr/local/python38/bin
Load environment variables
source /etc/profile
Add soft link
ln -s /usr/local/python38/bin/python3.8 /usr/bin/python38
ln -s /usr/local/python38/bin/pip3.8 /usr/bin/pip38
Upgrade pip
/usr/local/python38/bin/python3.8 -m pip install --upgrade pip
View python version
python38 -V
Output the following information:
Python 3.8.5
Install gerapy
pip38 install gerapy
The following message appears, indicating that the installation was successful
Successfully installed Automat-20.2.0 MarkupSafe-1.1.1 PyDispatcher-2.0.5 PyHamcrest-2.0.2 Twisted-20.3.0 appdirs-1.4.4 apscheduler-3.5.1 attrs-20.1.0 beautifulsoup4-4.9.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 constantly-15.1.0 cryptography-3.1 cssselect-1.1.0 django-1.11.29 django-apscheduler-0.3.0 django-cors-headers-3.2.0 djangorestframework-3.9.2 furl-2.1.0 gerapy-0.9.6 gevent-20.6.2 greenlet-0.4.16 hyperlink-20.0.1 idna-2.10 incremental-17.5.0 jinja2-2.10.1 lxml-4.5.2 orderedmultidict-1.0.1 parsel-1.6.0 protego-0.1.16 pyOpenSSL-19.1.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pyee-7.0.2 pymongo-3.11.0 pymysql-0.10.0 pyppeteer-0.2.2 pyquery-1.4.1 python-scrapyd-api-2.1.2 pytz-2020.1 queuelib-1.5.0 redis-3.5.3 requests-2.24.0 scrapy-1.8.0 scrapy-redis-0.6.8 scrapy-splash-0.7.2 service-identity-18.1.0 six-1.15.0 soupsieve-2.0.1 tqdm-4.48.2 tzlocal-2.1 urllib3-1.25.10 w3lib-1.22.0 websocket-0.2.1 websockets-8.1 zope.event-4.4 zope.interface-5.1.0
Create working directory
mkdir /data/gerapy_scrapy
gerapy initialization directory
[ root@localhost data]# gerapy init /data/gerapy_scrapy/
Initialized workspace /data/gerapy_scrapy/
gerapy initialize database
cd /data/gerapy_scrapy
[ root@localhost gerapy_scrapy]# gerapy migrate
Operations to perform:
Apply all migrations: admin, auth, authtoken, contenttypes, core, django_apscheduler, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying authtoken.0001_initial... OK
Applying authtoken.0002_auto_20160226_1747... OK
Applying core.0001_initial... OK
Applying core.0002_auto_20180119_1210... OK
Applying core.0003_auto_20180123_2304... OK
Applying core.0004_auto_20180124_0032... OK
Applying core.0005_auto_20180131_1210... OK
Applying core.0006_auto_20180131_1235... OK
Applying core.0007_task_trigger... OK
Applying core.0008_auto_20180703_2305... OK
Applying core.0009_auto_20180711_2332... OK
Applying core.0010_auto_20191027_2040... OK
Applying django_apscheduler.0001_initial... OK
Applying django_apscheduler.0002_auto_20180412_0758... OK
Applying sessions.0001_initial... OK
Create an administrator user, the password needs to meet the complexity
[ root@localhost gerapy_scrapy]# gerapy createsuperuser
Username (leave blank to use 'root'): admin
Email address: [email protected]
Password:
Password (again):
Superuser created successfully.
View selinux status
[ root@localhost gerapy_scrapy]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31
Close selinux and restart to take effect
vim /etc/selinux/config
SELINUX=disabled
Temporarily effective
setenforce 0
In order to facilitate the direct clear iptables strategy
iptables -F
Run gerapy
gerapy runserver 0.0.0.0:8000
Visit gerapy: ip address + port
http://192.168.18.131:8000/#/login
Django background login url (account password can be changed)
http://192.168.18.131:8000/admin/login/
Recommended Posts