Install Mesos tutorial under CentOS7
Based on centos7, install mesos and related examples to run the tutorial.
Please indicate the source for reprinting: https://blog.lzoro.com, thank you~
Introduction to Mesos###
Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos
kernel runs on every machine and provides applications(e.g., Hadoop, Spark, Kafka, Elasticsearch)with API’s for
resource management and scheduling across entire datacenter and cloud environments.
Mesos was created based on the same principles of the Linux kernel, but the difference is only at the level of abstraction. The Mesos kernel runs on each machine, and at the same time provides cross-data center and cloud resource management and scheduling capabilities (such as: Hadoop, Spark, Kafka, Elastic Search) for various applications through API.
Well, it's brief enough.
Architecture diagram
In layman's terms, it provides resource scheduling so that we can use the entire data center like a single machine.
Well, it's still not popular enough, forget it.
Mesos is the Open Source Distributed Resource Management Framework under Apache. Just remember this point. Anyway, this article is responsible for introducing the installation. After the specific usage and related concepts, there will be another chapter, Hia~ Hia~.
Environmental description###
1、 One centOS7 virtual machine(The configuration is better)2、mesos-1.6.13、xshell
The grid here uses win10 as the [host machine] (https://cloud.tencent.com/product/cdh?from=10680), installs VMware as the virtual machine platform, and then installs a centos7 virtual machine. It is recommended to allocate at least 2 core CPU and to the virtual machine 4G memory or more, otherwise the installation efficiency is relatively low, after all, it still consumes some resources.
After the virtual machine is installed and started, use xsell to connect (after all, the interface font looks more comfortable) to operate.
step###
1、 download####
Official website [download address] (http://mesos.apache.org/downloads/), please select a version you need to download.
Here you have several options
- Upload to the virtual machine via xftp after downloading on the Windows platform
Suitable for students whose virtual machines cannot connect to the Internet
- Download directly on the virtual machine
For example, download of version 1.6.1
wget http://archive.apache.org/dist/mesos/1.6.1/mesos-1.6.1.tar.gz
Suitable for students who can connect to the virtual machine, simple and convenient.
2、 System Requirements####
Mesos installation team system is required, the following instructions are excerpted from the official website.
The general meaning is as follows (make do and see, after all, it is not a professional translation, keke~)
Mesos runs on 64-bit Linux/MacOs. To build Mesos from source code, gcc 4.8.1 +/Clang 3.5 + environment is required.
On Linux, the kernel version must be higher than (inclusive) 2.628 during build/run. In order to fully support process isolation, the kernel version must be higher than (inclusive) 3.10.
The Mesos agent can also run on Windows, with additional instructions.
To allow full support of Docker's host networking function, please make sure that the host name can be resolved by DNS//etc/hosts, which is required for Mesos testing. When there is a problem, please verify that /etc/host contains the host name.
Mesos runs on Linux(64 Bit) and Mac OS X(64 Bit). To build Mesos from source, GCC 4.8.1+ or Clang 3.5+ is required.
On Linux, a kernel version >=2.6.28 is required at both build time and run time. For full support of process isolation under Linux a recent kernel >=3.10 is required.
The Mesos agent also runs on Windows. To build Mesos from source, follow the instructions in the Windows section.
Make sure your hostname is resolvable via DNS or via /etc/hosts to allow full support of Docker’s host-networking capabilities, needed for some of the Mesos tests. When in doubt, please validate that /etc/hosts contains your hostname.
For different systems, there will be different component requirements, mesos official website lists such as
- Ubuntu
- Mac OS
- centos
Let's take a look at the environmental requirements of centos.
I don’t know the version of my system.
cat /etc/redhat-release
# There will be similar output CentOS Linux release 7.5.1804(Core)
In addition, if you are not a root user, please remember to use the sudo command. This article is based on the operation of the root user, so the sudo command is omitted.
Install some utilities
yum install -y tar wget git
Obtain the Apache Maven repo file.
wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
Install REPL repo
yum install -y epel-release
Mesos versions higher than 0.21.0 require a development package of subversion> 1.8, which is not available in the default library, so it is in the file'/etc/yum.repos.d/wandisco-svn.repo' Add the following content pieces to install the correct version.
bash -c 'cat >/etc/yum.repos.d/wandisco-svn.repo <<EOF
[ WANdiscoSVN]
name=WANdisco SVN Repo 1.9
enabled=1
baseurl=http://opensource.wandisco.com/centos/7/svn-1.9/RPMS/\$basearch/
gpgcheck=1
gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
EOF'
Part of Mesos requires Systemd to operate, but Mesos only supports the systemd version that contains the Delegata logo.
This flag was first introduced in systemd version 218, and it is lower than the version installed by default under centos.
Fortunately, centos7.1 has a patch systemd<218 that contains the Delegate flag, and the following operation is to upgrade systemd.
yum update systemd
Install the necessary development tools
yum groupinstall -y "Development Tools"
Install other Mesos dependencies (it will take some time, if the machine and the network are relatively ordinary, you can go to make some tea first~)
yum install -y apache-maven python-devel python-six python-virtualenv java-1.8.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel
At this point, the system dependencies are basically processed.
3、 Compile and install
Remember where is the mesos-version-no.tar.gz you just downloaded (version-no is the version number you downloaded). If you forget, you may have to eat some walnuts to make a long memory~
Provide a search command, using find [path] -options [regx], you can use the specified options to find files that match the expression in the specified directory.
find /usr/local/-iname mesos*
If you remember where, go to the directory where the compressed package is located and unzip
# Switch directory
cd /usr/local/setup
# Unzip
tar zxvf mesos-1.6.1.tar.gz
If it is downloaded from the git repository, execute the following command first (skip if not)
. /bootstrap
Configure and build installation
mkdir build
cd build
# The above two commands can also be combined into
mkdir build & cd $_
# Configuration
.. /configure --prefix=/usr/local/mesos-1.6.1
# Has similar output
configure: Build option summary:
CXX: g++
CXXFLAGS:-g1 -O0 -Wno-unused-local-typedefs -std=c++11
CPPFLAGS:-I/usr/include/subversion-1-I/usr/include/apr-1-I/usr/include/apr-1.0
LDFLAGS:
LIBS:-lz -lsvn_delta-1-lsvn_subr-1-lsasl2 -lcurl -lapr-1-lrt
# Compile (depending on the machine configuration, the time is different)
make
# Not to mention, I'll go make some tea first.
.........
# Years later, there is output similar to the following
running install_scripts
creating build/bdist.linux-x86_64/wheel/mesos.scheduler-1.6.1.dist-info/WHEEL
make[2]: Leaving directory `/usr/local/setup/mesos-1.6.1/build/src'
make[1]: Leaving directory `/usr/local/setup/mesos-1.6.1/build/src'
# Run the test case(Optional, if you want to run the following example, this step cannot be omitted)
make check
# installation
make install
# After installation, there will be output similar to the following
for name in libfixed_resource_estimator \
libload_qos_controller \
liblogrotate_container_logger;do \
for lib in`cd //usr/local/mesos-1.6.1/lib/mesos/modules && ls ${name}*`;do \
rm -f $lib; \
ln -s mesos/modules/$lib $lib; \
done; \
done
make[4]: Leaving directory `/usr/local/setup/mesos-1.6.1/build/src'
make[3]: Leaving directory `/usr/local/setup/mesos-1.6.1/build/src'
make[2]: Leaving directory `/usr/local/setup/mesos-1.6.1/build/src'
make[1]: Leaving directory `/usr/local/setup/mesos-1.6.1/build/src'
Some examples###
Mesos is bundled with Java/Python/C++ framework examples, and the framework binaries are only available after running mmake check (run this command will compile the example framework), as described in the above section.
The following describes the use of examples.
# Enter the compilation directory
$ cd build
# Turn on Mesos master(Need to ensure the working directory/var/lib/mesos exists and has relevant permissions).
# If you are not visiting on this machine, allow intranet access, you can leave it alone--ip option
$ ./bin/mesos-master.sh --ip=127.0.0.1--work_dir=/var/lib/mesos
# Open Mesos agent(Need to ensure the working directory/var/lib/mesos exists and has relevant permissions).
$ ./bin/mesos-agent.sh --master=127.0.0.1:5050--work_dir=/var/lib/mesos
Run the framework example below
# Visit web page.
$ http://127.0.0.1:5050
The approximate interface is as shown below
Console
# Run C++frame(After the task runs successfully, it will exit).
$ ./src/test-framework --master=127.0.0.1:5050
# Run the Java framework(After the task runs successfully, it will exit).
$ ./src/examples/java/test-framework 127.0.0.1:5050
# Run the Python framework(After the task runs successfully, it will exit).
$ ./src/examples/python/test-framework 127.0.0.1:5050
The grid here only ran the Java framework example, there will be the following output
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0825 12:55:18.95426939142 sched.cpp:1719]**************************************************
Scheduler driver bound to loopback interface! Cannot communicate with remote master(s). You might want to set'LIBPROCESS_IP' environment variable to use a routable IP address.**************************************************
I0825 12:55:18.95689239142 sched.cpp:232] Version:1.6.1
I0825 12:55:18.96606139163 sched.cpp:336] New master detected at [email protected]:5050
I0825 12:55:18.96939339163 sched.cpp:351] No credentials provided. Attempting to register without authentication
I0825 12:55:18.99141139159 sched.cpp:749] Framework registered with 0d317686-be4b-42b8-9e3f-ff3e0068991f-0000
Registered! ID = 0d317686-be4b-42b8-9e3f-ff3e0068991f-0000
Received offer 0d317686-be4b-42b8-9e3f-ff3e0068991f-O0 with cpus:4.0 and mem:2750.0
Launching task 0 using offer 0d317686-be4b-42b8-9e3f-ff3e0068991f-O0
Launching task 1 using offer 0d317686-be4b-42b8-9e3f-ff3e0068991f-O0
Launching task 2 using offer 0d317686-be4b-42b8-9e3f-ff3e0068991f-O0
Launching task 3 using offer 0d317686-be4b-42b8-9e3f-ff3e0068991f-O0
Status update: task 2 is in state TASK_RUNNING
Status update: task 1 is in state TASK_RUNNING
Status update: task 0 is in state TASK_RUNNING
Status update: task 3 is in state TASK_RUNNING
Status update: task 2 is in state TASK_FINISHED
Finished tasks:1
Status update: task 1 is in state TASK_FINISHED
Finished tasks:2
Status update: task 0 is in state TASK_FINISHED
Finished tasks:3
Status update: task 3 is in state TASK_FINISHED
Finished tasks:4
Received offer 0d317686-be4b-42b8-9e3f-ff3e0068991f-O1 with cpus:4.0 and mem:2750.0
Launching task 4 using offer 0d317686-be4b-42b8-9e3f-ff3e0068991f-O1
Status update: task 4 is in state TASK_RUNNING
Status update: task 4 is in state TASK_FINISHED
Finished tasks:5
I0825 12:55:20.58519939160 sched.cpp:2013] Asked to stop the driver
I0825 12:55:20.58567639160 sched.cpp:1189] Stopping framework 0d317686-be4b-42b8-9e3f-ff3e0068991f-0000
I0825 12:55:20.58954839142 sched.cpp:2013] Asked to stop the driver
In addition, the console can see
Console-Java sample framework
Afterword
I learned about Mesos before, but I didn’t take the time to practice it. I took advantage of the rain on the weekend and did not go out. I tried to install and write some Hello World. After all, it’s too shallow on the paper, I definitely know this matter.
I slipped away, if it helps you a little, you can give me a thumbs up~3 grams Yo
Reference material: Mesos official website
Recommended Posts