Source Example

Apparently CentOS doesn’t have a terminal shortcut

2020-10-22T00:00:00+00:00

Since I was using Ubuntu, it seems that terminal cannot be opened with a shortcut such as Ctrl + Alt + t.

Click [Applications] → [System Tools] → [Settings] → [Devices] → [Keyboard] and click + at the bottom.

To add a shortcut
Name: terminal (anything is fine)
Command: gnome-terminal
Shortcut: Ctrl + Alt + t (if you like)

Press the add button

that’s all!

Reference url https://sironekosukaru.hatenablog.com/entry/2018/01/20/160510

I tried to easily put CentOS-7 in a PC that I no longer need

2020-10-22T00:00:00+00:00

at first

Things necessary
PC, USB (about 8GB)

Obtaining installation media
https://buildlogs.centos.org/rolling/7/isos/x86_64/CentOS-7-x86_64-Everything-1511.iso

Create USB installation media from ISO file
Use Rufus
https://github.com/pbatard/rufus/releases/download/v3.12/rufus-3.12.exe

Specify the iso file from the boot type selection and
Specify the partition configuration, target system, and file system and press the start button.

If you see this after pressing the start button, select Write as ISO image

Since such a warning message appears, if the data in the USB memory to be used remains, move it to a PC etc. before doing it.

When the writing is completed, a notification sound will sound, so this completes the USB installation media creation.

PC USB boot
After inserting the USB installation media into the PC you no longer need, start the PC and then press F12 (depending on the PC model) to enter the Boot Menu.

Select USB to enter the CentOS 7 installation screen

Select Install CentOS 7

Select language as Japan

I’ll go into the installation overview so what to do
・ Software selection (insert GUI)
-Select installation destination

Select the server (using GUI) in the software selection

Select installation destination
Select a PC disk
Partition configuration is automatic (manual is also acceptable)

If you press [Finish], you will get angry if there is no area, so select [Secure area].

Select [Delete All] → [Reuse Space]

When you return to the installation summary, the yellow display disappears and the minimum settings are completed, so select [Start Installation].

Set ROOT password and user during installation

Set ROOT password and complete

Set the full name, user name and password in user creation and complete

When the user settings are completed, a screen like this will appear. All you have to do is wait for the installation

Reboot when installation is complete

After restarting, you will be asked for a license, so agree and complete the setting

Done

Publish IP camera videos on web pages

2020-10-22T00:00:00+00:00

Publish a video of an IP camera installed at home on a web page.

Thanks to ahonomics, I am now a country of poor people without hiding, and I try to achieve this on the lowest possible budget. At this time, I rely on (1) cheap and high-quality IP cameras made in China, (2) free and high-quality free software (open source software), and (3) useful life. It is a used laptop computer that has passed.

As prerequisites, (A) a home LAN with an optical line and (B) a web server installed on the Internet are assumed. (A) is mandatory. It would be possible without (B), but the intended readers of this article will not complain even if we proceed as if it were.

1: IP camera

The IP camera used is a model called SV-801W-1080P-HX of the SV3C brand, and the main specifications are as follows.

–Aluminum die-cast housing, IP66 waterproof / dustproof
–Fixed angle of view 2 million pixels (1920 x 1080)
–Both wired / WiFi compatible
–Power supply by AC adapter
–Infrared night vision photography
–Motion detection
–Supports micro SD card (up to 64 GB)
–RTSP compatible
–ONVIF compatible

send e-mail
–FTP upload

The price is 4,899 yen (as of October 2020) on Amazon.co.jp, which is one of the cheapest, but at first glance it is robust and equipped with all the functions expected of this kind of surveillance camera. ing. There are other similar products, but I have the impression that the manuals on the support site are relatively substantial, so I choose the SV3C one.

It’s important to support ** RTSP ** and ** ONVIF **, but it’s rare for today’s IP cameras to support it.

1-1: IP camera setup

Set up the camera before installing it outdoors (on the edge of the house).

The following explanation is based on the SV3C SV-801W-1080P-HX, but I think that the procedure will be similar for cameras from other companies. I’m sorry if it’s different.

1-1-1: Micro SD card

First, attach the micro SD card (64GB, 1,596 yen) that you bought together to the camera.

This product is a little more difficult to work with because you can’t access the card slots on the board without removing the lens cover and pulling out the contents. However, once inserted, it is a card that will never be taken out, so I think this is fine.

The camera works without a card. You don’t need a card for video distribution, and even if you use it for crime prevention, you can omit the card when connecting to NVR (Network Video Recorder).

However, if you have a card installed, you can automatically record and save videos and snapshots. Besides, adding cards after installation is a hassle. It’s cheap, so it’s better to wear it from the beginning.

1-1-2: Connect to home LAN

Turn on the camera by connecting it to a router (or hub, etc.) on your home LAN with a LAN cable.

The router’s DHCP will now get the camera a dynamic IP address.

1-1-2 : CamHi

Install and launch an app called CamHi on your smartphone. By searching for and connecting to an IP camera connected to your home LAN with this app, you will be able to watch videos and access camera settings. Needless to say, smartphones need to be connected to the home LAN via WiFi in advance.

Subsequent settings can be done with CamHi, but it is easier to access the camera settings screen with a web browser, so just make a note of the camera’s IP address and CamHi will quit. (It will be used again later to check the image during outdoor installation work.)

1-1-3: Setting by web browser

Access the camera’s IP address with a web browser and configure the camera settings.

The browser uses ** Internet Explorer **. You will first be asked to install the ActiveX for video playback. I wonder what this is, but follow the instructions. It can be set with other browsers, but it is inconvenient because the image is not displayed. Give up and use IE.

There are a wide variety of items that can be set. For items that you do not understand well, leave the default settings and make the following initialization and setting changes that are considered to be the minimum necessary.

–Change the administrator password
–The IP address is a static address (here, 192.168.0.101)
–Set the time zone to GMT +09: 00
–Set the NTP server to ntp.nict.jp
–Initialize SD card

This time, we will go with a wired connection in consideration of the installation location and stability, so turn off WiFi. Also, since the purpose is different, it is set not to use functions for crime prevention such as infrared night vision, motion detection, alarm by e-mail or FTP.

1-2: Outdoor installation work of IP camera

Install an IP camera on the eaves of your home, where you will be shooting videos.

While checking the image with CamHi of the smart phone, decide the installation position and shooting direction and fix it. Since it is a wide-angle lens, it is inevitable that the peripheral part will be distorted, but since the human eye is sensitive to horizontal deviation, the angle of the camera is determined by paying close attention to the horizontal and vertical parts of the center of the image.

Since it is a place exposed to rain, the joint between the LAN cable and the power cable should be waterproofed with self-bonding tape.

1-3: Delivery of still images

With the SV3C HX series, it will be possible to deliver still images at this point.

There is a function to upload snapshots to an external FTP server on a regular basis, so if you take advantage of this and perform appropriate processing on the web server side, you can display almost real-time still images on a web page. You can.

This time, we will create a page to distribute still images in case the video distribution stops.

1-3-1: Client side (HTML + javascript)

<img id="live-img" width="950" height="534" 
  src="/livecam/snapshot" 
  alt="Live camera snapshot" />

The source of the image is the URL / livecam / snapshot. It makes the work on the server side easier than specifying it by file name like " snapshot.jpg ". You no longer have to worry about the negative effects of the file cache.

function updateImage() {
    $('#live-img').attr('src', '/livecam/snapshot');
}
var timer = setInterval(updateImage, 30000);

Then the javascript timer requests the image from the server every 30 seconds.

1-3-2: Server side

On the server side, the image data is sent to the client from the action corresponding to the URL / livecam / snapshot.

Since snapshots from the camera are being uploaded one after another to the directory specified in advance, select the newest file among them, read the contents, and add the header of 'Content-Type: image / jpeg'. And send it.

To put it in detail, it requires some troublesome processing such as resizing images and deleting unnecessary files, but I will omit the details.

2: Video distribution

2-1: Schematic diagram

In order to distribute videos from IP cameras, ** video conversion server ** and ** video distribution server ** are required in addition to the conventional web server, as shown in the figure.

First, the ** video conversion server ** converts the stream of video data output by the camera according to ** RTSP ** (Real Time Streaming Protocol) into data according to ** RTMP ** (Real Time Messaging Protocol). Convert. The substance of this video conversion server is an application program called ** ffmpeg **.

Then, the ** video distribution server ** that receives the output of ** RTMP ** from the video conversion server distributes the stream converted to the ** HLS ** (HTTP Live Streaming) protocol to the web client. The substance of this video distribution server is a web server program called ** NGINX ** equipped with ** nginx-rtmp-module **.

Although not supported this time and not shown in the figure, this video distribution server can distribute ** RTMP ** streams as they are, or ** MPEG-DASH ** protocol (DASH is Dynamic Adaptive Streaming over HTTP). It is also possible to convert it to) and distribute it.

2-2: Video server

Make a used laptop computer into a server that combines video conversion and video distribution, and install it on your home LAN.

In some cases, it may be better to have only a video conversion server on the home LAN and let a web server on the Internet also serve as video distribution.

2-2-1 : CentOS 8

Install CentOS 8 on your laptop. Nothing has changed in particular.

The reason for CentOS is simply because you’re used to it, and there’s no other reason. It should be any linux distribution.

The IP address of this video server on the home LAN is 192.168.0.99.

2-2-2 : ffmpeg

There are many articles on the net that build ffmpeg from source, but if you go there, you’ll usually be in distress. Until a decade ago, I would have had no choice but to do so, but now that I have a package, I don’t have to worry about it.

Specifically, install ffmpeg using the ** RPM Fusion ** repository.

First, the ʻEPEL` repository is required, so install it.

# dnf -y install epel-release

Then enable the repositories PowerTools and ʻepel-playground`.

# dnf config-manager --set-enabled PowerTools
# dnf config-manager --set-enabled epel-playground

Then install the RPM Fusion repository.

# rpm -ivh https://download1.rpmfusion.org/free/el/rpmfusion-free-release-8.noarch.rpm

You can now install ffmpeg.

# dnf install ffmpeg

For details, see ** “Install FFmpeg on CentOS 8 with RPMFusion repo” **. The title is English, but the article is in Japanese.

2-2-3 : NGINX + nginx-rtmp-module

If you only use NGINX, you can use the package, but in order to use nginx-rtmp-module, you need to get the source and build it.

** NGINX official website ](https://www.nginx.com) technical blog [ “Enabling Video Streaming for Remote Learning with NGINX and NGINX Plus” **.

First, install the tools you use to build.

$ sudo yum update
$ sudo yum groupinstall "Development Tools"
$ sudo yum install git

Next, install the dependent packages.

$ sudo yum groupinstall pcre-devel zlib-devel openssl-devel

Then, create an appropriate build directory, go into it, and clone the sources of nginx-rtmp-module and nginx from github to get them.

$ cd /path/to/build/dir
$ git clone https://github.com/arut/nginx-rtmp-module.git
$ git clone https://github.com/nginx/nginx.git

Then go into the nginx directory, add nginx-rtmp-module to your configuration, and run the build and install.

$ cd nginx
$ ./auto/configure --add-module=../nginx-rtmp-module
$ make
$ sudo make install

This will make NGINX work, so check it just in case.

$ sudo systemctl start nginx

When you access the home page of the video server (http://192.168.0.99/) from a personal computer connected to your home LAN, the NGINX welcome page should be displayed.

I can’t stream the video yet (naturally because I haven’t set it up yet), but once I put in ** ffmpeg ** and ** NGINX + nginx-rtmp-module **, it’s like winning. ..

The rest is the setting of video conversion by ffmpeg and the setting of video distribution by NGINX, but before that, I will finish the work of environment maintenance necessary for the operation of the video server.

2-2-4: Video server environment maintenance

2-2-4-1: Firewall

Open ports for http, https, RTSP, and RTMP.

$ sudo firewall-cmd --permanent --add-service=http 
$ sudo firewall-cmd --permanent --add-service=https 
$ sudo firewall-cmd --permanent --add-service=rtsp
$ sudo firewall-cmd --permanent --add-port=1935/tcp
$ sudo firewall-cmd --reload

1935 / tcp is the standard port for RTMP. RTSP uses 554 / tcp and 554 / udp as standard ports, but in CentOS 8 it can be set using --add-service = rtsp.

2-2-4-2: Router port mapping

Change the port mapping setting of the router of the home LAN so that the video server can be accessed from the Internet, and the incoming call to the WAN side global address is the local address of the LAN side video server (192.168.0.99). Try to go to.

You can just use http (80), https (443), rtmp (1935), but also map ssh (22) for remote management.

2-2-4-3 : DDNS (Dynamic DNS)

Use the DDNS (Dynamic DNS) service so that you can specify the video server by domain name instead of IP address.

Change server_name in nginx.conf to the DDNS domain name (assuming it is camsvr.on.ddns) and restart NGINX.

`nginx.conf`

http {
    server {
        server_name  camsvr.on.ddns;

At this point, I’d like to see if the router’s port mapping and DDNS are working as expected, but to do that I need to go out of my home LAN. When I access http://camsvr.on.ddns/ with a web browser on a PC connected to my home LAN, the NGINX welcome page of the video server is not displayed, and the home page of the router (http:: //192.168.0.1/) is displayed. In short, port mapping does not work for access from within the LAN. It’s a good idea to use a smartphone browser with WiFi turned off for testing.

2-2-4-4 : Let’s Encrypt

The plan is that the video distribution web page will be a mix of content from the new video server with the page provided by the existing web server. The web server is already SSL and provides the https page as a whole. If there is a http content that does not support SSL, it will be sick.

Therefore, let’s Encrypt is used to make the video server SSL.

For the procedure, please refer to ** “Simple free SSL conversion to CentOS8 (nginx) using certbot & SSL automatic update” **.

After completing the above (1) firewall, (2) router port mapping, and (3) DDNS settings, (4) start certbot with NGINX running. Then, at the same time as getting the certificate, it will make appropriate changes to nginx.conf and make the video server SSL.

`nginx.conf`

http {
    server {
        server_name  camsvr.on.ddns;
        listen 443 ssl; # managed by Certbot
        ssl_certificate /etc/letsencrypt/live/camsvr.on.ddns/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/camsvr.on.ddns/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    }
    server {
        if ($host = camsvr.on.ddns) {
            return 301 https://$host$request_uri;
        } # managed by Certbot
        listen 80;
        server_name  camsvr.on.ddns;
        return 404; # managed by Certbot
    }
}

The commented # managed by Certbot is where certbot rewrote or added.

2-2-4-5: CORS header

As mentioned just above, video distribution web pages will be a mix of content from the new video server with pages provided by existing web servers. At this time, the problem of CORS (Cross-Origin Resource Sharing) arises.

To deal with this, add a CORS-enabled header to the response from the video server.

`nginx.conf`

http {
    server {
        location / {
            # CORS start
            add_header 'Access-Control-Allow-Origin' 'https://my.web.server' always;
            # Allow CORS preflight requests
            if ($request_method = 'OPTIONS') {
                add_header 'Access-Control-Allow-Origin' 'https://my.web.server';
                add_header 'Access-Control-Max-Age' 1728000;
                add_header 'Content-Type' 'text/plain charset=UTF-8';
                add_header 'Content-Length' 0;
                return 204;
            }
            # CORS end
        }
    }
}

The setting is to allow only specific origins who know their identity. To be honest, I’m not sure about CORS. For the time being, shake it off to the safe side.

2-2-5: Video distribution by NGINX

** “Enabling Video Streaming for Remote Learning with NGINX and NGINX Plus” ** Add the video distribution settings to nginx.conf.

The following are the settings for HLS.

`nginx.conf`

http { 
    server { 
        location / { 
            root /tmp/hls; 
        } 
    }
    types {
        application/vnd.apple.mpegurl m3u8;
        video/mp2t ts;
        text/html html;
    } 
}
rtmp { 
    server { 
        listen 1935; 
        application livecam { 
            live on; 
            interleave on;
            hls on; 
            hls_path /tmp/hls; 
            hls_fragment 5s; 
        } 
    } 
} 

–http> server> location / root changed to / tmp / hls
–Additional settings for http> types blocks
–In addition to html, set the types of m3u8 and ts for HLS
–Additional settings for rtmp block
–hls_path should be / tmp / hls

The location> root setting on the http side and the hls_path setting on the rtmp side are paired. This video server is set to deliver HLS videos directly under / .

2-2-6: Video conversion by ffmpeg

Finally, ffmpeg converts the IP camera’s RTSP stream into an RTMP stream and sends it to the video distribution server.

$ ffmpeg -i rtsp://192.168.0.101/12.amp -c copy -an -f flv rtmp://192.168.0.99/livecam/std

–-i rtsp: //192.168.0.101/12.amp … Specify the RTSP stream of the IP camera as input.
–The 12 part depends on the IP camera used.
–SV3C SV-801W-1080P-HX provides two streams, 11 (1920 x 1080) and 12 (640 x 352). The latter is used here.
–-c copy … Leave the codec as input.
–-an … No audio output.
–-f flv … Specify the output format. FLV is Flash Video.
– rtmp: //192.168.0.99/livecam/std … Specify the output destination.
–livecam … application name.
–std … Stream name.
–The URL for RTMP will be rtmp: //camsvr.on.ddns/livecam/std.
–The URL for HLS will be https://camsvr.on.ddns/std.m3u8.

If you don’t need to process the video, you can use -c copy. It saves you the trouble of decoding the input and encoding the output, so there is almost no load on the CPU.

Resizing the image and adjusting the tint can be a bit more complicated and CPU-intensive.

$ ffmpeg -i rtsp://192.168.0.101/12.amp -c:v libx264 -b:v 512k -r 15 -vf hue=h=10 -an -f flv rtmp://192.168.0.99/livecam/std

–-c: v libx264 … Specify the video codec. It is H.264.
–-b: v 512k … Specify the bit rate of the video. 512kb / sec.
–-r 15 … Specifying the frame rate. 15 fps.
–-vf hue = h = 10 … Specify the video filter. Rotate the hue 10 degrees.

The final hue adjustment is to correct the redness of the image, whether it is a characteristic of this IP camera as a model or an individual.

The strength of ffmpeg is endless.

2-2-6-1: ffmpeg monitoring (obsoleted)

If the RTSP stream is disturbed due to a malfunction of the IP camera, ffmpeg may end. That’s not good as a video conversion server, so create a shell script that monitors the operation of ffmpeg and restarts it if it stops.

#!/usr/bin/bash
# The script runs for 18 hours (12 x 60 x 18 = 12960)
limit=12960
counter=0
while [ $counter -le $limit ]; do
    sleep 5
    alive=`ps -aef | grep ffmpeg | grep livecam | wc -l`
    # echo `date` ":ffmpeg alive ="$alive
    if [ $alive = 0 ]; then
        echo `date` " : ffmpeg is not running."
        echo "starting ffmpeg ..."
        ffmpeg -i rtsp://192.168.0.101/12.amp -c:v libx264 -an -r 15 -b:v 512k -vf hue=h=10 -f flv rtmp://192.168.0.99/livecam/std 2>>/var/log/nginx/ffmpeg.log &
        echo "done."
    fi
    ((counter++))
done

Check ffmpeg for survival every 5 seconds for 18 hours and launch if dead. Start this script with cron every day at 3:00 AM. It is a crazy setting that you can die between 9:00 PM and 3:00 AM.

2-2-6-2: Start ffmpeg on NGINX

Actually, you don’t have to start ffmpeg yourself and monitor it as described above. Just start ffmpeg on NGINX.

`nginx.conf`

        application livecam { 
            live on; 
            interleave on;
            hls on; 
            hls_path /tmp/hls; 
            hls_fragment 5s;
            exec_static /usr/bin/ffmpeg -i rtsp://192.168.0.101/12.amp -c:v libx264 -an -r 15 -b:v 512k -f flv rtmp: //192.168.0.99/livecam/std 2>>/var/log/nginx/ffmpeg-std.log;
        } 

I’m using ʻexec_static here because, for some reason, ʻexec_push doesn’t start ffmpeg as expected in my environment. This failure is also recorded as an open issue on github of nginx-rtmp-module, which seems to bother a lot of people.

Fortunately, ʻexec_static` will start successfully. And even if you kill the launched ffmpeg, another instance will be launched immediately, probably because NGINX is monitoring it. When you quit NGINX, ffmpeg also quits. This is fine.

2-3: Client side

On the client side, use the JavaScript video playback library ** VIDEO JS **.

<head>
    <link href="https://vjs.zencdn.net/7.8.4/video-js.css" rel="stylesheet" />
</head>

<body>
    <video id="live-cam" class="video-js" controls autoplay
            width="640" height="352" data-setup="{}">
        <source src="https://camsvr.on.ddns/std.m3u8" type="application/x-mpegURL" />
        <p class="vjs-no-js">
To watch this video, enable JavaScript and
Use a browser that supports the HTML5 video tag.
        </p>
    </video>

    <script src="https://vjs.zencdn.net/7.8.4/video.js"></script>
</body>

For details, see ** “Getting Started” ](https://videojs.com/getting-started) and documentation on [ VIDEO JS Official Site **. Please refer.

That’s it for publishing IP camera videos on web pages.

If there is something wrong, strange, or something that needs improvement, please let us know.

[Introduction] A collection of carefully selected links from how to get started on GitHub to how to solve errors when you encounter an error on Git.

2020-10-22T00:00:00+00:00

How to use Git such as Windows10, CentOS7

What you need to know before using GitHub How to use Git

[Git] Basic commands

How to resolve Git errors

Commit failed –Exit code 1 Received remedy When this error occurs, a file called .git is created in the file. Because it is done. So you can commit by showing the hidden files and deleting the .git files. Please refer to this site to display the hidden files of .git and delete them. By doing so, you can eliminate the error. j
When I run the git revert command, I get an “error: revert is not possible because you have unmerged files.”-How to use Git
The solution that says error: Your local changes to the following files would be overwritten by checkout in git checkout
git checkout didn’t understand
What to do when “Clone succeeded, but checkout failed” appears when cloning with Git

How to get started on GitHub

About Pull Request

Development process using pull request
Create a pull request from a fork (
https://docs.github.com/ja/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork)
[GitHub] Revert to a specific commit
git When you want to delete only a specific commit (revert)

GitHub application

[What is git submodule]
(https://qiita.com/sotarok/items/0d525e568a6088f6f6bb)
The git submodule is a mechanism for registering an external git repository as a subdirectory of your git repository and referencing a specific commit.
Command memo when you want to update only the git submodule
Github SSH communication settings
Procedure for ssh connection on GitHub ~ From generation of public / private key ~

How to apply GithHub to publish a simple program to the Internet

Configure a custom domain for your GitHub page site
https://help.github.com/en/github/working-with-github-pages/configuring-a-custom-domain-for-your-github-pages-site)
[Websites for you and your projects.]
(https://pages.github.com/)
Introduction to GitHub + CircleCI

How to set up Git and GitHub in PyCharm

Use ffmpeg or ffprobe with ffmpeg-aws-lambda-layer

2020-10-20T00:00:00+00:00

Use ffmpeg or ffprobe with ffmpeg-aws-lambda-layer

Introduction

This section describes the procedure for building a ffmpeg execution environment that is convenient for formatting and resizing videos and generating thumbnails using AWS Lambda functions and layers.
Here, we will use ffmpeg-aws-lambda-layer. By creating an AWS Lambda layer that includes ffmpeg and ffprobe separately from the Lambda function, these The command is called under / opt in the Lambda function execution environment. It can be called like a library without including the binary for command execution in the deployment package.

With Amazon Elastic Transcoder, you can easily format and resize videos, but it is expensive and requires machine power. If it is not a conversion, I think it is easier and better to process it with Lambda.
Transfer the ATOM Cam video used in the previously written article (Set up smart home camera ATOM Cam) to S3 via NAS and use Amazon Elastic Transcoder. When I created a process to convert the video format, I quickly consumed thousands of yen. It seems better to use ffmpeg for this purpose.

What you can do with this sample

If you deploy the sample project provided in ffmpeg-aws-lambda-layer, ffmpeg-layer-example-uploadbucket-xxxx in S3 And ffmpeg-layer-example-resultbucket-xxxx (xxxx is a random string, the same applies below) will be generated, and the connection with the Lambda function and the IAM role setting will be done automatically.
When a video file is uploaded to ffmpeg-layer-example-uploadbucket-xxxx, a thumbnail image is automatically generated in ffmpeg-layer-example-resultbucket-xxxx.

Things to prepare

-What is written in ffmpeg-aws-lambda-layer

- Unix Make environment
- AWS command line utilities (just for deployment)

This time, we confirmed the operation in the following environment.

$ sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H2

$ aws --version
aws-cli/2.0.57 Python/3.7.4 Darwin/19.6.0 exe/x86_64

–AWS account
–You need a role that can operate Lambda, S3, CloudFormation. There is no problem if you have Administrator privileges.

Reference information

-Create thumbnails from videos using FFmpeg on AWS Lambda
-Analyze video with ffprobe
-Encode with AWS Lambda (Node.js) + ffmpeg
-Try to generate and convert audio files with custom AWS Lambda runtime

Preparation

git clone ffmpeg-aws-lambda-layer

git clone https://github.com/serverlesspub/ffmpeg-aws-lambda-layer.git

Preparing a temporary S3 bucket

Creates a bucket in the same region as the region specified in the default profile in ~ / .aws / credentials.

Deploy sample project

Specify the bucket name of the temporary S3 bucket for the value of the DEPLOYMENT_BUCKET option.

cd ffmpeg-aws-lambda-layer
make deploy-example DEPLOYMENT_BUCKET=<BUCKET NAME>

When you execute the make command, the following log will be displayed.

cd example && \
		make deploy DEPLOYMENT_BUCKET=<BUCKET NAME> LAYER_STACK_NAME=ffmpeg-lambda-layer
mkdir -p build
aws cloudformation package --template-file template.yaml --output-template-file output.yaml --s3-bucket net.rev-system.nas
Uploading to d5886731828e201e755325e950d28eab  1742 / 1742.0  (100.00%)
Successfully packaged artifacts and wrote output template to file output.yaml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /tmp/ffmpeg-aws-lambda-layer/example/output.yaml --stack-name <YOUR STACK NAME>
aws cloudformation deploy --template-file output.yaml --stack-name ffmpeg-layer-example --capabilities CAPABILITY_IAM --parameter-overrides LambdaLayer=arn:aws:lambda:ap-northeast-1:************:layer:ffmpeg:1

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - ffmpeg-layer-example
aws cloudformation describe-stacks --stack-name ffmpeg-layer-example --query Stacks[].Outputs --output table
--------------------------------------------------------------------------------------------
|                                      DescribeStacks                                      |
+-------------------+----------------+-----------------------------------------------------+
|    Description    |   OutputKey    |                     OutputValue                     |
+-------------------+----------------+-----------------------------------------------------+
|  Upload S3 bucket |  UploadBucket  |  ffmpeg-layer-example-uploadbucket-xxxxxxxxxxx      |
|  Results S3 bucket|  ResultsBucket |  ffmpeg-layer-example-resultsbucket-yyyyyyyyyyyy    |
+-------------------+----------------+-----------------------------------------------------+

Here, the following resources are created.

Resource type	Name example
Lambda function	ffmpeg-layer-example-ConvertFileFunction-nnnnnnnnnnnn
Lambda layer	ffmpeg
S3 bucket(For upload)	ffmpeg-layer-example-uploadbucket-xxxxxxxxxxx
S3 bucket(Converted)	ffmpeg-layer-example-resultsbucket-yyyyyyyyyyyy
CloudFront stack	ffmpeg-layer-example
IAM roll	ffmpeg-layer-example-ConvertFileFunctionRole-zzzzzzzzzzzz

Thumbnail generation

Upload the appropriate video file to the S3 bucket ffmpeg-layer-example-uploadbucket-xxxxxxxxxxx.
After a while, the thumbnail image will be output to ffmpeg-layer-example-resultsbucket-yyyyyyyyyyyy.

Output other than thumbnails

If you want to format the video in a different format or output the meta information of the video file with ffprobe, you can try various things by changing ffmpeg-aws-lambda-layer / example / src / index.js and redeploying.

Create only Lambda layer

If you want to deploy only the Lambda layer without deploying the uploaded / converted S3 bucket or Lambda function, execute the following command.
Specify the bucket name of the temporary S3 bucket for the value of the DEPLOYMENT_BUCKET option.

cd ffmpeg-aws-lambda-layer
make deploy DEPLOYMENT_BUCKET=<BUCKET NAME>

Only the Lambda layer is deployed, so you can use it by adding this layer to any Lambda function.

CentOS 7 curl version upgrade

2020-10-20T00:00:00+00:00

I thought that if I followed the procedure introduced in some article, it would be completed smoothly, but I struggled unexpectedly, so I summarized my own procedure.

I hope it helps those who hit the same wall, saying, “I installed the latest version, but when I checked the version, it remained old.”

Install the required modules

Basically, I followed this article. I’m not sure what the required modules are. .. ..
https://sterfield.co.jp/programmer/curlの最新版をインストールする/

$ sudo yum install openssl-devel libmetalink-devel libssh2-devel c-ares-devel lbzip2

Get the latest version tar.gz download link address

Copy the latest curl-X.X.X.tar.gz link address at https://curl.haxx.se/download.html.

Download the latest version of tar.gz in any directory

After the installation is complete, I want to delete unnecessary folders and files, so download them to my home directory in an easy-to-understand manner.

$ cd
$wget [copied link address]

Extract the downloaded curl-X.X.X.tar.gz

After unpacking, a directory called curl-X.X.X will be created, so move the working directory.

$ tar xfvz curl-X.X.X.tar.gz
$ cd curl-X.X.X

Install the latest version of curl

To be honest, I ran it without knowing the meaning of these commands, but when I put together this article, I looked it up again and found an article that was explained carefully.
https://qiita.com/chihiro/items/f270744d7e09c58a50a5/
https://www.itmedia.co.jp/help/tips/linux/l0302.html

$ ./configure --enable-libcurl-option
$ make
$ sudo make install

Not the latest version …

Finally, I checked the version curl -V, but for some reason it wasn’t the latest version, so I checked the full path of the command with which -a curl.

$ which -a curl
/usr/local/bin/curl
/usr/bin/curl

After checking each version, it seems that the latest version is installed in / usr / local / bin / curl.

$ /usr/local/bin/curl -V
New version information

$ /usr/bin/curl -V
Old version information

Delete old version & create link for new version

I tried to copy “I made a mistake in the installation path, so I deleted the old one and made it a link” in this article, and I found that curl -V was the latest version!
https://m-shige1979.hatenablog.com/entry/2018/10/08/080000

$ sudo rm /usr/bin/curl
$ sudo ln -s /usr/local/bin/curl /usr/bin/curl
$ curl -V
New version information

802.1X authentication to the network of Bonding setting in CentOS7

2020-10-18T00:00:00+00:00

Network settings on a single NIC work, but …

There was a case where I wanted the installed server network to be ** Bonging using multiple NIC ports and join the network with 802.1X authentication **.
I was only able to participate in the network with a single NIC by myself, but with the cooperation of the people around me, I was able to participate with 802.1X authentication even in the case of Bonking.

Even if I searched on the net, I could not find the method of 802.1X authentication with “** Bonding setting **”, so I will publish it as an article here. (Thanks to everyone who cooperated!)

Environment / things to prepare

I can’t answer the device information of the server and network switch, but I hope you can think that the server is a racking type server such as Fujitsu, HP, IBM, and the network switch is done using products such as NETGEAR and Cisco. think.

We will proceed on the premise that the network bonding settings are created in CentOS 7 in advance.

OS:CentOS7.8
NIC port to be Bonding: Wired connection with two ports on Intel 10G NIC
Certificate file: ZZZZZZ.pem(Or it may be in crt format)
Private key file: ZZZZZZ.p12

① Convert the certificate file from crt format to pem format

Initially, I received a certificate in the crt file format from the user, but I don’t know the cause, but in the crt format, 802.1X authentication failed in the subsequent work.
So, convert from crt to pem format with the following command.

If the certificate file remains in pem format, this work is not necessary.

```Command line
openssl x509 -in ZZZZZZ.crt -out ZZZZZZ.pem

# (2) Migrate the storage directory of the certificate file and private key file
 Save ZZZZZZ.pem and ZZZZZZ.p12 in `/ etc / pki / CA / certs /`.

# ③ Edit the ifcfg file
 The bonding setting file is set as follows in `/ etc / sysconfig / network-scripts / ifcfg-bond0`.

BONDING_OPTS=”mode=4 miimon=100 xmit_hash_policy=layer2+3”
DEVICE=bond0
TYPE=Bond
BOOTPROTO=none
BONDING_MASTER=yes
NAME=bond0
ONBOOT=yes
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
MTU=9000
DNS1=aaa.bbb.ccc.ddd
DNS2=aaa.bbb.ccc.ddd
GATEWAY=aaa.bbb.ccc.ddd
IPADDR=aaa.bbb.ccc.ddd
NETMASK=aaa.bbb.ccc.ddd
PREFIX=yy

802.1X authentication settings

KEY_MGMT=IEEE8021X
IEEE_8021X_EAP_METHODS=TLS
IEEE_8021X_IDENTITY=XXXXXXXXX
IEEE_8021X_CA_CERT=/etc/pki/CA/certs/ZZZZZZ.pem #Specify the path where the pem format file is saved in ②
IEEE_8021X_PRIVATE_KEY=/etc/pki/CA/certs/ZZZZZZ.p12 #Specify the path where the p12 format file is saved in (2)
DOMAIN=zzzz.zzzz.zzzz

# ④ Edit /etc/sysconfig/wpa_supplicant and /etc/wpa_supplicant/wpa_supplicant.conf
 Edit the contents of `/etc/wpa_supplicant/wpa_supplicant.conf` as follows.

INTERFACES=”-iZZZZZZ”
DRIVERS=”-Dwired”
OTHER_ARGS=”-u -f /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid -t”


 Edit the contents of `/ etc / sysconfig / wpa_supplicant` as follows.

ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
network={
key_mgmt=IEEE8021X
eap=tls
identity=”ZZZZZZ”
ca_cert=”/etc/pki/CA/certs/ZZZZZZ.pem”
private_key=”/etc/pki/CA/certs/ZZZZZZ.p12”
private_key_passwd=”ZZZZZZ”
}

# (5) Disable the startup of NetworkManager when the server starts.
 If NetworkManager is started in advance, 802.1X authentication cannot be performed, so ** disable the service **.

chkconfig NetworkManager off

# ⑥ Enable and start wpa_supplicant
 Since wpa_supplicant must be started for 802.1X authentication, ** enable and start the service **.

chkconfig wpa_supplicant on
systemctl start wpa_supplicant

# ⑦ Start wpa_supplicant and check if Connection succeeds.

 If you get `bond0: CTRL-EVENT-CONNECTED --Connection to <MAC address> completed` with the following command, 802.1X authentication is successful.

wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -D wired -i bond0
```

⑧ Check if Supplicant authentication is successful on the network switch side

Log output and commands vary depending on the manufacturer and device of the network switch, so they are omitted.

Check the version of Cent OS

2020-10-17T00:00:00+00:00

You can check the version with the following command.

[root@sandbox ~]# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)

[root@sandbox ~]# bash --version
GNU bash,Version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+:GNU GPL version 3 or later<http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Until you can watch the HDR video taken with Sony’s mirrorless α6400 on Youtube (FFMpeg option story)

2020-10-16T00:00:00+00:00

It’s mainly about ffmpeg settings.

I bought a mirrorless interchangeable-lens camera this time. It is α6400.
I was told that I wouldn’t recommend a kit lens, so I chose a sharp one and chose Sirui’s 50mm F1.8 Anamorphic.
This time I decided to use the α6400 because it is possible to shoot HDR, so I took a test shot and tried it until it was uploaded to Youtube, so it is a memorandum.
I think that it is not necessary information especially for those who have a paid version of DaVinci or Adobe products.

Purpose

The α6400 shoots HDR video with HLG and saves it as mp4. First of all, I tried how to take this and post it on Youtube.
Also, since it is an anamorphic lens, it needs to be stretched sideways after shooting, so I will do this together as well.

Issue with PQ

If you are not an anamorphic lens, you do not need the -aspect 24:10 -s 3840x1600 section.

`ffmpeg -y -i C0001.MP4 -aspect 24:10 -s 3840x1600 -c:v libx265 -tag:v hvc1 -crf 22 -pix_fmt yuv420p10le -x265-params "colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc" C0001out.mp4`

 Click here for the uploaded video
https://youtu.be/uPuNuE5b0As
 I feel that I'm used to playing games because I often use it during HDR. When I posted it on Youtube, it was recognized as HDR, but the video that was automatically converted on Youtube, which is played when viewed in the SDR environment, is crushed and dirty. It looks better on an HDR monitor, but it still feels crushed.
### Issue with HLG
 If this is also not an anamorphic lens, the section -aspect 24:10 -s 3840x1600 is unnecessary.
 HLG is specified by color_trc 18.

#### **`ffmpeg -y -i C0001.MP4 -aspect 24:10 -s 3840x1600 -c:v libx265 -tag:v hvc1 -crf 22 -pix_fmt yuv420p10le -color_trc 18 -color_primaries 9 -colorspace 9 C0001out.mp4`**

Click here for the uploaded video
https://youtu.be/N0Rkmyfb2Ho
Both SDR and HDR can be seen in a safe form with less crushing than PQ, but worse, it feels a little sleepy. It’s in a state of being taken without grading, and I feel like this.

About resolution

For upload requirements
720p、1080p、1440p、2160p For best quality playback, use the UHD standard instead of the DCI standard (for example, use an aspect ratio of 3840x1600 instead of 4096x1716).``
So I chose 3840 * 1600.

To check if the uploaded video is what you intended

Right-click on the video and click Detailed Statistics

Then, the codec, color space, color gamut, etc. will be displayed in the upper right corner of the screen.

As a caveat, if you view it on a monitor or browser that is not in an HDR playback environment, the information of the automatically converted one will be displayed, so it is better to first check whether HDR is effective.

Finally

It seems that it is converted to vp9 at the time of uploading, so it may be more efficient to put it out with webm / vp9 from the beginning.
I’m still completely lacking in knowledge about HDR, so please let me know if you have any questions.
As an aside, I learned for the first time that Youtube uses ffmpeg for these upload conversions in the following words.

`If TRC is not supported by FFmpeg's color space conversion filter, YouTube will BT.Set it back to 709.`

```Set it back to 709.

From the color space section of the encoding settings in the reference.

reference

Options etc.
https://developers.google.com/media/vp9/hdr-encoding
Upload High Dynamic Range (HDR) videos
https://support.google.com/youtube/answer/7126552?hl=ja
Encoding settings
https://support.google.com/youtube/answer/1722171

Install PostgreSQL 12 on Centos8

2020-10-15T00:00:00+00:00

Introduction

Since I had only touched Mysql, I installed Postgresql using the installation method on the official website for verification.
This time, we assume a local postgres user connection without authentication.

environment

CentOS8
Postgresql12

Installation

dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
dnf -qy module disable postgresql
dnf install -y postgresql12-server
/usr/pgsql-12/bin/postgresql-12-setup initdb
systemctl enable postgresql-12
systemctl start postgresql-12

Select the one you want to install from the official website below and install it.
https://www.postgresql.org/download/linux/redhat/

setting change

Advance preparation

User change

After installation, the postgres user has been added, so change the user.
Basically, the operation is performed by the postgres user.

su - postgres

Directory transition

Change to the directory where the configuration file is.

cd /var/lib/pgsql/12/data

Add PATH

Make pg_ctl available in relative PATH.
You may add it to .bash_profile.

export PATH=$PATH:/usr/pgsql-12/bin/

postgresql.conf
A file for basic settings. Change the settings as appropriate.
This time it is a verification, so leave it as the default.

pg_hba.conf
A file that sets the network to connect to.
By changing from ident to trust, you can connect to the local host without authentication.
If postgresql.conf is the default, you may go to connect with ipv6, so change ipv6 as well.

Change before

# IPv4 local connections:
host    all             all             127.0.0.1/32            ident
# IPv6 local connections:
host    all             all             ::1/128                 ident

After change

# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

Reflection of settings

Apply the settings with the following command.

pg_ctl reload

Verification

Confirm that you can log in with the following command.

psql -h localhost

Password change (optional)

After connecting, change the password of the postgres user.

alter role postgres with password 'password';

Summary

I installed it on the official website.
I forgot to change the ipv6 settings, I didn’t know the initial password of the postgres user, or I was caught unexpectedly.
And the initial password for the postgres user is still unknown.

References

https://www.postgresql.org/download/linux/redhat/
PostgreSQL classroom for database beginners