Implemented cart pole with ML-Agents 1

Overview

Implement and train cart poles in ML-Agents. I hope it helps people who want to create models to train themselves using ML-Aegents.

Introduction

Create your own model like the ML-Agents sample. I explained the learning method in the previous article, so this time I focused on how to make a model to learn. I referred to the official Documents. Please check here for details.

Overall flow

There are four main steps in creating and learning a model.

–Create a 3D model with unity
–Write a script for the agent to be trained
–Set Behavior Parameter
–Learning

I will explain while actually creating it.
Since the article will be long, I will explain how to create a 3D model this time.

Create 3D model with unity

First, create a 3D model that you want to train with unity. This time we will implement a cart pole. A cart pole is a problem that controls the pole attached to the cart so that it does not fall over.
Finally, create the following model. We have multiple models to streamline learning.

Cart pole creation

First of all, I will explain how to create the following model.

Create a folder called CartPole in the Project at the bottom left as shown in the image below. Also, create a new scene from File / New Scene and save it in the created folder named Cart Pole.

Material contains items that set the material properties of carts, poles, and floors. Script contains a script that moves the created object. The last TFModels will be the folder to put the trained models.

Next, open the scene and add a 3D Object as shown below.

For the 3D Object to be added, add one cube, one sylinder, and one plane. Once added, rename the cube to Cart and the cylinder to Pole. Also, Cart and Pole should create an empty Object called TrainingArea and put it in it.
Please refer to the following images for each scale.

Next, we will add components.

Add Component

In unity, you can add various characteristics to the created object by adding a component. This time we will add a rigid body and a Hinge Joint. By adding a rigidbody, you can control the object by its physical properties. Specifically, gravity can be added, force can be applied, and air resistance can be set. Adding a Hinge Joint will hinge the Cart and Pole.

To add a component, select the object you want to add, click Add Component at the bottom right of the screen, and select the object you want to add.

Add rigid body

Select “rigid body”.

Add to Cart and Pole respectively. When added, it will look like the image below.

Set each parameter as shown in the image. (It may be appropriate)

Mass is the mass [Kg] of the object.

Drag stands for air resistance. The higher the value, the higher the resistance. If it is 0, it is in a vacuum state.

Angular Drag is the air resistance when an object rotates. The higher the value, the harder it is to rotate.

Constraints / Freeze Position restricts the movement of objects. This time, I want Cart to move in the horizontal direction (Z direction), so check the front direction (X direction).

Freeze Rotation limits the rotation of objects. Cart does not rotate this time, so please check all.

Addition of Hinge Joint

Add from Add Component / Physics / Hinge Joint.


Set the parameters as shown in the image. Also add Cart to the Connected Body.

Material settings

Next, set the Material of the object. Cart and Pole only set the color, so set them yourself. This time, set the friction with Plane. Add the Physic Material inside the Material folder and rename it Plane. I think you can get the yellow-green one in the middle.

Set the dynamic friction with Dynamic Friction. The value is 0 to inf, and the larger the value, the more the object will not move.
Static Friction sets static friction.
This time the friction parameter is good (Cart will not work if it is too large).

Add the created Physic Material to the Plane object. Just select the Plane object and drag and drop it there. It is OK if it looks like the mark on the image.

The 3D object is now complete.

Summary

This time, I created a 3D object Cart Pole with unity. The model to be trained is almost completed. Next time will write a script.