Estimate song key with Beat Saber

4 minute read

Key estimation library to use

libKeyFinder
I chose this library because the accuracy was good after using it.
I also tried SuperPowererdSDK, but libKeyFinder seems to be better. It was.
It has performance comparable to other paid apps.

(I’ll talk about the key estimation algorithm next time.
Is pitch estimation around here? Mainly removing overtone components? )

Precautions when assembling

Audio has a very large amount of data, so do not reserve it in the stack area.
(Since the stack area is 2MB on Windows, even 2-3 seconds of data will not survive)
Secure it dynamically, or add static to secure it in the static area.
(I still don’t fully understand how the stack / heap area works)

Also, the library has a memory limit, so don’t try to analyze long data at once.

Installation procedure

1. Create FFTW lib file

Note: Beat Saber works on 64bit, so create a 64bit library.

FFTW is a library of fast Fourier transforms, referenced by libKeyFinder.
Download FFTW from here and create a lib file by the method of linking item titles (up to ** Create the import library **).

You will be asked to use the developer console on the way, but you can trace it from the Windows folder search.
developercommand.png

2. Create a libKeyFinder wrapper dll file

Create a dll file by referring to How to call a C ++ function from C # in a Windows console application.
Call test from C #.
Note that the location where the C ++ library is output differs slightly depending on the version of Visual Studio.

Also note that the libKeyFinder sample code is slightly incorrect. Correctly below.


// Static because it retains useful resources for repeat use
static KeyFinder::KeyFinder k;

// Build an empty audio object
KeyFinder::AudioData a;

// Prepare the object for your audio stream
a.setFrameRate(yourAudioStream.framerate);
a.setChannels(yourAudioStream.channels);
a.addToSampleCount(yourAudioStream.length);

// Copy your audio into the object
for (int i = 0; i < yourAudioStream.length; i++) {
  a.setSample(i, yourAudioStream[i]);
}

// Run the analysis
KeyFinder::key_t r =  k.keyOfAudio(a); //<-This is different. key_use t

3. Run libKeyFinder.dll with Unity (optional)

Add this item because the difficulty of incorporating is lower if you take the step of experimenting with Unity first.
To call a C ++ function in Unity, create a Plugins / x86_64 folder directly under Asset and place the dll under it.

Sample code

        int dataLength = 44100 * 5;　//5 seconds of data
        int sampleRate = 44100;
        int bitDepth = 24;
        float ampMax = 0;

        
        AudioClip clip = this.transform.GetComponent<AudioSource>().clip;
        float[] clipData = new float[dataLength * clip.channels];
        clip.GetData(clipData, 0);


        double[] data = new double[dataLength];
        for(int i=0; i<dataLength; i++)
        {
            float monoData = clipData[i * 2] / 2f + clipData[i * 2 + 1] / 2f; //Processed in monaural for the time being
            data[i] = (double)monoData;

            if(Mathf.Abs(monoData) > ampMax)
            {
                ampMax = monoData; //AudioClip doesn't seem to have bit depth, so find the maximum value as a compromise.
            }
        }

        sampleRate = clip.frequency;

        int key = KeyFind(data, dataLength, ampMax, sampleRate);

4. Run libKeyFinder with Beat Saber

Placement of libKeyFinder.dll

Install libKeyFinder.dll below.
\<Beat Saber Folder>\Beat Saber_Data\Plugins\

Music acquisition

1. Get the music file path

The songs that are included by default cannot be analyzed because they are loaded in a compressed state.
Therefore, it is limited to Custom Level. (If you try to read it by force, this errorcoming out)
First, get the file path of the song added by Custom Level.

To get the CustomLevelPath, refer to here as follows.
In the following implementation, the process is executed only when it is Custom Level.


IDifficultyBeatmap diffBeatmap = BS_Utils.Plugin.LevelData.GameplayCoreSceneSetupData.difficultyBeatmap;
CustomPreviewBeatmapLevel customPreviewBeatmapLevel = diffBeatmap.level as CustomPreviewBeatmapLevel;

if (customPreviewBeatmapLevel != null)
{
    string customLevelPath = customPreviewBeatmapLevel.customLevelPath;
    string songFileName = customPreviewBeatmapLevel.standardLevelInfoSaveData.songFilename;
    string filepath = customLevelPath + "\\" + songFileName;
}
2. Read the file as AudioClip and get the waveform data

Use UnityWebRequset to load from the file path into AudioClip.

Sample code

        IEnumerator LoadAudioClipWithWebRequest(string filename)
        {
            AudioClip clip; 
           
            using (UnityWebRequest www = UnityWebRequestMultimedia.GetAudioClip("file://" + filename, AudioType.OGGVORBIS))
            {
                yield return www.SendWebRequest(); /*You have to wait for it to load*/
                clip = DownloadHandlerAudioClip.GetContent(www);
            }

            int key = KeyFinder.KeyFind(clip);
        }

I took www.result == UnityWebRequest.Result.ConnectionError because I got a syntax error.
If you confirm that the Load Status of AudioClip is Loaded, it should be OK for the time being.

(By the way, Load Status remains Unload while loading with WebRequest.
You have to wait for Load with yield return www.SendWebRequest ();.
Therefore, the use of coroutines is mandatory, but only by inheriting MonoBehaiviour)

Once you have the waveform data, pass it to libKeyFinder and you’re done.

Reference information

–AudioFile was easy to use as a library when reading Wave files for testing.
–It seems that there is a way to dynamically set the audio clip loading method in the program (https://kan-kikuchi.hatenablog.com/entry/AudioSettings) (AudioImporter))

–When searching for the default song, you can search for Automatic Music Score Library. discovered.
Maybe you can read the default song by referring to here?

Afterword

The entire sample code when embedding in Beat Saber is here

Tags:

Updated:

You May Also Enjoy