Rendering Skeletal Meshes (with Assimp)

Skeletal animation is one of those features everyone wants to support, but nobody wants to actually implement. Parsing animation data just feels messy and troublesome. In reality, it's probably less of a pain than you might imagine. Or rather, it can be, if you break it down into the right steps.

In this post, I'll walks through all the major steps that you need to take to implement skeletal mesh in your renderer. Load data -> skin mesh -> update BVH -> draw. That's it. Hopefully by the end, you too will have cool skeletal animations in your renderer.

Requirements

As per the title, I'll be assuming that you're working off data from the Assimp asset importer library. If you're using something else, there'll be some differences in the parsing aspects, but it'll be a similar process overall. In any case, you'll need:

Basic understanding of working with Assimp (static mesh loading, etc.)
Basic understanding of skinning (though I will briefly go through it again later).

The Big Picture: Skinning Pipeline

Skeletal animation ultimately boils down to three simple steps:

Load all necessary skeleton + mesh + animation data
Evaluate animation data
Update vertex positions

The most annoying part is really just step 1. There are many ways to do this, and there are many common pitfalls. Regardless of your importer of choice, transforming the data into a layout that is easy to work with is the key to reducing importer complexity.

Skinning Recap

There are some new pieces of data that needs to be parsed from a model file compared to static meshes. To better explain these (and as a revision), let's first briefly recap what skinning actually is.

For each bone $i$ , let $B_{i}$ be the local transform of bone relative to it's parent bone, and $G_{i}$ the global transform of the bone in the model space.

G_{i} = {B_{i}, G_{p (i)} B_{i}, if i is the root bone otherwise

What we need for transforming vertices to the skinned position is the bone's global transform, but what animation data gives us is often the local transform. Therefore, we store and animate each bone with their local transforms, and compute their global transform by evaluating the skeletal hierarchy during the animation process.

For each bone there is also the concept of an offset matrix $O_{i}$ , sometimes called the inverse bind matrix. $O_{i}$ transforms a vertex from it's local model space, into bone $i$ 's bone space, where the bone is at origin. You can visualize why we need this transformation as follows:

Item 1 of 3

There's also the weight $w_{i} (v)$ that specifies how much the current bone $i$ affects the current vertex $v$ . All weights affecting a single vertex should sum to 1, or at the very least, does not exceed 1. The input data may or may not guarantee this, so it's worth guarding against it just in case.

Finally, the skinned vertex position $v^{'}$ can be computed as:

v^{'} = i \in bones \sum w_{i} (v) G_{i} O_{i} v

With that out of the way, we can summarize the data we need to obtain from Assimp:

Vertex bone weights and indices - which bones affect the current vertex, and by how much
Bone Offset Matrix
Animation Keyframes (to compute the bone local / global matrix each frame, at a particular animation tick)

Loading Skeleton & Animation Data from Assimp

I highly recommend creating some new classes to store these data. This allows us to fully decouple from Assimp during rendering. As a side note, we really only need Assimp during our import / loading process.

skeleton.h

class SkeletonBone
{
    ...
    uint32_t m_ParentIndex;
    Matrix4x4 m_InverseBindMatrix;
};

struct SkeletonPose
{
    ...
    std::vector<Matrix4x4> m_GlobalBoneTransforms;
};

class Skeleton
{
    ... 
    std::vector<SkeletonBone> m_Bones;
};

animation.h

class AnimationData
{
    template <class T>
    struct Keyframe
    {
        float m_Time,
        T m_Value,
        InterpolationType m_InterpolationType;
    }

    ...
    Vector GetInterpolatedPosition(float tick) const;
    Quaternion GetInterpolatedRotation(float tick) const;
    Vector GetInterpolatedScale(float tick) const;
    ...
    std::vector<Keyframe<Vector>>     m_PositionKeyframes;
    std::vector<Keyframe<Quaternion>> m_RotationKeyframes;
    std::vector<Keyframe<Vector>>     m_ScaleKeyframes;
};

0. Assimp Import Flags

Before we do anything else, we need to make sure the import flags are setup correctly. Typically, one would be setting the following flags:

assetimporter.cpp

uint32_t importFlags = 0;
importFlags |= aiProcess_ConvertToLeftHanded;           // For DirectX left-handedness
importFlags |= aiProcess_TransformUVCoords;             // Handle UV transformations
importFlags |= aiProcessPreset_TargetRealtime_Quality;  // General realtime optimizations

Assimp::Importer importer;
aiScene* scene = importer.ReadFile(assetPath, importFlags);

There is one new flag that we want to add: aiProcess_PopulateArmatureData. This post process populates the aiBone->aiNode reference, as well as the aiBone->mArmature (root skeleton node) reference.

⚠️ You may have accidentally come across aiScene->mSkeleton. However, as of writing this post, this seems to be an undocumented, uncompleted, experimental feature that only has basic support for .fbx files. We won't be using this.

We also want to get rid of aiProcess_SplitLargeMeshes and aiProcess_PreTransformVertices, as they will destroy any skeletal/bone data. But wait, we aren't using them in the first place, right? Yes, we're not using aiProcess_PreTransformVertices, but aiProcess_SplitLargeMeshes is hiding in aiProcessPreset_TargetRealtime_Quality.

assimp/postprocess.h

#define aiProcessPreset_TargetRealtime_Quality ( \
    aiProcess_CalcTangentSpace              |  \
    aiProcess_GenSmoothNormals              |  \
    aiProcess_JoinIdenticalVertices         |  \
    aiProcess_ImproveCacheLocality          |  \
    aiProcess_LimitBoneWeights              |  \
    aiProcess_RemoveRedundantMaterials      |  \
    aiProcess_SplitLargeMeshes              |  \ <- Sneaky!
    aiProcess_Triangulate                   |  \
    aiProcess_GenUVCoords                   |  \
    aiProcess_SortByPType                   |  \
    aiProcess_FindDegenerates               |  \
    aiProcess_FindInvalidData               |  \
    0 )

So, our updated flags now looks like:

assetimporter.cpp

uint32_t importFlags = 0;
importFlags |= aiProcess_ConvertToLeftHanded;           // For DirectX left-handedness
importFlags |= aiProcess_TransformUVCoords;             // Handle UV transformations
importFlags |= aiProcessPreset_TargetRealtime_Quality;  // General realtime optimizations
importFlags |= aiProcess_PopulateArmatureData;          // Generate bone->node connections
importFlags &= ~aiProcess_SplitLargeMeshes;             // DON'T split large meshes

Assimp::Importer importer;
aiScene* scene = importer.ReadFile(assetPath, importFlags);

1. Vertex bone weights & indices

For skinned meshes, we need to assign the indices and weights of bones that affect this vertex. In Assimp, this data is stored in aiMesh->mBones. Each aiBone contains a list of vertexId-weight pair aiVertexWeight.

We'll create a new SkinnedVertex type to keep track of this information.

vertexformat.h

struct SkinnedVertex : public BaseVertex
{
    uint32_t m_BoneIndices[MAX_BONES_PER_VERTEX];
    float    m_BoneWeights[MAX_BONES_PER_VERTEX];
}

It is common to use fixed arrays of 4 weights in many renderers, but you can always support more. Do note that, if you ever want to implement compute shader skinning, you may need a separate buffer to store higher number of influences.

In the same place where you are already processing vertex data (position, normal, etc.) from aiMesh, we need to add a new loop to go through every aiBone in this aiMesh, and assign the weights found in aiVertexWeight.

assetimporter.h

// We'll populate this later
std::unordered_map<aiNode*, std::unique_ptr<Skeleton>> m_ArmatureRootToSkeletonMap;

assetimporter.cpp:FillVertexData()

aiBone* rootBone = assimpMesh->mBones[0]->mArmature;
if (rootBone != nullptr)
{
    const Skeleton& skeleton = m_ArmatureRootToSkeletonMap.at(rootBone);
    for (uint32_t i = 0; i < assimpMesh->mNumBones; ++i)
    {
        const aiBone* bone = assimpMesh->mBones[i];
        const uint32_t boneIndex = skeleton.GetBoneIndex(bone);

        // iterate through each "vertex" that this bone influences
        for (uint32_t vertexIndex = 0; vertexIndex < bone->mNumWeights; ++vertexIndex)
        {
            aiVertexWeight& vertexRef = bone->mWeights[vertexIndex];

            // Ignore bones that have very little or no weight. Optional.
            if (vertexRef.mWeight <= 0.01f)
                continue;

            // Find which weight slot is still available on the vertex
            // It may be necessary to sort and store the highest weight contributions instead
            for (uint32_t k = 0; k < MaxBonesPerVextex; ++k)
            {
                if (data[vertexRef.mVertexId].m_BoneIndices[k] == InvalidBoneIndex)
                {
                    data[vertexRef.mVertexId].m_BoneIndices[k] = boneIndex;
                    data[vertexRef.mVertexId].m_BoneWeights[k] = vertexRef.mWeight;
                    break;
                }
            }
        }
    }
}

⚠️ While it may be tempting to directly assign bone index as i from iterating assimpMesh->mBones, this is almost certainly different from the actual bone index in our skeleton class. A mismatch in bone indices is a common pitfall when implementing skinning.

This is all the data we need on a per-vertex level.

2. Offset Matrices

Bone related data in Assimp is store in objects of type aiBone, and they can be found in aiMesh->mBones, which contains all the bones that affects vertex in this mesh.

An important thing to note here is that, there may be bones that does not directly have vertex influence on this mesh, but is definitely part of the skeleton hierarchy and thus must be included in the Global Bone Matrix of it's children. Therefore, the bones in the mBones array should only be used as a hint as to which nodes are bones affect this mesh, not all bones that affect this mesh. We need to build the true skeleton hierarchy from traversing aiNodes in aiScene.

While traversing aiNode, we actually don't know which bone we're on since there is no aiNode->mBone reference. However, the only thing we really need from the bone is the offset matrix. So to make life easier, we'll create a node to matrix lookup table for convenience:

assetimporter.h

std::unordered_map<aiNode*, std::unique_ptr<Skeleton>> m_ArmatureRootToSkeletonMap;
std::unordered_map<aiNode*, Matrix4x4> m_OffsetMatrices;

assetimporter.cpp

void AssetImporter::PreprocessBones(const aiScene* assimpScene)
{
    for (int i = 0; i < assimpScene->mNumMeshes; ++i)
    {
        const aiMesh* mesh = assimpScene->mMeshes[i];

        if (!mesh->HasBones())
            continue;

        for (uint32_t boneIndex = 0; boneIndex < mesh->mNumBones; ++boneIndex)
        {
            aiBone* bone = mesh->mBones[boneIndex];
            m_OffsetMatrices.emplace(bone->mNode, ToOurMatrix4x4(bone->mOffsetMatrix));
        }
    }
}

3. Skeletons

With the offset matrices saved, we can traverse the nodes to build the our skeleton hierarchy:

assetimporter.cpp


void AssetImporter::ProcessSkeletons(const aiScene* assimpScene)
{
    for (int i = 0; i < assimpScene->mNumMeshes; ++i)
    {
        const aiMesh* mesh = assimpScene->mMeshes[i];

        if (!mesh->HasBones())
            continue;

        // We make the assumption that a mesh is only affected by a single skeleton.
        aiNode* rootNode = mesh->mBones[0]->mArmature;

        if (m_ArmatureRootToSkeletonMap.contains(rootNode))
        {
            // Skeleton already exist and has been processed
            continue;
        }

        auto [iter, inserted] = m_ArmatureRootToSkeletonMap.emplace(
          rootNode, 
          std::make_unique<Skeleton>()
        );
        Skeleton& skeleton = *iter->second;
        ProcessSkeleton(skeleton, rootNode);
    }
}

// Recursively add bones into our skeleton
void AssetImporter::ProcessSkeleton(Skeleton& skeleton, const aiNode* node, uint32_t parent) 
{
    const uint32_t currentBoneIndex = skeleton.NumBones();
    SkeletonBone bone(parent, GetOffsetMatrix(node));
    skeleton.AddBone(bone);

    for (uint32_t i = 0; i < node->mNumChildren; ++i)
    {
        aiNode* child = node->mChildren[i];
        ProcessSkeleton(skeleton, child, currentBoneIndex);
    };
}

We can call these new functions directly after importing the Assimp scene.

assetimporter.cpp

uint32_t importFlags = 0;
importFlags |= aiProcess_ConvertToLeftHanded;           // For DirectX left-handedness
importFlags |= aiProcess_TransformUVCoords;             // Handle UV transformations
importFlags |= aiProcessPreset_TargetRealtime_Quality;  // General realtime optimizations
importFlags |= aiProcess_PopulateArmatureData;          // Generate bone->node connections
importFlags &= ~aiProcess_SplitLargeMeshes;             // DON'T split large meshes

Assimp::Importer importer;
aiScene* scene = importer.ReadFile(assetPath, importFlags);

PreprocessBones(scene);
ProcessSkeletons(scene);

4. Animations

Getting the animations is simply a matter of getting the key value pairs from aiNodeAnim. There's really not much to say about this as it is a pretty straight forward process.

assetimporter.h

std::vector<AnimationData> m_Animations;

assetimporter.cpp

void AssetImporter::ProcessAnimations(const aiScene* assimpScene)
{
    for (uint32_t i = 0; i < assimpScene->mNumAnimations; ++i)
    {
        const std::string animName = animation->mName.C_Str();
        const float totalTicks = animation->mDuration;
        const float ticksPerSecond = animation->mTicksPerSecond;

        AnimationData animationData(animName, totalTicks, ticksPerSecond);

        for (uint32_t j = 0; j < animation->mNumChannels; ++j)
        {
            aiNodeAnim* animatedBone = animation->mChannels[j];
            const std::string boneName = animatedBone->mNodeName.C_Str();

            for (uint32_t k = 0; k < animatedBone->mNumPositionKeys; ++k)
            {
                animationData.m_PositionKeyframes.emplace(
                    animatedBone->mPositionKeys[k].mTime,
                    animatedBone->mPositionKeys[k].mValue,
                    animatedBone->mPositionKeys[k].mInterpolation
                );
            }

            for (uint32_t k = 0; k < animatedBone->mNumRotationKeys; ++k)
            {
                animationData.m_RotationKeyframes.emplace(
                    animatedBone->mRotationKeys[k].mTime,
                    animatedBone->mRotationKeys[k].mValue,
                    animatedBone->mRotationKeys[k].mInterpolation
                );
            }

            for (uint32_t k = 0; k < animatedBone->mNumScalingKeys; ++k)
            {
                animationData.m_ScaleKeyframes.emplace(
                    animatedBone->mScalingKeys[k].mTime,
                    animatedBone->mScalingKeys[k].mValue,
                    animatedBone->mScalingKeys[k].mInterpolation
                );
            }
        }

        m_Animations.emplace_back(animationData);
    }
}

assetimporter.cpp

uint32_t importFlags = 0;
importFlags |= aiProcess_ConvertToLeftHanded;           // For DirectX left-handedness
importFlags |= aiProcess_TransformUVCoords;             // Handle UV transformations
importFlags |= aiProcessPreset_TargetRealtime_Quality;  // General realtime optimizations
importFlags |= aiProcess_PopulateArmatureData;          // Generate bone->node connections
importFlags &= ~aiProcess_SplitLargeMeshes;             // DON'T split large meshes

Assimp::Importer importer;
aiScene* scene = importer.ReadFile(assetPath, importFlags);

PreprocessBones(scene);
ProcessSkeletons(scene);
ProcessAnimations(scene);

And with that, we've completed the entire parsing process. From this point onwards, we no longer need anything else from Assimp.

Skinning

Skinning is really a two part process:

Evaluate the animation data at a specified tick, and obtain a pose. This is the SkeletonPose that we've created in the beginning - a place to store bone matrices which tells us where all the bones are
Transform all the vertices into their skinned positions

Skinning is best performed in a vertex or compute shader. But for the sake of simplicity, this post will only skin vertices on the CPU. Compute Skinning is an exericse left for the reader.

We first compute all the global bone matrices at the current animation tick:

SkinningManager.cpp

SkeletonPose SkinningManager::CalculatePoseFromAnimation(
    const Skeleton& skeleton,
    const AnimationData& animation,
    float animTimeTicks) const
{
    SkeletonPose newPose;
    newPose.m_GlobalBoneTransforms.resize(skeleton.NumBones());

    // Since bones were populated in DFS fashion, every node is populated after its parent,
    // so it is save to assume parent's global transform has already been computed.
    for (uint32_t i = 0; i < skeleton.NumBones(); ++i)
    {
        const SkeletonBone& currentBone = skeleton.GetBone(i);
        const bool hasParent = currentBone.m_ParentIndex != InvalidBoneIndex;

        Vector interpolatedPosition = aniamtion->GetInterpolatedPosition(animTimeTicks);
        Quaternion interpolatedRotation = animation->GetInterpolatedRotation(animTimeTicks);
        Vector interpolatedScale = animation->GetInterpolatedScale(animTimeTicks);

        Matrix4x4 translation = Transform::GetTranslationMatrix(interpolatedPosition);
        Matrix4x4 rotation = Transform::GetRotationMatrix(interpolatedRotation);
        Matrix4x4 scale = Transform::GetScaleMatrix(interpolatedScale);

        Matrix4x4 localTransformation = translation * rotation * scale;
        Matrix4x4 parentTransformation = newPose.GetGlobalTransform(currentBone.m_ParentIndex);
        Matrix4x4 globalTransformation = parentTransformation * localTransformation;
        newPose.SetGlobalTransform(i, globalTransformation);
    }

    return newPose;
}

And then, we update the vertex positions to perform skinning:

SkinningManager.cpp

void SkinningManager::UpdateSkinnedMesh(
    SkinnedMesh& skinnedMesh,
    const Skeleton& skeleton,
    const AnimationData& animation)
{
    const float animationTime = Time::GetTimeSinceStartup();
    const float ticksPerSecond = animation.GetTicksPerSecond();
    const float timeInTicks = animationTime * ticksPerSecond;

    const SkeletonPose pose = CalculatePoseFromAnimation(skeleton, animation, timeInTicks);
    const std::vector<Matrix4x4> boneMatrices(skeleton.NumBones());
    
    // Pre compute the final matrices for each bone since this does not
    // change from vertex to vertex
    for (uint32_t b = 0; b < skeleton.NumBones(); ++b)
        boneMatrices[b] = pose.m_GlobalBoneTransforms[b] * 
                          skeleton.GetBone(b).m_InverseBindMatrix;

    std::vector<SkinnedVertex> skinningVertices = skinnedMesh.GetSkinningVertices();
    for (uint32_t i = 0; i < skinningVertices.size(); ++i)
    {
        SkinnedVertex& vertex = skinningVertices[i];
        Vector skinnedPosition, skinnedNormal;

        for (uint32_t j = 0; j < MaxBonesPerVextex; ++j)
        {
            skinnedPosition += vertex.m_BoneWeights[j] * 
                               boneMatrices[vertex.m_BoneIndices[j] * 
                               vertex.m_Position;
            skinnedNormal   += vertex.m_BoneWeights[j] * 
                               boneMatrices[vertex.m_BoneIndices[j] * 
                               vertex.m_Normal;
        }

        vertex.m_PrevPosition = vertex.m_Position; // For velocity
        vertex.m_Position = skinnedPosition;
        vertex.m_Normal = skinnedNormal;
    }
}

That's it! If you draw the vertices as is, they should now draw at the skinned position. Hopefully the vertices don't start flying all over the place.

Skinning

2132 Words

November 16, 2025