Some useful blender add-ons for SMPL skeleton's poses and global translation.

Overview

Blender add-ons for SMPL skeleton's poses and trans

There are two blender add-ons for SMPL skeleton's poses and trans.The first is for making an offline visual demo.The second is to make a live visual demo.

Offline Motion Capture

Show one demo image

You need to follow these steps to use the first add-on.

  1. Open a blender file and import the fbx file of a model with SMPL skeleton

I downloaded a model on Mixamo that happened to be an SMPL skeleton, and I changed the name of each bone so that the code could find the bones and act on them.You can find it in sources folder.

If you want to use your own model, make sure that the skeleton is SMPL skeleton and that the bones are named the same as in the code.

Also, you need to change the name of model to Armature1.

  1. Prepare the pkl file

You need to prepare a pkl file, datas can be read by pickling.load(f)['model'], where f stands for pkl file.

Datas are lists of [N,75], where N stands for the number of frames.The 75 elements represent SMPL's poses and global translation.The first 72 elements are SMPL poses, and the last 3 elements are global translation.

You then modify the file path in the code so that it can read your pkl file.

  1. Install the addon-on
  2. Make the skeleton and model active,then presh ctrl+R to run the add-on

Inserting keyframes takes a long time, you have to wait, and it's best not to insert more than 10000 frames at a time.

Real Time Motion Capture

ROMP outputs SMPL's poses and trans more than 20 times per second. With ROMP and the plugin, the output of ROMP can be input into Blender's skeleton in real time to achieve the effect of human-driven animation characters.

Of course, every model that outputs SMPL's poses and trans can use this plugin to drive animation characters in Blender.

Comments
  • How to run .fbx file to control the charater

    How to run .fbx file to control the charater

    Hello. I have successfully run the demo of ROMP, which exported .fbx file. And currently I want to use the .fbx to control the character. Can you provide steps for the video demo? I can only see the camera one

    opened by CheungBH 31
  • [simple-romp] How to use it?

    [simple-romp] How to use it?

    I tried to use it with simple-romp but it did not work. I already created an issue at the ROMP repository and described my problem here and here in detail.

    The author of the ROMP repository answered there:

    About live blender driving, please refer to this repo. https://github.com/yanch2116/CharacterDriven-BlenderAddon My colleague is responsible for maintaining this funciton now. Best regard.

    and closed the issue.

    @yanch2116 So how to solve it?

    My goal is to send the positions and quaternions of ROMP (or any other SMPL based solution) over the VMC protocol to other application (not Blender). Any ideas, whats the best way to do it?

    opened by vivi90 26
  • Could I ask for a detailed instruction on how to change the 3D Character?

    Could I ask for a detailed instruction on how to change the 3D Character?

    Also, I would like to know if there is a way to replace the 3D Character before I run the script and process the video so that I don't have to manually change it every time?

    opened by XXZhe 12
  • Can't connect ROMP into addon

    Can't connect ROMP into addon

    I use ROMP v1.1 on Windows machine. But CDBA works with version 1.0 of ROMP I read installation and using documentation of ROMP v1.0 but i can't figured out all How can i use this addon properly? I'm not good at programming i'm animator and interested in your project. Can you write steps how to install and use ROMP with this addon,please?

    image

    opened by hasanleiva 11
  • Hi, Looks like change another mixamo character not work

    Hi, Looks like change another mixamo character not work

    I have ROMP driven SMPL model looks OK, but when using CDBA (I using script locally) driven mixamo model, result looks wrong:

    image

    I am using this bone mapper in your repo:

    bones_mixamo_smpl_mapper = {
        "Hips": "Pelvis",
        "LeftUpLeg": "L_Hip",
        "RightUpLeg": "R_Hip",
        "Spine2": "Spine3",
        "Spine1": "Spine2",
        "Spine": "Spine1",
        "LeftLeg": "L_Knee",
        "RightLeg": "R_Knee",
        "LeftFoot": "L_Ankle",
        "RightFoot": "R_Ankle",
        "LeftToeBase": "L_Foot",
        "RightToeBase": "R_Foot",
        "Neck": "Neck",
        "LeftShoulder": "L_Collar",
        "RightShoulder": "R_Collar",
        "Head": "Head",
        "LeftArm": "L_Shoulder",
        "RightArm": "R_Shoulder",
        "LeftForeArm": "L_Elbow",
        "RightForeArm": "R_Elbow",
        "LeftHand": "L_Wrist",
        "RightHand": "R_Wrist",
        "LeftHandIndex1": "L_Hand",
        "LeftHandMiddle1": "L_Hand",
        "RightHandMiddle1": "R_Hand",
        "RightHandIndex1": "R_Hand",
    }
    bones_smpl_mixamo_mapper = {v: k for k, v in bones_mixamo_smpl_mapper.items()}
    bone_name_from_index_character = {
        k: bones_smpl_mixamo_mapper[v] for k, v in bone_name_from_index.items()
    }
    
    

    Do u know why?

    also the hand look not right.

    image

    opened by jinfagang 11
  • Detection variable: outputs = {'poses': poses, 'trans': trans[0]}

    Detection variable: outputs = {'poses': poses, 'trans': trans[0]}

    Hey, Yanchxx,

    May I ask, what is the output variable, I think "trans" is the translation between camera and object.

    What are the poses, is it the location x, y, z, or rotation x, y, z, degrees?

    Thank you 👍

    opened by zhangby2085 10
  • 导入fbx时骨骼方向乱了

    导入fbx时骨骼方向乱了

    @yanch2116 你好,非常酷的工作!我基本把整个项目跑通了,但我有两个问题想进一步请教一下你。 、 1.导入fbx时骨骼的方向错乱了,参见:https://www.bilibili.com/read/cv2520452 。但是这个文章里的方法没有完全解决方向错乱的问题,所以我想请问一下你,你是怎么解决这个问题的呢? 2.我想请教一下,如何将编辑骨架让其和smpl的骨架一致呢? 望不吝赐教!万分感谢!

    opened by syguan96 8
  • Use keyframes to prevent pose shaking

    Use keyframes to prevent pose shaking

    I found that the avatar will shake drastically when running the webcam demo. And I set the mode =1 to insert keyframe record the webcam results for playback.

    Now, if we set one more condition for inserting keyframe, only the frame_idx % 3 == 0 or frame_idx % 5 == 0. This would allow the avatar to move along these keyframes much smoother.

    However, is there a way to let the webcam demo runs in real-time using the keyframe strategy I said? This skip keyframe strategy only seems to work with the recorded playback. The character is still moving on each frame when we are actually running in real-time.

    opened by ZhengdiYu 4
  • Can I rotate the scene while running scripts?

    Can I rotate the scene while running scripts?

    Hi, I'm able to run to demo with blender now, but it turns out that the view is locked while the script is running.

    If there's a way to enable rotation?

    opened by anzisheng 4
  • Which version of Blender are you using?

    Which version of Blender are you using?

    I met the following error while running Beta.blend. I'm using blender 2.83.9. What's the expected version?

    Read blend: E:\Workspace\blender_test\addons\CharacterDriven-BlenderAddon-master\blender\Beta.blend 0 meshes freed Error: File written by newer Blender binary (290.0), expect loss of data!

    opened by sylyt62 4
  • multiple people

    multiple people

    Hey, yanch2116, very impressive job done in visualizing 3D characters. I tested and it works, one question to ask, the code romp_server.py has the setting.show_largest=True, I am thinking, what is the condition with multiple people in the webcamera, when there are many people, the blender character keeps on shifting. Are they ways to solve this? 1, keep on the object tracker in single object_ID 2, visualize multiple 3D characters as input in the camera

    opened by zhangby2085 3
Owner
犹在镜中
犹在镜中
Implementation for Shape from Polarization for Complex Scenes in the Wild

sfp-wild Implementation for Shape from Polarization for Complex Scenes in the Wild project website | paper Code and dataset will be released soon. Int

Chenyang LEI 41 Dec 23, 2022
A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets

HOW TO USE THIS PROJECT A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets Based on DeepLabCut toolbox, we run wit

1 Jan 10, 2022
This is an implementation for the CVPR2020 paper "Learning Invariant Representation for Unsupervised Image Restoration"

Learning Invariant Representation for Unsupervised Image Restoration (CVPR 2020) Introduction This is an implementation for the paper "Learning Invari

GarField 88 Nov 07, 2022
A MNIST-like fashion product database. Benchmark

Fashion-MNIST Table of Contents Why we made Fashion-MNIST Get the Data Usage Benchmark Visualization Contributing Contact Citing Fashion-MNIST License

Zalando Research 10.5k Jan 08, 2023
General Assembly Capstone: NBA Game Predictor

Project 6: Predicting NBA Games Problem Statement Can I predict the results of NBA games from the back-half of a season from the opening half of the s

Adam Muhammad Klesc 1 Jan 14, 2022
ESP32 python application to read data from a Tilt™ Hydrometer for homebrewing

TitlESP32 ESP32 MicroPython application to read and log data from a Tilt™ Hydrometer. Requirements A board with an ESP32 chip USB cable - USB A / micr

IoBeer 5 Dec 01, 2022
Efficient Householder transformation in PyTorch

Efficient Householder Transformation in PyTorch This repository implements the Householder transformation algorithm for calculating orthogonal matrice

Anton Obukhov 49 Nov 20, 2022
Face recognition with trained classifiers for detecting objects using OpenCV

Face_Detector Face recognition with trained classifiers for detecting objects using OpenCV Libraries required to be installed using pip Command: cv2 n

Chumui Tripura 0 Oct 31, 2021
Convolutional Neural Networks

Darknet Darknet is an open source neural network framework written in C and CUDA. It is fast, easy to install, and supports CPU and GPU computation. D

Joseph Redmon 23.7k Jan 05, 2023
Training deep models using anime, illustration images.

animeface deep models for anime images. Datasets anime-face-dataset Anime faces collected from Getchu.com. Based on Mckinsey666's dataset. 63.6K image

Tomoya Sawada 61 Dec 25, 2022
[CVPR21] LightTrack: Finding Lightweight Neural Network for Object Tracking via One-Shot Architecture Search

LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search The official implementation of the paper LightTra

Multimedia Research 290 Dec 24, 2022
Repository for "Improving evidential deep learning via multi-task learning," published in AAAI2022

Improving evidential deep learning via multi task learning It is a repository of AAAI2022 paper, “Improving evidential deep learning via multi-task le

deargen 11 Nov 19, 2022
Keyword spotting on Arm Cortex-M Microcontrollers

Keyword spotting for Microcontrollers This repository consists of the tensorflow models and training scripts used in the paper: Hello Edge: Keyword sp

Arm Software 1k Dec 30, 2022
Benchmark spaces - Benchmarks of how well different two dimensional spaces work for clustering algorithms

benchmark_spaces Benchmarks of how well different two dimensional spaces work fo

Bram Cohen 6 May 07, 2022
Storchastic is a PyTorch library for stochastic gradient estimation in Deep Learning

Storchastic is a PyTorch library for stochastic gradient estimation in Deep Learning

Emile van Krieken 140 Dec 30, 2022
OOD Generalization and Detection (ACL 2020)

Pretrained Transformers Improve Out-of-Distribution Robustness How does pretraining affect out-of-distribution robustness? We create an OOD benchmark

littleRound 57 Jan 09, 2023
PyTorch implementation of paper: AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer, ICCV 2021.

AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer [Paper] [PyTorch Implementation] [Paddle Implementation] Overview This reposit

148 Dec 30, 2022
Finding an Unsupervised Image Segmenter in each of your Deep Generative Models

Finding an Unsupervised Image Segmenter in each of your Deep Generative Models Description Recent research has shown that numerous human-interpretable

Luke Melas-Kyriazi 61 Oct 17, 2022
Implementation of ICCV 2021 oral paper -- A Novel Self-Supervised Learning for Gaussian Mixture Model

SS-GMM Implementation of ICCV 2021 oral paper -- Self-Supervised Image Prior Learning with GMM from a Single Noisy Image with supplementary material R

HUST-The Tan Lab 4 Dec 05, 2022
Companion code for the paper Theoretical characterization of uncertainty in high-dimensional linear classification

Companion code for the paper Theoretical characterization of uncertainty in high-dimensional linear classification Usage The required packages are lis

0 Feb 07, 2022