An addon uses SMPL's poses and global translation to drive cartoon character in Blender.

Overview

Blender addon for driving character

The addon drives the cartoon character by passing SMPL's poses and global translation into model's armature in Blender. Poses and global translation can be obtained from ROMP or any other 3D pose estimation model. If the model outputs poses and global translation at a high FPS, you can drive cartoon characters in Blender in real time.

Demo

image image

The first demo uses ROMP outputs from the video, which is stored in a file.

The second demo uses ROMP outputs from the webcam in real time.

How to Use the add-on

Data Requester

This addon is a data requester that sends a data request over TCP to 127.0.0.1:9999.It gets one data at a time from the data server.

After running the addon by pressing ctrl+E in Blender, it keeps asking for data until the server closes the TCP connection. You can also press A to close the TCP connection.

Data Server

The data server is bound to 127.0.0.1:9999. After receiving a request from the data requester, one data is sent to the requester.

I've written server.py as a example data server (you only need to know a little about Python TCP to understand it).

Real-time data server can be found in ROMP.You just need to run webcam_blender.sh.

Data Format

The data is a Python list of four elements in the form of [mode,poses,global translation,current keyframe id].

  1. Mode is an integer, 1 for insert keyframe, 0 for no insert keyframe. Insert keyframes and animation rendering can be carried out later. Keyframes are generally not inserted in real-time mode to make real-time driving cartoon characters smoother.
  2. Poses is a list of length 72.
  3. Global translation is a list of length 3. If you don't need global translation, just go into [0,0,0].
  4. Current keyframe id is an integer.If you insert keyframes, you should set it to the correct keyframe id. If you don't insert keyframes, just set it to 0.

Steps

  1. Install the addon in Blender
  2. Run data server
  3. Press Ctrl+E in Blender to run addon
  4. Press A in Blender to stop addon or wait until the data transfer is complete

In step 3, you'd better select Armature, otherwise bugs may occur. Also, the mouse must be placed in the 3D viewport area(where the model is), otherwise the addon will not run.

Something about Blender

If you're not familiar with Blender, I've placed a blender project in the resources folder to help you.All you need to do is open it and follow Steps to achieve the effect shown in the Demo.(It's better to know something about animation in Blender.)

If you need a video background in the demo, select Compositing in the top menu bar, click Open Clip in the Movie Clip, and select your video.

图 2

If you are familiar with Blender and want to use your own models, you should make sure it's armature is SMPL's skeleton. The armature should name Armature and each bone has the same name as the bones in demo model(Only the 24 bones of SMPL skeleton are needed, and the fingers don't need to change their names).

图 3

Comments
  • How to run .fbx file to control the charater

    How to run .fbx file to control the charater

    Hello. I have successfully run the demo of ROMP, which exported .fbx file. And currently I want to use the .fbx to control the character. Can you provide steps for the video demo? I can only see the camera one

    opened by CheungBH 31
  • [simple-romp] How to use it?

    [simple-romp] How to use it?

    I tried to use it with simple-romp but it did not work. I already created an issue at the ROMP repository and described my problem here and here in detail.

    The author of the ROMP repository answered there:

    About live blender driving, please refer to this repo. https://github.com/yanch2116/CharacterDriven-BlenderAddon My colleague is responsible for maintaining this funciton now. Best regard.

    and closed the issue.

    @yanch2116 So how to solve it?

    My goal is to send the positions and quaternions of ROMP (or any other SMPL based solution) over the VMC protocol to other application (not Blender). Any ideas, whats the best way to do it?

    opened by vivi90 26
  • Could I ask for a detailed instruction on how to change the 3D Character?

    Could I ask for a detailed instruction on how to change the 3D Character?

    Also, I would like to know if there is a way to replace the 3D Character before I run the script and process the video so that I don't have to manually change it every time?

    opened by XXZhe 12
  • Can't connect ROMP into addon

    Can't connect ROMP into addon

    I use ROMP v1.1 on Windows machine. But CDBA works with version 1.0 of ROMP I read installation and using documentation of ROMP v1.0 but i can't figured out all How can i use this addon properly? I'm not good at programming i'm animator and interested in your project. Can you write steps how to install and use ROMP with this addon,please?

    image

    opened by hasanleiva 11
  • Hi, Looks like change another mixamo character not work

    Hi, Looks like change another mixamo character not work

    I have ROMP driven SMPL model looks OK, but when using CDBA (I using script locally) driven mixamo model, result looks wrong:

    image

    I am using this bone mapper in your repo:

    bones_mixamo_smpl_mapper = {
        "Hips": "Pelvis",
        "LeftUpLeg": "L_Hip",
        "RightUpLeg": "R_Hip",
        "Spine2": "Spine3",
        "Spine1": "Spine2",
        "Spine": "Spine1",
        "LeftLeg": "L_Knee",
        "RightLeg": "R_Knee",
        "LeftFoot": "L_Ankle",
        "RightFoot": "R_Ankle",
        "LeftToeBase": "L_Foot",
        "RightToeBase": "R_Foot",
        "Neck": "Neck",
        "LeftShoulder": "L_Collar",
        "RightShoulder": "R_Collar",
        "Head": "Head",
        "LeftArm": "L_Shoulder",
        "RightArm": "R_Shoulder",
        "LeftForeArm": "L_Elbow",
        "RightForeArm": "R_Elbow",
        "LeftHand": "L_Wrist",
        "RightHand": "R_Wrist",
        "LeftHandIndex1": "L_Hand",
        "LeftHandMiddle1": "L_Hand",
        "RightHandMiddle1": "R_Hand",
        "RightHandIndex1": "R_Hand",
    }
    bones_smpl_mixamo_mapper = {v: k for k, v in bones_mixamo_smpl_mapper.items()}
    bone_name_from_index_character = {
        k: bones_smpl_mixamo_mapper[v] for k, v in bone_name_from_index.items()
    }
    
    

    Do u know why?

    also the hand look not right.

    image

    opened by jinfagang 11
  • Detection variable: outputs = {'poses': poses, 'trans': trans[0]}

    Detection variable: outputs = {'poses': poses, 'trans': trans[0]}

    Hey, Yanchxx,

    May I ask, what is the output variable, I think "trans" is the translation between camera and object.

    What are the poses, is it the location x, y, z, or rotation x, y, z, degrees?

    Thank you 👍

    opened by zhangby2085 10
  • 导入fbx时骨骼方向乱了

    导入fbx时骨骼方向乱了

    @yanch2116 你好,非常酷的工作!我基本把整个项目跑通了,但我有两个问题想进一步请教一下你。 、 1.导入fbx时骨骼的方向错乱了,参见:https://www.bilibili.com/read/cv2520452 。但是这个文章里的方法没有完全解决方向错乱的问题,所以我想请问一下你,你是怎么解决这个问题的呢? 2.我想请教一下,如何将编辑骨架让其和smpl的骨架一致呢? 望不吝赐教!万分感谢!

    opened by syguan96 8
  • Use keyframes to prevent pose shaking

    Use keyframes to prevent pose shaking

    I found that the avatar will shake drastically when running the webcam demo. And I set the mode =1 to insert keyframe record the webcam results for playback.

    Now, if we set one more condition for inserting keyframe, only the frame_idx % 3 == 0 or frame_idx % 5 == 0. This would allow the avatar to move along these keyframes much smoother.

    However, is there a way to let the webcam demo runs in real-time using the keyframe strategy I said? This skip keyframe strategy only seems to work with the recorded playback. The character is still moving on each frame when we are actually running in real-time.

    opened by ZhengdiYu 4
  • Can I rotate the scene while running scripts?

    Can I rotate the scene while running scripts?

    Hi, I'm able to run to demo with blender now, but it turns out that the view is locked while the script is running.

    If there's a way to enable rotation?

    opened by anzisheng 4
  • Which version of Blender are you using?

    Which version of Blender are you using?

    I met the following error while running Beta.blend. I'm using blender 2.83.9. What's the expected version?

    Read blend: E:\Workspace\blender_test\addons\CharacterDriven-BlenderAddon-master\blender\Beta.blend 0 meshes freed Error: File written by newer Blender binary (290.0), expect loss of data!

    opened by sylyt62 4
  • multiple people

    multiple people

    Hey, yanch2116, very impressive job done in visualizing 3D characters. I tested and it works, one question to ask, the code romp_server.py has the setting.show_largest=True, I am thinking, what is the condition with multiple people in the webcamera, when there are many people, the blender character keeps on shifting. Are they ways to solve this? 1, keep on the object tracker in single object_ID 2, visualize multiple 3D characters as input in the camera

    opened by zhangby2085 3
Owner
犹在镜中
犹在镜中
Implementation EfficientDet: Scalable and Efficient Object Detection in PyTorch

Implementation EfficientDet: Scalable and Efficient Object Detection in PyTorch

tonne 1.4k Dec 29, 2022
An implementation of based on pytorch and mmcv

FisherPruning-Pytorch An implementation of Group Fisher Pruning for Practical Network Compression based on pytorch and mmcv Main Functions Pruning f

Peng Lu 15 Dec 17, 2022
[CVPR 2022] Thin-Plate Spline Motion Model for Image Animation.

[CVPR2022] Thin-Plate Spline Motion Model for Image Animation Source code of the CVPR'2022 paper "Thin-Plate Spline Motion Model for Image Animation"

yoyo-nb 1.4k Dec 30, 2022
Anime Face Detector using mmdet and mmpose

Anime Face Detector This is an anime face detector using mmdetection and mmpose. (To avoid copyright issues, I use generated images by the TADNE model

198 Jan 07, 2023
Multiple custom object count and detection using YOLOv3-Tiny method

Electronic-Component-YOLOv3 Introduce This project created to detect, count, and recognize multiple custom object using YOLOv3-Tiny method. The target

Derwin Mahardika 2 Nov 14, 2022
An example showing how to use jax to train resnet50 on multi-node multi-GPU

jax-multi-gpu-resnet50-example This repo shows how to use jax for multi-node multi-GPU training. The example is adapted from the resnet50 example in d

Yangzihao Wang 20 Jul 04, 2022
50-days-of-Statistics-for-Data-Science - This repository consist of a 50-day program

50-days-of-Statistics-for-Data-Science - This repository consist of a 50-day program. All the statistics required for the complete understanding of data science will be uploaded in this repository.

komal_lamba 22 Dec 09, 2022
🐥A PyTorch implementation of OpenAI's finetuned transformer language model with a script to import the weights pre-trained by OpenAI

PyTorch implementation of OpenAI's Finetuned Transformer Language Model This is a PyTorch implementation of the TensorFlow code provided with OpenAI's

Hugging Face 1.4k Jan 05, 2023
Official repository of ICCV21 paper "Viewpoint Invariant Dense Matching for Visual Geolocalization"

Viewpoint Invariant Dense Matching for Visual Geolocalization: PyTorch implementation This is the implementation of the ICCV21 paper: G Berton, C. Mas

Gabriele Berton 44 Jan 03, 2023
Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh

Arjun Majumdar 44 Dec 14, 2022
Unified learning approach for egocentric hand gesture recognition and fingertip detection

Unified Gesture Recognition and Fingertip Detection A unified convolutional neural network (CNN) algorithm for both hand gesture recognition and finge

Mohammad 227 Dec 25, 2022
Convolutional 2D Knowledge Graph Embeddings resources

ConvE Convolutional 2D Knowledge Graph Embeddings resources. Paper: Convolutional 2D Knowledge Graph Embeddings Used in the paper, but do not use thes

Tim Dettmers 586 Dec 24, 2022
The official implementation of paper "Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networks" (IJCV under review).

DGMS This is the code of the paper "Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networks". Installation Our code works with Pytho

Runpei Dong 3 Aug 28, 2022
基于PaddleClas实现垃圾分类,并转换为inference格式用PaddleHub服务端部署

百度网盘链接及提取码: 链接:https://pan.baidu.com/s/1HKpgakNx1hNlOuZJuW6T1w 提取码:wylx 一个垃圾分类项目带你玩转飞桨多个产品(1) 基于PaddleClas实现垃圾分类,导出inference模型并利用PaddleHub Serving进行服务

thomas-yanxin 22 Jul 12, 2022
Pytorch implementation of the paper: "A Unified Framework for Separating Superimposed Images", in CVPR 2020.

Deep Adversarial Decomposition PDF | Supp | 1min-DemoVideo Pytorch implementation of the paper: "Deep Adversarial Decomposition: A Unified Framework f

Zhengxia Zou 72 Dec 18, 2022
PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition, CVPR 2018

PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place

Mikaela Uy 294 Dec 12, 2022
Python parser for DTED data.

DTED Parser This is a package written in pure python (with help from numpy) to parse and investigate Digital Terrain Elevation Data (DTED) files. This

Ben Bonenfant 12 Dec 18, 2022
NAVER BoostCamp Final Project

CV 14조 final project Super Resolution and Deblur module Inference code & Pretrained weight Repo SwinIR Deblur 실행 방법 streamlit run WebServer/Server_SRD

JiSeong Kim 5 Sep 06, 2022
deep learning for image processing including classification and object-detection etc.

深度学习在图像处理中的应用教程 前言 本教程是对本人研究生期间的研究内容进行整理总结,总结的同时也希望能够帮助更多的小伙伴。后期如果有学习到新的知识也会与大家一起分享。 本教程会以视频的方式进行分享,教学流程如下: 1)介绍网络的结构与创新点 2)使用Pytorch进行网络的搭建与训练 3)使用Te

WuZhe 13.6k Jan 04, 2023
PixelPick This is an official implementation of the paper "All you need are a few pixels: semantic segmentation with PixelPick."

PixelPick This is an official implementation of the paper "All you need are a few pixels: semantic segmentation with PixelPick." [Project page] [Paper

Gyungin Shin 59 Sep 25, 2022