This is a project very important for the younger me who wanted to be an artist and wished to have a tool like this.
Why did I create this?
I really wanted to be a storyboard artist when I was a kid. They were so cool! Story artists built the underlying story. They had to understand composition, perspective, anatomy, and timing.
So I started studying to become one. It was tough learning, but I improved rapidly. During this process there were a few tools I really wished existed, and this project is focused on one particular pain point: the difficulty of finding a pose idea I have in my head online.
The worst feeling is having a great composition in your head, wanting a particular pose, but not having the ability to ensure anatomical accuracy by finding a close enough reference. Google tries its best, but its inherently difficult to use words to describe a pose.
Take a person sitting. There’s so many ways a person can sit, but it’s so difficult to describe the exact angle, the way the legs should interact with each other, etc. It is difficult to translate a pose into words.
That’s why I created this project! It’s inspired by a preexisting tool called pose search, but adds a sketch-to-pose and photo-to-pose pipeline and uses a new pose inference method called HSMR to allow for more anatomical accuracy. It uses a custom blender skeleton modified from this free skeleton vtuber skin I found online to include the ulna-radius movement for realistic forearm pronation and supination.
To preface, this is a personal research project with no intentions of monetary gain. Please enjoy reading this short blog about my creation process!
The Website
The bulk of my time for this project was actually focused on the research side of it, but I figured I should have a short section dedicated to the resulting website.
Design
My first project was bogged down by signups. I learned my lesson. I wanted to make sure this website was easy to use for someone who randomly stumbles upon the site.
Thus I maintained a simple interface, with clear instructions.
For the searcher itself, I wanted the image layout to be reminiscent of Pinterest. In addition, I needed it to be easy to swap between different datasets and inputs. So I used the masonry layout and a dropdown that toggles the dataset.
On the input section, the user can choose between modifying the skeleton and uploading a sketch or photograph.
The primary issue I found with preexisting pose searchers was the ability to control the pose. I hated spending time wrestling with the software. For most sites it’s difficult to get both the ability to make broad changes to the pose, as well as, small modifications. That’s why I have the sketch-to-pose and photo-to-pose models separate from the skeleton angle modifier.
The upload allows users to very quickly establish the rough idea of their pose, while the skeleton angle modifier lets the user choose the exact angle of each anatomical body part.
Systems Design
Moving on to the internal system, I had a few functional and non-functional requirements.
There are two core entities. The dataset schema and the pose schema. Every dataset is composed of a collections of poses. Searching is specific to a dataset. The poses contain all the inferred data from my services.
At a high level the system is very simple. The user fires a query from the frontend, depending on the input type (sketch, photo, or skeleton) I either run the ML-services on the upload to get the bone angles or I take the bone angles and camera rotation directly from the skeleton. I take this input and search through the database.
Diving deeper into the search system itself, typical pose searchers using MediaPipe have precomputed embeddings. They use the Euclidean distance between the target pose and the embedding to find the closest poses. Because the HSMR model returns two primary outputs the SKEL angles and the 3D joint position, I decided to do a hybrid search.
The query takes two vectors: 46 SKEL angles and 54 bone-direction floats. I do not use the raw position because there can be confounding effects caused by torso and joint length, instead I use the direction of each bone. The system retreives a small pool with two HNSW queries, then rescores that pool using a combination of both signals.
HNSW is a graph index method for approximate nearest neighbors built into the pgvector extension that works faster than a full scan. I decided to use it because I wanted the search to scale as dataset size increases.
A fun problem I ran into using the angles as a search metric was the fact that between two camera angles, the joint angles stayed the same. That is, limb angles do not change if you walk around a skeleton, but how the pose looks does.
In order to fix this, I passed a 3x3 camera rotation from the Three.js orbit. Since the body direction primarily follows the box of the pelvis, I only needed to rotate the pelvis params to capture the camera angle.
Another feature of the site is how it stores state. Because I wanted to have the ability to save a specific search, I figured it would be best to have the url translate to the actual embedding. Since the pose is too big to put in the URL as JSON, the binary is zipped and then converted to base 64.
Research
This section covers the more researchy part of the project. I realized that my projects didn’t have to be only focused on the SWE side and that I could use my personal background to create some custom components.
I’ll go over the sketch-to-pose models first and then the distilled model.
Sketch Models
There are preexisting sketch-to-pose models like Sketch2PoseNet and Sketch2Pose, but the Sketch2PoseNet’s models aren’t publicly released and I wanted to specialize my model toward one type of sketch input.
Because the goal was to use this sketch assisted pose searching on the website, I figured the sketch needed to be easy and quick to draw using the cursor.
So I wanted the input to look like a stick figure and still contain enough information about the pose. Since there wasn’t preexisting data for my goal online, I artificially created the data using the COCO 2014 dataset (27,237 images) for training and LSP-EXTENDED dataset (9,428 images) for evaluation.
Architecture
Results
Distilled Model
Architecture
Results
Development
/// improvements to development practices, 9 stage ci/cd thing
Reflection
Conclusion
Future Plans
Project Documentation
A collection of notes of design ideas I had when I was actually working on the project. Dates are sometimes included, you can see the swap from Mediapipe to HSMR in real time!