CS 184 Final Project Proposal

Face Morphing

Daiwei Liu, Provi Zhang



Summary

We want to map person A’s face onto person B’s face seamlessly in 2D and in 3D.


Problem Description

The problem is not technically important. We do it primarily for entertainment purpose.

  • Challenge 1: How to detect face on photo
  • Challenge 2: How to merge two face together naturally, despite different shapes of faces and poses.
  • Challenge 3: How to construct a 3D head model from photo(s).

    • Basic idea of approaching the problem

    Baseline - 2D case:

    Find an open source face detection API to detect faces and extract a set of feature points on the faces (eye, nose, lips, etc.). The points on the two faces should have an one-to-one mapping relationship. If the API does not generate enough points, consider manually adding feature points on GUI. Using the feature points, construct a triangle mesh for each of the faces, and pair corresponding individual triangle mashes. Then merge one of the mesh onto the other by alpha blending each pixel in each triangle mesh using barycentric coordinates.

    Aspiration - 3D case:

  • 3D Head Modeling

    Get an unisex baseline human head mesh, and select a few (around 50) key feature points on the mesh to be control points. Get the front-view photo and side-view photo of a head, and extract the corresponding feature points on these pictures and then distort the baseline mesh (using the key feature points) to fit them. Texture map the front-view photo and side-view photo onto the distorted mesh to get a 3D model of the head. Apply the same logic on the second head.

  • 3D Head Morphing

    Blend the two head meshes together. The texture of the morphed head mesh is calculated by alpha belnding each pair of corresponding triangles from the two source head meshes, and each vertex of the morphed head mesh is calculated as the average of each pair of corresponding vertices of the two source head meshes.


    • Goal & Deliverables

    The quality that we are aiming for in this project

  • Baseline Deliverables

    1. Given two photos A and B that are shot in a controlled environment (lighting, pose, size), generate a blended image of morphing face of A with the face of B.

    2. The result should look real and natural, which means viewers should not be able to easily tell it is two faces merged together.

  • Extra Deliverables if things goes well

    1. Given a set of photos taken in different angles that are shot in a controlled environment, generate a 3D head mesh.

    2. Given two head meshes with the same topology, generate a blended mesh of morphing head mesh of A with the head mesh of B.


    • Schedule

  • Week 1:

    Layout the ground work and get the face detection API working. Get reliable feature points from faces. Construct triangle mesh from points. Blend the texture on the two meshes using barycentric coordinates. (Baseline goal almost done)

  • Week 2:

    Optimize for realistic result (Baseline goal done). Try to do the 3D head modeling part of extra deliverable if possible.

  • Week 3:

    Try to do the 3D head morphing part of extra deliverables if possible.

  • Week 4:

    Buffer week


    • Resources

    OpenCV & OpenGL

    Inspiration of 2D face morphing: Face Morph Using OpenCV

    Inspiration of 3D head modeling: Convert Picture into 3D Model