Latent-Identity Tuning in Text-to-Image Personalization Models

Abstract

Generating and editing a person's face demands high precision, as even minor modifications can significantly alter a subject’s perceived identity. Current personalization and editing methods built on general-purpose text-to-image models, however, often lack the precision required for fine-grained facial edits. We present a method for fine-grained identity tuning in text-to-image personalization models. Unlike standard image editing, which operates on a given image, identity tuning modifies the latent representation of a specific identity, enabling the generation of diverse images that consistently depict the same edited identity. To enable fine-grained latent identity tuning, we explore the latent space of an encoder trained for text-to-image personalization. This space consists of a set of latent tokens that play distinct roles in capturing different aspects of an identity and often correspond to specific spatial or semantic facial regions. We show that meaningful directions can be identified within this space and within subspaces defined by selected tokens, enabling localized, fine-grained, and semantically coherent edits. We validate our approach through qualitative and quantitative experiments that demonstrate diverse localized facial edits while preserving cross-image identity consistency.

Token behavior in the personalization encoder

Visualizing the attention maps of the personalization encoder reveals that identity tokens capture different facial regions.
Different tokens specialize in capturing specific features such as eyes, nose, and mouth, demonstrating the spatial and semantic structure of the identity space.

How does it work?

An input image is encoded using a pre-trained personalization encoder into a set of identity tokens.
Each token captures a distinct aspect of the face (e.g., eyes, nose, or hair), which we identify using patch editing as described later.
These tokens reside in an identity space, where meaningful directions (e.g., curly hair) can be discovered and traversed to achieve targeted edits.
The modified identity tokens are then passed to a text-to-image model, which generates images of the edited identity in diverse contexts.

Supervised directions

Using a labeled dataset of identity attributes, we discover directions for specific attributes (e.g., bald, beard, rosy cheeks) in the identity space through two supervised approaches.
The first approach calculates the mean identity representation with and without an attribute, then uses the difference between them as a direction to control that attribute.
The second approach trains an SVM to classify identity attributes, then uses the normal to the SVM hyperplane as the direction that corresponds to the attribute.
Each row shows a different source identity (leftmost column) and its edits along three directions. The images in each column are generated by adding the same direction. For each edit, we show the main result with two alternative prompts stacked vertically to the right.

Unsupervised directions

We apply PCA on the identity space to discover directions that capture the most variance in facial appearance.
PCA can be applied either globally (on all tokens) or locally (on a specific subset of tokens).
To achieve localized edits, we first identify which tokens correspond to a specific facial region, then apply PCA only on those tokens.
The resulting principal components serve as edit directions.
Each row shows a different source identity (leftmost column) and its transformations along three unsupervised directions. From left to right, we show edits applied via: a direction from a single token, a direction from all tokens, and a direction from a subset of tokens.

Identity-Based Directions

We can transfer specific facial features from one identity to another by blending their token representations.
First, we encode both a main identity and a donor identity into their respective identity tokens.
Then, we replace specific tokens in the main identity (e.g., those corresponding to the lips) with the corresponding tokens from the donor, effectively transferring that facial feature.
In each grid, the leftmost column shows the main identity, and each subsequent column shows the result of transferring one facial part from the donor shown at the top. The left grid demonstrates transfers of the periocular region (eyelids and eyebrows), while the right grid shows lip transfers. All other facial attributes and the background remain unchanged.

Patch Editing

To identify which tokens correspond to a specific facial feature, we use a patch-based approach.
We create multiple variants of an identity by pasting different patches (e.g., different lips) onto the same face before encoding.
By observing which tokens change the most across these variants, we can identify the tokens responsible for that facial feature.
Interestingly, this approach can also be used directly for editing: by pasting a desired facial element (e.g., a specific beard style) onto an identity before encoding, we can seamlessly add that element to the generated images.

BibTeX

TBD