VAE for perfectly square pixels

This is a trained VAE to enforce that pixels are always 4x4 in size and solid in color. It replicates an existing pixelation model while being built into the model itself.

Compability

Most architectures uses an unmodified VAE and does not retrain it, so it can be used accross different generations of models.

• SDXL: Can be used for all SDXL based models, such as Pony, Illustrious, and NoobXL. They all use the original VAE shipped with SDXL.

• Anima: Can be used for Anima and Qwen Image (untested).

How VAEs works

VAE's are used to reduce the working resolution of the model so they don't use as much VRAM. SDXL for example runs in a 8 times smaller resolution, then uses the VAE decoder to upscale it to full resolution. This model exploits that to enforce that this upscale process always results in square pixels.

Further work

The VAE has been trained to produce pixelated images from non-pixelated images. This means it isn't as reliable if the original image produced is close to pixelated, but contains errors. Training so it keeps pixelated images exactly the same, or potentially also fixes images with have inconsistent pixel sizes could improve the output.

For high contrast areas, some ringing can happen. Training with syntetic data will likely help.

Only the VAE decoder has been trained. It is likely possible to train the encoder to remove the pixelation as well, which could potentially make it easier to train Loras.