SHI Labs

Neighborhood Attention Transformer

Neighborhood Attention (NA) is the first efficient and scalable sliding-window attention mechanism for vision. NA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels, and therefore enjoys a linear time and space complexity. The sliding-window pattern allows NA's receptive field to grow without needing extra pixel shifts, and preserves translational equivariance.

Dilated Neighborhood Attention Transformer

Dilated Neighborhood Attention (DiNA) extends NA's sliding-window local attention to sparse global attention at no additional cost. Combinations of NA/DiNA are capable of preserving locality, maintaining translational equivariance, expanding the receptive field exponentially, and capturing longer-range inter-dependencies, leading to significant performance boosts in downstream vision tasks.

OneFormer

OneFormer is the first multi-task universal image segmentation framework. It achieves state-of-the-art performance across semantic, instance and panoptic segmentation tasks with a single task-dynamic model that is jointly trained across tasks. OneFormer reduces underlying resource requirement for segmentation and makes image segmentation more universal and accessible.

Versatile Diffusion

We built Versatile Diffusion (VD), the first unified multi-flow multimodal diffusion framework, as a step towards Universal Generative AI. Versatile Diffusion can natively support image-to-text, image-variation, text-to-image, and text-variation, and can be further extended to other applications such as semantic-style disentanglement, image-text dual-guided generation, latent image-to-text-to-image editing, and more. Future versions will support more modalities such as speech, music, video and 3D.

StyleNAT

StyleNAT is a flexible and efficient state-of-the-art image generation framework. It is Style-based GAN that exploits Neighborhood Attention to extend the power of localized attention heads to capture long range features and maximize information gain within the generative process. The flexibility of the the system allows it to be adapted to various environments and datasets. StyleNAT attains a new SOTA FID score on FFHQ-256 with 2.046, beating prior arts with convolutional models such as StyleGAN-XL and transformers such as HIT and StyleSwin. These results show a 6.4% improvement on FFHQ-256 scores when compared to StyleGAN-XL with a 28% reduction in the number of parameters and 56% improvement in sampling throughput.

Layer-wise Image Vectorization

Layer-wise Image Vectorization (LIVE) is a new state-of-the-art image vectorization method to progressively generate a SVG that fits the raster image in a layer-wise fashion. Given an arbitrary input image, LIVE recursively learns the visual concepts by adding new optimizable closed bezier paths and optimizing all these paths