Automated Segmentation of Cell Images with CycleGAN
Cycle-Consistent Generative Adversarial Networks applied on Retinal Pigment Epithelium
Sorbonne Université · Télécom Paris · Institut Pasteur
Abstract
Segmenting cells in microscopy images normally requires many hand-prepared image–mask pairs. This project explores generative adversarial networks to reduce that need, with two objectives: generating synthetic cell images for data augmentation, and segmenting cells with a cycle-consistent GAN that can be trained without paired images and masks.
A DCGAN generates new retinal pigment epithelium (RPE) images, and a CycleGAN learns to translate between cell images and segmentation masks using adversarial, cycle-consistency and identity losses. Experiments use two public datasets: 1,032 RPE images from NIST and U2OS cells from the Broad Bioimage Benchmark Collection.
On held-out RPE images, the unpaired CycleGAN reaches Dice similarity coefficients between 0.79 and 0.87, with Hausdorff distances around 9.4–9.7.
Keywords: GAN · CycleGAN · DCGAN · unpaired image-to-image translation · cell segmentation · microscopy
Method
A GAN pairs a generator, which produces new samples, with a discriminator, which tells real samples from generated ones. CycleGAN learns a mapping between two image domains (cell images and segmentation masks) without paired examples.
- Adversarial losses push each generator to produce images its discriminator cannot tell from real ones.
- Cycle-consistency loss requires that translating an image to a mask and back recovers the original image.
- Identity loss keeps an image unchanged when it already belongs to the target domain.
2.1Generating cell images (DCGAN)
Following the DCGAN guidelines, the generator uses strided transposed convolutions, batch normalization, ReLU activations and a Tanh output; the discriminator uses strided convolutions and LeakyReLU (slope 0.2). Implemented in PyTorch and trained for 300 epochs with Adam (learning rate 0.0002, β₁ = 0.5), mini-batches of 128 and weights initialized from N(0, 0.02).
2.2Segmentation with CycleGAN
Each generator has an encoder of convolutions, a transformer of residual blocks and a decoder of transposed convolutions; the discriminators are PatchGANs. Two generators G: X → Y and F: Y → X are trained with adversarial losses plus a cycle-consistency loss, ‖F(G(x)) − x‖₁ + ‖G(F(y)) − y‖₁, weighted by λ.
Data
Table 1: Datasets.
| Dataset | Images | Notes |
|---|---|---|
| Human RPE cells (NIST) | 1,032 absorbance microscopy images, 256 × 256 | Used for generation and segmentation |
| Human U2OS cells (BBBC) | 55k+ images, 768 image–mask pairs | In- and out-of-focus high-content screening images |
Training
Results
Table 2: Segmentation quality on three test images
| Test image | Dice similarity coefficient | Hausdorff distance | Perceptual similarity |
|---|---|---|---|
| Image 1 | 0.85066 | 9.5082 | 0.398 |
| Image 2 | 0.86987 | 9.4055 | 0.324 |
| Image 3 | 0.7897 | 9.7022 | 0.453 |
Key findings
- Unpaired data works: real-world problems where paired data is not available can be treated with CycleGANs.
- Training is hard: training a GAN is not easy, and the equilibrium point is rarely reached.
- Better losses: other loss functions, such as an identity loss, can be used to build better models.
Report & slides
Project report (PDF, 4 MB) · Presentation slides (PDF, 2 MB) · Code on GitHub.
