Most explanations of deep learning list network types as if they were interchangeable. In industrial machine vision they are not. The right architecture depends on the question the inspection has to answer, how much labeled data exists, how fast a decision is needed and what hardware it must run on. This article compares the deep learning architectures used in machine vision: CNNs, vision transformers, detection and segmentation networks, anomaly detection models, GANs for synthetic data, and 3D networks, from the point of view of building systems that work on a production line.
Deep learning architectures at a glance
| Architecture | What it does | Where it fits in industrial vision | Watch-outs |
|---|---|---|---|
| CNN classifier | Labels a whole image or region | Pass/fail and defect-type sorting on fixtured parts | Needs examples of every class; no location output |
| Object detector | Finds and boxes objects or defects | Counting, locating parts, multiple defects per image | Box labels are slower to create; small defects need high resolution |
| Segmentation network | Labels every pixel | Measuring defect area, coverage, coatings, cells | Pixel masks are the most expensive labels |
| Anomaly detection | Learns what good looks like and flags differences | Rare or unpredictable defects; few defect samples | Flags any change, including harmless variation |
| Vision transformer | Uses attention across the whole image | Large pretrained models, global context, complex scenes | More data and compute; usually used pretrained |
| GAN or diffusion model | Generates synthetic images | Augmenting rare defect classes | Synthetic artifacts can mislead training |
| 3D CNN or point-cloud network | Learns from volumes or 3D points | CT, microscopy stacks, 3D sensor data | Memory and compute grow quickly |
| Recurrent network (RNN, LSTM) | Models sequences over time | Rare in modern vision; mostly replaced | Slower to train; temporal CNNs or transformers usually win |
Start with the question, not the network
The first decision is the output the system needs, because it fixes the labels you must create and the architecture family you use. A sorter that only needs pass or fail can use a classifier. A system that must report where a defect is and how large it is needs detection or segmentation. If defects are rare or impossible to predict, anomaly detection trained only on good parts is often the only practical start. See AI anomaly detection and deep learning vs rule-based machine vision for when deep learning is worth it at all.
CNNs: still the workhorse
Convolutional neural networks learn small filters that respond to edges, textures and shapes, and stack them so deeper layers recognize larger structures. Two properties make them a natural fit for inspection: they look for the same feature anywhere in the image, and they are efficient enough to run in milliseconds on an edge GPU. Most production classifiers, detectors and segmentation networks are still built on CNN backbones, often pretrained on large image collections and fine-tuned on the customer’s parts.
Design choices that matter more than the architecture name: input resolution (a defect must span enough pixels to be learned), the receptive field relative to defect size, and how the image is tiled for large parts. Many failed deep learning projects used a sound network on images where the defect simply was not visible.
Vision transformers vs CNNs
Vision transformers split an image into patches and use attention to relate every patch to every other one. That gives them global context a CNN builds only gradually, and it is why large foundation models are transformer-based. The cost is data and compute: trained from scratch, transformers need far more images than a CNN to learn the local patterns a convolution captures by design.
| CNN | Vision transformer | |
|---|---|---|
| Data needed from scratch | Moderate | Large |
| With a pretrained backbone | Fine-tunes well on hundreds of images | Fine-tunes well on hundreds of images |
| Strength | Local texture and small defects | Global context, complex scenes, varied parts |
| Edge inference cost | Low to moderate | Moderate to high |
| Typical industrial use | Most fixtured inspection | Varied products, open-ended visual questions |
In practice the choice is rarely pure. Hybrid models use convolutional early layers with attention later, and attention modules are added to CNNs to focus on relevant regions. For a typical fixtured inspection with a few hundred to a few thousand images, a pretrained CNN remains the safe default; transformers earn their place when parts vary widely or the system must answer more open questions, as in vision language models.
Transfer learning: how small datasets work
Very few industrial projects have the data to train a network from scratch. Transfer learning starts from a network pretrained on millions of general images, keeps the early layers that detect generic edges and textures, and retrains the later layers on the customer’s parts. With good imaging, a few hundred labeled examples per class can be enough for a first model. The usual practice is to freeze most of the backbone at first, then unfreeze more layers as data grows, and to validate on parts from different lots and shifts than the training set. See how much training data AI visual inspection needs.
GANs and synthetic defect images
Generative adversarial networks pit a generator that creates images against a discriminator that tries to tell them from real ones. In inspection, generative models (GANs and, increasingly, diffusion models) are used to create extra examples of rare defects, so a classifier sees more than the handful that occurred in production.
- Use them to augment, not replace: synthetic images widen coverage of a defect class that already has real examples.
- Validate only on real defects: a model can learn the generator’s artifacts and look better than it is.
- Check realism with inspectors: if an expert can spot the fakes, the network probably can too.
- Consider simpler augmentation first: rotation, lighting, blur and cut-and-paste defects often deliver most of the gain.
3D data: 3D CNNs and point clouds
When the data is a volume, such as CT scans, confocal or light sheet stacks, 3D CNNs extend convolution into depth and learn features across slices. For 3D sensor data, point-cloud networks work directly on measured points. Both cost much more memory and compute than 2D networks. A common engineering shortcut is 2.5D: converting a height map or a few slices into image channels and using a 2D network, which is often accurate enough and far cheaper. See 3D vision systems for automated inspection.
What happened to RNNs?
Recurrent networks such as LSTMs were once the standard for sequences, including video and sensor streams. In vision they have largely been replaced: temporal convolutions and transformers train faster and handle long sequences better. For time-series problems in a plant, such as vibration or process signals, see AI predictive maintenance.
Deployment constraints decide more than accuracy
A model that scores well offline still has to meet the line rate, run on available hardware and stay accurate as conditions drift. Inference time, model size and quantization often push the choice toward a smaller CNN over a larger, slightly more accurate model. Hardware matters too; see GPU vs CPU vs FPGA for image processing.
How Sciotex chooses an architecture
Frequently asked questions
The main families are CNN classifiers, object detectors, segmentation networks, anomaly detection models, vision transformers, generative models for synthetic data, and 3D networks for volumetric or point-cloud data. The task and the available data decide which fits.
Not by default. Pretrained CNNs remain the practical choice for most fixtured inspection with limited data. Vision transformers help when parts vary widely, global context matters or a large pretrained model can be fine-tuned.
With transfer learning and good imaging, a first model can often be trained on a few hundred labeled examples per class. Production models typically need more, covering lots, shifts and borderline parts.
No. Synthetic images from GANs or diffusion models can augment rare classes, but models must be validated on real defects, because they can learn artifacts of the generator.