Benchmarking Deep Learning Inference Optimization Strategies on NVIDIA Jetson Edge Devices
Supervisor: Florens
Description:
Deploying deep learning models on NVIDIA Jetson edge devices requires balancing inference speed against the limited compute and power resources available compared to datacenter GPUs. Common optimization strategies include model-level techniques (quantization, pruning, layer fusion via TensorRT) as well as deployment-level choices such as running natively on the device versus inside containerized environments like Docker. The goal of this thesis is to systematically benchmark deep learning inference performance on NVIDIA Jetson hardware across different optimization techniques (e.g. quantization, TensorRT) and deployment configurations (native vs. containerized).
More details
Tasks:
- Literature and tooling review: Jetson architecture, TensorRT and related optimization frameworks, containerization on embedded devices
- Select representative model(s) and task(s) (e.g. image classification, object detection) for benchmarking
- Implement optimization pipeline: convert/optimize models (quantization, pruning, graph optimization)
- Implement two deployment variants: native and Docker-based
- Design and run a benchmarking matrix (optimization level × deployment strategy) measuring latency, throughput, and resource consumption
- Analyze results, identify best-performing configurations per use case, and report limitations