Architecture
This document describes the safeuploads validation pipeline, component responsibilities, and data flow for each file type.
Component Overview
safeuploads/
├── file_validator.py # FileValidator — orchestrator
├── config.py # FileSecurityConfig, SecurityLimits
├── protocols.py # SeekableFile, UploadFileProtocol
├── exceptions.py # Exception hierarchy, ErrorCode
├── enums.py # Threat categories, patterns
├── audit.py # SecurityAuditLogger, correlation IDs
├── utils.py # ResourceMonitor
├── validators/
│ ├── base.py # BaseValidator interface
│ ├── unicode_validator.py # Unicode normalization & checks
│ ├── extension_validator.py # Extension allow/block rules
│ ├── filename_validator.py # Path and length rules for filenames
│ ├── signature_validator.py # File header signature checks
│ ├── mime_validator.py # Content MIME detection and enforcement
│ ├── windows_validator.py # Windows reserved name checks
│ ├── fit_validator.py # FIT container structure checks
│ ├── image_dimension_validator.py # Pixel count from image headers
│ ├── compression_validator.py # ZIP bomb detection
│ └── xml_validator.py # XXE-safe XML parsing
└── inspectors/
├── base.py # BaseInspector interface
├── zip_inspector.py # Deep ZIP content analysis
├── gzip_inspector.py # Gzip bomb detection
├── pdf_inspector.py # PDF structure, active content, stream bombs
└── content_inspector.py # Malware/polyglot scanning
Roles
| Component | Responsibility |
|---|---|
FileValidator |
Orchestrates validation for each file type. Manages streaming, resource monitoring and audit events, and delegates to validators/inspectors. The async methods offload blocking inspection to a worker thread (or a configured executor); the *_sync methods run the same pipeline in the calling thread and refuse to run on an event-loop thread. Exposes them as filename_validator, signature_validator, mime_validator and the other validator attributes; magic_mime and magic_available are properties that delegate to mime_validator. |
FileSecurityConfig |
Centralizes all configuration: allowed MIME types mapped to the extensions each permits, blocked extensions, Unicode characters, Windows reserved names, and the PDF names (BLOCKED_PDF_NAMES) that reject a PDF. |
SecurityLimits |
Holds numeric thresholds: file sizes, compression ratios, timeouts, entry limits, resource caps. |
BaseValidator |
Abstract base class; validators inherit and implement validate(). |
UnicodeSecurityValidator |
NFC normalization and dangerous character detection (rejects with UnicodeSecurityError). |
ExtensionSecurityValidator |
Blocked, dangerous and compound extension checks. validate_allowed checks an extension against the allowlist and the blocked list. |
WindowsSecurityValidator |
Rejects Windows reserved device names. |
FilenameSecurityValidator |
Strips path components and dangerous characters from a filename and limits its length to max_sanitized_name_length. |
FileSignatureValidator |
Checks that the file header matches the signatures of the expected type (image: JPEG, PNG, WebP, or HEIC: an ISOBMFF ftyp box naming the heic or heix brand; ZIP, gzip, activity, FIT, or PDF with %PDF- at offset 0). |
MimeTypeValidator |
Detects the MIME type from content with libmagic (serialized by a lock, never falling back to the filename), enforces the allowed MIME types and requires the filename's extension to be one listed for the detected type. |
CompressionSecurityValidator |
Checks compression ratio, uncompressed size, entry count, and nested archive limits, and that the listed entries fill the file exactly. |
XmlSecurityValidator |
Parses XML with defusedxml, blocking DTDs and external entities (raises XmlSecurityError). |
FitSecurityValidator |
Checks the FIT container: headers, header CRC, data size, CRC-16 per section, and that the sections cover the file exactly. |
ImageDimensionValidator |
Reads width and height from the PNG IHDR chunk, the first JPEG frame header (SOF), the first WebP chunk (VP8, VP8L or VP8X canvas) or the HEIC primary item's ispe property without decoding pixels, and rejects an image whose pixel count exceeds max_image_pixels (raises ImageDimensionError). For HEIC, every ispe in the file is checked, and a grid's descriptor and tiles must match its ispe. A missing or malformed dimension header raises FileSignatureError with IMAGE_INVALID_STRUCTURE. |
ZipContentInspector |
Iterates ZIP entries checking for traversal, executables, scripts, symlinks, and recursive structures, and runs the top-level Unicode, Windows reserved-name and blocked-extension checks on each entry name. With allow_nested_archives, inspects every inner ZIP like the top level. |
GzipContentInspector |
Streams gzip decompression, checking ratio and size progressively. |
PdfContentInspector |
Tokenizes the whole PDF and verifies the structure readers rely on (header, trailing startxref and %%EOF, every cross-reference entry, every stream /Length, every object stream offset), so no reader parses an object the scan did not see. Rejects encryption, names in BLOCKED_PDF_NAMES after #xx decoding (including inside object streams), external stream data, unsupported filters and Flate output past max_pdf_uncompressed_size or max_compression_ratio (raises PdfSecurityError). |
ContentSecurityInspector |
Scans raw bytes for structurally verified executable headers, web shells, and polyglot patterns (image, activity and PDF uploads; polyglot signatures for image and activity only). |
ResourceMonitor |
Context manager enforcing wall-clock time and peak-RSS memory limits at loop boundaries and on exit. |
SecurityAuditLogger |
Emits structured audit events under safeuploads.audit with correlation IDs, escaping and length-capping attacker-controlled fields. |
Validation Pipelines
The validate_*_file_sync methods run the identical steps listed below in the calling thread, without the worker-thread offload. Every method returns a ValidationResult with the sanitized filename and the detected MIME type.
Image Validation (validate_image_file)
UploadFile
│
▼
┌─────────────────────────────┐
│ 1. Audit: VALIDATION_START │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 2. Filename Validation │
│ ├── Unicode normalization │
│ ├── Dangerous characters │
│ ├── Windows reserved name │
│ ├── Extension blocklist │
│ ├── Length limit, recheck │
│ └── Extension allowlist │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 3. ResourceMonitor START │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 4. Stream to SpooledTempFile│
│ (size checked per chunk; │
│ memory < 10MB → disk) │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 5. MIME Type Check │
│ (python-magic, first 8KB)│
│ ├── Type allowed │
│ └── Extension listed for │
│ the detected type │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 6. File Signature Check │
│ (magic bytes) │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 7. Dimension Check │
│ (header only, no decode) │
│ ├── PNG IHDR │
│ ├── JPEG frame header │
│ ├── WebP VP8/VP8L/VP8X │
│ ├── HEIC ispe, grid │
│ └── width × height ≤ │
│ max_image_pixels │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 8. Content Analysis │
│ (if enabled) │
│ ├── Executable signatures│
│ ├── Script injection │
│ └── Polyglot detection │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 9. ResourceMonitor END │
└─────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 10. Audit: VALIDATION_SUCCESS│
└──────────────────────────────┘
ZIP Validation (validate_zip_file)
UploadFile
│
▼
┌─────────────────────────────┐
│ 1. Audit: VALIDATION_START │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 2. Filename Validation │
│ (same as image) │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 3. ResourceMonitor START │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 4. File Size Check │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 5. Stream to SpooledTempFile│
│ (memory < 10MB → disk) │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 6. MIME + Signature Check │
│ (extension listed for │
│ the detected type) │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 7. Compression Validation │
│ ├── Ratio check │
│ ├── Uncompressed size │
│ ├── Entry count │
│ ├── Nested archive check │
│ └── Exact file layout │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 8. ZIP Content Inspection │
│ ├── Entry name checks │
│ │ ├── Null bytes │
│ │ ├── Path traversal │
│ │ ├── Absolute paths │
│ │ ├── Symlinks │
│ │ └── Top-level name │
│ │ checks (Unicode, │
│ │ Windows names, │
│ │ blocked exts) │
│ ├── Dangerous entry types│
│ ├── Binary signatures │
│ ├── Nested archive magic │
│ ├── Script patterns │
│ ├── Gzip entry start │
│ ├── Unreadable entries │
│ └── Recursive inspection │
│ ├── Depth tracking │
│ ├── Hash tracking │
│ ├── Entry count cap │
│ └── Each inner ZIP: │
│ same checks as │
│ top level (steps │
│ 7 and 8), if │
│ nested allowed │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 9. ResourceMonitor END │
└─────────────┬───────────────┘
│
▼
┌─────────────────────────────┐
│ 10. Audit: VALIDATION_SUCCESS│
└──────────────────────────────┘
Activity File Validation (validate_activity_file)
UploadFile (.gpx, .tcx, .fit)
│
▼
┌──────────────────────────────┐
│ 1. Audit: VALIDATION_START │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 2. Filename Validation │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 3. ResourceMonitor START │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 4. File Size Check │
│ (max_activity_file_size) │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 5. MIME + Signature Check │
│ ├── FIT: binary signature │
│ │ (.FIT at bytes 8-11) │
│ ├── GPX/TCX: XML header │
│ │ (<?xml) │
│ └── Extension listed for │
│ the detected type │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 6. Structure Validation │
│ ├── GPX/TCX: XML security │
│ │ ├── defusedxml parse │
│ │ ├── DTD forbidden │
│ │ └── Entity expansion │
│ │ block │
│ └── FIT: container check │
│ ├── Header + CRCs │
│ └── Sections cover │
│ the file exactly │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 7. Content Analysis │
│ (if enabled) │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 8. ResourceMonitor END │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 9. Audit: VALIDATION_SUCCESS │
└──────────────────────────────┘
Gzip Validation (validate_gzip_file)
UploadFile (.gz)
│
▼
┌──────────────────────────────┐
│ 1. Audit: VALIDATION_START │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 2. Filename Validation │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 3. ResourceMonitor START │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 4. File Size Check │
│ (max_gzip_size) │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 5. MIME + Signature Check │
│ (gzip magic: \x1f\x8b) │
│ (extension listed for │
│ the detected type) │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 6. Gzip Content Inspection │
│ ├── Chunked decompression │
│ ├── Progressive ratio chk │
│ └── Progressive size chk │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 7. ResourceMonitor END │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 8. Audit: VALIDATION_SUCCESS │
└──────────────────────────────┘
PDF Validation (validate_pdf_file)
UploadFile (.pdf)
│
▼
┌──────────────────────────────┐
│ 1. Audit: VALIDATION_START │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 2. Filename Validation │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 3. ResourceMonitor START │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 4. File Size Check │
│ (max_pdf_size) │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 5. MIME + Signature Check │
│ (%PDF- at offset 0) │
│ (extension listed for │
│ the detected type) │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 6. PDF Inspection │
│ ├── Header version and │
│ │ startxref + %%EOF end │
│ ├── Tokenize whole file │
│ │ ├── Blocked names │
│ │ │ (#xx decoded) │
│ │ └── /Encrypt │
│ ├── Cross-reference chain │
│ │ (table, stream, /Prev,│
│ │ /XRefStm): every entry│
│ │ at a parsed object │
│ ├── Object streams parsed │
│ │ and scanned │
│ ├── Stream /Length ends │
│ │ at endstream │
│ └── Flate output within │
│ size and ratio limits │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 7. Content Analysis │
│ (if enabled) │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 8. ResourceMonitor END │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ 9. Audit: VALIDATION_SUCCESS │
└──────────────────────────────┘
Data Flow: Where File Content Is Read
| Stage | What is read | Buffer size |
|---|---|---|
| Stream to temp (size check) | Chunked reads (chunk_size, 64 KB default) to SpooledTemporaryFile, counted against the type's size limit |
In-memory up to max_memory_buffer_size (10 MB), then disk |
| MIME detection | First 8 KB via python-magic |
8 KB |
| Signature check | First 4-12 bytes depending on format | Minimal |
| Compression validation | ZIP central directory, local headers and data descriptors | Metadata; entries with a data descriptor are inflated in chunk_size chunks |
| ZIP content inspection | Per-entry read for binary/text scan | First 512 bytes per entry |
| Nested ZIP inspection | Each inner ZIP streamed to a spooled temp file | chunk_size at a time; one spooled file (up to max_memory_buffer_size) per nesting level |
| Gzip inspection | Chunked streaming decompression | chunk_size at a time |
| XML validation | Chunked feed to defusedxml's parser, no element tree built |
chunk_size at a time |
| FIT validation | Chunked CRC-16 over each section | chunk_size at a time |
| PDF inspection | The whole file, tokenized outside stream data; Flate streams inflated, object and xref streams kept decoded to parse them | Whole file (bounded by max_pdf_size); inflation chunk_size at a time, kept output bounded by max_pdf_uncompressed_size |
| Image dimension check | PNG and WebP: first 30 bytes. JPEG: the marker and length of each segment up to the frame header, skipping segment payloads by seek. HEIC: the header of each top-level box, skipping payloads by seek, then the whole meta box and the grid descriptor |
Minimal for PNG, JPEG and WebP; a run of 0xFF fill bytes is read chunk_size at a time. HEIC: the meta box (bounded by max_image_size, a few KB in camera files) |
| Content analysis | Byte scan of image, activity and PDF uploads (not ZIP) | Bounded by content_scan_max_size (50 MB); about twice the sample while matching |
Exception Hierarchy
Exception
├── FileSecurityConfigurationError
└── FileSecurityError
├── FileValidationError
│ ├── FilenameSecurityError
│ │ ├── UnicodeSecurityError
│ │ ├── ExtensionSecurityError
│ │ └── WindowsReservedNameError
│ ├── FileSizeError
│ ├── MimeTypeError
│ ├── FileSignatureError
│ ├── ImageDimensionError
│ ├── CompressionSecurityError
│ │ ├── ZipBombError
│ │ └── ZipContentError
│ ├── XmlSecurityError
│ ├── PdfSecurityError
│ └── ContentThreatError
└── FileProcessingError
└── ResourceLimitError
All FileSecurityError subclasses carry an error_code from
ErrorCode for machine-readable classification.
FileSecurityConfigurationError is separate and exposes its
errors list of ConfigValidationIssue records instead. When no code is passed, the default is:
| Exception | Default ErrorCode |
|---|---|
FileSecurityError, FileProcessingError |
PROCESSING_ERROR |
FileValidationError |
VALIDATION_FAILED |
FilenameSecurityError |
FILENAME_INVALID |
UnicodeSecurityError |
UNICODE_DANGEROUS_CHARS |
ExtensionSecurityError |
EXTENSION_BLOCKED |
WindowsReservedNameError |
WINDOWS_RESERVED_NAME |
FileSizeError |
FILE_TOO_LARGE |
MimeTypeError |
MIME_TYPE_INVALID (uploads whose detected MIME is not allowed are reported as MIME_TYPE_MISMATCH, and an extension not listed for the detected MIME as EXTENSION_MIME_MISMATCH) |
FileSignatureError |
FILE_SIGNATURE_MISMATCH (a missing or invalid FIT structure is reported as FIT_INVALID_STRUCTURE, and a missing or invalid image dimension header as IMAGE_INVALID_STRUCTURE) |
ImageDimensionError |
IMAGE_DIMENSIONS_EXCEEDED |
CompressionSecurityError |
ZIP_INVALID_STRUCTURE |
ZipBombError |
ZIP_BOMB_DETECTED |
ZipContentError |
ZIP_CONTENT_THREAT (the ZIP inspector passes the code of the first threat found) |
XmlSecurityError |
XML_MALFORMED |
PdfSecurityError |
PDF_INVALID_STRUCTURE (an encrypted PDF is reported as PDF_ENCRYPTED, a blocked name or external stream data as PDF_ACTIVE_CONTENT, and Flate output over the limits as PDF_STREAM_BOMB) |
ContentThreatError |
CONTENT_THREAT_DETECTED |
ResourceLimitError |
RESOURCE_LIMIT_EXCEEDED |
FileProcessingError covers internal errors and its subclass
ResourceLimitError only; validation failures, including XML,
PDF and content threats, are FileValidationError subclasses.
Audit Event Flow
FileValidator.validate_*() (_validation_run)
│
├── bind_correlation_id() # reuses the host's ID, if set
├── audit.start() # VALIDATION_START
│
├── [validation pipeline]
│ │
│ ├── ZipContentInspector ──► audit.threat()
│ ├── GzipContentInspector ─► audit.threat()
│ ├── PdfContentInspector ──► audit.threat()
│ ├── CompressionSecurityValidator ► audit.threat()
│ └── ContentSecurityInspector ► audit.threat()
│
├── On success: audit.success() # VALIDATION_SUCCESS
├── On failure: audit.failure() # VALIDATION_FAILURE
├── On resource limit: audit.resource_limit() # RESOURCE_LIMIT
│
└── restore the previous correlation ID
start, success, failure, threat and resource_limit are methods of
SecurityAuditLogger; each builds an AuditEvent and passes it
to log_event. Audit records carry the correlation ID as
audit_correlation_id in extra, together with the
audit_event_type, audit_filename, audit_result,
audit_details, audit_duration_ms and audit_source_ip
fields. Other log records from validators and inspectors add
correlation_id to extra through log_extra() where they call
it, enabling request-level tracing across those sub-components.
log_event escapes control, format, surrogate, private-use,
unassigned and line or paragraph separator characters in the
correlation ID, filename, result, details and source IP fields
with repr()-style escapes, and caps each field at 1,024
characters, marking a cut with ...[truncated]. log_extra()
adds the escaped correlation ID.