Database

Design Safe Data Import Pipelines for CSV Files

Process large CSV imports with validation, bounded batches, encoding checks, error reporting and restartable execution.

CSV imports are deceptively complex. Files may contain unexpected encodings, delimiters, line endings, malformed quotes, duplicate rows or values that violate business rules. A reliable importer treats the file as untrusted input and processes it in bounded steps.

Validate the file before processing

Enforce file size limits, allowed extensions and a controlled storage location. Detect or require an expected encoding and delimiter rather than assuming every file is identical. Do not trust the browser-provided MIME type or original filename.

Stream rows and bound memory

Use a streaming parser such as PHP's fgetcsv rather than loading the entire file into memory. Validate each row and write in batches sized for the database and hosting environment. Keep transaction size bounded so one malformed record does not require an enormous rollback.

Make imports restartable

Assign an import identifier and track progress, processed rows and errors. For large jobs, persist a checkpoint or staging state so a worker can resume safely. Use unique keys or idempotent upsert rules to avoid duplicating records after retries.

Report errors safely

Provide row numbers and clear validation messages without exposing database internals. Separate rejected rows from accepted rows and define whether partial imports are allowed. Keep the original file only as long as operationally necessary.

Checklist

  • Treat uploaded files and row values as untrusted.
  • Stream rows instead of loading the whole file.
  • Validate and commit in bounded batches.
  • Make retries and restarts idempotent.
  • Report rejected rows without leaking internal errors.
Share this page Help other developers discover this resource.