Importing rows from CSV file stored on SFTP

Using SFTP agent to download file from SFTP and then parse it to get a list of items inside the CSV

This scenario pulls CSV files from an SFTP folder, turns each row into data you can use elsewhere, and remembers which files were already processed so you never import the same file twice.

What you provide

  • path (required): the SFTP folder to scan.
  • source (required): a short code to track which system the files come from (used for de-duplication).
  • Optional: filePrefix to only pick up files that start with a certain string.
  • Optional CSV settings if your files are not standard: csvDelimiter (default ;), csvEncoding (default LATIN-1), and csvUseHeader (default true).

How it runs (in order)

  1. List files in the SFTP folder.
  2. Keep only files that match the optional filePrefix.
  3. If nothing matches, return empty results right away.
  4. For each remaining file:
    • Check the ImportedFile log to see if this exact file (by name, modified time, and source) was processed before.
    • If already processed, skip it and note the reason.
    • If new, download the file, parse the CSV using your delimiter/encoding/header settings, then record it in ImportedFile so repeats are skipped next time.
  5. Return both per-file details and an all-records list.

Corner cases we handle

  • Duplicate protection: Files already marked in ImportedFile are skipped; updates with a different modified time are treated as new and get processed.
  • Prefix filtering: Prevents picking up unrelated files when multiple feeds share a folder.
  • No files: Returns zero counts gracefully instead of failing.
  • CSV quirks: Custom delimiter, encoding, and header handling cover non-standard exports.
  • Counts you can trust: Totals show how many files were processed vs. skipped and the total number of rows parsed.

What you get back

  • files[]: each file with whether it was processed or skipped and any parsed rows.
  • allRecords[]: every row from all processed files in one flat list.
  • processedCount / skippedCount: how many files were new vs. already handled.
  • totalRecordCount: total rows across processed files.