Remove Duplicates

Removes duplicate items — compared by the whole item or selected fields — keeping the first or last occurrence.

Finding it in the library

Search the builder's node library for Remove Duplicates (it lives under Processing). A single click opens the in-editor docs panel shown here — description, ports, and every property, without leaving the canvas. Double-click (or drag) to add it to the workflow.

Wired up in the builder

Remove Duplicates in a real, runnable flow — captured live from the Studio editor, exactly as it looks on your canvas. This is the same workflow used for the example input & output below.

How it’s configured

The node’s Configure panel as it opens in the builder when you select the step — every setting laid out with real values. Click any field to edit it.

Ports

Ports are the node’s contract with its neighbours. In the editor a port label renders bold when wired and italic when optional; ◈ ports accept attachment carriers rather than data wires.

How data flows through it

Remove Duplicates consumes the content of the incoming envelope — when it is fed directly by a trigger, the trigger’s wrapper is unwrapped at the node boundary so the node sees the actual data, not the metadata shell. Its output becomes the payload for the next node, while the envelope (trace ids, correlation, binary refs) rides along untouched. In the Runs view you always see the whole envelope for both sides of this node.

Expressions in the config

None of this node’s properties are string-typed, so {{ }} expressions don’t apply here — JSON- and code-typed fields are always taken literally.

Build it with AI

Every node in this reference is reachable through Flowdrome’s AI Copilot and the MCP tools — say what you want, and the graph surgery happens server-side. Node types resolve fuzzily, so the catalog label (Remove Duplicates) works as well as the exact type id (processing.dedupe).

In the Copilot panel (or any connected AI):

add a remove duplicates node after the trigger

As a step in a create_chain_workflow call:

{"type":"Remove Duplicates","config":{}}

curl -s -X POST http://localhost:48170/mcp -H "content-type: application/json" -d '{ "jsonrpc": "2.0", "id": "1", "method": "tools/call", "params": { "name": "add_node", "arguments": { "workflowId": "<id>", "type": "Remove Duplicates" } } }'

Example input & output

Captured from a real test run of the workflow above — this is what you see in the run data panel after pressing Test workflow.

Input — what the node received

Output — what the node produced

Property reference

Every setting, with its type and default — the same fields shown configured in the panel above.

Related nodes

The rest of the Processing group — the same folder you’d scan in the editor’s library.

• Validate JSON Schema Validates input with an inline or context-provided JSON Schema.
• Map Fields Builds a new JSON object by copying source paths to destination paths.
• Modify Set, rename, remove, or keep fields with {{ }} expressions, type coercion, and dot-paths.
• Edit Fields (Set) Set output fields from literals or JS {{ }} expressions, per item
• JSON To XML Converts a JSON value to an XML document with a configured root element.
• XML To JSON Parses an XML string into a JSON object (xml2js-style: attributes under $, repeated elements become arrays).
• Check Checks incoming data, including request headers, and routes to valid or invalid.
• Digest Accumulates the events that flow through it and releases them as one batch — when a count is reached (collect)

This page is generated from the node registry by gen-node-docs.mjs on every site build — ports, properties, defaults and visibility rules cannot drift from the code. The screenshots and example data are captured from a live Flowdrome by npm run shots:nodes and npm run gen:examples.