{
  "workflow": {
    "id": 5730,
    "name": "Deduplicate data records using JavaScript array methods",
    "views": 243,
    "recentViews": 0,
    "totalViews": 243,
    "createdAt": "2025-07-07T03:48:43.299Z",
    "description": " ## How It Works – Data Deduplication in n8n\n\nThis tutorial demonstrates how to remove duplicate records from a dataset using JavaScript logic inside n8n's Code nodes. It simulates real-world data cleaning by generating sample user data with intentional duplicates (based on email addresses) and walks you through the process of deduplication step-by-step.\n\n**The process includes**:\n\nCreating Sample Data with duplicates.\nFiltering Out Duplicates using filter() and findIndex() based on email.\nDisplaying Cleaned Results with simple statistics for before-and-after comparison.\nThis is ideal for scenarios like CRM imports, ETL processes, and general data hygiene.\n\n## ⚙️ Set-Up Steps\n\n🔹 Step 1: Manual Trigger\nNode: When clicking 'Test workflow'\nPurpose: Initiates the workflow manually for testing.\n\n🔹 Step 2: Generate Sample Data\nNode: Create Sample Data (Code node)\nWhat it does:\n\nCreates 6 users, including 2 intentional duplicates (by email).\nOutputs data as usersJson with metadata (totalCount, message).\nMimics real-world messy datasets.\n🔹 Step 3: Deduplicate the Data\nNode: Deduplicate Users (Code node)\nWhat it does:\n\nParses usersJson.\nUses .filter() + .findIndex() to keep only the first instance of each email.\nLogs total, unique, and removed counts.\nOutputs clean user list as separate items.\n🔹 Step 4: Display Results\nNode: Display Results (Code node)\nWhat it does:\n\n**Outputs structured summary**:\nUnique users\nStatus\nTimestamp\nPrepares results for review or downstream use.\n📈 Sample Output\n\nOriginal count: 6 users\nDeduplicated count: 4 users\nDuplicates removed: 2 users\n🎯 Learning Objectives\n\n**You'll learn how to**:\n\nUse .filter() and .findIndex() in n8n Code nodes\nClean JSON data within workflows\nCreate simple, effective deduplication pipelines\nOutput structured summaries for reporting or integration\n🧠**Best Practices**\n\nValidate input format (e.g., JSON schema)\nHandle null or missing fields gracefully\nUse logging for visibility\nAdd error handling for production use\nUse pagination/chunking for large datasets",
    "workflow": {
      "id": "hQXDUS1zEZFI37kU",
      "meta": {
        "instanceId": "2000c64071c20843606b95c63795bb0797c41036047055a6586498e855b96efc"
      },
      "name": "Javascript Masterclass2",
      "tags": [],
      "nodes": [
        {
          "id": "81dd117a-81f3-4dff-a7c7-664bc161987e",
          "name": "When clicking 'Test workflow'",
          "type": "n8n-nodes-base.manualTrigger",
          "position": [
            -420,
            80
          ],
          "parameters": {},
          "typeVersion": 1
        },
        {
          "id": "6f507b8c-8a8d-4c89-acdf-f0f4eda9fa94",
          "name": "Sticky Note",
          "type": "n8n-nodes-base.stickyNote",
          "position": [
            -660,
            -260
          ],
          "parameters": {
            "color": 7,
            "width": 400,
            "height": 580,
            "content": "# 🧱 Data Deduplication Tutorial\n\n**Author:** David Olusola\n\n## 🚀 n8n Coaching\nUnlock the full potential of n8n with personalized, one-on-one coaching. Whether you're just getting started or need help solving a specific challenge, I'll guide you step-by-step so you can build confidently and efficiently.\n\n👉 [Book Coaching Session](mailto:david@daexai.com?subject=n8n%20Coaching%20Request)\n\n## 🔧 n8n Consulting\nHave a complex automation project, unique integration needs, or want a custom workflow built from scratch? Let's collaborate to design and implement a robust solution tailored to your business.\n\n📩 [Inquire About Consulting Services](mailto:david@daexai.com?subject=n8n%20Consultation%20Request)"
          },
          "typeVersion": 1
        },
        {
          "id": "eadab384-f0d6-48b1-b4dd-e0eb106228ac",
          "name": "Sticky Note1",
          "type": "n8n-nodes-base.stickyNote",
          "position": [
            -240,
            220
          ],
          "parameters": {
            "color": 7,
            "width": 300,
            "height": 360,
            "content": "## 📊 Sample Data\n\nThis node creates sample user data with intentional duplicates based on email addresses.\n\n**What it does:**\n- Creates an array of users\n- Includes duplicate entries (same email)\n- Simulates real-world messy data\n\n**Next Step:** Pass this data to the deduplication code node"
          },
          "typeVersion": 1
        },
        {
          "id": "745ba692-ffa8-4ac6-851f-fb9dde8392f2",
          "name": "Sticky Note2",
          "type": "n8n-nodes-base.stickyNote",
          "position": [
            100,
            220
          ],
          "parameters": {
            "color": 7,
            "width": 300,
            "height": 380,
            "content": "## 🧹 Deduplication Logic\n\n**Key Concepts:**\n- `filter()` - Creates new array with filtered elements\n- `findIndex()` - Returns index of first matching element\n- `index === self.findIndex()` - Keeps only first occurrence\n\n**Algorithm:**\n1. Parse JSON data\n2. Filter array keeping first occurrence of each email\n3. Transform back to n8n format\n4. Return deduplicated results"
          },
          "typeVersion": 1
        },
        {
          "id": "8bc4bd06-b031-4909-8210-5ec1f11366ff",
          "name": "Sticky Note3",
          "type": "n8n-nodes-base.stickyNote",
          "position": [
            460,
            220
          ],
          "parameters": {
            "color": 7,
            "width": 300,
            "height": 380,
            "content": "## 📈 Results\n\n**What you'll see:**\n- Original count: 6 items\n- Deduplicated count: 4 items\n- Duplicates removed: 2 items\n\n**Use Cases:**\n- Clean data before CRM import\n- Prevent duplicate database entries\n- Data quality improvement\n- ETL pipeline cleanup"
          },
          "typeVersion": 1
        },
        {
          "id": "75a624be-73e9-4b35-ae58-33bc45a0208c",
          "name": "Create Sample Data",
          "type": "n8n-nodes-base.code",
          "position": [
            -200,
            80
          ],
          "parameters": {
            "jsCode": "// Create sample data with intentional duplicates\nconst usersWithDuplicates = [\n  { id: 1, name: \"John Doe\", email: \"user@example.com\", department: \"Engineering\" },\n  { id: 2, name: \"Jane Smith\", email: \"user@example.com\", department: \"Marketing\" },\n  { id: 3, name: \"John Doe\", email: \"user@example.com\", department: \"Engineering\" }, // Duplicate\n  { id: 4, name: \"Bob Johnson\", email: \"user@example.com\", department: \"Sales\" },\n  { id: 5, name: \"Alice Brown\", email: \"user@example.com\", department: \"HR\" },\n  { id: 6, name: \"Jane Smith Updated\", email: \"user@example.com\", department: \"Marketing\" } // Duplicate\n];\n\n// Return the sample data as a single item with usersJson property\nreturn [{\n  json: {\n    usersJson: JSON.stringify(usersWithDuplicates),\n    totalCount: usersWithDuplicates.length,\n    message: \"Sample data with duplicates created\"\n  }\n}];"
          },
          "typeVersion": 2
        },
        {
          "id": "909634f2-52ad-4f98-aad0-7c3d8a8b8e43",
          "name": "Deduplicate Users",
          "type": "n8n-nodes-base.code",
          "position": [
            140,
            80
          ],
          "parameters": {
            "jsCode": "// 🧱 Data Deduplication Code Node\n// Use Case: Clean up duplicates before inserting into DB or CRM\n\n// Parse the users JSON from the previous node\nconst users = JSON.parse(items[0].json.usersJson);\n\n// Deduplicate users based on email address\n// This keeps the first occurrence of each unique email\nconst uniqueUsers = users.filter(\n  (user, index, self) => \n    index === self.findIndex(u => u.email === user.email)\n);\n\n// Log the deduplication results\nconsole.log(`Original count: ${users.length}`);\nconsole.log(`Deduplicated count: ${uniqueUsers.length}`);\nconsole.log(`Duplicates removed: ${users.length - uniqueUsers.length}`);\n\n// Return the deduplicated users in n8n format\n// Each user becomes a separate item in the workflow\nreturn uniqueUsers.map(user => ({ json: user }));"
          },
          "typeVersion": 2
        },
        {
          "id": "59855549-51e4-42df-b586-730ff1e59e67",
          "name": "Display Results",
          "type": "n8n-nodes-base.code",
          "position": [
            500,
            80
          ],
          "parameters": {
            "jsCode": "// Display the final results with statistics\nconst currentItems = items;\nconst uniqueCount = currentItems.length;\n\n// Create a summary of the deduplication process\nconst summary = {\n  deduplicated_users: currentItems.map(item => item.json),\n  statistics: {\n    unique_users_count: uniqueCount,\n    process_completed: true,\n    timestamp: new Date().toISOString()\n  },\n  message: `Successfully deduplicated data - ${uniqueCount} unique users remaining`\n};\n\nreturn [{ json: summary }];"
          },
          "typeVersion": 2
        },
        {
          "id": "2a5c40fc-ff56-4e54-b06e-4c56e2c4da0e",
          "name": "Sticky Note4",
          "type": "n8n-nodes-base.stickyNote",
          "position": [
            -140,
            -240
          ],
          "parameters": {
            "color": 7,
            "width": 430,
            "height": 330,
            "content": "## 🎯 Learning Objectives\n\n**After this tutorial you'll understand:**\n\n✅ How to use JavaScript array methods in n8n\n✅ Data deduplication techniques\n✅ Working with JSON data in Code nodes\n✅ Transforming data between n8n items\n✅ Practical use cases for data cleaning\n\n**Best Practices:**\n- Always validate input data\n- Consider performance for large datasets\n- Test with various duplicate scenarios\n- Add error handling for production use"
          },
          "typeVersion": 1
        }
      ],
      "active": false,
      "pinData": {},
      "settings": {
        "executionOrder": "v1"
      },
      "versionId": "03170dda-ff22-468b-be1c-e32424a27c66",
      "connections": {
        "Deduplicate Users": {
          "main": [
            [
              {
                "node": "Display Results",
                "type": "main",
                "index": 0
              }
            ]
          ]
        },
        "Create Sample Data": {
          "main": [
            [
              {
                "node": "Deduplicate Users",
                "type": "main",
                "index": 0
              }
            ]
          ]
        },
        "When clicking 'Test workflow'": {
          "main": [
            [
              {
                "node": "Create Sample Data",
                "type": "main",
                "index": 0
              }
            ]
          ]
        }
      }
    },
    "lastUpdatedBy": 29,
    "workflowInfo": {
      "nodeCount": 9,
      "nodeTypes": {
        "n8n-nodes-base.code": {
          "count": 3
        },
        "n8n-nodes-base.stickyNote": {
          "count": 5
        },
        "n8n-nodes-base.manualTrigger": {
          "count": 1
        }
      }
    },
    "status": "published",
    "user": {
      "name": "David Olusola",
      "username": "dae221",
      "bio": "I design enterprise-grade automation systems that eliminate bottlenecks, cut manual work by 70%+, and unlock millions in lost revenue opportunities. My clients — typically achieve 5–10x ROI within the first 90 days. Currently partnering with select high-growth companies ready to scale faster with AI automation.  david@daexai.com",
      "verified": true,
      "links": [
        ""
      ],
      "avatar": "https://gravatar.com/avatar/2abbc4c9e828c54a8d8838beda1382e8e4a66684ef9fa8451dbeb3e72b0c9f9c?r=pg&d=retro&size=200"
    },
    "nodes": [
      {
        "id": 565,
        "icon": "fa:sticky-note",
        "name": "n8n-nodes-base.stickyNote",
        "codex": {
          "data": {
            "alias": [
              "Comments",
              "Notes",
              "Sticky"
            ],
            "categories": [
              "Core Nodes"
            ],
            "nodeVersion": "1.0",
            "codexVersion": "1.0",
            "subcategories": {
              "Core Nodes": [
                "Helpers"
              ]
            }
          }
        },
        "group": "[\"input\"]",
        "defaults": {
          "name": "Sticky Note",
          "color": "#FFD233"
        },
        "iconData": {
          "icon": "sticky-note",
          "type": "icon"
        },
        "displayName": "Sticky Note",
        "typeVersion": 1,
        "nodeCategories": [
          {
            "id": 9,
            "name": "Core Nodes"
          }
        ]
      },
      {
        "id": 834,
        "icon": "file:code.svg",
        "name": "n8n-nodes-base.code",
        "codex": {
          "data": {
            "alias": [
              "cpde",
              "Javascript",
              "JS",
              "Python",
              "Script",
              "Custom Code",
              "Function"
            ],
            "details": "The Code node allows you to execute JavaScript in your workflow.",
            "resources": {
              "primaryDocumentation": [
                {
                  "url": "https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code/"
                }
              ]
            },
            "categories": [
              "Development",
              "Core Nodes"
            ],
            "nodeVersion": "1.0",
            "codexVersion": "1.0",
            "subcategories": {
              "Core Nodes": [
                "Helpers",
                "Data Transformation"
              ]
            }
          }
        },
        "group": "[\"transform\"]",
        "defaults": {
          "name": "Code"
        },
        "iconData": {
          "type": "file",
          "fileBuffer": "data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNTEyIiBoZWlnaHQ9IjUxMiIgdmlld0JveD0iMCAwIDUxMiA1MTIiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxnIGNsaXAtcGF0aD0idXJsKCNjbGlwMF8xMTcxXzQ0MSkiPgo8cGF0aCBkPSJNMTcwLjI4MyA0OEgxOTYuNUMyMDMuMTI3IDQ4IDIwOC41IDQyLjYyNzQgMjA4LjUgMzZWMTJDMjA4LjUgNS4zNzI1OCAyMDMuMTI3IDAgMTk2LjUgMEgxNzAuMjgzQzEyNi4xIDAgOTAuMjgzIDM1LjgxNzIgOTAuMjgzIDgwVjE3NkM5MC4yODMgMjA2LjkyOCA2NS4yMTA5IDIzMiAzNC4yODMgMjMySDIzQzE2LjM3MjYgMjMyIDExIDIzNy4zNzIgMTEgMjQ0VjI2OEMxMSAyNzQuNjI3IDE2LjM3MjQgMjgwIDIyLjk5OTYgMjgwTDM0LjI4MyAyODBDNjUuMjEwOSAyODAgOTAuMjgzIDMwNS4wNzIgOTAuMjgzIDMzNlY0NDBDOTAuMjgzIDQ3OS43NjQgMTIyLjUxOCA1MTIgMTYyLjI4MyA1MTJIMTk2LjVDMjAzLjEyNyA1MTIgMjA4LjUgNTA2LjYyNyAyMDguNSA1MDBWNDc2QzIwOC41IDQ2OS4zNzMgMjAzLjEyNyA0NjQgMTk2LjUgNDY0SDE2Mi4yODNDMTQ5LjAyOCA0NjQgMTM4LjI4MyA0NTMuMjU1IDEzOC4yODMgNDQwVjMzNkMxMzguMjgzIDMwOS4wMjIgMTI4LjAxMSAyODQuNDQzIDExMS4xNjQgMjY1Ljk2MUMxMDYuMTA5IDI2MC40MTYgMTA2LjEwOSAyNTEuNTg0IDExMS4xNjQgMjQ2LjAzOUMxMjguMDExIDIyNy41NTcgMTM4LjI4MyAyMDIuOTc4IDEzOC4yODMgMTc2VjgwQzEzOC4yODMgNjIuMzI2OSAxNTIuNjEgNDggMTcwLjI4MyA0OFoiIGZpbGw9IiNGRjk5MjIiLz4KPHBhdGggZD0iTTMwNSAzNkMzMDUgNDIuNjI3NCAzMTAuMzczIDQ4IDMxNyA0OEgzNDIuOTc5QzM2MC42NTIgNDggMzc0Ljk3OCA2Mi4zMjY5IDM3NC45NzggODBWMTc2QzM3NC45NzggMjAyLjk3OCAzODUuMjUxIDIyNy41NTcgNDAyLjA5OCAyNDYuMDM5QzQwNy4xNTMgMjUxLjU4NCA0MDcuMTUzIDI2MC40MTYgNDAyLjA5OCAyNjUuOTYxQzM4NS4yNTEgMjg0LjQ0MyAzNzQuOTc4IDMwOS4wMjIgMzc0Ljk3OCAzMzZWNDMyQzM3NC45NzggNDQ5LjY3MyAzNjAuNjUyIDQ2NCAzNDIuOTc5IDQ2NEgzMTdDMzEwLjM3MyA0NjQgMzA1IDQ2OS4zNzMgMzA1IDQ3NlY1MDBDMzA1IDUwNi42MjcgMzEwLjM3MyA1MTIgMzE3IDUxMkgzNDIuOTc5QzM4Ny4xNjEgNTEyIDQyMi45NzggNDc2LjE4MyA0MjIuOTc4IDQzMlYzMzZDNDIyLjk3OCAzMDUuMDcyIDQ0OC4wNTEgMjgwIDQ3OC45NzkgMjgwSDQ5MEM0OTYuNjI3IDI4MCA1MDIgMjc0LjYyOCA1MDIgMjY4VjI0NEM1MDIgMjM3LjM3MyA0OTYuNjI4IDIzMiA0OTAgMjMyTDQ3OC45NzkgMjMyQzQ0OC4wNTEgMjMyIDQyMi45NzggMjA2LjkyOCA0MjIuOTc4IDE3NlY4MEM0MjIuOTc4IDM1LjgxNzIgMzg3LjE2MSAwIDM0Mi45NzkgMEgzMTdDMzEwLjM3MyAwIDMwNSA1LjM3MjU4IDMwNSAxMlYzNloiIGZpbGw9IiNGRjk5MjIiLz4KPC9nPgo8ZGVmcz4KPGNsaXBQYXRoIGlkPSJjbGlwMF8xMTcxXzQ0MSI+CjxyZWN0IHdpZHRoPSI1MTIiIGhlaWdodD0iNTEyIiBmaWxsPSJ3aGl0ZSIvPgo8L2NsaXBQYXRoPgo8L2RlZnM+Cjwvc3ZnPgo="
        },
        "displayName": "Code",
        "typeVersion": 2,
        "nodeCategories": [
          {
            "id": 5,
            "name": "Development"
          },
          {
            "id": 9,
            "name": "Core Nodes"
          }
        ]
      },
      {
        "id": 838,
        "icon": "fa:mouse-pointer",
        "name": "n8n-nodes-base.manualTrigger",
        "codex": {
          "data": {
            "resources": {
              "generic": [],
              "primaryDocumentation": [
                {
                  "url": "https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.manualworkflowtrigger/"
                }
              ]
            },
            "categories": [
              "Core Nodes"
            ],
            "nodeVersion": "1.0",
            "codexVersion": "1.0"
          }
        },
        "group": "[\"trigger\"]",
        "defaults": {
          "name": "When clicking ‘Execute workflow’",
          "color": "#909298"
        },
        "iconData": {
          "icon": "mouse-pointer",
          "type": "icon"
        },
        "displayName": "Manual Trigger",
        "typeVersion": 1,
        "nodeCategories": [
          {
            "id": 9,
            "name": "Core Nodes"
          }
        ]
      }
    ],
    "categories": [
      {
        "id": 5,
        "name": "Engineering"
      }
    ],
    "image": []
  }
}