CelinQ Insights · No. 73

Detecting Duplicate Applications and Components in Sparx EA

Nobody sets out to model the same application twice. It happens anyway, quietly, across teams and years, until the portfolio is telling two slightly different stories about the same system.

Ask any architect who has inherited a mature Enterprise Architect repository whether they trust the application portfolio it describes, and the honest answer is usually a qualified one. They trust most of it. They are less sure about a particular corner where two components, modelled years apart by different teams, seem to describe the same underlying system under different names and with slightly different sets of relationships. Nobody remembers exactly why both exist. Nobody has had the time to sit down, compare them property by property, and decide whether they really are the same thing wearing two labels, or two genuinely distinct systems that happen to share a similar name by coincidence. So both stay in the model, both get counted separately in whatever portfolio statistics get reported upward, and the repository quietly overstates the size and complexity of the landscape it is supposed to describe accurately.

This is not a rare failure mode; it is closer to an inevitability. A repository maintained by more than one person, across more than one initiative, over more than a couple of years, will accumulate duplication the way any large, collaboratively maintained dataset does. A team modelling an integration project creates a component to represent a system they are connecting to, because from where they sit that system did not previously exist in the repository — and they are often right, because nobody had modelled it yet. A separate team, a year later, working on a different initiative, needs to represent the same underlying system and, not knowing the earlier component exists, creates their own. Neither team did anything wrong. The repository simply grew in the way real organisations grow: in parallel, by people who cannot be expected to have perfect visibility into everything everyone else has modelled before them.

What duplication actually costs

The cost of undetected duplication is not abstract. An application portfolio that counts the same system twice overstates technology risk assessments, understates the real degree of consolidation opportunity, and produces dependency analyses that are subtly wrong in ways that are hard to spot precisely because each half of the duplicate looks individually correct. A cost model built on a duplicated portfolio double-counts licensing or hosting spend for a system that only exists once in production. A rationalisation programme that is meant to identify genuine consolidation candidates gets noise injected into its input data from duplicates that were never candidates for consolidation because they were never actually two things. And every one of these downstream distortions compounds the longer the duplication goes unnoticed, because reports get built on top of reports, and by the time someone finally notices the discrepancy, untangling which numbers were affected and by how much is its own investigation.

The trust cost is arguably worse than any single number being wrong. Once a governance board discovers, halfway through a presentation, that two line items on a portfolio dashboard were actually one system counted twice, the natural reaction is not to correct that single figure and move on. It is to start quietly doubting every other figure the dashboard has ever produced, and rebuilding that confidence takes far longer than the original error took to introduce. A repository is only as useful as the willingness of the people making decisions from it to trust what it says, and duplication is one of the more common, and more preventable, ways that trust erodes.

A worked example: the same claims system, twice

Concreteness helps here more than abstraction does. Picture an insurance-sector repository where a claims-processing team modelled ClaimsProcessor three years ago as part of an early digitisation project, complete with relationships to a policy-lookup interface, a fraud-scoring service and a payout-authorisation workflow. A separate team, working on a later customer-portal initiative and drawing its own picture of the systems the portal depends on, modelled ClaimProcessingEngine a year and a half later — the same underlying application, described from the outside by people who had never seen the first team's package and had no reason to go looking for it. The second team's version carries a relationship to a document-storage service that the first team's version never captured, and a tagged value recording a hosting region that the first version also lacks, because that information simply was not relevant to the earlier project's concerns.

Run against this repository, exact-name matching finds nothing, because the two names share no substring worth flagging. Similar-name scoring picks up a moderate signal from the shared "Claim" and "Process" fragments, but on its own that signal is weak enough to appear alongside several genuinely unrelated pairs that happen to share common architectural vocabulary. What tips this particular pair into a strong candidate is the combination: both connect to the same policy-lookup interface and the same fraud-scoring service, a relationship-overlap signal that is unlikely to occur by chance between two genuinely distinct systems, and both carry a tagged value recording the same underlying insurance product line. None of these signals alone would justify surfacing the pair with confidence. Combined, they produce a candidate score high enough to place the pair near the top of a review list — at which point a person familiar with the claims domain looks at both, recognises immediately that they describe the same system, and proceeds to the actual consolidation with the confidence that comes from having the evidence laid out rather than from a hunch.

The kinds of evidence that point at a duplicate

Before any detection can happen, it is worth being precise about what actually distinguishes a real duplicate from two genuinely different systems that merely resemble each other on the surface, because the naive version of this problem — flag anything with a similar name — produces far too many false positives to be useful. Several independent signals matter, and they matter more in combination than any one of them does alone. An exact-name match is the strongest and simplest signal, though it is rarer than you would expect, because most accidental duplication happens precisely because the second author did not know the first component's name to begin with. A similar-name match — different capitalisation, a missing or added word, an abbreviation versus a spelled-out form, "Payment Gateway" against "PaymentGW" — is a weaker but still meaningful signal, and it is the one that catches the largest share of real-world duplicates, because people naturally converge on similar but not identical names for the same real thing.

A shared external identifier is, where it exists, close to conclusive: if both elements carry the same value in a tagged value that records an external system ID, an ITSM CMDB reference, or a vendor product code, that is strong evidence they represent the same underlying system regardless of what their names happen to be. Overlapping relationships are a softer but still valuable signal — two components that both connect to the same order-management system, the same identity provider and the same payment settlement service, in a portfolio where that combination is otherwise rare, are more likely than chance to be the same system modelled twice. Similar tagged values more broadly — matching values for hosting environment, technology stack, business owner — add further weight without being conclusive on their own, because plenty of genuinely distinct systems legitimately share a hosting platform or a business owner.

Combining evidence into a candidate score

No single signal above is reliable enough, by itself, to justify treating two elements as duplicates. Two payment-related components with similar names could be the actual gateway and a genuinely separate reconciliation service that happens to have "payment" in both names. Two components sharing a business owner could simply be two of the dozen systems that owner is responsible for. The practical approach is to combine several weaker signals into a single candidate score for each pair of elements worth comparing, weighting exact matches — shared external identifiers, identical names — more heavily than softer signals like relationship overlap or tagged-value similarity, and surfacing pairs whose combined score crosses a threshold as candidates worth a person's attention, rather than either ignoring weak signals entirely or treating every partial match as a confirmed duplicate.

This scoring runs deterministically against the model, through the Analyze Selected Package entry point in the EA add-in — the same analysis entry point used for broader model-quality checks — scoped to a package or to the whole repository depending on what an architect is trying to clean up. It requires no external service and produces the same candidate list given the same model state every time, which matters for a task like this precisely because the output is going to inform decisions about merging real architectural entities, and a non-reproducible detector would be a poor foundation for that kind of decision.

Where optional AI similarity adds a further signal

Deterministic scoring based on names, identifiers, relationships and tagged values catches a large share of real duplication, but it has a known blind spot: two components that describe the same system in genuinely different vocabulary — "Customer Onboarding Service" and "New Client Intake" — will not score highly on name similarity even though a person reading both descriptions would immediately recognise them as the same thing. This is the kind of semantic gap that an optional AI-assisted similarity pass can help close, by comparing the descriptive text, notes and broader context of two candidate elements for meaning rather than surface string similarity, and folding that signal into the same candidate score alongside the deterministic evidence.

The pipeline this runs through follows the same shape as every other AI-assisted feature in the product: CelinQ retrieves the relevant model context for the candidate pair — names, notes, relationships, tagged values, nothing beyond what a workspace's policy permits — and an AI-assisted step proposes a similarity assessment, which is then combined with the deterministic score rather than replacing it, deterministically validated, and presented to a human as a ranked candidate list rather than an automatic merge. Nothing about this step writes to the model. It only ever changes where a candidate pair sits in a list an architect is going to review anyway, and it is entirely optional — the deterministic scoring alone produces a usable, reproducible candidate list without it, and a workspace can choose to run duplicate detection on names, identifiers and relationships only, with the AI similarity pass switched off, and lose nothing except the ability to catch duplicates whose only real signal is a difference in vocabulary.

A candidate list is not a verdict. The whole value of scoring duplicates deterministically, with an optional semantic pass on top, is that it turns "did anyone notice this yet" into a ranked list someone actually looks at — not that it decides, on your behalf, which two elements were always the same thing.

Review before merge, every time

Nothing produced by duplicate detection merges anything automatically, and this is not a cautious default that a workspace can turn off — there is no path in CelinQ from a candidate score to a merged model that does not pass through an architect looking at the specific pair and deciding. The reasoning is straightforward: a false positive in duplicate detection is not merely an annoyance, it is a proposal to destroy the distinctness of two things that were never actually the same, and the cost of getting that wrong is high enough that no confidence score, however high, is treated as sufficient justification on its own. A pair that scores highly on every signal — matching external identifier, identical relationships, overlapping tagged values — is still presented as a candidate for review, with the evidence that produced the score shown alongside it, rather than merged silently in the background because the score cleared some threshold.

When an architect does decide two candidates genuinely represent the same system, the actual consolidation runs through the same refactoring pipeline used for any deliberate model restructuring: a proposed change set showing exactly which relationships and tagged values survive under the merged identity, a preview of the full footprint before anything is written, and a single Fusion transaction once approved, so that the merge lands in the workspace's revision history as one coherent, attributable, reversible operation rather than a silent disappearance of one of the two elements. Duplicate detection finds the candidates. Refactoring, with a human decision at the centre of it, is what actually acts on them.

Building an audit trail out of the review itself

Every candidate pair that gets reviewed, whether the outcome is a merge or a decision that the two elements are genuinely distinct, is worth recording as a decision in its own right, not just acting on silently when the answer happens to be yes. A candidate pair dismissed as "these are different systems that happen to share a vendor" is useful information the next time duplicate detection runs and resurfaces the same pair — it tells the tool, and more importantly the next architect who looks at the list, that this particular near-match has already been considered and rejected, rather than making every subsequent scan re-raise a question that was already settled. This is part of why the review step matters beyond the immediate safety concern: it turns duplicate detection from a one-off cleanup exercise into a standing discipline that gets more useful, not more repetitive, the more often it runs against a portfolio that keeps growing.

Duplicates that are not really duplicates

It is worth dwelling on the false-positive case specifically, because a detector that is tuned too aggressively toward finding duplicates is arguably more damaging than one that misses a few genuine ones. Two components can legitimately share a name, a technology stack, a business owner and several relationships without being the same system — a large organisation running regional variants of the same platform, deliberately kept as separate deployments for compliance or data-residency reasons, will produce exactly the signal pattern that a naive detector treats as conclusive evidence of accidental duplication. This is precisely why the review step is not a rubber stamp on the way to an automatic merge: an architect who understands why the regional variants exist can dismiss the candidate pair in seconds, and that dismissal, recorded rather than discarded, keeps the same pair from being flagged as urgent every time the scan runs again. A scoring system that could not be corrected by a human, or that treated a high score as sufficient grounds to act, would eventually train a team to stop trusting its output altogether — which would defeat the entire purpose of running it in the first place.

AI stays optional and off by default

As with every AI-assisted capability in CelinQ, the similarity pass described above is off by default in every workspace, is enabled explicitly by an administrator through the Control Plane's AI settings, and can be scoped to specific capabilities rather than switched on wholesale. A workspace operating under sovereign mode has no path to external AI at all, and duplicate detection in that workspace runs on deterministic evidence only — names, identifiers, relationships, tagged values — which is enough, on its own, to surface the majority of accidental duplication in most portfolios, even without the semantic pass that catches vocabulary mismatches. A regulated public-sector or financial-services deployment can run duplicate detection as a routine part of portfolio governance, on real production data, without any model content ever leaving the organisation's own infrastructure, and without the review-before-merge discipline described above changing in any way.

The honest limits

Scoring is only as good as the evidence available to score against, and a repository that is thin on tagged values, sparse on relationships, and inconsistent about recording external identifiers will produce a shorter and less confident candidate list than a well-documented one — not because the detection logic is weaker in that repository, but because there is genuinely less evidence to work from. This is worth naming honestly rather than glossing over: duplicate detection cannot manufacture signal that was never captured in the model, and a portfolio's underlying documentation discipline sets a real ceiling on how much it can find. In practice this means duplicate detection tends to work best on exactly the portfolios that already have reasonably good modelling hygiene, which is a slightly uncomfortable irony, though not a surprising one — the same discipline that keeps duplication from accumulating in the first place is also what makes it easier to catch when it does.

There is also a genuine risk of false negatives that no amount of scoring tuning eliminates entirely: two systems that really are duplicates but were modelled with completely different names, no shared external identifier, and different relationship sets because the two teams that created them understood the same underlying system in different terms, will simply not surface as a strong candidate, deterministic or AI-assisted, because there is not enough overlapping evidence to distinguish them from two genuinely unrelated systems. Duplicate detection is a way of catching a large share of accidental duplication with reasonable confidence and no risk of silent, unreviewed consolidation. It is not, and should not be presented as, a guarantee that a portfolio contains no undetected duplicates at all. A periodic, deliberate review by someone who actually knows the domain remains valuable precisely because the tool's blind spots are real ones, not merely theoretical.

Finally, scoring thresholds are themselves a judgement call that has to be made once, deliberately, and then revisited as a workspace's experience with the tool grows. A threshold set too low buries genuine candidates under a long tail of coincidental near-matches, and a review list nobody has time to work through is barely better than no list at all. A threshold set too high misses the borderline cases that were, in practice, often the most valuable ones to catch, because a borderline duplicate is exactly the kind two people would disagree about without evidence in front of them. There is no universally correct setting, and any vendor claiming otherwise is glossing over a real trade-off; the honest position is that the threshold should be tuned against a specific portfolio's actual review capacity and revisited once an architect has seen how the first few review cycles actually played out, not treated as a value chosen once and never questioned again.