How to compare products fairly: what to keep in mind

A fair comparison starts with the job, not with the list of products. Write down what the software must do, decide which criteria are must-haves, and score every option on the same criteria, with the same plan, platform and data size. Weight the criteria by how much they matter, and treat the result as a shortlist to test, not a verdict. The video above shows the method in about 80 seconds; this page adds the detail and the limits.

Start with the job

Most unfair comparisons begin with a grid of logos. A better start is one sentence that names the job and the person it serves: "Store logins for my family and fill them in on a phone and a laptop," or "Keep meeting notes that I can search, on my own disk." The sentence settles what the software has to do before any product is named.

The job also rules products out. A team that needs shared folders can skip a tool built for one person's notes, however good those notes are. The password manager ranking and the note-taking apps ranking group products by the job they do, so start from the ranking for your job and read its criteria before its order. The guide how to choose a password manager applies the same steps to one category.

Separate must-haves from nice-to-haves

A must-have is a requirement that fails the job without it: the product runs on the devices you use, or it exports your data in a format you can open. A nice-to-have improves the experience without changing whether the job gets done, such as a theme or a sharing feature you might use one day.

Cap the list. Keep the must-haves to five at most. A scoring table with four or five criteria is enough for most choices, because a long list dilutes the ones that matter. Be wary of feature-count tables in particular. A grid with forty ticks invites a count of ticks, and a count treats every feature as equally important. A tick also says that a feature exists, not that it works well. Two products with the same number of ticks can differ on the one feature your job depends on.

Compare like with like

A fair comparison puts the same things side by side. Three things matter most.

• Plan tier. Compare the free plan with the free plan, and the paid plan with the paid plan that includes the features you need. The password manager overview on this site notes that Bitwarden, Proton Pass and KeePassXC are free for personal use with no time limit, and that 1Password has no free plan after its 14-day trial. A trial and a free plan are different things.
• Platform. Check that each product is officially supported on the devices you use. A product that runs on your system only through a third-party wrapper is a different case, and a fair comparison records it that way.
• Data volume. A tool that feels quick with two hundred notes may behave differently with twenty thousand. Test with a volume close to yours, or the comparison measures a different job.

Count the cost over the whole period

The headline price is rarely the whole bill. Compare totals over one period that you choose and use for every option. Example figures only: suppose a plan costs 5 a month and a sync add-on costs 3 a month. The headline is 5 a month, or 120 over two years (5 times 24). With the add-on the bill is 8 a month, or 192 over the same two years (8 times 24). The headline figure understates the cost by 72.

Count the costs that do not appear on the price page as well. These include setup time, any hardware the product needs, and the cost of leaving. The lock-in and exit costs lesson covers the last one. Always confirm the current price on the vendor's own pricing page, because prices change faster than a comparison can follow.

Check upkeep, data location and exit

Look for signs of upkeep on the vendor's own site: a changelog or release notes, the date of the latest update, and whether security fixes are announced. A product with no visible update for a long time may still work, but it deserves a closer look. The date on a listme.name entry is when that entry was last checked, which is not the same as the vendor's last release, so read the vendor's dates directly.

Then ask where the data lives and how it leaves. Note whether the data sits on the vendor's servers, on your device, or in a file you manage. Then check the export: which formats it produces, whether it includes attachments and history, and whether you can run it yourself without asking support. A product that cannot export what you put in it is not a fair option for a long-term need.

Finally, test with your own files. Export a small sample from your current tool, import it into a trial of the candidate, export it again and compare the two. The trial lesson gives the steps.

Weigh the criteria with a worked example

A weighted score turns several criteria into one total. Each criterion gets a weight that says how much it matters to the job, and each option gets a score on every criterion. The total for an option is the sum of weight times score across all criteria. This is the weighted sum model, and it only works when every score uses the same scale and the same direction, so that higher is better on every row.

The table below uses example figures, not measurements. Weights are points that add up to ten. Scores run from zero to ten, and for cost a higher score means a lower cost to you.

Criterion	Weight	Option A	Option B
Fits the job	4	9	7
Total cost	3	5	9
Export options	2	8	4
Support	1	6	8

Option A totals 73, from 4 times 9, plus 3 times 5, plus 2 times 8, plus 1 times 6. Option B totals 71, from 4 times 7, plus 3 times 9, plus 2 times 4, plus 1 times 8. Option A leads by two points. A gap that small is a reason to run a trial, not a decision. Change the weights and the result changes: if Fits the job weighs 2 and Total cost weighs 5, with the other weights unchanged, the totals become 65 for Option A and 75 for Option B. The weights are where your judgment enters, so write them down before you look at the scores.

How listme.name applies these rules

The methodology page states the rules that every listme.name ranking follows. Each product gets an editorial score from 0 to 100, written for that list against the criteria shown on it. Those scores are judgments, not benchmarks, user ratings or survey results. The same product can score differently on different lists, because the question changes. No product scores above 97, and no two products on a list share a score.

Lists are ordered by score, with one exception. Products connected to the publisher are marked with an asterisk and receive a capped lift and a placement floor, as described on the methodology page and on the disclosure page. At the time of writing, listme.name has no paid placements, no advertising and no affiliate links. Use a ranking as a shortlist, then open the head-to-head comparisons for the top products and the category index for every ranked list. Apply the criteria in this lesson to the shortlist yourself, because your job is not the same as the typical reader's.

Sources

• Weighted sum model, a reference for the method, including its requirement that every score is expressed in the same unit.