Maintenance: Deleting Products with Missing Data

darrin

— 3 min read

When products are synced into the excise tax system, some may arrive with incomplete information and get flagged with missingData=true in their details JSON. There may be some cases where we may need to remove these manually (as was the case when BigDVapor.net re-SKU'd all of their products - making these notifications obsolete).

The hq-delete-missing-data-products maintenance script finds these products and optionally removes them.

How it works

The script queries excise_tax_product_by_app_domain for a given app domain, parses the details JSON for each row, and identifies products where missingData is truthy. It uses Cassandra WRITETIME to determine when each product was created, which enables date-based filtering.

Products can be filtered by age (--older-than), a start date (--startDate), an end date (--endDate), or any combination of these.

Usage

node scripts/maintenance/hq-delete-missing-data-products [options] <appDomain ...>

Multiple app domains can be provided and will be processed sequentially.

Options

Flag Description
-x, --execute Actually delete the products (default is dry-run)
-o, --older-than <days> Only include products older than this many days
-s, --startDate <date> Only include products created on or after this date (YYYY-MM-DD)
-e, --endDate <date> Only include products created on or before this date (YYYY-MM-DD)

Modes

Flags Behavior
(none) Dry-run. Lists all matching products but does not delete anything.
--execute Deletes all matching products (concurrency of 5).

Step 1: Dry-run scan

Start with a dry-run to see what would be deleted:

node scripts/maintenance/hq-delete-missing-data-products acme.example.com

Example output:

[acme.example.com] Found 42 products with missingData.
  [DRY RUN]  sku=WIDGET-001                     created=2024-06-15  Widget Deluxe
  [DRY RUN]  sku=GIZMO-002                      created=2024-08-22  Gizmo Pro
  [DRY RUN]  sku=GADGET-003                     created=2025-01-10  Gadget Mini
  ...

[acme.example.com] Total: 42 missing-data products.
[acme.example.com] Dry run complete. Use --execute to actually delete these products.

Step 2: Narrow with date filters (optional)

If you only want to clean up older products, use the date filters:

# Only products older than 90 days
node scripts/maintenance/hq-delete-missing-data-products --older-than 90 acme.example.com

# Only products created in a specific date range
node scripts/maintenance/hq-delete-missing-data-products --startDate 2024-01-01 --endDate 2024-06-30 acme.example.com

# Combine: older than 30 days AND within a date range
node scripts/maintenance/hq-delete-missing-data-products --older-than 30 --startDate 2024-01-01 acme.example.com

Date filters are cumulative — a product must satisfy all specified filters to be included.

Step 3: Execute

Once you've confirmed the dry-run output looks correct:

node scripts/maintenance/hq-delete-missing-data-products --execute acme.example.com

Example output:

[acme.example.com] Found 42 products with missingData.
  [DELETE]  sku=WIDGET-001                     created=2024-06-15  Widget Deluxe
  [DELETE]  sku=GIZMO-002                      created=2024-08-22  Gizmo Pro
  ...

[acme.example.com] Total: 42 missing-data products.
[acme.example.com] Deleting 42 products...
[acme.example.com] Successfully deleted 42 missing-data products.

Processing multiple domains

You can pass multiple app domains in a single invocation. They are processed one at a time:

node scripts/maintenance/hq-delete-missing-data-products --execute acme.example.com bigco.example.com

How deletion works

Deletions are performed via the exciseTaxProductService.deleteProduct() method with a concurrency of 5. This script does not back up rows before deletion. Make sure you are confident in the dry-run output before using --execute.

Troubleshooting

No output / immediate exit — Running without any arguments prints the help text and exits. Make sure you provide at least one app domain.

"ERROR deleting ..." — Individual product deletion failures are logged but cause the script to stop processing that domain. Check the error message for Cassandra connectivity or consistency issues.

Unexpected products in the list — The script matches any product where details.missingData is truthy. Use date filters to narrow the scope, or inspect the details JSON for specific products to understand why they are flagged.

Debug logging

The script uses the debug module. To see detailed logging:

DEBUG=delete-missing-data-products* node scripts/maintenance/hq-delete-missing-data-products acme.example.com

Read more