App Instrumentation — Data Collection & Pipeline

Crawl jobs are run for each of the 3 apps to instrument their pages and collect raw data.

What is collected:

• XML Dumps — Raw XML response data from the app's pages, capturing product listings, categories, and metadata.
• Screenshots — Visual screenshots of the crawled URLs, used for validation and QA purposes.

Output location: [path/to/crawl/output/]

The crawl is configured per app using the seed file (see Step 2). The frequency and scope of crawls can be adjusted based on the app's update cycle.

Each app has a corresponding seed file that drives the crawl and extraction process. The seed file contains two key fields:

The seed file is loaded at the start of both the crawl and the extraction phases. It acts as the source of truth for what data to collect and how to label it.

Seed file location: [path/to/seed_file]

This step transforms the raw XML dumps into structured product data using the CMv3 extraction framework. The process runs locally and involves the following sub-steps:

1. Fetch regex configuration from CMv3 API — extractor.py makes an API call to the CMv3 service to retrieve the regex patterns associated with the app. These patterns define how product fields (title, price, URL, etc.) are identified within the XML.
2. Apply regex on XML dump — The regex patterns are applied on the collected XML dump for each seed_url. This parses out the relevant product attributes from the raw markup.
3. Output JSON file — The extracted data is written to a JSON file. The file contains all products grouped by keyword or category, as defined in the seed file's meta_info.

Key Component: extractor.py

Sample JSON Output Structure

{
  "keyword": "running shoes",
  "category": "shoes",
  "app": "App 1",
  "products": [
    {
      "title": "Nike Air Zoom",
      "price": "₹5,499",
      "url": "https://app.example.com/product/nike-air-zoom",
      "rank": 1
    },
    ...
  ]
}

Once extraction is complete, the output JSON file is pushed to the workspace. The workspace acts as a staging area where files are organized by date before being consumed by the Kafka pipeline.

Directory structure in workspace:

workspace/
└── data/
    └── YYYY-MM-DD/          ← today's data folder
        ├── app1_keyword1.json
        ├── app1_keyword2.json
        ├── app2_keyword1.json
        └── ...

Files are named to reflect the app and keyword/category for easy identification and traceability.

A crontab job is configured in the workspace environment to periodically run commands that push data to Kafka. The job is triggered whenever a new JSON file lands in today's data folder.

How it works

• The crontab runs at a defined interval (e.g., every 15 minutes or hourly — update with actual schedule).
• It scans the workspace/data/YYYY-MM-DD/ folder for new or unprocessed files.
• For each new file detected, the corresponding Kafka push command is executed.
• Data is published to the Kafka topic associated with the app and category.

Crontab Configuration (example)

# Run every 15 minutes — push new JSON files to Kafka
*/15 * * * * /path/to/scripts/kafka_push.sh /workspace/data/$(date +\%Y-\%m-\%d)/

Kafka Topic Mapping