🕸️ Site Update — Hybrid Web Extraction & Orchestration Node for Android

A mobile-first, on-device data extraction engine. It loads real web pages in a (hidden) Android WebView, runs scripted interactions, extracts exactly the data you care about, tracks changes over time, and delivers the result as a notification, a log entry, or an encrypted webhook to your own backend.

📑 Table of Contents

1. Overview
2. Why This Exists
3. Feature Summary
4. Real-World Use Cases
5. Architecture & Execution Pipeline
6. Feature Deep Dive
7. UI Walkthrough
8. Custom JavaScript Guide
9. Backend Integration (Webhook + Encryption)
10. Log Export Format
11. Security Model
12. Design Decisions
13. Known Limitations
14. Responsible Use
15. Roadmap
16. Engineering Workflow & Role
17. Source Code & Contact

🔭 Overview

Site Update turns an Android phone into a small, programmable data-collection node. Instead of relying on a cloud scraper that must be hosted, paid for, and kept alive, the app runs the whole pipeline on the device:

Schedule → Load page → Wait → Click sequence → Extract → Clean → Compare → Notify / Log / Webhook

Every monitor is configured on the phone, runs in the background, and can end in any combination of:

• a native Android notification,
• an entry in a per-monitor history log (exportable as text),
• an optional webhook / API call whose body can be encrypted end-to-end (AES-256-GCM) for your own backend.

The same app handles a one-line CSS selector for a parcel tracker and a full JavaScript routine that walks a job board and returns a formatted result.

💡 Why This Exists

The project grew out of real problems hit while scraping and monitoring sites by hand:

✨ Feature Summary

🎯 Real-World Use Cases

• Package tracking: watch the latest milestone on courier and e-commerce tracking pages (Amazon, Bluedart, local couriers) and get the newest status as a notification.
• Job and exam notices: monitor recruitment and exam portals (e.g. staff-selection, teacher-eligibility, and state public-service pages) for new vacancy or notification text.
• Price watching: track a product price and alert when the value changes or contains a target string.
• Weather and status pages: pull a single line (temperature, "feels like") from a page that offers no alerts.
• Personal data pipeline: forward each extracted value to your own API, store it in PostgreSQL, and analyze change history over time.

🏗️ Architecture & Execution Pipeline

High-level flow of one monitor run

flowchart TD
    A["Scheduler<br/>interval + jitter"] --> B["Load target URL"]
    B --> C{"Hidden WebView<br/>enabled?"}
    C -- "No" --> D["Fetch page content"]
    C -- "Yes" --> E["Render in background WebView<br/>(with imported session, if any)"]
    D --> F["Wait after load<br/>(fixed seconds)"]
    E --> F
    F --> G["Click sequence<br/>click → wait → click → wait"]
    G --> S["Apply search scope<br/>Whole Page / CSS / XPath / JS scope"]
    S --> H["Extraction rule<br/>CSS / XPath / Regex / JS / Parent traversal"]
    H --> I["Sanitize output<br/>strip tags, normalize whitespace"]
    I --> J{"Changed since last run<br/>and trigger rule matched?"}
    J -- "No / duplicate" --> K["Stop (no alert)"]
    J -- "Yes" --> L["Save to history log"]
    L --> M["Android notification<br/>(template with $s)"]
    L --> N{"Webhook<br/>enabled?"}
    N -- "Yes" --> O["Build body from $s<br/>optionally encrypt AES-256-GCM"]
    O --> P["POST to your API"]

Components

🔬 Feature Deep Dive

1. Four-step monitor configuration

Each monitor is defined in four logical steps:

1. Search Scope: choose the Base Element: Whole Page (default), CSS Selector Scope (e.g. table#data), XPath Scope (e.g. //table), or JavaScript Code Scope. Everything after this step searches only inside that region.
2. Search Target: a keyword to anchor on, plus a numeric index to pick the n-th occurrence inside the scope.
3. What to Extract: the extraction rule, its selector / XPath / pattern, an Index to pick which match to use, and the output format (Cleaned Text: no tags, spaces fixed by default).
4. Alert Conditions: trigger rule, case sensitivity, duplicate suppression, interval, jitter, and the notification message.

The top of the form holds the URL, Keep Auth Session (with a session status indicator and a Login to Site button), Use Hidden WebView (with a Wait (s) field), the optional Auto-Click Sequence, and a Test & Download Rendered HTML button for quick debugging.

2. Extraction modes

3. Hierarchical scoping and indexing

Selector ambiguity is a common cause of wrong data on busy pages. The Base Element setting limits the engine to one region of the DOM before any extraction happens:

Two Index fields then pick exactly which occurrence to use: one beside the Target Keyword (Step 2) and one beside the Selector/XPath (Step 3), for example the third match inside a given table.

4. Click sequences and timing

Many pages only reveal data after an interaction. Each monitor can define:

• Wait (s), set next to Use Hidden WebView, so the page can finish rendering before anything else happens.
• An optional Auto-Click Sequence: use Add Click Action to append steps. Each step is a CSS selector plus a wait time (seconds). Steps run in order, each followed by its own wait, then extraction starts.

Waits are deliberately strict durations, not "wait until visible" (see Design Decisions).

5. Hidden WebView and authenticated sessions

• Hidden background WebView: for JavaScript-driven sites, the app creates an invisible Android WebView inside a background service, lets the page render and run its scripts, then runs the extraction.
• Keep Auth Session: preserves the logged-in state between runs for sites that require login. A status indicator shows whether a session exists (for example No Session), and Login to Site lets you sign in from within the app.
• JSON session import: log in on a desktop browser (handling 2FA/CAPTCHA there), export the cookies as a JSON array, and paste them into the app. The WebView starts already authenticated, with no brittle automated login flow.

6. Scheduling and jitter

• Interval (min): base time between runs.
• Jitter (min): a random offset added to each run, so requests do not land at perfectly regular moments and a fleet of monitors does not fire simultaneously.
• Each monitor has an independent on/off toggle on the dashboard.

7. Change detection and alert conditions

• Stateful delta tracking: the last extracted value is stored per monitor and compared on every run.
• Trigger rule: for example Contains Target, with an optional case-sensitive switch.
• Ignore duplicate alerts: if the value has not changed, no notification is sent. This also prevents repeated alerts for an unchanged message.

8. Notification templating

The Notification Message field accepts free text plus the $s placeholder, which is replaced with the cleaned extracted text. For example:

Parcel update: $s

9. History log and export

• Every detected change is stored with a timestamp in a per-monitor history dialog (with a Visit Website shortcut and copy-to-clipboard).
• The log can be exported as plain text (format documented below). The export is intentionally raw; encrypt it separately if you need to share it.

10. Webhook / API call

Enable Trigger Webhook / API Call on a monitor to forward each detected change:

Sending the body as plain text means no character in the extracted value can break a JSON structure.

11. Payload encryption

When Encrypt Payload is enabled, the body is encrypted before it leaves the device.

The receiver needs the same passphrase to decrypt. An in-app Decryption Guides screen provides ready-to-copy snippets for Python, Node.js, and PHP.

12. Encrypted backup (export / import)

• Export asks for a password and writes an encrypted file.
• Import requires the exact same password; a wrong password or a modified file fails authentication and nothing is imported.

📸 UI Walkthrough

1. Main Orchestration Dashboard

2. Task Initialization

3. Multi-Modal Extraction Rules

4. Granular Searching Scope

5. Advanced JavaScript Injection

6. Auto-Click Sequences

7. Alert Conditions, Jitter & Scheduling

8. Cross-Environment Session Import

9. Authenticated Hidden WebView

10. Extraction History Log

11. Android Notifications

12. Webhook / API Configuration

13. In-App Decryption Guides

14. Session, Hidden WebView & Auto-Click (latest editor)

15. Search Scope Options

16. Search Target & Extraction Rules

🧩 Custom JavaScript Guide

The JavaScript rule runs inside the loaded page. The last evaluated expression is the result, and it should be a string.

Conventions

• Return a string. It becomes $s in the notification, the log entry, and the webhook body.
• Return an empty string for "nothing to report", which simply produces no alert.
• Handle failures inside the script. Because every site is different, the script decides what counts as a problem and what the alert should say (for example 'Layout changed: tracker container not found'). The app does not impose a generic error policy.
• Normalize whitespace yourself when needed (.replace(/\s+/g, ' ').trim()).

Example: latest delivery milestone from a tracking page

var tempElement = document.querySelector('#progressTracker-container');

var outputstring = tempElement
    ? [...tempElement.querySelectorAll('.milestone')]
        .reverse()
        .map(el => {
            var status = el.querySelector('.milestone-primaryMessage')?.innerText
                .replace(/\s+/g, ' ').trim() || '';

            var details = el.querySelector('.milestone-explanation')?.innerText
                .replace(/\s+/g, ' ').trim() || '';

            return details ? `${status} - ${details}` : '';
        })
        .find(text => text.includes(' - ')) || ''
    : '';

outputstring;

This is an illustration of the technique. Class names on third-party sites change, so selectors in examples may need updating.

Example: explicit "layout changed" reporting

var box = document.querySelector('#results');
if (!box) {
    'Layout changed: #results not found';
} else {
    box.innerText.replace(/\s+/g, ' ').trim();
}

🔌 Backend Integration (Webhook + Encryption)

Request shape

POST https://your-server.example/events
Content-Type: text/plain; charset=utf-8
Authorization: Bearer <token>
X-Monitor: Amazon Tracking

<body: extracted text, or the encrypted string when Encrypt Payload is on>

Keep header values ASCII-only. Put free-form text in the body.

Decryption snippets

import base64
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
from cryptography.hazmat.primitives import hashes

def decrypt_payload(encrypted_str, password):
    parts = encrypted_str.strip().split(']')
    if len(parts) != 3:
        raise ValueError("unexpected payload format")
    salt = base64.b64decode(parts[0])
    nonce = base64.b64decode(parts[1])
    ciphertext = base64.b64decode(parts[2])  # includes the 16-byte tag

    kdf = PBKDF2HMAC(algorithm=hashes.SHA256(), length=32, salt=salt, iterations=10000)
    secret_key = kdf.derive(password.encode('utf-8'))

    return AESGCM(secret_key).decrypt(nonce, ciphertext, None).decode('utf-8')

const crypto = require('crypto');

function decryptPayload(encryptedStr, password) {
    const parts = encryptedStr.trim().split(']');
    const salt = Buffer.from(parts[0], 'base64');
    const nonce = Buffer.from(parts[1], 'base64');
    const ciphertextBuffer = Buffer.from(parts[2], 'base64');

    const derivedKey = crypto.pbkdf2Sync(password, salt, 10000, 32, 'sha256');

    const authTag = ciphertextBuffer.slice(-16);
    const actualCiphertext = ciphertextBuffer.slice(0, -16);

    const decipher = crypto.createDecipheriv('aes-256-gcm', derivedKey, nonce);
    decipher.setAuthTag(authTag);

    let decrypted = decipher.update(actualCiphertext, undefined, 'utf8');
    decrypted += decipher.final('utf8');
    return decrypted;
}

function decryptPayload($encryptedStr, $password) {
    $parts = explode(']', trim($encryptedStr));
    $salt = base64_decode($parts[0]);
    $nonce = base64_decode($parts[1]);
    $ciphertext = base64_decode($parts[2]);

    $derivedKey = hash_pbkdf2('sha256', $password, $salt, 10000, 32, true);

    $authTag = substr($ciphertext, -16);
    $actualCiphertext = substr($ciphertext, 0, -16);

    return openssl_decrypt($actualCiphertext, 'aes-256-gcm', $derivedKey,
                           OPENSSL_RAW_DATA, $nonce, $authTag);
}

Example receiver: FastAPI + PostgreSQL

A minimal reference implementation that accepts the webhook, decrypts, and stores each event.

CREATE TABLE monitor_events (
    id          BIGSERIAL PRIMARY KEY,
    monitor     TEXT        NOT NULL,
    value       TEXT        NOT NULL,
    received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

import os
import psycopg
from fastapi import FastAPI, Request, Header, HTTPException

app = FastAPI()
PASSWORD = os.environ["PAYLOAD_PASSPHRASE"]
DSN = os.environ["DATABASE_URL"]

@app.post("/events")
async def ingest(request: Request, x_monitor: str = Header(default="unknown")):
    body = (await request.body()).decode("utf-8").strip()
    try:
        text = decrypt_payload(body, PASSWORD)   # from the Python snippet above
    except Exception:
        raise HTTPException(status_code=400, detail="decryption failed")

    text = text.replace("\x00", "")              # PostgreSQL text cannot store NUL
    with psycopg.connect(DSN) as conn:
        conn.execute(
            "INSERT INTO monitor_events (monitor, value) VALUES (%s, %s)",
            (x_monitor, text),
        )
    return {"status": "ok"}

Receiver checklist: read the raw body (do not parse as JSON), trim whitespace before splitting, validate there are three parts, and treat decryption failure as a normal 400 response.

📄 Log Export Format

The text export is human-readable and consistent, so a single parser works for every file.

=== Log Export for Amazon Tracking ===
URL: https://example.eu/d/00V

[Oct 21, 2026 - 03:41 pm]
Package arrived at Delhi Airport at 4:15 am Wednesday, 21 October

--------------------------------------------------

Each entry starts with a [timestamp] line, followed by the extracted value, followed by a dashed separator. If the format ever changes, a version marker will be added to the header so parsers can tell the formats apart; files without one are the original format.

import re

text = open("export.txt", encoding="utf-8").read()
entries = re.findall(r"^\[(.+?)\]\n(.*?)(?=^\[|\Z)", text, re.S | re.M)
rows = [(ts, val.replace("-" * 50, "").strip()) for ts, val in entries]

🔒 Security Model

Notes:

• The encryption passphrase is shared between the app and the receiver; keep it out of source control and load it from an environment variable or secret store on the server.
• Treat exports and backups like any file that may contain API URLs, tokens, and extracted data.

🧠 Design Decisions

• Selectors over screen positions. Targets are described by what the element is (CSS/XPath/keyword), not where it appears, so extraction survives layout shifts.
• Strict time waits over "wait until visible". If a target element has been removed from a page, a visibility wait could hang forever. Fixed waits give every run a known maximum duration, which matters for a battery-conscious background job.
• Failure handling lives in the script. Different monitors need different definitions of failure (some do not care about outages at all). Letting each JavaScript routine return its own message avoids forcing hard-coded "site unavailable" alerts on every monitor.
• Plain-text webhook body. Sending raw UTF-8 text avoids escaping problems entirely; no character in scraped content can invalidate the body.
• Raw text log export. Logs are not always tabular; exporting text and shaping it with JavaScript or downstream scripts keeps the app flexible.
• Local-first. Scheduling, rendering, extraction, and storage all happen on the device; no cloud service is required.

⚠️ Known Limitations

• Selector dependence. Extraction depends on the target site's markup. If a site restructures its page, the selector or script must be updated.
• Polling, not push. Updates are detected on the configured interval; there is no real-time subscription.
• Android background scheduling. Timing can be affected by device battery optimization and vendor-specific restrictions, so very short intervals are not guaranteed.
• Login-protected sites rely on a valid imported session; sessions expire and must be refreshed.
• Third-party sites may restrict automated access. Behavior depends on each site's own policies and defenses.

🤝 Responsible Use

This tool is built for personal monitoring of information you are entitled to see, such as your own parcels, public notices, and prices.

• Respect each website's terms of service and robots.txt.
• Use reasonable intervals; do not hammer servers.
• Use session import only with your own accounts.
• Do not use the tool to circumvent access controls or to collect data you are not permitted to access.

🗺️ Roadmap

Ideas under consideration:

• Send test payload button to verify URL, headers, and decryption in one step
• Versioned payload header (including the key-derivation iteration count) for future-proof decryption
• JSON Lines log export for direct database ingestion
• Retry queue for webhook deliveries that fail while offline
• "Simple" vs "Advanced" configuration modes to shorten the form for basic selector monitors
• Per-step failure messages for click sequences ("step 2: element not found")

🛠️ Engineering Workflow & Role

Role: Product and system design, feature specification, data flow, and orchestration logic.

Approach: The feature set comes directly from problems encountered while scraping and monitoring real sites. The system behavior (extraction modes, timing model, change detection, webhook and encryption design, backup format) was specified and tested by me; AI-assisted workflows were used to accelerate the native Android implementation.

Testing: Features were verified on a physical device against live sites, including webhook delivery, payload encryption and decryption, encrypted backup round-trips, and log export.

📬 Source Code & Contact

The source code is currently private. This repository documents the architecture, features, and integration formats.

I am happy to provide a live technical walkthrough of the Android background services, extraction engine, and encryption design over a screen share.

• Name: Soumyadip Ghosh
• Email: ***
• LinkedIn: www.linkedin.com/in/SamDip
• GitHub: https://github.com/rotten-repo