Skip to main content

Documents

Composable building blocks for document workflows: pull text and layout out of a file, scan that text for sensitive patterns, turn matched strings into page geometry, burn redactions into a flattened PDF, remember decisions by content hash, and expand an uploaded archive into table rows.

None of these types require a connector. Each is a small, single-purpose block designed to be chained — most often with a Tool Pipeline or an AutessaScript orchestrator — so one extraction can feed several detectors and model calls without re-reading the file. The extract, locate-spans, and redaction-burn blocks take an s3Uri pointing at a file in Autessa storage (an s3:// URI is presigned automatically) or an existing https:// URL. They are pure with respect to your data — nothing is written to an AutessaDB row unless you add a follow-up step that does so.

A typical redaction flow chains these blocks in order: Extract Text to get the document's text, then Detect Patterns and/or model calls to decide what is sensitive, then Locate Spans to turn those strings into boxes, then Burn Redactions to produce the final flattened PDF. The Decision Cache blocks let a later run of the same document reuse the earlier verdict instead of repeating the work, and Bulk Intake is the front door that turns an uploaded archive of documents into rows ready to be processed.

Extract Text TOOLDOCUMENT_EXTRACT​

Extracts text and per-page layout from the document at s3Uri. Native-text PDFs are read directly; scanned or image pages are rasterized and read with vision so image-only documents still come back as text. Handles PDF, DOCX, TXT, MD, CSV, XLSX, EML, MSG, and common image formats. maxPages caps how many pages are read (default: all), and rasterDpi overrides the rasterization resolution for image pages. This block does not write anywhere — persist its result with a follow-up step if you need it stored.

The result is {"sha256": ..., "mime": ..., "text": ..., "pages": [{ "n": ..., "text": ..., "width": ..., "height": ... }, ...], "metadata": {...}} — the full text, a per-page breakdown with dimensions, the document's MIME type, and the SHA-256 of its canonical bytes (the key the Decision Cache is keyed on).

Example​

Tool config
{
  "toolType": "DOCUMENT_EXTRACT",
  "s3Uri": "@@{DOCUMENT_S3_URI}",
  "maxPages": 100
}

Configuration fields

  • Name
    s3Uri
    Type
    string
    Description
    s3:// URI or presigned https URL of the source document. Required.
  • Name
    maxPages
    Type
    integer · optional
    Description
    Optional cap on how many pages to extract. Null = all pages.
  • Name
    rasterDpi
    Type
    integer · optional
    Description
    Optional rasterization DPI for image/scanned pages. Null = service default.
  • Name
    canonicalOutputS3Uri
    Type
    string · optional
    Description
    Optional s3:// URI the compute service renders the document's CANONICAL PDF to (native-layout via LibreOffice for office docs; all sheets/pages become consecutive PDF pages). When set, the executor presigns the PUT and passes an `output` block to Python, and the success envelope gains `canonicalS3Uri` + `canonicalPageCount`. When null/blank, extract behaves as before (no canonical render, no output block). The caller supplies the target URI — nothing is derived.
  • Name
    convertedXlsxOutputS3Uri
    Type
    string · optional
    Description
    Optional s3:// URI the compute service writes the LibreOffice-converted `.xlsx` twin to when the source is a LEGACY spreadsheet (.xls / .ods). Harmless to supply for anything else: the service simply does not convert, omits the `converted_xlsx` response block, and nothing is written — so callers may set it unconditionally rather than sniffing file types on the Java side. When the conversion happened, the success envelope gains `convertedXlsxS3Uri` so the caller can persist `converted_xlsx_file` without re-deriving the uri. That artifact is the ONE spreadsheet surface a legacy workbook has — grid viewer, reviewer cell marks and the openpyxl scrub all read it — which is exactly why it is converted once here and stored, never re-derived later: two LibreOffice passes may disagree, and cell coordinates must not move under a mark.
  • Name
    timeoutSeconds
    Type
    integer
    Description
    Per-call HTTP timeout (seconds) for the compute service. Absent: DERIVED from the source's size (see DocumentExtractToolExecutor#timeoutSeconds), 60..1800 s. Cap 21600 (6h) when given explicitly.

Detect Patterns TOOLDOCUMENT_DETECT​

Scans a piece of text for sensitive patterns deterministically — no model involved. This is the "find the obvious matches" primitive that a classification workflow composes alongside model-based judgment. presets selects from a catalog of built-in detectors: SSN, CREDIT_CARD (validated with the Luhn checksum), US_DRIVERS_LICENSE, BANK_ROUTING_ACCOUNT, PHONE, EMAIL, GPS_COORDS, GPS_NEAR_TERMS, and EXIF_GPS. customRegexes adds your own detectors — each a name, a pattern, optional flags (a subset of imsux), optional requiredContextKeywords (a match only counts when one of these words appears nearby), a code label, an optional target (TEXT, the default, or FILENAME to match against filename), and an optional luhn flag to require the checksum. Rules can instead live as rows of an AutessaDB table via rulesTableId, so editing a row changes the very next document scanned with no redeploy; a term or name list can be supplied through termsTableId and termsColumn. filename and metadata are optional companion inputs some detectors read (for example, EXIF_GPS fires on a truthy GPS signal in metadata).

The result lists the hits found — each with the matched text, the detector or rule that fired, its label code, and the location within the text — so a downstream step can decide what to do with them.

Example​

Tool config
{
  "toolType": "DOCUMENT_DETECT",
  "text": "@@{steps.extract.text}",
  "presets": ["SSN", "CREDIT_CARD", "EMAIL", "PHONE"],
  "customRegexes": [
    {
      "name": "employee_id",
      "pattern": "EMP-\\d{6}",
      "code": "INTERNAL_ID"
    }
  ]
}

Configuration fields

  • Name
    text
    Type
    string
    Description
    The text to scan. Fed from a prior DOCUMENT_EXTRACT step. Required (s3Uri not yet supported).
  • Name
    filename
    Type
    string · optional
    Description
    Optional filename to scan (fed via `@@{filename`}). Custom rules whose `CustomRegex#target` is `FILENAME` match against this rather than `#text`, mirroring the Python filename prefilter rules (bwc/wolf_depredation/personnel).
  • Name
    metadata
    Type
    string · optional
    Description
    Optional document metadata (fed via `@@{metadata`}) — a JSON object string or a plain value. The `EXIF_GPS` preset fires when this metadata carries a truthy `exifGps` (a JSON field `exifGps` that is true/non-empty, or the whole value being truthy). Detect operates on text/metadata only — it never parses image bytes itself.
  • Name
    s3Uri
    Type
    string
    Description
    Reserved: scanning straight from a document is not yet supported; use DOCUMENT_EXTRACT first.
  • Name
    presets
    Type
    array<string>
    Description
    Preset detector names from the catalog: SSN, CREDIT_CARD, US_DRIVERS_LICENSE, BANK_ROUTING_ACCOUNT, PHONE, EMAIL, GPS_COORDS, GPS_NEAR_TERMS, EXIF_GPS. `GPS_NEAR_TERMS` — a GPS coordinate within ±200 chars of any term drawn from `#termsTableId`/`#termsColumn` (needs both set). `EXIF_GPS` — a truthy `exifGps` signal in `#metadata`.
  • Name
    customRegexes
    Type
    array<object>
    Description
    Customer-authored custom regex detectors baked into this tool's config.
  • Name
    rulesTableId
    Type
    string (ID) · optional
    Description
    Optional AutessaDB table of rule rows (name, pattern, flags, context_keywords, code, enabled). References an AutessaDB table by ID.
  • Name
    termsTableId
    Type
    string (ID) · optional
    Description
    Optional AutessaDB table whose `#termsColumn` values are matched as a term/name list. References an AutessaDB table by ID.
  • Name
    termsColumn
    Type
    string
    Description
    Column of `#termsTableId` holding the term/name to match.
  • Name
    name
    Type
    string
    Description
    Required string value.
  • Name
    pattern
    Type
    string
    Description
    Required string value.
  • Name
    flags
    Type
    string
    Description
    Required string value.
  • Name
    requiredContextKeywords
    Type
    array<string>
    Description
    Required array<string> value.
  • Name
    code
    Type
    string
    Description
    Required string value.
  • Name
    target
    Type
    string
    Description
    Which input this rule scans: `TEXT` (default) matches against the extracted document text; `FILENAME` matches against `@@{filename`}. Case-insensitive; a blank/absent value means TEXT. FILENAME rules mirror the Python filename prefilters.
  • Name
    luhn
    Type
    boolean
    Description
    When true, a raw match only counts if it passes the Luhn checksum (the credit-card check). Lets a custom card-like rule opt into the same validation the `CREDIT_CARD` preset uses. Default false/absent = no Luhn gate.

Locate Spans TOOLDOCUMENT_LOCATE_SPANS​

Finds every occurrence of a set of verbatim text spans in the document at s3Uri and returns geometry boxes for them. This is the bridge between "these strings are sensitive" — the output of the detectors and model calls — and "here are the boxes to draw or burn." Each entry in spans is the literal text to locate, plus an optional code label and a reason. This block writes nothing; it just returns geometry.

The result is a list of located boxes — each carrying the page number, position and size, the matched text, and the code/reason carried through — in the shape Burn Redactions accepts directly, so a pipeline can splice this output straight into that block's redactions.

Example​

Tool config
{
  "toolType": "DOCUMENT_LOCATE_SPANS",
  "s3Uri": "@@{DOCUMENT_S3_URI}",
  "spans": [
    { "text": "123-45-6789", "code": "SSN", "reason": "Social security number" },
    { "text": "jane@example.com", "code": "EMAIL", "reason": "Personal email" }
  ]
}

Configuration fields

  • Name
    s3Uri
    Type
    string
    Description
    s3:// URI or presigned https URL of the source document. Required.
  • Name
    spans
    Type
    array<object>
    Description
    Text spans to locate. Required (non-empty).
  • Name
    timeoutSeconds
    Type
    integer
    Description
    Per-call HTTP timeout (seconds). Absent: DERIVED from `totalPages` and the measured per-page locate cost (`ComputeCostModel#pageLinearTimeoutSeconds`, 60..1800 s), the 1800 s ceiling when the page count is unknown. Cap 21600 (6h) when given explicitly.
  • Name
    totalPages
    Type
    integer
    Description
    The surface's page count, when the caller knows it (the orchestrator does, from extract). Sizes the timeout (a hinted span's search can still widen ring by ring to the whole document, so the document, not the hint, is the worst case), and a surface longer than one locate slice is located in parallel page slices and merged (DocumentLocateSpansToolExecutor#locateSliced). Absent: one whole-document call, as before.
  • Name
    text
    Type
    string
    Description
    Required string value.
  • Name
    code
    Type
    string
    Description
    Required string value.
  • Name
    reason
    Type
    string
    Description
    Required string value.
  • Name
    pageFirst
    Type
    integer · optional
    Description
    PAGE HINT (1-based, inclusive): the page range of the window whose proposer found this span. Optional — both null means "search the whole document", the behaviour every caller had before hints existed. A hinted span is searched nearest-first, outward from the hint, and stops at its first hit; compute clamps a nonsensical range to the document rather than failing the call.
  • Name
    pageLast
    Type
    integer · optional
    Description
    PAGE HINT (1-based, inclusive): the page range of the window whose proposer found this span. Optional — both null means "search the whole document", the behaviour every caller had before hints existed. A hinted span is searched nearest-first, outward from the hint, and stops at its first hit; compute clamps a nonsensical range to the document rather than failing the call.

Burn Redactions TOOLREDACTION_BURN​

Burns a set of redaction boxes into the document at s3Uri, removing the underlying content (not just hiding it) and writing a flattened PDF to outputS3Uri. Each box becomes a solid black block, and — unless showRuleText is set to false — a label naming the triggering rule (its code) is drawn over the block. redactions is the list of boxes to burn, in the same shape Locate Spans emits, so the two chain directly.

The result reports the written output document — the flattened PDF at outputS3Uri, with the sensitive content permanently removed.

Example​

Tool config
{
  "toolType": "REDACTION_BURN",
  "s3Uri": "@@{DOCUMENT_S3_URI}",
  "outputS3Uri": "@@{OUTPUT_S3_URI}",
  "redactions": "@@{steps.locate.boxes}",
  "showRuleText": true
}

Configuration fields

  • Name
    s3Uri
    Type
    string
    Description
    s3:// URI or presigned https URL of the source document. Required.
  • Name
    outputS3Uri
    Type
    string
    Description
    s3:// URI the burned PDF is written to; the executor presigns the PUT. Required.
  • Name
    redactions
    Type
    array<object>
    Description
    Merged redaction boxes to burn: same shape DOCUMENT_LOCATE_SPANS emits ({page, x, y, w, h, text, code, reason, status}). Left as a raw list so pipeline mapping can splice locate-spans output straight through.
  • Name
    showRuleText
    Type
    boolean
    Description
    Whether the white overlay label names the rule/citation on the block. Default true.
  • Name
    timeoutSeconds
    Type
    integer
    Description
    Per-call HTTP timeout (seconds). Absent: DERIVED from the pages this call burns (the page range, else `totalPages`) and the measured per-page burn cost (`ComputeCostModel#pageLinearTimeoutSeconds`, 60..1800 s), and the 1800 s ceiling when the page count is unknown. Cap 21600 (6h) when given explicitly.
  • Name
    pageRange
    Type
    object
    Description
    Burn only pages `[start, end)` (0-based, end exclusive) and write ONLY those pages to `outputS3Uri`: one SLICE of a long document's burn (burn_and_attach.js joins the slices). Box `page` numbers stay the ORIGINAL 1-based document pages; the compute service maps them. Absent: the whole document, exactly as before. Requires `totalPages`.
  • Name
    totalPages
    Type
    integer
    Description
    The whole reviewed canonical PDF's page count. When present the compute service validates EVERY box against it (not only the slice's), so a box past the end fails a slice exactly as it failed the whole burn. Also the page basis of the derived timeout when no range is given.
  • Name
    metadataRedactions
    Type
    array<string> · optional
    Description
    Dotted metadata paths a reviewer marked REDACT (the FirstPass `metadata_redactions` cell's `fields`: e.g. `["exif.gps", "doc.author"]`). The compute service strips these from the burned PDF's metadata best-effort — a scrub failure never fails the burn. Optional; empty/absent means no metadata scrub.
  • Name
    rasterizeOutput
    Type
    boolean
    Description
    Whether the released hard PDF is RASTERIZED to an image-only document (the print-and-scan guarantee: no text layer anywhere, so nothing is extractable). Default TRUE — this is the safety-critical release mode. Set false for a searchable vector release (covered content is still removed by the vector burn; only the rest keeps a live text layer). Absent => default.
  • Name
    rasterDpi
    Type
    integer
    Description
    DPI for the rasterized release (default 200 in the compute service; clamped 72..600). Higher = crisper but larger. Only meaningful when `rasterizeOutput` is true. Absent => default.
  • Name
    ocrTextLayer
    Type
    boolean
    Description
    Lay an invisible OCR text layer over the RASTERIZED release (the searchable-image production shape): Tesseract reads the burned raster only — black blocks included — so the layer cannot contain covered content, while the release becomes searchable/selectable again. Best-effort in the compute service (missing Tesseract degrades to the image-only release; the response's `ocr_text_pages` says how many pages got the layer). Only meaningful when `rasterizeOutput` is true. Absent/false => image-only release, wire-identical to before.
  • Name
    stripAllLinks
    Type
    boolean
    Description
    Break EVERY hyperlink in the release (the FirstPass "Break all links" document flag, `links_sanitized`): after the box burn, the compute service deletes every remaining link annotation on every page — external URIs and internal GoTo links alike — and counts them in `links_stripped`. No box, no exemption code: this is not a redaction of content, only of the live links. A rasterized release carries no annotations anyway, so it matters for the vector release. Absent/false => only links under an applied redaction are removed, wire-identical to before.

Look Up Decision Cache TOOLDOCUMENT_CACHE_LOOKUP​

Looks up a document's prior decisions by its sha256 content hash. When an earlier run already classified and marked up an identical document, a lookup returns those decisions so a later run reuses them instead of repeating the analysis. Decisions are stored per company and encrypted at rest.

The result is {"hit": ..., "classifications": ..., "redactions": ..., "version": ..., "provenance": ..., "hitCount": ...}. On a miss, hit is false and the decision fields are empty. On a hit, classifications and redactions carry the stored verdicts, version is the current revision of the cached record, provenance describes who last reviewed it and when, and hitCount reflects how often it has been reused.

Example​

Tool config
{
  "toolType": "DOCUMENT_CACHE_LOOKUP",
  "sha256": "@@{steps.extract.sha256}"
}

Configuration fields

  • Name
    sha256
    Type
    string
    Description
    SHA-256 hex of the canonical document bytes. Required.

Store Decision Cache TOOLDOCUMENT_CACHE_STORE​

Stores a document's decisions under its sha256 content hash so a later run of the same document can reuse them. classificationsJson and redactionsJson are opaque JSON strings the caller defines — the cache round-trips them unchanged and encrypts them at rest. expiresInDays sets a retention window (omit for no expiry). Providing provenanceJson — non-sensitive review metadata such as who adjudicated the document and when — records an adjudication history: each store keeps the decisions versioned and provenance-aware. Setting supersede to true replaces an existing record for the same hash and bumps its version, which is how an edited-after-the-fact decision overwrites an earlier one; without it, a store is a plain write. sourceRequestId optionally tags the decisions with the request that produced them.

The result is {"stored": ..., "version": ..., "superseded": ...} — whether the write landed, the resulting record version, and whether it replaced a prior record.

Example​

Tool config
{
  "toolType": "DOCUMENT_CACHE_STORE",
  "sha256": "@@{steps.extract.sha256}",
  "classificationsJson": "@@{steps.classify.result}",
  "redactionsJson": "@@{steps.locate.boxes}",
  "provenanceJson": "{\"reviewedBy\":\"@@{REVIEWER}\",\"adjudicatedAt\":\"@@{currentDate}\"}",
  "expiresInDays": 365
}

Configuration fields

  • Name
    sha256
    Type
    string
    Description
    SHA-256 hex of the canonical document bytes. Required.
  • Name
    classificationsJson
    Type
    string
    Description
    Classification verdicts as a JSON string. Stored encrypted.
  • Name
    redactionsJson
    Type
    string
    Description
    Soft-redaction boxes as a JSON string. Stored encrypted.
  • Name
    sourceRequestId
    Type
    string · optional
    Description
    UUID of the request that produced these decisions (optional, for provenance).
  • Name
    expiresInDays
    Type
    integer
    Description
    Agency retention: expires_at = now + N days. Null = never expires.
  • Name
    provenanceJson
    Type
    string
    Description
    E9 learning loop: non-sensitive provenance JSON ({sourceRequestId, reviewedBy, adjudicatedAt, …}) stored plaintext for the UI. When present (or `#supersede` is set), the store routes through `storeAdjudicated` (version/provenance-aware) instead of the plain backstop `store`.
  • Name
    supersede
    Type
    boolean
    Description
    E9: when true, an existing learned record for this hash is SUPERSEDED — its version is bumped and decisions/provenance are replaced (set by finalize when the doc was edited after a cache hit, or a prior cache row already exists). Absent/false → plain store (no version bump).
  • Name
    adjudicationTier
    Type
    string
    Description
    Adjudication tier for this entry: `HUMAN_ADJUDICATED` (a reviewer acted on the doc) or `AI_ONLY` (a pure AI proposal). Finalize derives it per document. The tier only ratchets up — a HUMAN_ADJUDICATED value upgrades an existing AI_ONLY entry; an AI_ONLY/absent value never downgrades an existing HUMAN_ADJUDICATED entry. Absent/blank ⇒ defaults to AI_ONLY on a new row, leaves an existing row's tier alone. When present the store routes through `storeAdjudicated`.

Bulk Intake TOOLDOCUMENT_BULK_INTAKE​

Expands one uploaded archive (a .zip) at archiveS3Uri into rows of a target AutessaDB table — the front door for turning a delivery of many documents into records ready to process. Each archive entry is streamed to storage (the archive is never buffered whole), its SHA-256 is computed as it streams, and the rows are inserted in chunks. columnMapping names which target columns receive which per-entry facts: fileColumn (a FILE column, required) gets the stored file, and the optional nameColumn, pathColumn, shaColumn, sizeColumn, and mimeColumn receive the entry's name, path within the archive, content hash, byte size, and MIME type; extraStatic writes fixed values (each a template) on every row. Email entries (.eml) are expanded so their attachments become their own rows, with parentShaColumn linking an attachment back to its parent. allowedExtensions narrows which file types are accepted (a sensible document/spreadsheet/email/image default applies otherwise); entries that are the wrong type or over maxEntryBytes are skipped and reported per-entry rather than failing the whole intake. Only the columns you map are written, so rows stay user-editable. Set enableIndexing to true to index rows for semantic search as they land, and uploadsTableId to record one lifecycle row per archive (its filename, size, status, document count, and any error).

Intake is crash-safe and resumable: a large archive is expanded as a chain of independent slices (bounded by chunkSize, batchEntriesPerRun, timeBudgetSeconds, and timeoutSeconds), each checkpointing its progress before the next begins. Re-running or retrying an interrupted intake is idempotent — already-inserted entries are reconciled and skipped, so no document is ingested twice. Long intakes are typically run as work queue items.

The result reports the intake's progress — how many entries were expanded into rows, how many were skipped, and any per-entry errors.

Example​

Tool config
{
  "toolType": "DOCUMENT_BULK_INTAKE",
  "archiveS3Uri": "@@{ARCHIVE_S3_URI}",
  "targetTableId": "V1StGXR8Z5",
  "columnMapping": {
    "fileColumn": "document",
    "nameColumn": "file_name",
    "shaColumn": "sha256",
    "sizeColumn": "size_bytes",
    "extraStatic": { "batch_id": "@@{BATCH_ID}" }
  },
  "uploadsTableId": "3oPtLQmZ7v",
  "enableIndexing": false
}

Configuration fields

  • Name
    archiveS3Uri
    Type
    string
    Description
    s3:// URI (or presigned https URL) of the archive to expand. Required; typically `@@{archiveS3Uri`}.
  • Name
    targetTableId
    Type
    string (ID)
    Description
    The AutessaDB table each archive entry becomes a row of. Required. References an AutessaDB table by ID.
  • Name
    columnMapping
    Type
    object
    Description
    Which target-table columns receive which per-entry facts. Required (`fileColumn` at minimum).
  • Name
    uploadsTableId
    Type
    string (ID) · optional
    Description
    Optional per-archive lifecycle table (filename/size/status/n_documents/error columns by name). References an AutessaDB table by ID.
  • Name
    chunkSize
    Type
    integer
    Description
    Rows per bulk-insert chunk. Null ⇒ {@value #DEFAULT_CHUNK_SIZE}. Clamped to at least 1.
  • Name
    allowedExtensions
    Type
    array<string> · optional
    Description
    Optional allow-list of lower-case file extensions (no dot). When non-empty, an entry whose extension is not listed is skipped and reported per-entry. Null/empty ⇒ the executor falls back to `#DEFAULT_ALLOWED_EXTENSIONS` (common document/spreadsheet/email/image types); a non-empty configured list wins.
  • Name
    enableIndexing
    Type
    boolean
    Description
    Whether inserted rows are indexed. Default false ⇒ indexingMode SKIP (indexing deferred to a later touchpoint); true ⇒ IMMEDIATE.
  • Name
    timeoutSeconds
    Type
    integer
    Description
    Per-invocation timeout (seconds). Null ⇒ {@value #DEFAULT_TIMEOUT_SECONDS}; clamped to [1, {@value #MAX_TIMEOUT_SECONDS}].
  • Name
    batchEntriesPerRun
    Type
    integer
    Description
    LEGACY (pre-split slice model): zip entries expanded per run. Still parsed so in-flight payloads deserialize across deploy; the phase model reads `#enqueueBatchSize` instead.
  • Name
    timeBudgetSeconds
    Type
    integer
    Description
    Soft wall-clock budget (seconds) for a run; a run that exceeds it stops at the next chunk boundary and continues in a fresh slice. Null ⇒ {@value #DEFAULT_TIME_BUDGET_SECONDS} (45min). Keep it under `#timeoutSeconds` so a run checkpoints rather than being killed.
  • Name
    maxArchiveBytes
    Type
    integer
    Description
    Max archive size accepted (bytes). Null ⇒ {@value #DEFAULT_MAX_ARCHIVE_BYTES} (20 GiB).
  • Name
    maxEntryBytes
    Type
    integer
    Description
    Per-entry size cap (bytes). Null ⇒ {@value #DEFAULT_MAX_ENTRY_BYTES} (2 GiB). Top-level entries stream to S3 so this is a runaway backstop, not a working ceiling; an over-cap entry is skipped with a per-entry error row rather than failing the whole intake.
  • Name
    queueId
    Type
    string
    Description
    The work queue (UUID string) continuation slices are enqueued onto. Usually resolved at runtime from the executing item's own queue (`AUTESSA_WORK_QUEUE_ID`); this config value is the fallback for a direct (non-work-item) invocation that still wants self-chaining.
  • Name
    entryOffset
    Type
    integer
    Description
    LEGACY (pre-split slice model): zero-based entry offset a slice started at. Still parsed so an in-flight pre-split payload deserializes across deploy; the INDEX phase treats it as an `#enqueueOffset` floor so such an item resumes rather than re-walking from zero.
  • Name
    uploadRowId
    Type
    string (ID)
    Description
    The lifecycle (uploads) row id shared across all slices of one archive. The first slice inserts the row and stamps its id into every continuation payload (`@@{uploadRowId`}); later slices update that same row (progress + terminal status) instead of inserting a new one. References an AutessaDB table by ID.
  • Name
    groupKey
    Type
    string
    Description
    The queue group key continuation slices join (same group counters as the parent). Usually resolved from the executing item's own group (`AUTESSA_WORK_GROUP_KEY`); this config/payload value is the fallback.
  • Name
    ingestUnmatched
    Type
    boolean
    Description
    NEVER-IGNORE mode. When true, an entry whose extension is NOT in `#allowedExtensions` is INGESTED as an ordinary document row (bytes staged to S3, name/sha/size/mime recorded) instead of being recorded-and-skipped — the tenant's downstream pipeline decides what to do with it (in FirstPass: route it to human review rather than drop it). The SAFETY guards are unaffected and still skip: a zip-slip / absolute / `..` entry path and an entry over `#maxEntryBytes`. Those are attack and runaway-size guards, not "we don't recognise this type" — conflating the two would let this flag re-open a path-traversal hole. Null/false ⇒ unchanged platform behaviour (unknown extensions are counted skips).
  • Name
    maxArchiveDepth
    Type
    integer
    Description
    How many levels of ARCHIVE this intake expands. Null ⇒ {@value #DEFAULT_ARCHIVE_DEPTH} (the uploaded archive only, the historical behaviour). With `maxArchiveDepth > 1`, a `.zip` entry found at depth `d < maxArchiveDepth` is staged to S3 and gets its OWN intake work item (its own lifecycle row, its own `uploadRowId`, hence its own deterministic doc-key space) enqueued on the same queue/group with `archiveDepth = d + 1` — the same self-chaining mechanism continuation slices use. A zip AT the max depth is not expanded: it is ingested as a document row when `#ingestUnmatched` is on, else recorded-and-skipped.
  • Name
    archiveDepth
    Type
    integer
    Description
    The archive-nesting depth of THIS run: 1 for the uploaded archive, `d+1` for an archive expanded out of a depth-`d` archive. Carried in the nested work item's payload (`@@{archiveDepth`}); absent/blank ⇒ 1.
  • Name
    phase
    Type
    string
    Description
    Which phase this invocation runs: `#PHASE_INDEX` enumerates the archive's central directory once and enqueues one `#PHASE_EXTRACT` work item per inner file (self-chaining when the walk exceeds its budget), each EXTRACT lands exactly one entry (bounded ranged GET → inflate → S3 → one row insert, whose BULK_INSERT trigger enqueues the document's pipeline immediately), and one `#PHASE_FINALIZE` item converges the archive to its terminal lifecycle status once every EXTRACT is done. Absent/blank ⇒ INDEX — which is also what a pre-split payload (no phase at all) resolves to, keeping in-flight items valid across deploy.
  • Name
    entryIndex
    Type
    integer
    Description
    EXTRACT: this entry's zero-based ordinal in the NAME-SORTED entry list — the `entryIndex` half of the deterministic doc key `"{uploadRowId`:{entryIndex}"}. Assigned once at INDEX time and carried here, so it is stable regardless of extraction order or retries.
  • Name
    entryName
    Type
    string
    Description
    EXTRACT: the raw central-directory entry name (sanitized again at extract time).
  • Name
    dataOffset
    Type
    integer
    Description
    LEGACY EXTRACT: absolute byte offset of this entry's compressed DATA within the archive object. INDEX no longer populates this — resolving it required commons-compress to parse every LOCAL FILE HEADER at open, which walks essentially the whole archive over ranged S3 reads (~10 silent minutes for a 7.3 GB zip). INDEX now opens central-directory-only and carries `#localHeaderOffset` instead; EXTRACT derives the data offset from ONE 64-byte ranged GET. Still parsed so an in-flight pre-deploy payload deserializes and extracts correctly: when it carries a usable `dataOffset` EXTRACT uses it directly and skips the LFH read.
  • Name
    localHeaderOffset
    Type
    integer
    Description
    EXTRACT: absolute byte offset of this entry's LOCAL FILE HEADER within the archive object, read straight off the central directory (no local-header parse needed, ZIP64 included). The extract run reads 64 bytes there, parses the LFH's own name/extra lengths, and derives `dataOffset = localHeaderOffset + 30 + nameLen + extraLen`. The LFH's extra length routinely DIFFERS from the central directory's (zip writers pad the local extra for alignment, and ZIP64/Unicode extra fields are frequently present in only one of the two) — which is exactly why the data offset must be derived from the LOCAL header and never from the central-directory extra length.
  • Name
    compressedSize
    Type
    integer
    Description
    EXTRACT: this entry's compressed byte count (the bounded ranged GET's length).
  • Name
    uncompressedSize
    Type
    integer
    Description
    EXTRACT: this entry's uncompressed byte count (the S3 put's content length).
  • Name
    compressionMethod
    Type
    integer
    Description
    EXTRACT: zip compression method — 0 STORED (pass-through) or 8 DEFLATED (raw-deflate inflate).
  • Name
    entryLastModifiedMillis
    Type
    integer · optional
    Description
    EXTRACT: this entry's LAST-MODIFIED time as epoch millis, read from the zip central directory by the INDEX walk; absent when the archive did not carry one, or for a whole-object entry (a single-file upload / manifest member, which was never inside a container). Carried in the payload because it exists ONLY at INDEX. That walk opens the central directory — the one place a per-entry timestamp lives — and EXTRACT reads a byte window, never a directory record. Without this field a document's own date would be unrecoverable for every non-email document in the corpus, and the ingestion time (which says nothing about when the record happened) would be the only thing left to write. Optional and additive: a payload from a pre-deploy INDEX run simply lacks it and that document falls to the next rung of the date ladder, exactly as before.
  • Name
    enqueueOffset
    Type
    integer
    Description
    INDEX: the entry ordinal the enqueue walk resumes from. Checkpointed to the lifecycle row after every enqueue batch (the row is authoritative over this payload value, exactly like the legacy `entryOffset` rule), so a crashed/re-leased INDEX run re-enqueues at most one batch — and re-enqueued EXTRACTs no-op against their doc key anyway.
  • Name
    totalEntries
    Type
    integer
    Description
    INDEX continuation / FINALIZE: total file entries in the archive, once its directory is read.
  • Name
    finalizeAttempt
    Type
    integer
    Description
    FINALIZE: self-chain attempt counter driving the re-check delay backoff.
  • Name
    enqueueBatchSize
    Type
    integer
    Description
    INDEX: EXTRACT items enqueued per batch. Null ⇒ {@value #DEFAULT_ENQUEUE_BATCH_SIZE}.
  • Name
    manifestS3Uri
    Type
    string · optional
    Description
    INDEX (MANIFEST variant): s3:// URI of a JSON manifest naming objects ALREADY UPLOADED to S3, which the walk enumerates INSTEAD of a zip central directory. This is how a whole FOLDER is taken in: the browser uploads each file with the ordinary multipart upload and then names them all in one manifest, rather than zipping tens of thousands of files in a tab. Shape — `{"files":[{"s3Uri":"s3://…","relativePath":"case-7/photos/a.jpg", "sizeBytes":12345`]}}. `relativePath` plays EXACTLY the role a zip entry name plays: it is the `source_path` of the row, the input to the zip-slip sanitizer, the extension the allow-list is checked against, and the name an OS-artifact skip is decided by. That equivalence is the whole design — the named-skip contract, the ordinal (name-sorted), the doc-key formula and the lifecycle bookkeeping all carry over from the zip walk with no second implementation. `sizeBytes` is optional (the per-entry cap is simply not decidable at index time without it, and EXTRACT's own bounded read still enforces it). A URI, never inline JSON: a folder can be tens of thousands of files, and a work-item payload is not a place to put a megabyte of paths. The walk itself is chunked across self-chained INDEX items on the same time budget as the zip walk, so there is no file-count cap. When set, `#archiveS3Uri` is still required (it identifies the upload's lifecycle row and is what `findUploadRowByArchive` resolves on) but is never opened as a zip.
  • Name
    documentTagsTableId
    Type
    string (ID)
    Description
    The tag MEMBERSHIP table (`firstpass_document_tags`) — one row per (document, tag). When set (together with a resolvable `#uploadTagLabel`), every document row this intake lands is tagged with the name of the UPLOAD it came from, so a reviewer can filter a corpus down to "everything out of PRR-23276-batch2.zip" with the same indexed read a hand-applied tag uses. The rows written here are IDENTICAL in shape to the ones the app's tag op service writes — same columns, same slug fold, same display mirror on the document — because a second tag mechanism would be a second thing every tag reader has to know about. Absent ⇒ no tagging at all, silently and correctly: a tenant provisioned before the tag tables existed keeps ingesting exactly as it did. References an AutessaDB table by ID.
  • Name
    tagsTableId
    Type
    string (ID)
    Description
    The tag VOCABULARY table (`firstpass_tags`) — one row per tag, per request, carrying the human `label` the picker renders. Set alongside `#documentTagsTableId` so the upload's tag is a FIRST-CLASS tag: it appears in the tag picker and the facet list beside hand-made ones rather than existing only as membership rows nothing can name. The vocabulary row is upserted once per upload by the INDEX phase. References an AutessaDB table by ID.
  • Name
    uploadTagLabel
    Type
    string
    Description
    The upload's display name, as the tag LABEL every row of this upload carries (its slug is derived from it). Resolved ONCE by the INDEX phase — which is the only run that can see the whole source (an archive's own filename, a folder manifest's common top-level directory, a single file's name) — and carried on every EXTRACT payload from there, exactly as the entry byte-window facts are. Deriving it per-EXTRACT instead would mean re-reading the manifest once per file and would let two files of one folder disagree about which upload they came from.
  • Name
    fileColumn
    Type
    string
    Description
    The FILE column that receives each uploaded entry's S3 URI. Required.
  • Name
    nameColumn
    Type
    string · optional
    Description
    Optional column receiving the entry's file name (basename).
  • Name
    nameKeyColumn
    Type
    string · optional
    Description
    Optional column receiving the entry's file name NORMALIZED for search (lower-cased, trimmed). The indexed twin of `nameColumn`: same value, written in the same pass, so a filename predicate runs against an index instead of scanning. Omit it and nothing is written — the mapping is optional exactly like `uploadKeyColumn`.
  • Name
    pathColumn
    Type
    string · optional
    Description
    Optional column receiving the entry's path within the archive.
  • Name
    shaColumn
    Type
    string · optional
    Description
    Optional column receiving the entry's sha256 hex.
  • Name
    sizeColumn
    Type
    string · optional
    Description
    Optional column receiving the entry's size in bytes (INTEGER).
  • Name
    mimeColumn
    Type
    string · optional
    Description
    Optional column receiving the entry's guessed MIME type.
  • Name
    parentShaColumn
    Type
    string · optional
    Description
    Optional column receiving the sha256 of the parent .eml for an extracted attachment row (empty for top-level entries). Wires the intake output to the documents table's `parent_sha256` column for email attachment linkage.
  • Name
    parentDocKeyColumn
    Type
    string · optional
    Description
    Optional column receiving the PARENT's `doc_key` for an extracted attachment row (empty for top-level entries) — the stable attachment→email edge. Distinct from `#parentShaColumn`, which carries the parent's CONTENT hash: byte-identical emails (the same message in several custodians' mailboxes) share a sha256, so it cannot identify WHICH parent row a child belongs to. The parent's doc key is deterministic and unique per row, so it can.
  • Name
    docKeyColumn
    Type
    string · optional
    Description
    Optional column receiving the deterministic per-entry doc key `"{uploadRowId`:{entryIndex}"} (attachments: `"{uploadRowId`:{entryIndex}:a{n}"}). This is the crash-safe idempotency key: a resume/retry reconciles the boundary chunk against existing values of this column and skips any already-inserted key, so a crash between an insert and the offset write re-inserts nothing.
  • Name
    intakeUploadIdColumn
    Type
    string · optional
    Description
    Optional column receiving the LIFECYCLE ROW ID of the archive this row expanded from — the durable per-archive join key, written on parents and attachments alike (children are built by the same method, so they inherit it without a second code path). Distinct from `#docKeyColumn`, which embeds the same id as the first segment of a composite text key: parsing an id back out of a doc key is a string operation on every row, whereas its own column answers "which documents came from this zip?" directly.
  • Name
    uploadKeyColumn
    Type
    string · optional
    Description
    Optional SECOND column receiving the same archive row id as `#intakeUploadIdColumn`, written in the same pass onto the same rows. Two columns for one value is not duplication for its own sake: on the FirstPass documents table the older column is unindexed and can never become indexed (column provisioning is additive by name — it adds what is missing and cannot retype or re-index what is there), while a filter over "everything that came from this upload" must run against an INDEXED column or it scans the whole corpus. Writing both here is what keeps the indexed twin populated from ingestion onward, instead of only from whenever a backfill last ran.
  • Name
    extraStatic
    Type
    object · optional
    Description
    Optional fixed columns written on every row: column name → value template resolved from the tool's `@@{...`} invocation variables (already substituted by the parser).