PIMP — Polling Inverse
Messaging Protocol

Conventions & Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 / RFC 8174.

Motivation

Serverless platforms often restrict response bodies to valid UTF-8 text strings. PIMP enables binary delivery over these transports using chunked polling, with an optimized "Text Pass-through" mode to avoid unnecessary encoding overhead for text-based streams.

Encoding Strategies

To balance platform safety with performance, PIMP supports two encoding modes defined at the job level via contentEncoding in the Metadata Chunk.

1. Base64 Mode (Binary-Safe)

Used for non-textual data (images, PDFs, protobufs). Each data chunk value is a standard Base64-encoded string (RFC 4648 §4). The consumer MUST decode each chunk value before appending to the output buffer.

2. Identity Mode (Text Pass-through)

Used for textual data (JSON, Markdown, CSV). Each data chunk value is a raw UTF-8 string. No decoding is required. This avoids the ~33% bandwidth overhead and the computational cost of double-parsing on the consumer.

If a chunk value contains characters outside the valid UTF-8 range, the sender MUST fall back to Base64 mode for the entire job. Partial mode switching within a job is not permitted.

Metadata Chunk (Index 0)

The first chunk (index 0) is reserved for job configuration. It MUST be written to the buffer immediately upon job start. Its value is a JSON object with the following fields:

// Example Metadata Chunk (index 0)
{
  "contentType": "application/json",
  "contentEncoding": "identity",
  "compression": "none",
  "contentLength": 48210,
  "meta": { "requestId": "abc-123" }
}

Protocol Envelope

Every chunk stored in the buffer MUST be wrapped in the following envelope structure:

{
  "jobId":     string,  // UUIDv4 — unique per job
  "index":     number,  // 0-based sequential chunk index
  "value":     string,  // chunk payload (metadata JSON for 0, encoded data for 1+)
  "done":      boolean, // true on the final chunk only
  "error":     string | null, // non-null if job failed — see §9
  "createdAt": number   // Unix epoch milliseconds
}

Field Rules

• index MUST be monotonically increasing with no gaps. Index 0 is always the Metadata Chunk.
• done MUST be false on all chunks except the last. The final chunk MAY have an empty value.
• error MUST be null on all non-error chunks. When non-null, the consumer MUST treat the job as failed.
• value for index 0 is a JSON-serialized Metadata object (§4). For index ≥ 1, it is an encoded data segment.

Job Creation & Handoff

The consumer is responsible for creating the job. It generates a jobId (UUIDv4) and passes it — along with the buffer's write URL — to the sender as part of the trigger request.

Trigger Request

The consumer initiates work by calling the sender's endpoint with at minimum:

{
  "jobId":    string,  // generated by the consumer
  "writeUrl": string,  // buffer endpoint the sender will POST chunks to
  "writeKey": string,  // short-lived auth token scoped to this job
  "payload":  object   // application-specific input (e.g. LLM prompt)
}

The sender MUST respond immediately (HTTP 202 Accepted) to acknowledge the job, then begin producing chunks asynchronously. The consumer does not wait for the sender to finish — it begins polling the buffer for chunks.

Sequence

  Consumer                 Sender                  Buffer
     │                        │                        │
     │── POST trigger ───────▸│                        │
     │◂── 202 Accepted ──────│                        │
     │                        │── POST chunk 0 ──────▸│
     │── GET /pimp/{id} ────────────────────────────▸│
     │◂── { chunks: [0] } ──────────────────────────│
     │                        │── POST chunk 1..N ──▸│
     │── GET /pimp/{id} ────────────────────────────▸│
     │◂── { chunks: [1..N] } ───────────────────────│
     │                        │── POST final (done) ─▸│
     │── GET /pimp/{id} ────────────────────────────▸│
     │◂── { chunks: [done] } ───────────────────────│
     ▼                        ▼                        ▼
    

Write API (Sender → Buffer)

The sender pushes chunks to the buffer store via HTTP POST. This is the counterpart to the read/poll API in §9.

Write Request

POST /pimp/{jobId}/chunks
Authorization: Bearer {writeKey}
Content-Type: application/json

{
  "chunks": [
    /* one or more envelope objects (§5) */
  ]
}

The buffer store MUST validate that:

• The writeKey is valid and scoped to the given jobId.
• Each chunk's jobId matches the URL parameter.
• Chunk indices have not already been written (idempotency — see below).

Write Response

// Success
201 Created
{ "written": 3 }  // number of new chunks accepted

// Duplicate chunks (already stored) — not an error
200 OK
{ "written": 0, "duplicates": 3 }

// Auth failure
401 Unauthorized

Idempotency

If the sender retries a write (e.g. after a network timeout), the buffer MUST accept the request without error. Chunks with an index that already exists for the job MUST be silently ignored. This makes write operations safe to retry.

Sender Behavior

Immediate Metadata Flush

The Metadata Chunk (index 0) MUST NOT be buffered. It MUST be written to the buffer immediately upon job creation, before any data chunks, to allow the consumer to negotiate decoding strategies.

Data Batching

Subsequent chunks (index ≥ 1) are buffered locally and flushed to the buffer store when either condition is met:

• The number of buffered chunks reaches sender.maxChunksPerBatch, OR
• sender.flushIntervalMs milliseconds have elapsed since the last flush.

End of Stream

After the last data segment, the sender MUST write a final chunk with done: true. If the final chunk carries data, done is set on that chunk. Otherwise, an empty sentinel chunk (value: "", done: true) MUST be appended.

Max Chunk Size

Individual chunk value strings SHOULD NOT exceed 256 KiB (262,144 bytes) to remain within common platform limits. Implementations MAY support a configurable ceiling.

Consumer Behavior & Polling API

Poll Request

The consumer requests chunks by providing jobId and the next expected index:

GET /pimp/{jobId}?from={index}

The buffer store MUST return all available chunks with index >= from, ordered by index. If no new chunks are available, the store MUST return an empty array with HTTP 200.

Poll Response

{
  "chunks": [ /* array of envelope objects */ ],
  "nextIndex": number  // = max(returned index) + 1
}

Polling Cadence

The consumer begins polling at consumer.pollIntervalMs. Implementations SHOULD use exponential backoff when receiving consecutive empty responses, up to a maximum of consumer.maxPollIntervalMs. Upon receiving data, the interval MUST reset to consumer.pollIntervalMs.

Completion

When the consumer receives a chunk with done: true, it MUST stop polling. The reassembled payload is the concatenation of all value fields from index 1 through the final chunk, decoded according to contentEncoding, then decompressed per compression.

Job Lifecycle & States

A job transitions through the following states:

  ┌──────────┐    index 0 written     ┌──────────┐
  │ CREATED  │ ──────────────────▸ │ ACTIVE   │
  └──────────┘                        └────┬─────┘
                                           │
                          ┌────────────────┬┴────────────────┐
                          ▼                ▼                  ▼
                   ┌────────────┐   ┌────────────┐   ┌────────────┐
                   │ COMPLETED  │   │ FAILED     │   │ EXPIRED    │
                   └────────────┘   └────────────┘   └────────────┘
                   done: true       error != null    ttl exceeded
    

Error Handling

Sender-Side Failure

If the sender encounters an unrecoverable error, it MUST write a final chunk with done: true and error set to a human-readable message. The value field MAY be empty.

// Error sentinel chunk
{
  "jobId": "550e8400-...",
  "index": 4,
  "value": "",
  "done": true,
  "error": "upstream timeout after 30s",
  "createdAt": 1713600000000
}

Consumer-Side Resilience

• If the consumer receives an HTTP error (5xx) from the buffer store, it SHOULD retry with exponential backoff before treating the job as failed.
• If no new chunks arrive for longer than consumer.stallTimeoutMs, the consumer SHOULD treat the job as failed with a timeout error.
• Duplicate chunks (same index received twice) MUST be silently discarded.
• Out-of-order chunks MUST be buffered and reassembled in index order.

Missing Chunks

If the consumer detects a gap (e.g. receives index 5 without having index 4), it MUST NOT advance reassembly past the gap. It SHOULD continue polling with from set to the missing index.

Configuration & Defaults

Implementations MUST provide these defaults to ensure out-of-the-box functionality. All timing values are integers in milliseconds.

Security Considerations

Write Authentication

The buffer's write endpoint MUST require a writeKey that is scoped to a single jobId. The consumer generates this key at job creation time and passes it to the sender in the trigger request. The key SHOULD be a cryptographically random token (≥ 32 bytes) or a signed JWT with a jobId claim and a short expiry (≤ job.ttlMs).

Read Access

The poll endpoint MAY be unauthenticated if jobId values are sufficiently unguessable (UUIDv4 provides 122 bits of entropy). For higher-security deployments, implementations SHOULD require a separate read token or tie access to the originating session.

Transport

All endpoints (trigger, write, poll) MUST be served over HTTPS. Implementations MUST NOT transmit writeKey values over unencrypted channels.

Payload Size Limits

Buffer implementations SHOULD enforce a maximum total job size (e.g. 50 MiB) and a maximum chunk count to prevent abuse. Senders exceeding these limits SHOULD receive a 413 Payload Too Large response.

End-to-End Examples

Example A — Identity / JSON Stream

Sender streams a JSON array of records to a consumer.

// Chunk 0 — Metadata (written immediately)
{ "jobId": "a1b2c3...", "index": 0,
  "value": "{\"contentType\":\"application/json\",\"contentEncoding\":\"identity\",\"compression\":\"none\"}",
  "done": false, "error": null, "createdAt": 1713600000000 }

// Chunk 1 — Data
{ "jobId": "a1b2c3...", "index": 1,
  "value": "[{\"id\":1,\"name\":\"Alice\"},{\"id\":2,\"name\":",
  "done": false, "error": null, "createdAt": 1713600000100 }

// Chunk 2 — Final data
{ "jobId": "a1b2c3...", "index": 2,
  "value": "\"Bob\"}]",
  "done": true, "error": null, "createdAt": 1713600000200 }

// Consumer reassembles: value[1] + value[2]
// → [{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]

Example B — Base64 / Binary Image

Sender streams a small PNG to a consumer.

// Chunk 0 — Metadata
{ "jobId": "d4e5f6...", "index": 0,
  "value": "{\"contentType\":\"image/png\",\"contentEncoding\":\"base64\",\"compression\":\"none\",\"contentLength\":1024}",
  "done": false, "error": null, "createdAt": 1713600001000 }

// Chunk 1 — Base64-encoded segment
{ "jobId": "d4e5f6...", "index": 1,
  "value": "iVBORw0KGgoAAAANSUhEUgAA...",
  "done": false, "error": null, "createdAt": 1713600001050 }

// Chunk 2 — Final segment
{ "jobId": "d4e5f6...", "index": 2,
  "value": "AAAElFTkSuQmCC",
  "done": true, "error": null, "createdAt": 1713600001100 }

// Consumer reassembles: base64Decode(value[1] + value[2])
// → <binary PNG bytes>