# LogTap — full documentation (English & 中文)

> LogTap is an open-source (MIT) high-throughput observability data gateway: natively compatible with the Sentry SDK reporting protocol, combining structured log search, event tracking with funnel & retention analytics, health probing, and multi-channel alerting. Built on Go + NSQ, it keeps single-node write P95 latency steadily under 2 ms and deploys privately with one `docker compose` command.

LogTap is built on Go + Gin + NSQ: a stateless HTTP ingest gateway handles auth, deserialization, and queue publishing only, while NSQ decouples it from batch consumer workers that write to PostgreSQL / TimescaleDB (open-source edition) or ClickHouse (enterprise / cloud edition). Official single-node benchmarks (32-core/64 GB, co-located) show P95 write latency of 1.7–1.9 ms at 10–120 concurrent users with 5,600+ EPS, peaking at 11,200 EPS.

This site is available in Chinese (https://logtap-website.autotask.run/) and English (https://logtap-website.autotask.run/en/). Every documentation page below is also served as raw Markdown at the same URL with a `.md` suffix, and `llms-full.txt` next to this file contains all documentation in both languages.

This file concatenates the landing-page summary and every documentation page (English first, then 中文), separated by `========` markers.


========

# Benchmarks & quick facts (landing page)

## Benchmarks (open-source edition, single node)

| Concurrency | Actual throughput | P95 | P99 |
| --- | --- | --- | --- |
| 10 VU | 467 EPS | 1.8 ms | 2.4 ms |
| 40 VU | 1,867 EPS | 1.9 ms | 3.3 ms |
| 120 VU | 5,600 EPS | 1.7 ms | 2.8 ms |
| 160 VU (knee point) | 7,463 EPS | 163.9 ms | 556.2 ms |

- Peak ingest: 10,000–20,000 EPS (open source) vs 100,000–300,000+ EPS (enterprise/cloud)
- Storage compression: 1:1.5–1:2 (PostgreSQL row store) vs 1:5–1:10 (ClickHouse column store, LZ4/ZSTD)
- Analytics over 100M+ events: 3–10 s (open source) vs 0.1–0.5 s (enterprise, vectorized engine)

## Quick start

```bash
git clone https://github.com/aak1247/logtap.git
cd logtap/deploy
bash ../scripts/gen-auth-secret.sh
docker compose up -d
```

========

# Overview (English)

---
title: Overview
description: What LogTap is and what it does — Sentry-compatible error monitoring plus structured logs, event analytics, and alerting in one observability gateway.
---

# Overview

LogTap is an open-source (MIT) high-throughput observability data gateway (Go + Gin + NSQ + TimescaleDB/Postgres) that brings error monitoring, structured logs, event tracking, and multi-channel alerting together in a single service, with a React console for querying and analysis.

## What you can do with it

- **Error monitoring**: natively compatible with the Sentry `store` / `envelope` reporting protocols — existing Sentry SDKs work unchanged; just point the DSN at LogTap
- **Structured logs**: a unified HTTP endpoint with batch ingestion (single object or JSON array) and gzip; keyword search runs on pg_trgm trigram indexes, listings on time-descending indexes
- **Event analytics**: report via `/track/` and the console gives you DAU/MAU, top events, property breakdowns, retention cohorts, and custom funnels
- **Health probing & alerting**: built-in active HTTP/TCP probes; alerts via WeCom, DingTalk, and Feishu webhooks, SMTP email, and Alibaba/Tencent Cloud SMS, with deduplication and noise suppression
- **GeoIP resolution**: an offline-bundled MaxMind GeoLite2 database resolves reporting IPs to country, city, and ASN with no external lookups

## Architecture at a glance

Ingestion and storage are decoupled by NSQ: the gateway is stateless and horizontally scalable, and slow downstream queries never block reporting.

1. **Client reporting** — a Sentry SDK or any HTTP client sends POSTs; single items and batches are accepted, batches may be gzipped
2. **Gateway validates and enqueues** — the Go + Gin gateway only deserializes, authenticates, and publishes to NSQ; it never touches the database, keeping latency stable
3. **Queue absorbs spikes** — events and logs use separate topics and channels, so database jitter does not affect the ingest path
4. **Batch writes** — consumers batch-write to TimescaleDB every 500 rows or 20 ms, trading throughput for IO efficiency

## Two storage engines, pick by scale

| Dimension | Open source (PostgreSQL / TimescaleDB) | Enterprise / Cloud (ClickHouse engine) |
| --- | --- | --- |
| Peak ingest EPS | 10,000 ~ 20,000 | 100,000 ~ 300,000+ |
| Write P95 latency | 1.7 ~ 1.9 ms (constant within 120 VU) | < 1.5 ms (distributed parallel ingest) |
| Storage compression | 1 : 1.5 ~ 1 : 2 (row store, B-tree) | 1 : 5 ~ 1 : 10 (column store, LZ4/ZSTD) |
| 100M+ event analytics | 3 ~ 10 s | 0.1 ~ 0.5 s (vectorized engine) |
| Operational complexity | Minimal (Postgres + Redis + NSQ) | Distributed (ClickHouse cluster) |

Both editions share the same API and console, so you can migrate when scale demands it. Official single-node benchmarks (32-core/64 GB, co-located): P95 write latency steady at 1.7 ~ 1.9 ms across 10 ~ 120 concurrent users, 5,600+ EPS sustained, 11,200 EPS peak.

## Next steps

- [Quick start](https://logtap-website.autotask.run/en/docs/quickstart/): three commands to run it locally
- [Deployment](https://logtap-website.autotask.run/en/docs/deployment/): production setups and environment variables
- [Ingest API](https://logtap-website.autotask.run/en/docs/ingest-api/): the HTTP protocol for logs and events
- [Sentry compatibility](https://logtap-website.autotask.run/en/docs/sentry-compat/): zero-change adoption of existing Sentry SDKs
- [SDKs](https://logtap-website.autotask.run/en/docs/sdks/): JavaScript / Go / Flutter clients

========

# Quick Start (English)

---
title: Quick Start
description: Bring up LogTap with three Docker Compose commands and send your first structured log.
---

# Quick Start

## Prerequisites

- Docker and Docker Compose
- (Production) a server with at least 2 vCPU / 4 GB RAM

## 1. Clone and start

```bash
git clone https://github.com/aak1247/logtap.git
cd logtap/deploy

# Generate AUTH_SECRET (base64, >= 32 bytes decoded)
bash ../scripts/gen-auth-secret.sh

# Start — ships with Postgres/TimescaleDB, Redis, NSQ
docker compose up -d
```

Once started:

- Gateway & console: `http://localhost:8080`
- NSQ Admin: `http://localhost:4171`
- Postgres: `localhost:5432` (default database `logtap`)

The `docker-compose.yml` includes TimescaleDB, Redis, NSQ (lookupd/nsqd/admin), and the LogTap gateway — ready in about a minute. You can also try the [online demo console](https://logtap.autotask.run/) first.

## 2. Access with auth enabled

Once `AUTH_SECRET` is set:

- The console requires login
- Reporting must carry a project key: `X-Project-Key: pk_...` (create one on the project page), or the Sentry SDK's `sentry_key`

On PowerShell, use `powershell -ExecutionPolicy Bypass -File scripts/gen-auth-secret.ps1`, or `task auth:secret` from the Taskfile.

## 3. Send your first log

Create a project in the console, then grab the project ID and the `pk_`-prefixed project key:

```bash
curl -X POST http://localhost:8080/api/1/logs/ \
  -H "Content-Type: application/json" \
  -H "X-Project-Key: pk_xxx" \
  -d '{
    "level": "info",
    "message": "hello logtap",
    "user": { "id": "u1" },
    "fields": { "env": "dev" }
  }'
```

A `202 Accepted` response means the event is queued. Back in the console you will see the log, and from there you can keep exploring:

- Search: keyword search on the logs page
- Tracking: POST events to `/api/1/track/` and inspect DAU, funnels, and retention
- Errors: point any Sentry SDK's DSN at LogTap — see [Sentry compatibility](https://logtap-website.autotask.run/en/docs/sentry-compat/)

## Optional: enable GeoIP

Requires a MaxMind GeoLite2 download key: set `MAXMIND_LICENSE_KEY` in `.env`, and Compose downloads and caches the mmdb files into the `/data/geoip/` volume for country/city/ASN breakdowns.

## Next steps

- [Deployment](https://logtap-website.autotask.run/en/docs/deployment/): non-Docker setups, systemd, and production advice
- [Ingest API](https://logtap-website.autotask.run/en/docs/ingest-api/): the full data model and batch reporting
- [SDKs](https://logtap-website.autotask.run/en/docs/sdks/): report from application code with an SDK

========

# Ingest API (English)

---
title: Ingest API
description: The HTTP reporting protocol for custom logs and tracking events — endpoints, auth, data model, batching, and gzip.
---

# Ingest API

LogTap's client SDKs (Browser/Node, Go, Flutter) all use the HTTP endpoints and JSON data model below for "log" and "event/tracking" reporting. Languages without an SDK can call the HTTP API directly.

## Conventions

- **Base URL**: your LogTap gateway address, e.g. `http://localhost:8080`
- **Project ID**: visible on the project page in the console; used as a path parameter `/api/:projectId/...`
- **Auth** (enforced once `AUTH_SECRET` is set): reporting must carry a project key — any of:
  - `X-Project-Key: pk_...`
  - `Authorization: Bearer pk_...`
  - the Sentry SDK's `X-Sentry-Auth` (`sentry_key=...`)
- **Batching**: the same endpoint accepts a single object or a JSON array
- **Compression**: `Content-Encoding: gzip` is supported
- **Success response**: `202 Accepted` (the gateway only validates and enqueues; consumers write to the database asynchronously)

## 1. Structured logs

Endpoint: `POST /api/:projectId/logs/`

```json
{
  "level": "info",
  "message": "hello",
  "timestamp": "2025-01-01T00:00:00Z",
  "device_id": "d1",
  "trace_id": "t1",
  "span_id": "s1",
  "fields": { "k": "v" },
  "tags": { "env": "prod" },
  "user": { "id": "u1" },
  "contexts": { "app": { "version": "1.2.3" } }
}
```

Field notes:

- `message`: required
- `level`: suggested values `debug / info / warn / error / fatal`; defaults to `info` server-side when empty
- `timestamp`: RFC3339; the server fills in the current UTC time when omitted
- `user.id`: used for unique-user metrics (DAU/MAU, funnels); falls back to `device_id` when absent
- `fields`: custom structured fields — put business dimensions here
- `tags` / `contexts`: labels and app context stored with the log and filterable in the console

Batch example (same endpoint, JSON array):

```json
[
  { "level": "info", "message": "hello-1", "user": { "id": "u1" }, "fields": { "k": "v1" } },
  { "level": "info", "message": "hello-2", "user": { "id": "u1" }, "fields": { "k": "v2" } }
]
```

## 2. Events / tracking

Endpoint: `POST /api/:projectId/track/`

```json
{
  "name": "signup",
  "timestamp": "2025-01-01T00:00:00Z",
  "user": { "id": "u1" },
  "device_id": "d1",
  "properties": { "plan": "pro", "ab": "b" }
}
```

- `name`: event name, required
- `properties`: event properties, used for funnel filtering and property breakdowns

Storage rule: events are written to the `logs` table (`level = "event"`, `message = name`, `fields = properties`), so event analytics only count `level="event"` rows and stay unpolluted by regular logs.

Batch example:

```json
[
  { "name": "signup", "user": { "id": "u1" }, "properties": { "plan": "pro" } },
  { "name": "purchase", "user": { "id": "u1" }, "properties": { "sku": "s1" } }
]
```

## 3. Search and queries

- Log/event queries, keyword full-text search, time-series aggregation, funnels, and retention are all done through the console
- Sentry error reporting uses its own protocol — see [Sentry compatibility](https://logtap-website.autotask.run/en/docs/sentry-compat/)

## Recommendations

- Keep "logs" and "tracking events" separate: logs go to `/logs/`, events to `/track/`
- For high-volume scenarios prefer batch arrays + gzip to cut request counts and network overhead
- Put business dimensions into `fields` / `properties` — console filtering and funnels are built on structured fields

========

# Sentry Compatibility (English)

---
title: Sentry Compatibility
description: Adopt LogTap with zero code changes — swap one line of DSN. Store / Envelope protocols implemented natively, with historical data migration.
---

# Sentry Compatibility

LogTap natively implements the Sentry Store and Envelope reporting protocols, taking over the `/api/:projectId/store/` and `/api/:projectId/envelope/` endpoints. SDKs like `@sentry/browser` and `@sentry/node` work without swapping libraries — stack traces, breadcrumbs, tags, and context are stored as-is.

## How to adopt

Replace the DSN host in your existing Sentry SDK initialization with your LogTap gateway:

```js
// Before
Sentry.init({
  dsn: "https://examplePublicKey@o0.ingest.sentry.io/0",
});

// After: change only the host and project ID, keep everything else
Sentry.init({
  dsn: "https://examplePublicKey@your-logtap-host/api/1/",
});
```

Key points:

- The project ID in the DSN maps to the project ID in the LogTap console
- Auth reuses Sentry's `sentry_key`; `X-Project-Key` works too
- SDK-side sampling, environment, tags, and user-context configuration all keep working

## Data handling

- Both Store and Envelope protocols are implemented per the Sentry spec; events land in the same storage as logs and are queried in one console
- Reporting IPs are resolved to country, city, and ASN via the offline GeoLite2 database (when GeoIP is enabled)

## Migrating historical data from Sentry

LogTap ships a migration tool that imports historical events from an existing Sentry instance, so switching over doesn't mean discarding old data. See the tool and its usage notes in the `logtap` source repository.

## FAQ

- **Do I need to change business code?** No — changing the DSN is enough; the SDK and reporting path stay untouched.
- **How complete is the compatibility?** The Store and Envelope endpoints are implemented natively per the Sentry protocol and cover what SDKs report by default. If you hit a missing advanced feature, please open a GitHub issue.
- **Can I still use custom logs and events?** Yes — Sentry errors and custom data share the same project system. See the [Ingest API](https://logtap-website.autotask.run/en/docs/ingest-api/).

========

# SDKs (English)

---
title: SDKs
description: Official LogTap client SDKs — JavaScript/TypeScript (Browser/Node), Go, and Flutter, with batch buffering, gzip, and automatic retries.
---

# SDKs

LogTap ships official SDKs for three platforms that report "structured logs" and "events" to the gateway, wrapping batch buffering, gzip compression, retry-on-failure, and context merging. Other languages can call the HTTP API directly — see the [Ingest API](https://logtap-website.autotask.run/en/docs/ingest-api/).

## Common configuration

- `baseUrl`: gateway address
- `projectId`: project ID
- `projectKey`: required when `AUTH_SECRET` is enabled (`X-Project-Key: pk_...`)
- `gzip`: optional; automatically downgrades on platforms that don't support it

Shared capabilities: an internal queue flushes on `flushIntervalMs` or as soon as `minBatchSize` items accumulate; failures retry with exponential backoff (capped at 30 s); `globalTags` / `globalContexts` merge into every payload; `identify()` sets user context; a `beforeSend` hook supports redaction or dropping.

## JavaScript / TypeScript (Browser / Node)

Package: `logtap-sdk` (repo directory `sdks/js/logtap/`)

```bash
npm i logtap-sdk
```

```ts
import { LogtapClient } from "logtap-sdk";

const client = new LogtapClient({
  baseUrl: "https://your-logtap-host",
  projectId: 1,
  projectKey: "pk_xxx",
  gzip: true,
  flushIntervalMs: 5000, // max delay (also sends when the timer fires)
  minBatchSize: 20,      // send immediately once this many items queue up
  immediateEvents: ["purchase", "payment_succeeded"],
  globalTags: { env: "prod" },
  globalContexts: { app: { version: "1.2.3" } },
});

client.captureBrowserErrors(); // optional in browsers
client.captureNodeErrors();    // optional in Node

client.identify("u1", { plan: "pro" });
client.info("hello", { k: "v" }, { tags: { req: "1" } });
client.track("signup", { from: "landing" });

await client.close();
```

## Go

```bash
go get github.com/aak1247/logtap/sdks/go/logtap
```

```go
client, err := logtap.NewClient(logtap.ClientOptions{
	BaseURL:    "https://your-logtap-host",
	ProjectID:  1,
	ProjectKey: "pk_xxx",
	Gzip:       true,
	GlobalTags: map[string]string{"env": "prod"},
	GlobalContexts: map[string]any{
		"app": map[string]any{"version": "1.2.3"},
	},
})
if err != nil {
	panic(err)
}
defer client.Close(context.Background())
defer client.Recover(true) // optional: report panics

client.Identify("u1", map[string]any{"plan": "pro"})
client.Info("hello", map[string]any{"k": "v"}, nil)
client.Track("signup", map[string]any{"from": "landing"}, nil)
```

## Flutter

Package: `logtap_flutter` (repo directory `sdks/flutter/logtap_flutter/`)

```dart
final client = await LogtapClient.create(
  const LogtapClientOptions(
    baseUrl: "https://your-logtap-host",
    projectId: 1,
    projectKey: "pk_xxx",
    gzip: true, // Web automatically downgrades to non-gzip
    globalTags: {"env": "prod"},
    globalContexts: {"app": {"version": "1.2.3"}},
  ),
);

client.captureFlutterErrors(); // optional: capture unhandled errors

client.identify("u1", {"plan": "pro"});
client.info("hello", {"k": "v"});
client.track("signup", {"from": "landing"});

await client.close();
```

## Other languages

You don't need an SDK — report over plain HTTP (see the [Ingest API](https://logtap-website.autotask.run/en/docs/ingest-api/)). A Python example:

```python
import requests

requests.post(
    "https://your-logtap-host/api/1/logs/",
    headers={"X-Project-Key": "pk_xxx"},
    json=[{"level": "info", "message": "hello", "user": {"id": "u1"}}],
)
```

More language integration examples (Python, Java, PHP, Ruby, .NET) live in the repository under `docs/integrations/`.

## What about error monitoring?

For error monitoring prefer the Sentry SDK (zero changes except the DSN — see [Sentry compatibility](https://logtap-website.autotask.run/en/docs/sentry-compat/)); the per-platform `capture*Errors()` helpers fit lightweight scenarios where no Sentry SDK is present.

========

# Deployment (English)

---
title: Deployment
description: Deploying LogTap to production — Docker Compose, non-Docker (systemd), environment variables, and production advice.
---

# Deployment

This guide covers two deployment paths: Docker Compose for local/testing, and either non-Docker (systemd) or custom containerized setups for production servers.

## Dependencies

- PostgreSQL / TimescaleDB (event and log storage)
- NSQ (async writes and spike absorption)
- (Optional) Redis (some metrics/caching capabilities, depending on project configuration)

## Option 1: Docker Compose

```bash
git clone https://github.com/aak1247/logtap.git
cd logtap/deploy
bash ../scripts/gen-auth-secret.sh   # generate AUTH_SECRET
docker compose up -d
```

The Compose file includes TimescaleDB, Redis, NSQ (lookupd/nsqd/admin), and the LogTap gateway. GeoIP is optional: set `MAXMIND_LICENSE_KEY` and the mmdb files are downloaded and cached into the `/data/geoip/` volume. If the mmdb files exist and `GEOIP_CITY_MMDB` / `GEOIP_ASN_MMDB` are not explicitly set, the startup script automatically uses `/data/geoip/GeoLite2-City.mmdb` and `/data/geoip/GeoLite2-ASN.mmdb`.

## Option 2: non-Docker (production servers)

### Build the gateway

Requires Go 1.22+:

```bash
go build -o gateway ./cmd/gateway
```

Upload the `gateway` binary to your server (e.g. `/opt/logtap/gateway`).

### Minimal environment variables

See `.env.example` in the repository for the full reference; the common ones:

| Variable | Description |
| --- | --- |
| `HTTP_ADDR` | HTTP listen address, e.g. `:8080` |
| `NSQD_ADDRESS` | NSQ address, e.g. `127.0.0.1:4150` |
| `RUN_CONSUMERS` | Whether to run DB-writing consumers (`true` in production) |
| `POSTGRES_URL` | Required when `RUN_CONSUMERS=true`, e.g. `postgres://logtap:logtap@127.0.0.1:5432/logtap?sslmode=disable` |
| `AUTH_SECRET` | Enables console login and reporting auth (recommended) |

On startup the gateway runs GORM `AutoMigrate` to create tables and indexes automatically.

### systemd (optional templates)

Templates live in `deploy/systemd/`:

- `logtap-gateway.service` — gateway
- `nsqd.service` — NSQ
- `logtap-alert-worker.service` — optional: standalone alert delivery worker

Suggested approach: create `/opt/logtap/`, place the binary and `.env` in fixed locations, then `systemctl enable --now logtap-gateway`.

Alert delivery worker: if you don't enable `RUN_ALERT_WORKER=true` (the in-gateway worker), a standalone `alert-worker` process can consume `alert_deliveries`; see `deploy/alert-worker.env.example` for its environment variables.

## Console (web)

The console is a separate frontend (`web/`, Vite + React + Tailwind); the production build lands in `web/dist/` and is served by Nginx or any static host:

```bash
cd web
npm install
npm run build
```

For production, use Nginx/Traefik for HTTPS and reverse-proxy `/api/` to the gateway.

## Production advice

- **Reverse proxy**: Nginx/Traefik for HTTPS, `/api/` proxied to the gateway
- **Persistence**: plan the Postgres data disk, NSQ directories, and log retention up front
- **Resource isolation**: run DB-writing consumers separately from the gateway (different processes/instances) so slow queries never affect reporting
- **Alert worker**: under heavy traffic use the standalone `alert-worker` process to avoid competing with the gateway

========

# 概览（中文）

---
title: 概览
description: LogTap 是什么、能做什么：Sentry 协议兼容的异常监控 + 结构化日志 + 事件埋点 + 告警，一体化可观测性网关。
---

# 概览

LogTap 是一个采用 MIT 协议开源的高吞吐可观测性数据网关（Go + Gin + NSQ + TimescaleDB/Postgres），把异常监控、结构化日志、事件埋点与多通道告警收口到同一个服务，并提供 React 控制台用于查询与分析。

## 你能用它做什么

- **异常监控**：原生兼容 Sentry 的 `store` / `envelope` 上报协议，现有 Sentry SDK 不换库，改一行 DSN 即可接入
- **结构化日志**：统一 HTTP 接口批量上报（单条对象或 JSON 数组），支持 gzip；关键词检索走 pg_trgm 三元组索引，列表查询走时间倒序索引
- **事件分析**：`/track/` 埋点上报后，控制台直接出 DAU/MAU、事件 Top、属性分布、留存队列和自定义漏斗
- **健康巡检与告警**：内置 HTTP/TCP 主动拨测；告警支持企业微信、钉钉、飞书 Webhook、邮件 SMTP 与阿里云/腾讯云短信，带去重与降噪
- **GeoIP 解析**：离线打包 MaxMind GeoLite2，把上报 IP 解析到国家、城市与 ASN，无外部查询依赖

## 架构一览

写入和落库被 NSQ 隔开：网关无状态可横向扩展，下游慢查询拖不垮上报链路。

1. **客户端上报** — Sentry SDK 或任意 HTTP 客户端发起 POST，单条与批量都收，批量可带 gzip
2. **网关校验入队** — Go + Gin 网关只做反序列化、鉴权和 NSQ 投递，不碰数据库，响应延迟稳定
3. **队列削峰** — 事件与日志分 topic 走不同 channel，数据库抖动不影响上报端
4. **批量落库** — 消费者按 500 条或 20ms 攒批写入 TimescaleDB，吞吐换 IO 效率

## 两种存储引擎，按量级选

| 维度 | 开源版（PostgreSQL / TimescaleDB） | 企业版 / 云端版（ClickHouse 引擎） |
| --- | --- | --- |
| 接入峰值 EPS | 10,000 ~ 20,000 | 100,000 ~ 300,000+ |
| 写入 P95 延迟 | 1.7 ~ 1.9 ms（120 VU 内恒定） | < 1.5 ms（分布式并行接入） |
| 存储压缩比 | 1 : 1.5 ~ 1 : 2（行存 B 树） | 1 : 5 ~ 1 : 10（列存 LZ4/ZSTD） |
| 亿级多维分析 | 3 ~ 10 s | 0.1 ~ 0.5 s（向量化引擎） |
| 运维复杂度 | 极简（Postgres + Redis + NSQ） | 分布式（ClickHouse 集群） |

两边 API 和控制台一致，量级上来了再迁移。官方单节点实测（32 核 64G 同机部署）：10 ~ 120 并发下 P95 写入延迟稳定在 1.7 ~ 1.9 ms，5,600+ EPS，峰值 11,200 EPS。

## 下一步

- [快速开始](https://logtap-website.autotask.run/docs/quickstart/)：三条命令在本地跑起来
- [部署指南](https://logtap-website.autotask.run/docs/deployment/)：生产环境部署与环境变量
- [上报接口](https://logtap-website.autotask.run/docs/ingest-api/)：日志与埋点的 HTTP 协议
- [Sentry 兼容](https://logtap-website.autotask.run/docs/sentry-compat/)：零改动接入现有 Sentry SDK
- [SDK](https://logtap-website.autotask.run/docs/sdks/)：JS / Go / Flutter 客户端

========

# 快速开始（中文）

---
title: 快速开始
description: 用 Docker Compose 三条命令把 LogTap 跑起来，发出第一条结构化日志。
---

# 快速开始

## 前置条件

- Docker 与 Docker Compose
- （生产环境）一台 2C4G 以上的服务器

## 1. 克隆并启动

```bash
git clone https://github.com/aak1247/logtap.git
cd logtap/deploy

# 生成 AUTH_SECRET（base64，解码后 >= 32 字节）
bash ../scripts/gen-auth-secret.sh

# 启动，自带 Postgres/TimescaleDB、Redis、NSQ
docker compose up -d
```

启动后：

- 网关与控制台：`http://localhost:8080`
- NSQ Admin：`http://localhost:4171`
- Postgres：`localhost:5432`（默认库 `logtap`）

`docker-compose.yml` 已包含 TimescaleDB、Redis、NSQ（lookupd/nsqd/admin）与 logtap 网关，约 1 分钟即可就绪。也可以先在[在线演示控制台](https://logtap.autotask.run/)体验。

## 2. 开启鉴权后的访问方式

设置了 `AUTH_SECRET` 后：

- 控制台需要登录
- 上报必须携带项目 Key：`X-Project-Key: pk_...`（项目页可创建），或 Sentry SDK 的 `sentry_key`

PowerShell 环境可用 `powershell -ExecutionPolicy Bypass -File scripts/gen-auth-secret.ps1`，或 Taskfile 的 `task auth:secret`。

## 3. 发出第一条日志

先在控制台创建项目，拿到项目 ID 与 `pk_` 开头的项目 Key：

```bash
curl -X POST http://localhost:8080/api/1/logs/ \
  -H "Content-Type: application/json" \
  -H "X-Project-Key: pk_xxx" \
  -d '{
    "level": "info",
    "message": "hello logtap",
    "user": { "id": "u1" },
    "fields": { "env": "dev" }
  }'
```

返回 `202 Accepted` 即入队成功。回到控制台即可看到这条日志，并可以继续尝试：

- 搜索：日志页关键词检索
- 埋点：`POST /api/1/track/` 上报事件后查看 DAU、漏斗与留存
- 异常：把任意 Sentry SDK 的 DSN 指到 LogTap，见[Sentry 兼容](https://logtap-website.autotask.run/docs/sentry-compat/)

## 可选：启用 GeoIP

需要 MaxMind GeoLite2 下载密钥：在 `.env` 中设置 `MAXMIND_LICENSE_KEY`，Compose 会把 mmdb 下载并缓存到 volume `/data/geoip/`，用于来源国家/城市/ASN 分布分析。

## 下一步

- [部署指南](https://logtap-website.autotask.run/docs/deployment/)：无 Docker 部署、systemd 与生产建议
- [上报接口](https://logtap-website.autotask.run/docs/ingest-api/)：完整的数据模型与批量上报
- [SDK](https://logtap-website.autotask.run/docs/sdks/)：在业务代码里用 SDK 上报

========

# 上报接口（中文）

---
title: 上报接口
description: LogTap 自定义日志与埋点事件的 HTTP 上报协议：端点、鉴权、数据模型、批量与 gzip。
---

# 上报接口

LogTap 的客户端 SDK（Browser/Node、Go、Flutter）统一使用如下 HTTP 接口与 JSON 数据模型进行「日志」与「埋点/事件」上报。没有 SDK 的语言直接按本文协议调用 HTTP 即可。

## 基本约定

- **Base URL**：你的 LogTap 网关地址，例如 `http://localhost:8080`
- **Project ID**：项目 ID（控制台项目页可见），作为路径参数 `/api/:projectId/...`
- **鉴权**（设置 `AUTH_SECRET` 后强制）：上报必须携带项目 Key，以下三种方式任选：
  - `X-Project-Key: pk_...`
  - `Authorization: Bearer pk_...`
  - Sentry SDK 的 `X-Sentry-Auth`（`sentry_key=...`）
- **批量**：同一路径支持「单条对象」或「JSON 数组」
- **压缩**：支持 `Content-Encoding: gzip`
- **成功响应**：`202 Accepted`（网关只做校验和入队，写库由消费者异步完成）

## 1. 结构化日志

端点：`POST /api/:projectId/logs/`

```json
{
  "level": "info",
  "message": "hello",
  "timestamp": "2025-01-01T00:00:00Z",
  "device_id": "d1",
  "trace_id": "t1",
  "span_id": "s1",
  "fields": { "k": "v" },
  "tags": { "env": "prod" },
  "user": { "id": "u1" },
  "contexts": { "app": { "version": "1.2.3" } }
}
```

字段说明：

- `message`：必填
- `level`：建议值 `debug / info / warn / error / fatal`，为空时服务端默认 `info`
- `timestamp`：RFC3339；不传则服务端补当前时间（UTC）
- `user.id`：用于用户去重口径（DAU/MAU、漏斗等）；不传时回退到 `device_id`
- `fields`：自定义结构化字段，建议把业务维度放这里
- `tags` / `contexts`：标签与应用上下文，随日志入库可在控制台过滤

批量上报示例（同一路径，JSON 数组）：

```json
[
  { "level": "info", "message": "hello-1", "user": { "id": "u1" }, "fields": { "k": "v1" } },
  { "level": "info", "message": "hello-2", "user": { "id": "u1" }, "fields": { "k": "v2" } }
]
```

## 2. 埋点/事件

端点：`POST /api/:projectId/track/`

```json
{
  "name": "signup",
  "timestamp": "2025-01-01T00:00:00Z",
  "user": { "id": "u1" },
  "device_id": "d1",
  "properties": { "plan": "pro", "ab": "b" }
}
```

- `name`：事件名，必填
- `properties`：事件属性，用于漏斗过滤与属性分布分析

服务端落库规则：事件写入 `logs` 表（`level = "event"`，`message = name`，`fields = properties`），因此事件分析只统计 `level="event"` 的记录，不会被普通日志污染。

批量上报示例：

```json
[
  { "name": "signup", "user": { "id": "u1" }, "properties": { "plan": "pro" } },
  { "name": "purchase", "user": { "id": "u1" }, "properties": { "sku": "s1" } }
]
```

## 3. 检索与查询

- 日志与事件查询、关键词全文检索、时序聚合、漏斗与留存分析均通过控制台完成
- Sentry 错误上报走独立协议，见[Sentry 兼容](https://logtap-website.autotask.run/docs/sentry-compat/)

## 建议

- 「日志」与「埋点事件」分开上报：日志走 `/logs/`，埋点走 `/track/`
- 高频场景优先使用批量数组 + gzip，减少请求数与网络开销
- 把业务维度放进 `fields` / `properties`，控制台的过滤与漏斗都基于结构化字段

========

# Sentry 兼容（中文）

---
title: Sentry 兼容
description: 零改动接入现有 Sentry SDK：改一行 DSN 即可，Store / Envelope 协议原生实现，历史数据可迁移。
---

# Sentry 兼容

LogTap 原生实现了 Sentry 的 Store 与 Envelope 上报协议，接管 `/api/:projectId/store/` 与 `/api/:projectId/envelope/` 两个端点。`@sentry/browser`、`@sentry/node` 等 SDK 不需要换库，异常堆栈、面包屑、标签与上下文原样入库。

## 接入方式

把现有 Sentry SDK 初始化代码中的 DSN 主机替换为 LogTap 网关地址即可：

```js
// 之前
Sentry.init({
  dsn: "https://examplePublicKey@o0.ingest.sentry.io/0",
});

// 之后：只改主机与项目 ID，其余不变
Sentry.init({
  dsn: "https://examplePublicKey@your-logtap-host/api/1/",
});
```

要点：

- DSN 中的项目 ID 对应 LogTap 控制台里的项目 ID
- 鉴权沿用 Sentry 的 `sentry_key`，也可以用 `X-Project-Key`
- SDK 端的采样、环境、标签、用户上下文等配置全部继续生效

## 数据处理

- Store / Envelope 两种协议都按 Sentry 协议实现，事件进入与日志相同的存储，控制台统一查询
- 上报 IP 经离线 GeoLite2 解析出国家、城市与 ASN（启用 GeoIP 时）

## 从 Sentry 迁移历史数据

LogTap 附带迁移工具，可以从现有 Sentry 实例导入历史事件，切换过程不需要丢弃旧数据。工具与使用说明见仓库 `logtap` 源码目录。

## 常见问题

- **需要改业务代码吗？** 不需要，改 DSN 即可；SDK 与上报链路零改动。
- **协议兼容度？** Store 与 Envelope 端点按 Sentry 协议原生实现，覆盖 SDK 默认上报所需的能力；如遇到高级特性缺失，欢迎在 GitHub 提 issue。
- **还能继续用自定义日志/埋点吗？** 可以，Sentry 异常与自定义数据是同一套项目体系，见[上报接口](https://logtap-website.autotask.run/docs/ingest-api/)。

========

# SDK（中文）

---
title: SDK
description: LogTap 官方客户端 SDK：JavaScript/TypeScript（Browser/Node）、Go、Flutter，批量缓冲、gzip、自动重试。
---

# SDK

LogTap 提供三端官方 SDK，用于向网关上报「结构化日志」与「埋点/事件」，统一封装批量缓冲、gzip 压缩、失败重试与上下文合并。其他语言直接按[上报接口](https://logtap-website.autotask.run/docs/ingest-api/)调用 HTTP 即可。

## 共同配置

- `baseUrl`：网关地址
- `projectId`：项目 ID
- `projectKey`：启用 `AUTH_SECRET` 时必填（`X-Project-Key: pk_...`）
- `gzip`：可选，不支持的平台自动降级

统一能力：内部队列按 `flushIntervalMs` 定时或攒够 `minBatchSize` 立即发送；失败自动指数退避重试（上限 30s）；`globalTags` / `globalContexts` 合并进每条 payload；`identify()` 设置用户上下文；`beforeSend` 钩子可用于脱敏或丢弃。

## JavaScript / TypeScript（Browser / Node）

包名：`logtap-sdk`（仓库目录 `sdks/js/logtap/`）

```bash
npm i logtap-sdk
```

```ts
import { LogtapClient } from "logtap-sdk";

const client = new LogtapClient({
  baseUrl: "https://your-logtap-host",
  projectId: 1,
  projectKey: "pk_xxx",
  gzip: true,
  flushIntervalMs: 5000, // 最大延迟（到时间也会发）
  minBatchSize: 20,      // 达到条数立即发
  immediateEvents: ["purchase", "payment_succeeded"],
  globalTags: { env: "prod" },
  globalContexts: { app: { version: "1.2.3" } },
});

client.captureBrowserErrors(); // browser 可选
client.captureNodeErrors();    // node 可选

client.identify("u1", { plan: "pro" });
client.info("hello", { k: "v" }, { tags: { req: "1" } });
client.track("signup", { from: "landing" });

await client.close();
```

## Go

```bash
go get github.com/aak1247/logtap/sdks/go/logtap
```

```go
client, err := logtap.NewClient(logtap.ClientOptions{
	BaseURL:    "https://your-logtap-host",
	ProjectID:  1,
	ProjectKey: "pk_xxx",
	Gzip:       true,
	GlobalTags: map[string]string{"env": "prod"},
	GlobalContexts: map[string]any{
		"app": map[string]any{"version": "1.2.3"},
	},
})
if err != nil {
	panic(err)
}
defer client.Close(context.Background())
defer client.Recover(true) // 可选：捕获 panic 上报

client.Identify("u1", map[string]any{"plan": "pro"})
client.Info("hello", map[string]any{"k": "v"}, nil)
client.Track("signup", map[string]any{"from": "landing"}, nil)
```

## Flutter

包名：`logtap_flutter`（仓库目录 `sdks/flutter/logtap_flutter/`）

```dart
final client = await LogtapClient.create(
  const LogtapClientOptions(
    baseUrl: "https://your-logtap-host",
    projectId: 1,
    projectKey: "pk_xxx",
    gzip: true, // Web 自动降级为非 gzip
    globalTags: {"env": "prod"},
    globalContexts: {"app": {"version": "1.2.3"}},
  ),
);

client.captureFlutterErrors(); // 可选：捕获未处理异常

client.identify("u1", {"plan": "pro"});
client.info("hello", {"k": "v"});
client.track("signup", {"from": "landing"});

await client.close();
```

## 其他语言

不引入 SDK 也可以直接用 HTTP 上报（见[上报接口](https://logtap-website.autotask.run/docs/ingest-api/)）。以 Python 为例：

```python
import requests

requests.post(
    "https://your-logtap-host/api/1/logs/",
    headers={"X-Project-Key": "pk_xxx"},
    json=[{"level": "info", "message": "hello", "user": {"id": "u1"}}],
)
```

更多语言集成示例（Python、Java、PHP、Ruby、.NET）见仓库 `docs/integrations/` 目录。

## 异常监控怎么办？

异常监控优先复用 Sentry SDK（零改动改 DSN，见[Sentry 兼容](https://logtap-website.autotask.run/docs/sentry-compat/)）；各端 SDK 的 `capture*Errors()` 适合在没有 Sentry SDK 的轻量场景里捕获未处理异常。

========

# 部署指南（中文）

---
title: 部署指南
description: LogTap 生产部署：Docker Compose、无 Docker（systemd）、环境变量与生产建议。
---

# 部署指南

本文覆盖两种部署方式：本地/测试用 Docker Compose 一键启动；服务器/生产可以无 Docker（systemd）或自定义容器化部署。

## 依赖组件

- PostgreSQL / TimescaleDB（事件与日志存储）
- NSQ（异步写库与削峰）
- （可选）Redis（部分指标/缓存能力，按项目配置而定）

## 方式一：Docker Compose

```bash
git clone https://github.com/aak1247/logtap.git
cd logtap/deploy
bash ../scripts/gen-auth-secret.sh   # 生成 AUTH_SECRET
docker compose up -d
```

Compose 已包含 TimescaleDB、Redis、NSQ（lookupd/nsqd/admin）与 logtap 网关。GeoIP 可选启用：设置 `MAXMIND_LICENSE_KEY` 后，mmdb 自动下载并缓存到 volume `/data/geoip/`；若 mmdb 已存在且未显式配置 `GEOIP_CITY_MMDB` / `GEOIP_ASN_MMDB`，启动脚本会自动使用 `/data/geoip/GeoLite2-City.mmdb` 与 `/data/geoip/GeoLite2-ASN.mmdb`。

## 方式二：无 Docker（服务器/生产）

### 构建网关

需要 Go 1.22+：

```bash
go build -o gateway ./cmd/gateway
```

把 `gateway` 上传到服务器（例如 `/opt/logtap/gateway`）。

### 最小环境变量

环境变量参考仓库 `.env.example`，常用项：

| 变量 | 说明 |
| --- | --- |
| `HTTP_ADDR` | HTTP 监听地址，例如 `:8080` |
| `NSQD_ADDRESS` | NSQ 地址，例如 `127.0.0.1:4150` |
| `RUN_CONSUMERS` | 是否启动消费者写库（生产通常 `true`） |
| `POSTGRES_URL` | `RUN_CONSUMERS=true` 时必填，例如 `postgres://logtap:logtap@127.0.0.1:5432/logtap?sslmode=disable` |
| `AUTH_SECRET` | 开启登录与上报鉴权（推荐） |

网关启动时使用 GORM `AutoMigrate` 自动建表/建索引。

### systemd（可选模板）

模板在 `deploy/systemd/`：

- `logtap-gateway.service` — 网关
- `nsqd.service` — NSQ
- `logtap-alert-worker.service` — 可选：独立告警投递 worker

建议做法：创建 `/opt/logtap/` 目录，把二进制与 `.env` 放到固定目录，然后 `systemctl enable --now logtap-gateway`。

告警投递 worker：若不启用 `RUN_ALERT_WORKER=true`（网关内嵌 worker），可用独立进程 `alert-worker` 消费 `alert_deliveries`，环境变量示例见 `deploy/alert-worker.env.example`。

## 控制台（Web）

控制台是独立前端（`web/`，Vite + React + Tailwind），生产构建产物在 `web/dist/`，用 Nginx/静态托管部署：

```bash
cd web
npm install
npm run build
```

生产建议用 Nginx/Traefik 提供 HTTPS，并把 `/api/` 反代到网关。

## 生产建议

- **反向代理**：Nginx/Traefik 提供 HTTPS，`/api/` 反代到网关
- **数据持久化**：Postgres 数据盘、NSQ 目录与日志保留策略提前规划
- **资源隔离**：消费者写库与网关可拆开部署（不同进程/不同实例），数据库慢查询不影响上报
- **告警 worker**：大流量下用独立 `alert-worker` 进程投递，避免与网关争资源

========

# Links

- [GitHub repository](https://github.com/aak1247/logtap)
- [Technical design docs](https://github.com/aak1247/logtap/blob/main/docs/OVERVIEW.md)
- [Issues & feedback](https://github.com/aak1247/logtap/issues)
- [Online console](https://logtap.autotask.run/)