Output from command in shell 0d5e98:
total 16
drwxr-xr-x  12 root1  staff   384 Jun 25 06:31 .
drwx------@  9 root1  staff   288 Jun 24 02:53 ..
-rw-r--r--@  1 root1  staff  6148 Jun 22 23:45 .DS_Store
drwxr-xr-x  36 root1  staff  1152 Jun 24 14:13 .playwright-mcp
drwxr-xr-x   2 root1  staff    64 Jun 25 06:31 clickhouse-github
drwxr-xr-x   5 root1  staff   160 Jun 19 00:27 deep-research
drwxr-xr-x   7 root1  staff   224 Jun 22 23:47 glm5.2-inference
drwxr-xr-x   7 root1  staff   224 Jun 24 02:35 housing
drwxr-xr-x  13 root1  staff   416 Jun 24 03:52 stockington
drwxr-xr-x  11 root1  staff   352 Jun 24 01:36 tweetington
drwxr-xr-x  12 root1  staff   384 Jun 19 14:11 twitter-scraper
drwxr-xr-x   2 root1  staff    64 Jun 24 00:10 x-hall
---
/Users/root1/downloads/projects


Exit code: 0

Output from command in shell f9e606:
total 0
drwxr-xr-x   2 root1  staff   64 Jun 25 06:31 .
drwxr-xr-x  12 root1  staff  384 Jun 25 06:31 ..
---glm5.2---
total 8496
drwxr-xr-x   3 root1  staff       96 Jun 23 00:36 __pycache__
drwxr-xr-x   7 root1  staff      224 Jun 22 23:47 .
drwxr-xr-x  12 root1  staff      384 Jun 25 06:31 ..
-rw-r--r--@  1 root1  staff  4327723 Jun 22 23:42 100ktps.ipynb
-rw-r--r--   1 root1  staff     8488 Jun 23 16:09 modal_inference.py
-rw-r--r--   1 root1  staff     3274 Jun 23 16:12 README.md
-rw-r--r--   1 root1  staff       72 Jun 22 23:43 requirements.txt


Exit code: 0

<file-view path="/Users/root1/Downloads/projects/glm5.2-inference/README.md" start_line="1" end_line="75" total_lines="75">
  1|# Qwen3-8B Inference Throughput on Modal (B200)
  2|
  3|Modal app that benchmarks **Qwen3-8B** throughput with TensorRT-LLM on a single
  4|NVIDIA **B200**, using **NVFP4** weights and the **PyTorch backend** (no engine
  5|build required).
  6|
  7|## Pipeline
  8|
  9|`modal_inference.py` defines three steps, orchestrated by `run_full_pipeline`:
 10|
 11|1. **`generate_synthetic_dataset`** — builds a synthetic `token-norm-dist`
 12|   dataset with `prepare_dataset.py` (configurable ISL/OSL and request count).
 13|2. **`download_model`** — fetches `nvidia/Qwen3-8B-NVFP4` into a Modal volume.
 14|3. **`run_benchmark_pytorch`** — runs `trtllm-bench throughput` on the PyTorch
 15|   backend and returns the JSON report.
 16|
 17|## Run
 18|
 19|```bash
 20|pip install -r requirements.txt
 21|modal secret create huggingface HF_TOKEN=your_token_here   # one-time
 22|python modal_inference.py
 23|```
 24|
 25|The `__main__` block runs the full pipeline and prints the throughput report.
 26|
 27|## Results (1× B200, Qwen3-8B)
 28|
 29|| Quantization / backend | Output tps (decode) | Total tps (in+out) |
 30|| --- | --- | --- |
 31|| FP8 / TensorRT engine | ~41k | ~76–86k |
 32|| FP8 / PyTorch | ~42k | ~83k |
 33|| **NVFP4 / PyTorch + CUDA graphs** | **~47–50k** | **~96–101k** (balanced 64/64) |
 34|| **NVFP4 / PyTorch, 128/64 workload** | ~39k | **~117k** |
 35|
 36|Key facts established while tuning:
 37|
 38|- **Output (decode) throughput is the hard ceiling (~50k tok/s)** for a dense
 39|  8B at NVFP4 on one B200. CUDA graphs are already engaged for the large
 40|  steady-state batch (verified — forcing explicit graph batch sizes did not
 41|  change throughput), so decode is genuinely compute-bound, not launch-bound.
 42|- With a **balanced** ISL=OSL workload, total throughput = ~2× output ≈ 100k,
 43|  landing right on the line with ~4% run-to-run instance variance.
 44|- The single biggest win was **FP8 → NVFP4** (B200 native FP4 tensor cores),
 45|  which lifted total from ~83k to ~100k.
 46|- To **reliably exceed 100k** on one B200, use an input-heavier workload
 47|  (prefill is compute-efficient): the default `128/64` config reports ~117k.
 48|  Going substantially higher needs more GPUs (TP/DP).
 49|
 50|## Key configuration
 51|
 52|`run_benchmark_pytorch` exposes the throughput knobs:
 53|
 54|- `kv_cache_free_gpu_mem_fraction` (default `0.85`) — higher = more concurrent
 55|  decode, but too high (≥0.9) OOMs during runtime allocation.
 56|- `max_batch_size` / `concurrency` (default `10240`) — output throughput peaks
 57|  around batch ~8–10k; larger batches OOM or lose efficiency.
 58|- `max_num_tokens` (default `16384`).
 59|- CUDA graphs are captured for an explicit batch-size list covering the full
 60|  `max_batch_size` so steady-state decode is always graph-accelerated.
 61|
 62|`generate_synthetic_dataset` exposes `input_mean` / `output_mean` (ISL/OSL) and
 63|`num_requests`.
 64|
 65|## Notes / gotchas handled
 66|
 67|- **PMIX/MPI crash** inside Modal containers is fixed with
 68|  `PMIX_MCA_gds=hash`.
 69|- `nvidia/Qwen3-8B-NVFP4` is non-gated; the original notebook's
 70|  `meta-llama/Llama-3.2-1B` is gated and 401s without access.
 71|- The PyTorch backend's `--model_path` must be a **local directory**
 72|  (a HF repo id passed directly trips a `trtllm-bench` path-validation bug),
 73|  so the model is pre-downloaded to the volume.
 74|- `trtllm-bench` 1.2.0rc1 uses `--extra_llm_api_options` (the `--config` alias
 75|  is not yet available in this version).
</file-view>

Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
Current todo list:
1. [~] Research ClickHouse Lexer token types and syntax elements [in_progress]
2. [ ] Design project structure and trajectory format spec [pending]
3. [ ] Launch 5 parallel GLM 5.2 subagents to generate 20 trajectories each (100 total) [pending]
4. [ ] Implement ClickHouse tokenizer with lookbehind copies for identifiers/strings [pending]
5. [ ] Implement GPT-based SLM model (~10MB, CPU-optimized) [pending]
6. [ ] Implement training pipeline [pending]
7. [ ] Train model on generated trajectories [pending]
8. [ ] Implement inference/autocomplete with syntactic correctness sampling [pending]
9. [ ] Implement C++ export for ClickHouse embedding and WASM export [pending]

# Web Search Results for "ClickHouse Lexer.h token types TokenType enum source code"

## 1. src/Parsers/Lexer.h at 77a7d29d · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/77a7d29d/src/Parsers/Lexer.h

#define APPLY_FOR_TOKENS(M) \
    M(Whitespace) \
    M(Comment) \
    \
    M(BareWord)               /** Either keyword (SELECT) or identifier (column) */ \
    \
    M(Number)                 /** Always non-negative. No leading plus. 123 or something like 123.456e12, 0x123p12 */ \
    M(StringLiteral)          /** 'hello word', 'hello''word', 'hello\'word\\' */ \
    \
    M(QuotedIdentifier)       /** "x", `x` */ \
    \
    M(OpeningRoundBracket) \
    M(ClosingRoundBracket) \
    \
    M(OpeningSquareBracket) \
    M(ClosingSquareBracket) \
    \
    M(OpeningCurlyBrace) \
    M(ClosingCurlyBrace) \
    \
    M(Comma) \
    M(Semicolon) \
    M(VerticalDelimiter)      /** Vertical delimiter \G */ \
    M(Dot)                    /** Compound identifiers, like a.b or tuple access operator a.1, (x, y).2. */ \
                              /** Need to be distinguished from floating point number with omitted integer part: .1 */ \
    \
    M(Asterisk)               /** Could be used as multiplication operator or on it's own: "SELECT *" */ \
    \
    M(HereDoc) \
    \
    M(DollarSign) \
    M(Plus) \
    M(Minus) \
    M(Slash) \
    M(Percent) \
    M(Arrow)                  /** ->. Should be distinguished from minus operator. */ \
    M(QuestionMark) \
    M(Colon) \
    M(Caret) \
    M(DoubleColon) \
    M(Equals) \
    M(NotEquals) \
    M(Less) \
    M(Greater) \
    M(LessOrEquals) \
    M(GreaterOrEquals) \
    M(Spaceship)              /** <=>. Used in MySQL for NULL-safe equality comparison. */ \
    M(PipeMark) \
    M(Concatenation)          /** String concatenation operator: || */ \
    \
    M(At)                     /** @. Used for specifying user names and also for MySQL-style variables. */ \
    M(DoubleAt)               /** @@. Used for MySQL-style global variables. */ \
    \
    /** Order is important. EndOfStream goes after all usual tokens, and special error tokens goes after EndOfStream. */ \
    \
    M(EndOfStream) \
    \
    /** Somethi...

## 2. Lexer.h source code [ClickHouse/dbms/src/Parsers/Lexer.h] - Woboq Code Browser
URL: https://blog.weghos.com/clickhouse/ClickHouse/dbms/src/Parsers/Lexer.h.html

| 1 | #pragma once |
| --- | --- |
| 2 |
| 3 | #include<stddef.h> |
| 4 |
| 5 |
| 6 | namespace DB |
| 7 | { |
| 8 |
| 9 | #define APPLY_FOR_TOKENS(M) \ |
| 10 | M(Whitespace) \ |
| 11 | M(Comment) \ |
| 12 | \ |
| 13 | M(BareWord
...
/** Either keyword (SELECT) or identifier (column) */ \ |
| 14 | \ |
| 15 |
...
123
...
something like 123.4
...
6e12,
...
62
...
\ |
| 63
...
M(ErrorSinglePipeMark) \ |
| 64 | M(ErrorWrongNumber) \ |
| 65 | M(ErrorMaxQuerySizeExceeded) \ |
| 66 |
| 67 |
| 68 | enum class TokenType |
| 69 | { |
| 70 | #define M(TOKEN) TOKEN, |
| 71 | APPLY_FOR_TOKENS(M) |
| 72 | #undef M |
| 73 | }; |
| 74 |
| 75 | const char * getTokenName(TokenType type); |
| 76 | const char * getErrorTokenDescription(TokenType type); |
| 77 |
| 78 |
| 79 | struct Token |
| 80 | { |
| 81 | TokenType type; |
| 82 | const char * begin; |
| 83 | const char * end; |
| 84 |
| 85 | size_t size() const { return end - begin; } |
| 86 |
| 87 | Token() = default; |
| 88 | Token(TokenType type_, const char * begin_, const char * end_) : type(type_), begin(begin_), end(end_) {} |
| 89 |
| 90 | bool isSignificant() const { return type != TokenType::Whitespace && type != TokenType::Comment; } |
| 91 | bool isError() const { return type > TokenType::EndOfStream; } |
| 92 | bool isEnd() const { return type == TokenType::EndOfStream; } |
| 93 | }; |
| 94 |
| 95 |
| 96 | class Lexer |
| 97 | { |
| 98 | public: |
| 99 | Lexer(const char * begin_, const char * end_, size_t max_query_size_ = 0) |
| 100 | : begin(begin_), pos(begin_), end(end_), max_query_size(max_query_size_) {} |
| 101 | Token nextToken(); |
| 102 |
| 103 | private: |
| 104 | const char * const begin; |
| 105 | const char * pos; |
| 106 | const char * const end; |
| 107 |
| 108 | const size_t max_query_size; |
| 109 |
| 110 | Token nextTokenImpl(); |
| 111 |
| 112 | /// This is needed to disambiguate tuple access operator from floating point number (.1). |
| 113 | TokenType prev_significant_token_type = TokenType::White...

## 3. src/Parsers/Lexer.cpp at 77a7d29d · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/77a7d29d/src/Parsers/Lexer.cpp

'\n'>(
...
(*pos == '/')
...
Token(TokenType::Slash
...
case '%':
            return Token(TokenType::Percent
...
token_begin
...
++pos);
        case '=':   /// =, ==
        {
            ++pos;
            if (pos < end && *pos == '=')
                ++pos;
            return Token(TokenType::Equals, token
...
pos);
        }
        case '!':   /// !=
        {
            ++pos;
...
if (pos < end
...
*pos == '=
...
return Token(TokenType::NotEquals
...
return Token(TokenType::Spaceship, token
...
begin, pos);
            }
            if (pos < end && *pos == '=')
                return Token(TokenType::LessOrEquals, token
...
begin, ++pos);
            if (pos < end && *pos == '>')
                return Token(TokenType::NotEquals, token
...
}
        case '>
...
GreaterOrEquals
...
return Token(TokenType::Greater
...
begin, pos);
        }
        case '?':
            return Token(TokenType::QuestionMark, token_begin, ++pos);
        case '^':
            return Token(TokenType::Caret
...
case '|':
...
return Token(TokenType::Concatenation
...
);
            return Token(TokenType::Pipe
...
);
        }
        case
...
return Token(TokenType::
...
(TokenType type)
...
{
    switch (
...
const char * getErrorTokenDescription(TokenType type)
...
switch (type)
    {
        case TokenType::Error:
            return "Unrecognized token";
        case TokenType::ErrorMultilineCommentIsNotClosed:
            return "Multiline comment is not closed
...
case TokenType::ErrorSingleQuoteIsNotClosed:
            return "Single quoted string is not closed
...
case TokenType::ErrorDoubleQuoteIsNotClosed:
            return "Double quoted string is not closed
...
case TokenType::ErrorBackQuoteIsNotClosed:
            return "Back quoted string is not closed
...
case TokenType::ErrorSingleExclamationMark:
            return "Exclamation mark can only occur in != operator
...
case TokenType::ErrorSinglePipeMark:
            return "Pipe symbol could only occur in || opera...

## 4. src/Functions/QueryTokenizationImpl.h at 880f38dc · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/880f38dc/src/Functions/QueryTokenizationImpl.h

/// Helper: build a DataTypeEnum8 from an X-macro list.
/// Before calling, temporarily #define ENUM_TYPE to the target enum.
/// Example:
///   #define ENUM_TYPE TokenType
///   return MAKE_ENUM8_TYPE(APPLY_FOR_TOKENS);
///   #undef ENUM_TYPE
...
#define ENUM8_ENTRY_(NAME) {#NAME, static_cast<Int8>(ENUM_TYPE::NAME)},
#define MAKE_ENUM8_TYPE(APPLY_MACRO) \
    std::make_shared<DataTypeEnum8>(DataTypeEnum8::Values{APPLY_MACRO(ENUM8_ENTRY_)})
...
const auto &
...
Ref();
...
= settings[Setting::
...
_parser_depth];
            parser_settings.max_parser_backtracks = settings[Setting::max_parser_backtracks];
            parser_settings.implicit_select = settings[Setting::implicit_select];
        }
    }

    String getName() const override { return name; }
    size_t getNumberOfArguments() const override { return 1; }
    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
    bool useDefaultImplementationForConstants() const override { return true; }

    DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
    {
        FunctionArgumentDescriptors args{{"query", static_cast<FunctionArgumentDescriptor::TypeValidator>(&isString), nullptr, "String"}};
        validateFunctionArguments(*this, arguments, args);

        DataTypes types{std::make_shared<DataTypeUInt64>(), std::make_shared<DataTypeUInt64>(), Impl::makeEnumType()};
        Strings names{"begin", "end", "type"};
        return std::make_shared<DataTypeArray>(std::make_shared<DataTypeTuple>(std::move(types), std::move(names)));
    }

    ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
    {
        const ColumnString & col_query = assert_cast<const ColumnString &>(*arguments[0].column);

        auto col_begin = ColumnUInt64::create();
        auto col_end = ColumnUInt64::create();
        auto col_type = ColumnInt8::create();
        auto ...

## 5. src/Functions/QueryTokenizationImpl.h at 139d3796 · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/139d3796/src/Functions/QueryTokenizationImpl.h

/// Helper: build a DataTypeEnum8 from an X-macro list.
/// Before calling, temporarily #define ENUM_TYPE to the target enum.
/// Example:
///   #define ENUM_TYPE TokenType
///   return MAKE_ENUM8_TYPE(APPLY_FOR_TOKENS);
///   #undef ENUM_TYPE
...
#define ENUM8_ENTRY_(NAME) {#NAME, static_cast<Int8>(ENUM_TYPE::NAME)},
#define MAKE_ENUM8_TYPE(APPLY_MACRO) \
    std::make_shared<DataTypeEnum8>(DataTypeEnum8::Values{APPLY_MACRO(ENUM8_ENTRY_)})
...
const auto &
...
Ref();
...
= settings[Setting::
...
_parser_depth];
            parser_settings.max_parser_
...
tracks = settings[Setting::max_parser_backtracks];
            parser_settings.implicit_select = settings[Setting::implicit_select];
        }
    }

    String getName() const override { return name; }
    size_t getNumberOfArguments() const override { return 1; }
    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
    bool useDefaultImplementationForConstants() const override { return true; }

    DataTypePtr getReturnTypeImpl(const ColumnsW… (42 chars truncated)
… (17 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/3d5bcf31/content.txt
</truncation_notice>

Output from command in shell a4c78c:
Python 3.12.8
numpy                                    2.2.6
pytorch-lightning                        2.6.0
pytorch-metric-learning                  2.9.0
tokenizers                               0.22.2
torch                                    2.10.0          2
torch-audiomentations                    0.12.0
torch_pitch_shift                        1.2.5
torchaudio                               2.11.0
torchmetrics                             1.8.2
torchvision                              0.25.0
transformers                             4.57.6
---
/usr/bin/clang++
/usr/bin/g++
/Users/root1/.cargo/bin/cargo


Exit code: 1

# Web Search Results for "ClickHouse Parser.h keywords list APPLY_FOR_KEYWORDS SELECT FROM WHERE"

## 1. utils/antlr/ClickHouseLexer.g4
URL: https://github.com/ClickHouse/ClickHouse/blob/master/utils/antlr/ClickHouseLexer.g4

// NOTE: don't forget to add new keywords to the parser rule "keyword"!
...
ACCESS: A C C E S S
...
ELSE: E L S E;
...
FETCHES: F E T
...
H E S;
...
FOR: F O R
...
FROM: F R O M;
...
KEY: K E Y;
...
SELECT: S E L E C T;
...
SOURCE: S O U R C E;
...
WHERE: W H E R E;

## 2. src/Parsers/IParser.h at 139d3796 · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/139d3796/src/Parsers/IParser.h

/// Highlight types for syntax highlighting of parsed queries.
/// APPLY_FOR_HIGHLIGHTS lists the publicly visible types (used in the output of `highlightQuery`).
/// string_like and string_regexp are internal types used by the parser before expansion.
#define APPLY_FOR_HIGHLIGHTS(M) \
    M(none) \
    M(keyword) \
    M(identifier) \
    M(function) \
    M(alias) \
    M(substitution) \
    M(number) \
    M(string) \
    M(string_escape) \
    M(string_metacharacter)

enum class Highlight : uint8_t
{
#define M(NAME) NAME,
    APPLY_FOR_HIGHLIGHTS(M)
#undef M
    /// These two are used internally by the parser to mark LIKE/REGEXP string ranges.
    /// They are expanded into string/string_escape/string_metacharacter by `expandHighlights`.
    string_like,
    string_regexp,
};
...
/// Expand string_like and string_regexp ranges into character-level sub-ranges
/// with string, string_escape, and string_metacharacter highlight types.
/// Other ranges are passed through unchanged.
std::vector<HighlightedRange> expandHighlights(const std::set<HighlightedRange> & highlights);
...
/** Parse piece of text from position `pos`, but not beyond end of line (`end` - position after end of line),
      * move pointer `pos` to the maximum position to which it was possible to parse,
      * in case of success return `true` and the result in `node` if it is needed, otherwise false,
      * in `expected` write what was expected in
...
maximum position,
      *  to which it was possible to parse if parsing was unsuccessful,
      *  or what this parser parse if parsing was successful.
      * The string to which the [begin, end) range is included may be not 0-terminated.
      */
    virtual bool parse(Pos & pos, ASTPtr & node, Expected & expected) =
...
0;

    bool ignore(Pos & pos, Expected & expected)
    {
        ASTPtr ignore_node;
        return parse(pos, ignore_node, expected);
    }

    bool ignore(Pos & pos)
    {
        Expected expected;
        return ignore(pos, e...

## 3. src/Parsers/Lexer.h at 77a7d29d · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/77a7d29d/src/Parsers/Lexer.h

#define APPLY_FOR_TOKENS(M) \
    M(Whitespace) \
    M(Comment) \
    \
    M(BareWord)               /** Either keyword (SELECT) or identifier (column) */ \
    \
    M(Number)                 /** Always non-negative. No leading plus. 123 or something like 123.456e12, 0x123p12 */ \
    M(StringLiteral)          /** 'hello word', 'hello''word', 'hello\'word\\' */ \
    \
    M(QuotedIdentifier)       /** "x", `x` */ \
    \
    M(OpeningRoundBracket) \
    M(ClosingRoundBracket) \
    \
    M(OpeningSquareBracket) \
    M(ClosingSquareBracket) \
    \
    M(OpeningCurlyBrace) \
    M(ClosingCurlyBrace) \
    \
    M(Comma) \
    M(Semicolon) \
    M(VerticalDelimiter)      /** Vertical delimiter \G */ \
    M(Dot)                    /** Compound identifiers, like a.b or tuple access operator a.1, (x, y).2. */ \
                              /** Need to be distinguished from floating point number with omitted integer part: .1 */ \
    \
    M(Asterisk)               /** Could be used as multiplication operator or on it's own: "SELECT *" */ \
    \
    M(HereDoc) \
    \
    M(DollarSign) \
    M(Plus) \
    M(Minus) \
    M(Slash) \
    M(Percent) \
    M(Arrow)                  /** ->. Should be distinguished from minus operator. */ \
    M(QuestionMark) \
    M(Colon) \
    M(Caret) \
    M(DoubleColon) \
    M(Equals) \
    M(NotEquals) \
    M(Less) \
    M(Greater) \
    M(LessOrEquals) \
    M(GreaterOrEquals) \
    M(Spaceship)              /** <=>. Used in MySQL for NULL-safe equality comparison. */ \
    M(PipeMark) \
    M(Concatenation)          /** String concatenation operator: || */ \
    \
    M(At)                     /** @. Used for specifying user names and also for MySQL-style variables. */ \
    M(DoubleAt)               /** @@. Used for MySQL-style global variables. */ \
    \
    /** Order is important. EndOfStream goes after all usual tokens, and special error tokens goes after EndOfStream. */ \
    \
    M(EndOfStream) \
    \
    /** Somethi...

Output from command in shell e5e6c5:
export
inference
model
tests
tokenizer
trajectories

/Users/root1/Downloads/projects/clickhouse-github/export:

/Users/root1/Downloads/projects/clickhouse-github/inference:

/Users/root1/Downloads/projects/clickhouse-github/model:

/Users/root1/Downloads/projects/clickhouse-github/tests:

/Users/root1/Downloads/projects/clickhouse-github/tokenizer:

/Users/root1/Downloads/projects/clickhouse-github/trajectories:
data

/Users/root1/Downloads/projects/clickhouse-github/trajectories/data:


Exit code: 0

# Web Content from https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/utils/antlr/ClickHouseLexer.g4

lexer grammar ClickHouseLexer;

// NOTE: don't forget to add new keywords to the parser rule "keyword"!

// Keywords

ACCESS: A C C E S S;
ADD: A D D;
AFTER: A F T E R;
ALIAS: A L I A S;
ALL: A L L;
ALTER: A L T E R;
AND: A N D;
ANTI: A N T I;
ANY: A N Y;
ARRAY: A R R A Y;
AS: A S;
ASCENDING: A S C | A S C E N D I N G;
ASOF: A S O F;
AST: A S T;
ASYNC: A S Y N C;
ATTACH: A T T A C H;
BETWEEN: B E T W E E N;
BOTH: B O T H;
BY: B Y;
CACHES: C A C H E S;
CASE: C A S E;
CAST: C A S T;
CHECK: C H E C K;
CLEAR: C L E A R;
CLUSTER: C L U S T E R;
CLUSTERS: C L U S T E R S;
CODEC: C O D E C;
COLLATE: C O L L A T E;
COLUMN: C O L U M N;
COLUMNS: C O L U M N S;
COMMENT: C O M M E N T;
CONSTRAINT: C O N S T R A I N T;
CREATE: C R E A T E;
CROSS: C R O S S;
CUBE: C U B E;
CURRENT: C U R R E N T;
CURRENT_USER: C U R R E N T UNDERSCORE U S E R;
CHANGED: C H A N G E D;
DATABASE: D A T A B A S E;
DATABASES: D A T A B A S E S;
DATE: D A T E;
DAY: D A Y;
DEDUPLICATE: D E D U P L I C A T E;
DEFAULT: D E F A U L T;
DELAY: D E L A Y;
DELETE: D E L E T E;
DESC: D E S C;
DESCENDING: D E S C E N D I N G;
DESCRIBE: D E S C R I B E;
DETACH: D E T A C H;
DICTIONARIES: D I C T I O N A R I E S;
DICTIONARY: D I C T I O N A R Y;
DISK: D I S K;
DISTINCT: D I S T I N C T;
DISTRIBUTED: D I S T R I B U T E D;
DROP: D R O P;
ELSE: E L S E;
ENABLED: E N A B L E D;
END: E N D;
ENGINE: E N G I N E;
ENGINES: E N G I N E S;
ESTIMATE: E S T I M A T E;
EVENTS: E V E N T S;
EXCEPT: E X C E P T;
EXISTS: E X I S T S;
EXPLAIN: E X P L A I N;
EXPRESSION: E X P R E S S I O N;
EXTENDED: E X T E N D E D;
EXTRACT: E X T R A C T;
FETCHES: F E T C H E S;
FIELDS: F I E L D S;
FILESYSTEM: F I L E S Y S T E M;
FILL: F I L L;
FINAL: F I N A L;
FIRST: F I R S T;
FLUSH: F L U S H;
FOLLOWING: F O L L O W I N G;
FOR: F O R;
FORMAT: F O R M A T;
FREEZE: F R E E Z E;
FROM: F R O M;
FULL: F U L L;
FUNCTION: F U N C T I O N;
FUNCTIONS: F U N C T I O N S;
GLOBAL: G L O B A L;
GRANULARITY: G R A N U L A R I T Y;
GRANTS: G R A N T S;
GROUP: G R O U P;
GROUPING: G R O U P I N G;
HAVING: H A V I N G;
HIERARCHICAL: H I E R A R C H I C A L;
HOUR: H O U R;
ID: I D;
IF: I F;
ILIKE: I L I K E;
IMPLICIT: I M P L I C I T;
IN: I N;
INDEX: I N D E X;
INDEXES: I N D E X E S;
INDICES: I N D I C E S;
INF: I N F | I N F I N I T Y;
INJECTIVE: I N J E C T I V E;
INNER: I N N E R;
INSERT: I N S E R T;
INTERPOLATE: I N T E R P O L A T E;
INTERVAL: I N T E R V A L;
INTO: I N T O;
IS: I S;
IS_OBJECT_ID: I S UNDERSCORE O B J E C T UNDERSCORE I D;
JOIN: J O I N;
KEY: K E Y;
KEYS: K E Y S;
KILL: K I L L;
LAST: L A S T;
LAYOUT: L A Y O U T;
LEADING: L E A D I N G;
LEFT: L E F T;
LIFETIME: L I F E T I M E;
LIKE: L I K E;
LIMIT: L I M I T;
LIVE: L I V E;
LOCAL: L O C A L;
LOGS: L O G S;
MATERIALIZE: M A T E R I A L I Z E;
MATERIALIZED: M A T E R I A L I Z E D;
MAX: M A X;
MERGES: M E R G E S;
MICROSECOND: M I C R O S E C O N D;
MILLISECOND: M I L L I S E C O N D;
MIN: M I N;
MINUTE: M I N U T E;
MODIFY: M O D I F Y;
MONTH: M O N T H;
MOVE: M O V E;
MUTATION: M U T A T I O N;
NAN_SQL: N A N; // conflicts with macro NAN
NANOSECOND: N A N O S E C O N D;
NO: N O;
NOT: N O T;
NULL_SQL: N U L L; // conflicts with macro NULL
NULLS: N U L L S;
OFFSET: O F F S E T;
ON: O N;
OPTIMIZE: O P T I M I Z E;
OR: O R;
ORDER: O R D E R;
OUTER: O U T E R;
OUTFILE: O U T F I L E;
OVER: O V E R;
OVERRIDE: O V E R R I D E;
PARTITION: P A R T I T I O N;
PIPELINE: P I P E L I N E;
PLAN: P L A N;
POLICY: P O L I C Y;
POLICIES: P O L I C I E S;
POPULATE: P O P U L A T E;
PRECEDING: P R E C E D I N G;
PREWHERE: P R E W H E R E;
PRIMARY: P R I M A R Y;
PRIVILEGES: P R I V I L E G E S;
PROCESSLIST: P R O C E S S L I S T;
PROFILE: P R O F I L E;
PROFILES: P R O F I L E S;
PROJECTION: P R O J E C T I O N;
QUARTER: Q U A R T E R;
QUOTA: Q U O T A;
QUOTAS: Q U O T A S;
RANGE: R A N G E;
RECURSIVE: R E C U R S I V E;
RELOAD: R E L O A D;
REMOVE: R E M O V E;
RENAME: R E N A M E;
REPLACE: R E P L A C E;
REPLICA: R E P L I C A;
REPLICATED: R E P L I C A T E D;
RIGHT: R I G H T;
ROLE: R O L E;
ROLES: R O L E S;
ROLLUP: R O L L U P;
ROW: R O W;
ROWS: R O W S;
SAMPLE: S A M P L E;
SECOND: S E C O N D;
SELECT: S E L E C T;
SEMI: S E M I;
SENDS: S E N D S;
SET: S E T;
SETS: S E T S;
SETTING: S E T T I N G;
SETTINGS: S E T T I N G S;
SHOW: S H O W;
SOURCE: S O U R C E;
START: S T A R T;
STOP: S T O P;
SUBSTRING: S U B S T R I N G;
SYNC: S Y N C;
SYNTAX: S Y N T A X;
SYSTEM: S Y S T E M;
STEP: S T E P;
TABLE: T A B L E;
TABLES: T A B L E S;
TEMPORARY: T E M P O R A R Y;
TEST: T E S T;
THEN: T H E N;
TIES: T I E S;
TIMEOUT: T I M E O U T;
TIMESTAMP: T I M E S T A M P;
TO: T O;
TOP: T O P;
TOTALS: T O T A L S;
TRAILING: T R A I L I N G;
TREE: T R E E;
TRIM: T R I M;
TRUNCATE: T R U N C A T E;
TTL: T T L;
TYPE: T Y P E;
UNBOUNDED: U N B O U N D E D;
UNION: U N I O N;
UPDATE: U P D A T E;
USE: U S E;
USER: U S E R;
USERS: U S E R S;
USING: U S I N G;
UUID: U U I D;
VALUES: V A L U E S;
VIEW: V I E W;
VOLUME: V O L U M E;
WATCH: W A T C H;
WEEK: W E E K;
WHEN: W H E N;
WHERE: W H E R E;
WINDOW: W I N D O W;
WITH: W I T H;
YEAR: Y E A R | Y Y Y Y;

JSON_FALSE: F A L S E;
JSON_TRUE: T R U E;

// Tokens

HEXADECIMAL_STRING_LITERAL: X QUOTE_SINGLE HEX_DIGIT+ QUOTE_SINGLE;
BINARY_STRING_LITERAL: B QUOTE_SINGLE BINARY_DIGIT+ QUOTE_SINGLE;

IDENTIFIER
    : (LETTER | UNDERSCORE) (LETTER | UNDERSCORE | DEC_DIGIT)*
    | BACKQUOTE ( ~([\\`]) | (BACKSLASH .) | (BACKQUOTE BACKQUOTE) )* BACKQUOTE
    | QUOTE_DOUBLE ( ~([\\"]) | (BACKSLASH .) | (QUOTE_DOUBLE QUOTE_DOUBLE) )* QUOTE_DOUBLE
    ;
FLOATING_LITERAL
    : HEXADECIMAL_NUMERIC_LITERAL DOT HEX_DIGIT* (P | E) (PLUS | DASH)? DEC_DIGIT+
    | HEXADECIMAL_NUMERIC_LITERAL (P | E) (PLUS | DASH)? DEC_DIGIT+
    | DECIMAL_LITERAL DOT DEC_DIGIT* E (PLUS | DASH)? DEC_DIGIT+
    | DOT DECIMAL_LITERAL E (PLUS | DASH)? DEC_DIGIT+
    | DECIMAL_LITERAL E (PLUS | DASH)? DEC_DIGIT+
    ;
OCTAL_LITERAL: '0' OCT_DIGIT+;
DECIMAL_LITERAL: DEC_DIGIT+;
HEXADECIMAL_NUMERIC_LITERAL: '0' X HEX_DIGIT+;
BINARY_NUMERIC_LITERAL: '0' B BINARY_DIGIT+;

// It's important that quote-symbol is a single character.
STRING_LITERAL: QUOTE_SINGLE ( ~([\\']) | (BACKSLASH .) | (QUOTE_SINGLE QUOTE_SINGLE) )* QUOTE_SINGLE;

// Alphabet and allowed symbols

fragment A: [aA];
fragment B: [bB];
fragment C: [cC];
fragment D: [dD];
fragment E: [eE];
fragment F: [fF];
fragment G: [gG];
fragment H: [hH];
fragment I: [iI];
fragment J: [jJ];
fragment K: [kK];
fragment L: [lL];
fragment M: [mM];
fragment N: [nN];
fragment O: [oO];
fragment P: [pP];
fragment Q: [qQ];
fragment R: [rR];
fragment S: [sS];
fragment T: [tT];
fragment U: [uU];
fragment V: [vV];
fragment W: [wW];
fragment X: [xX];
fragment Y: [yY];
fragment Z: [zZ];

fragment LETTER: [a-zA-Z];
fragment OCT_DIGIT: [0-7];
fragment DEC_DIGIT: [0-9];
fragment HEX_DIGIT: [0-9a-fA-F];
fragment BINARY_DIGIT: [0-1];

ARROW: '->';
ASTERISK: '*';
BACKQUOTE: '`';
BACKSLASH: '\\';
DOUBLE_COLON: '::';
COLON: ':';
COMMA: ',';
CONCAT: '||';
DASH: '-';
DOT: '.';
EQ_DOUBLE: '==';
EQ_SINGLE: '=';
GE: '>=';
GT: '>';
LBRACE: '{';
LBRACKET: '[';
LE: '<=';
LPAREN: '(';
LT: '<';
NOT_EQ: '!=' | '<>';
PERCENT: '%';
PLUS: '+';
QUERY: '?';
QUOTE_DOUBLE: '"';
QUOTE_SINGLE: '\'';
RBRACE: '}';
RBRACKET: ']';
RPAREN: ')';
SEMICOLON: ';';
SLASH: '/';
UNDERSCORE: '_';

// Comments and whitespace

MULTI_LINE_COMMENT: '/*' .*? '*/' -> skip;
SINGLE_LINE_COMMENT: '--' ~('\n'|'\r')* ('\n' | '\r' | EOF) -> skip;
WHITESPACE: [ \u000B\u000C\t\r\n] -> skip;  // '\n' can be part of multiline single query

Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
Current todo list:
1. [x] Research ClickHouse Lexer token types and syntax elements [completed]
2. [~] Design project structure and trajectory format spec [in_progress]
3. [ ] Launch 5 parallel GLM 5.2 subagents to generate 20 trajectories each (100 total) [pending]
4. [ ] Implement ClickHouse tokenizer with lookbehind copies for identifiers/strings [pending]
5. [ ] Implement GPT-based SLM model (~10MB, CPU-optimized) [pending]
6. [ ] Implement training pipeline [pending]
7. [ ] Train model on generated trajectories [pending]
8. [ ] Implement inference/autocomplete with syntactic correctness sampling [pending]
9. [ ] Implement C++ export for ClickHouse embedding and WASM export [pending]

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/trajectories/trajectory_format_spec.md

Background subagent started with agent_id=43eb9708. You can wait for this agent to finish using the read_subagent tool, otherwise you will automatically be notified with a <subagent_completion_notification> when it completes.

Background subagent started with agent_id=c359f399. You can wait for this agent to finish using the read_subagent tool, otherwise you will automatically be notified with a <subagent_completion_notification> when it completes.

Background subagent started with agent_id=06759edc. You can wait for this agent to finish using the read_subagent tool, otherwise you will automatically be notified with a <subagent_completion_notification> when it completes.

Background subagent started with agent_id=75196d0f. You can wait for this agent to finish using the read_subagent tool, otherwise you will automatically be notified with a <subagent_completion_notification> when it completes.

Background subagent started with agent_id=ee6aeaf1. You can wait for this agent to finish using the read_subagent tool, otherwise you will automatically be notified with a <subagent_completion_notification> when it completes.

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/trajectory_format_spec.md" start_line="1" end_line="100" total_lines="100">
  1|# ClickHouse Autocomplete Trajectory Format Specification
  2|
  3|## Overview
  4|
  5|Each trajectory is a JSON object (one per line in JSONL format) representing a
  6|rich ClickHouse query autocomplete example. Trajectories are used to train a
  7|small GPT-based language model (~10 MB) that autocompletes ClickHouse queries.
  8|
  9|## Trajectory JSON Schema
 10|
 11|```json
 12|{
 13|  "id": "traj_001",
 14|  "category": "aggregation",
 15|  "subcategory": "group_by_having",
 16|  "description": "Aggregation with GROUP BY, HAVING, and ORDER BY",
 17|  "query": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY avg_price DESC LIMIT 20",
 18|  "tables": ["orders"],
 19|  "columns": {
 20|    "orders": ["id", "user_id", "price", "status", "city", "region", "created_at"]
 21|  },
 22|  "dictionaries": [],
 23|  "settings": {},
 24|  "completion_points": [
 25|    {
 26|      "prefix": "SELECT ",
 27|      "completion": "city, count() AS cnt, avg(price) AS avg_price ",
 28|      "next_token": "FROM"
 29|    },
 30|    {
 31|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM ",
 32|      "completion": "orders ",
 33|      "next_token": "WHERE"
 34|    },
 35|    {
 36|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE ",
 37|      "completion": "status = 'completed' ",
 38|      "next_token": "GROUP"
 39|    },
 40|    {
 41|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY ",
 42|      "completion": "city ",
 43|      "next_token": "HAVING"
 44|    },
 45|    {
 46|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING ",
 47|      "completion": "cnt > 10 ",
 48|      "next_token": "ORDER"
 49|    },
 50|    {
 51|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY ",
 52|      "completion": "avg_price DESC ",
 53|      "next_token": "LIMIT"
 54|    }
 55|  ]
 56|}
 57|```
 58|
 59|## Fields
 60|
 61|| Field | Type | Description |
 62||---|---|---|
 63|| `id` | string | Unique trajectory ID, format: `traj_NNN` |
 64|| `category` | string | Broad category (see Categories below) |
 65|| `subcategory` | string | More specific subcategory |
 66|| `description` | string | Human-readable description of what the query demonstrates |
 67|| `query` | string | The complete, syntactically valid ClickHouse query |
 68|| `tables` | string[] | Table names referenced in the query (for dictionary-based sampling) |
 69|| `columns` | object | Map of table name -> column names available on that table |
 70|| `dictionaries` | string[] | ClickHouse dictionary names available (if any) |
 71|| `settings` | object | ClickHouse SETTINGS clauses if relevant (e.g. `{"max_threads": "8"}`) |
 72|| `completion_points` | array | Array of prefix/completion pairs for training autocomplete |
 73|
 74|## Completion Points
 75|
 76|Each completion point represents a natural autocomplete boundary — a place
 77|where a user would pause and the model should predict the next segment.
 78|
 79|- `prefix`: The partial query up to this point (ends with a space or token boundary)
 80|- `completion`: The text that should be predicted next (up to the next natural boundary)
 81|- `next_token`: The first keyword/token of the completion (for constrained sampling)
 82|
 83|## Categories
 84|
 85|1. **basic_select** — Simple SELECT, WHERE, ORDER BY, LIMIT
 86|2. **aggregation** — GROUP BY, HAVING, aggregate functions, window functions
 87|3. **joins** — INNER/LEFT/RIGHT/FULL JOIN, ASOF JOIN, ARRAY JOIN, subqueries
 88|4. **ddl_dml** — CREATE TABLE, ALTER, DROP, INSERT, TRUNCATE, system queries
 89|5. **advanced** — Arrays, maps, JSON, materialized views, projections, dictionaries, TTL, CTEs, sampling
 90|
 91|## Rules
 92|
 93|1. All queries MUST be syntactically valid ClickHouse SQL
 94|2. Use realistic table and column names (not foo/bar)
 95|3. Include diverse ClickHouse-specific features (array functions, dictGet, etc.)
 96|4. Each trajectory should have 4-10 completion points
 97|5. Completion points should be at natural syntactic boundaries
 98|6. Table/column definitions must be consistent with the query
 99|7. Vary query complexity — some simple, some complex
100|8. Use ClickHouse-specific functions: count(), avg(), sum(), dictGet(), arrayJoin(), etc.
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/trajectory_format_spec.md" start_line="1" end_line="100" total_lines="100">
  1|# ClickHouse Autocomplete Trajectory Format Specification
  2|
  3|## Overview
  4|
  5|Each trajectory is a JSON object (one per line in JSONL format) representing a
  6|rich ClickHouse query autocomplete example. Trajectories are used to train a
  7|small GPT-based language model (~10 MB) that autocompletes ClickHouse queries.
  8|
  9|## Trajectory JSON Schema
 10|
 11|```json
 12|{
 13|  "id": "traj_001",
 14|  "category": "aggregation",
 15|  "subcategory": "group_by_having",
 16|  "description": "Aggregation with GROUP BY, HAVING, and ORDER BY",
 17|  "query": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY avg_price DESC LIMIT 20",
 18|  "tables": ["orders"],
 19|  "columns": {
 20|    "orders": ["id", "user_id", "price", "status", "city", "region", "created_at"]
 21|  },
 22|  "dictionaries": [],
 23|  "settings": {},
 24|  "completion_points": [
 25|    {
 26|      "prefix": "SELECT ",
 27|      "completion": "city, count() AS cnt, avg(price) AS avg_price ",
 28|      "next_token": "FROM"
 29|    },
 30|    {
 31|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM ",
 32|      "completion": "orders ",
 33|      "next_token": "WHERE"
 34|    },
 35|    {
 36|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE ",
 37|      "completion": "status = 'completed' ",
 38|      "next_token": "GROUP"
 39|    },
 40|    {
 41|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY ",
 42|      "completion": "city ",
 43|      "next_token": "HAVING"
 44|    },
 45|    {
 46|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING ",
 47|      "completion": "cnt > 10 ",
 48|      "next_token": "ORDER"
 49|    },
 50|    {
 51|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY ",
 52|      "completion": "avg_price DESC ",
 53|      "next_token": "LIMIT"
 54|    }
 55|  ]
 56|}
 57|```
 58|
 59|## Fields
 60|
 61|| Field | Type | Description |
 62||---|---|---|
 63|| `id` | string | Unique trajectory ID, format: `traj_NNN` |
 64|| `category` | string | Broad category (see Categories below) |
 65|| `subcategory` | string | More specific subcategory |
 66|| `description` | string | Human-readable description of what the query demonstrates |
 67|| `query` | string | The complete, syntactically valid ClickHouse query |
 68|| `tables` | string[] | Table names referenced in the query (for dictionary-based sampling) |
 69|| `columns` | object | Map of table name -> column names available on that table |
 70|| `dictionaries` | string[] | ClickHouse dictionary names available (if any) |
 71|| `settings` | object | ClickHouse SETTINGS clauses if relevant (e.g. `{"max_threads": "8"}`) |
 72|| `completion_points` | array | Array of prefix/completion pairs for training autocomplete |
 73|
 74|## Completion Points
 75|
 76|Each completion point represents a natural autocomplete boundary — a place
 77|where a user would pause and the model should predict the next segment.
 78|
 79|- `prefix`: The partial query up to this point (ends with a space or token boundary)
 80|- `completion`: The text that should be predicted next (up to the next natural boundary)
 81|- `next_token`: The first keyword/token of the completion (for constrained sampling)
 82|
 83|## Categories
 84|
 85|1. **basic_select** — Simple SELECT, WHERE, ORDER BY, LIMIT
 86|2. **aggregation** — GROUP BY, HAVING, aggregate functions, window functions
 87|3. **joins** — INNER/LEFT/RIGHT/FULL JOIN, ASOF JOIN, ARRAY JOIN, subqueries
 88|4. **ddl_dml** — CREATE TABLE, ALTER, DROP, INSERT, TRUNCATE, system queries
 89|5. **advanced** — Arrays, maps, JSON, materialized views, projections, dictionaries, TTL, CTEs, sampling
 90|
 91|## Rules
 92|
 93|1. All queries MUST be syntactically valid ClickHouse SQL
 94|2. Use realistic table and column names (not foo/bar)
 95|3. Include diverse ClickHouse-specific features (array functions, dictGet, etc.)
 96|4. Each trajectory should have 4-10 completion points
 97|5. Completion points should be at natural syntactic boundaries
 98|6. Table/column definitions must be consistent with the query
 99|7. Vary query complexity — some simple, some complex
100|8. Use ClickHouse-specific functions: count(), avg(), sum(), dictGet(), arrayJoin(), etc.
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/trajectory_format_spec.md" start_line="1" end_line="100" total_lines="100">
  1|# ClickHouse Autocomplete Trajectory Format Specification
  2|
  3|## Overview
  4|
  5|Each trajectory is a JSON object (one per line in JSONL format) representing a
  6|rich ClickHouse query autocomplete example. Trajectories are used to train a
  7|small GPT-based language model (~10 MB) that autocompletes ClickHouse queries.
  8|
  9|## Trajectory JSON Schema
 10|
 11|```json
 12|{
 13|  "id": "traj_001",
 14|  "category": "aggregation",
 15|  "subcategory": "group_by_having",
 16|  "description": "Aggregation with GROUP BY, HAVING, and ORDER BY",
 17|  "query": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY avg_price DESC LIMIT 20",
 18|  "tables": ["orders"],
 19|  "columns": {
 20|    "orders": ["id", "user_id", "price", "status", "city", "region", "created_at"]
 21|  },
 22|  "dictionaries": [],
 23|  "settings": {},
 24|  "completion_points": [
 25|    {
 26|      "prefix": "SELECT ",
 27|      "completion": "city, count() AS cnt, avg(price) AS avg_price ",
 28|      "next_token": "FROM"
 29|    },
 30|    {
 31|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM ",
 32|      "completion": "orders ",
 33|      "next_token": "WHERE"
 34|    },
 35|    {
 36|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE ",
 37|      "completion": "status = 'completed' ",
 38|      "next_token": "GROUP"
 39|    },
 40|    {
 41|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY ",
 42|      "completion": "city ",
 43|      "next_token": "HAVING"
 44|    },
 45|    {
 46|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING ",
 47|      "completion": "cnt > 10 ",
 48|      "next_token": "ORDER"
 49|    },
 50|    {
 51|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY ",
 52|      "completion": "avg_price DESC ",
 53|      "next_token": "LIMIT"
 54|    }
 55|  ]
 56|}
 57|```
 58|
 59|## Fields
 60|
 61|| Field | Type | Description |
 62||---|---|---|
 63|| `id` | string | Unique trajectory ID, format: `traj_NNN` |
 64|| `category` | string | Broad category (see Categories below) |
 65|| `subcategory` | string | More specific subcategory |
 66|| `description` | string | Human-readable description of what the query demonstrates |
 67|| `query` | string | The complete, syntactically valid ClickHouse query |
 68|| `tables` | string[] | Table names referenced in the query (for dictionary-based sampling) |
 69|| `columns` | object | Map of table name -> column names available on that table |
 70|| `dictionaries` | string[] | ClickHouse dictionary names available (if any) |
 71|| `settings` | object | ClickHouse SETTINGS clauses if relevant (e.g. `{"max_threads": "8"}`) |
 72|| `completion_points` | array | Array of prefix/completion pairs for training autocomplete |
 73|
 74|## Completion Points
 75|
 76|Each completion point represents a natural autocomplete boundary — a place
 77|where a user would pause and the model should predict the next segment.
 78|
 79|- `prefix`: The partial query up to this point (ends with a space or token boundary)
 80|- `completion`: The text that should be predicted next (up to the next natural boundary)
 81|- `next_token`: The first keyword/token of the completion (for constrained sampling)
 82|
 83|## Categories
 84|
 85|1. **basic_select** — Simple SELECT, WHERE, ORDER BY, LIMIT
 86|2. **aggregation** — GROUP BY, HAVING, aggregate functions, window functions
 87|3. **joins** — INNER/LEFT/RIGHT/FULL JOIN, ASOF JOIN, ARRAY JOIN, subqueries
 88|4. **ddl_dml** — CREATE TABLE, ALTER, DROP, INSERT, TRUNCATE, system queries
 89|5. **advanced** — Arrays, maps, JSON, materialized views, projections, dictionaries, TTL, CTEs, sampling
 90|
 91|## Rules
 92|
 93|1. All queries MUST be syntactically valid ClickHouse SQL
 94|2. Use realistic table and column names (not foo/bar)
 95|3. Include diverse ClickHouse-specific features (array functions, dictGet, etc.)
 96|4. Each trajectory should have 4-10 completion points
 97|5. Completion points should be at natural syntactic boundaries
 98|6. Table/column definitions must be consistent with the query
 99|7. Vary query complexity — some simple, some complex
100|8. Use ClickHouse-specific functions: count(), avg(), sum(), dictGet(), arrayJoin(), etc.
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/trajectory_format_spec.md" start_line="1" end_line="100" total_lines="100">
  1|# ClickHouse Autocomplete Trajectory Format Specification
  2|
  3|## Overview
  4|
  5|Each trajectory is a JSON object (one per line in JSONL format) representing a
  6|rich ClickHouse query autocomplete example. Trajectories are used to train a
  7|small GPT-based language model (~10 MB) that autocompletes ClickHouse queries.
  8|
  9|## Trajectory JSON Schema
 10|
 11|```json
 12|{
 13|  "id": "traj_001",
 14|  "category": "aggregation",
 15|  "subcategory": "group_by_having",
 16|  "description": "Aggregation with GROUP BY, HAVING, and ORDER BY",
 17|  "query": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY avg_price DESC LIMIT 20",
 18|  "tables": ["orders"],
 19|  "columns": {
 20|    "orders": ["id", "user_id", "price", "status", "city", "region", "created_at"]
 21|  },
 22|  "dictionaries": [],
 23|  "settings": {},
 24|  "completion_points": [
 25|    {
 26|      "prefix": "SELECT ",
 27|      "completion": "city, count() AS cnt, avg(price) AS avg_price ",
 28|      "next_token": "FROM"
 29|    },
 30|    {
 31|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM ",
 32|      "completion": "orders ",
 33|      "next_token": "WHERE"
 34|    },
 35|    {
 36|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE ",
 37|      "completion": "status = 'completed' ",
 38|      "next_token": "GROUP"
 39|    },
 40|    {
 41|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY ",
 42|      "completion": "city ",
 43|      "next_token": "HAVING"
 44|    },
 45|    {
 46|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING ",
 47|      "completion": "cnt > 10 ",
 48|      "next_token": "ORDER"
 49|    },
 50|    {
 51|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY ",
 52|      "completion": "avg_price DESC ",
 53|      "next_token": "LIMIT"
 54|    }
 55|  ]
 56|}
 57|```
 58|
 59|## Fields
 60|
 61|| Field | Type | Description |
 62||---|---|---|
 63|| `id` | string | Unique trajectory ID, format: `traj_NNN` |
 64|| `category` | string | Broad category (see Categories below) |
 65|| `subcategory` | string | More specific subcategory |
 66|| `description` | string | Human-readable description of what the query demonstrates |
 67|| `query` | string | The complete, syntactically valid ClickHouse query |
 68|| `tables` | string[] | Table names referenced in the query (for dictionary-based sampling) |
 69|| `columns` | object | Map of table name -> column names available on that table |
 70|| `dictionaries` | string[] | ClickHouse dictionary names available (if any) |
 71|| `settings` | object | ClickHouse SETTINGS clauses if relevant (e.g. `{"max_threads": "8"}`) |
 72|| `completion_points` | array | Array of prefix/completion pairs for training autocomplete |
 73|
 74|## Completion Points
 75|
 76|Each completion point represents a natural autocomplete boundary — a place
 77|where a user would pause and the model should predict the next segment.
 78|
 79|- `prefix`: The partial query up to this point (ends with a space or token boundary)
 80|- `completion`: The text that should be predicted next (up to the next natural boundary)
 81|- `next_token`: The first keyword/token of the completion (for constrained sampling)
 82|
 83|## Categories
 84|
 85|1. **basic_select** — Simple SELECT, WHERE, ORDER BY, LIMIT
 86|2. **aggregation** — GROUP BY, HAVING, aggregate functions, window functions
 87|3. **joins** — INNER/LEFT/RIGHT/FULL JOIN, ASOF JOIN, ARRAY JOIN, subqueries
 88|4. **ddl_dml** — CREATE TABLE, ALTER, DROP, INSERT, TRUNCATE, system queries
 89|5. **advanced** — Arrays, maps, JSON, materialized views, projections, dictionaries, TTL, CTEs, sampling
 90|
 91|## Rules
 92|
 93|1. All queries MUST be syntactically valid ClickHouse SQL
 94|2. Use realistic table and column names (not foo/bar)
 95|3. Include diverse ClickHouse-specific features (array functions, dictGet, etc.)
 96|4. Each trajectory should have 4-10 completion points
 97|5. Completion points should be at natural syntactic boundaries
 98|6. Table/column definitions must be consistent with the query
 99|7. Vary query complexity — some simple, some complex
100|8. Use ClickHouse-specific functions: count(), avg(), sum(), dictGet(), arrayJoin(), etc.
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/trajectory_format_spec.md" start_line="1" end_line="100" total_lines="100">
  1|# ClickHouse Autocomplete Trajectory Format Specification
  2|
  3|## Overview
  4|
  5|Each trajectory is a JSON object (one per line in JSONL format) representing a
  6|rich ClickHouse query autocomplete example. Trajectories are used to train a
  7|small GPT-based language model (~10 MB) that autocompletes ClickHouse queries.
  8|
  9|## Trajectory JSON Schema
 10|
 11|```json
 12|{
 13|  "id": "traj_001",
 14|  "category": "aggregation",
 15|  "subcategory": "group_by_having",
 16|  "description": "Aggregation with GROUP BY, HAVING, and ORDER BY",
 17|  "query": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY avg_price DESC LIMIT 20",
 18|  "tables": ["orders"],
 19|  "columns": {
 20|    "orders": ["id", "user_id", "price", "status", "city", "region", "created_at"]
 21|  },
 22|  "dictionaries": [],
 23|  "settings": {},
 24|  "completion_points": [
 25|    {
 26|      "prefix": "SELECT ",
 27|      "completion": "city, count() AS cnt, avg(price) AS avg_price ",
 28|      "next_token": "FROM"
 29|    },
 30|    {
 31|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM ",
 32|      "completion": "orders ",
 33|      "next_token": "WHERE"
 34|    },
 35|    {
 36|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE ",
 37|      "completion": "status = 'completed' ",
 38|      "next_token": "GROUP"
 39|    },
 40|    {
 41|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY ",
 42|      "completion": "city ",
 43|      "next_token": "HAVING"
 44|    },
 45|    {
 46|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING ",
 47|      "completion": "cnt > 10 ",
 48|      "next_token": "ORDER"
 49|    },
 50|    {
 51|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY ",
 52|      "completion": "avg_price DESC ",
 53|      "next_token": "LIMIT"
 54|    }
 55|  ]
 56|}
 57|```
 58|
 59|## Fields
 60|
 61|| Field | Type | Description |
 62||---|---|---|
 63|| `id` | string | Unique trajectory ID, format: `traj_NNN` |
 64|| `category` | string | Broad category (see Categories below) |
 65|| `subcategory` | string | More specific subcategory |
 66|| `description` | string | Human-readable description of what the query demonstrates |
 67|| `query` | string | The complete, syntactically valid ClickHouse query |
 68|| `tables` | string[] | Table names referenced in the query (for dictionary-based sampling) |
 69|| `columns` | object | Map of table name -> column names available on that table |
 70|| `dictionaries` | string[] | ClickHouse dictionary names available (if any) |
 71|| `settings` | object | ClickHouse SETTINGS clauses if relevant (e.g. `{"max_threads": "8"}`) |
 72|| `completion_points` | array | Array of prefix/completion pairs for training autocomplete |
 73|
 74|## Completion Points
 75|
 76|Each completion point represents a natural autocomplete boundary — a place
 77|where a user would pause and the model should predict the next segment.
 78|
 79|- `prefix`: The partial query up to this point (ends with a space or token boundary)
 80|- `completion`: The text that should be predicted next (up to the next natural boundary)
 81|- `next_token`: The first keyword/token of the completion (for constrained sampling)
 82|
 83|## Categories
 84|
 85|1. **basic_select** — Simple SELECT, WHERE, ORDER BY, LIMIT
 86|2. **aggregation** — GROUP BY, HAVING, aggregate functions, window functions
 87|3. **joins** — INNER/LEFT/RIGHT/FULL JOIN, ASOF JOIN, ARRAY JOIN, subqueries
 88|4. **ddl_dml** — CREATE TABLE, ALTER, DROP, INSERT, TRUNCATE, system queries
 89|5. **advanced** — Arrays, maps, JSON, materialized views, projections, dictionaries, TTL, CTEs, sampling
 90|
 91|## Rules
 92|
 93|1. All queries MUST be syntactically valid ClickHouse SQL
 94|2. Use realistic table and column names (not foo/bar)
 95|3. Include diverse ClickHouse-specific features (array functions, dictGet, etc.)
 96|4. Each trajectory should have 4-10 completion points
 97|5. Completion points should be at natural syntactic boundaries
 98|6. Table/column definitions must be consistent with the query
 99|7. Vary query complexity — some simple, some complex
100|8. Use ClickHouse-specific functions: count(), avg(), sum(), dictGet(), arrayJoin(), etc.
</file-view>

Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
Current todo list:
1. [x] Research ClickHouse Lexer token types and syntax elements [completed]
2. [x] Design project structure and trajectory format spec [completed]
3. [x] Launch 5 parallel GLM 5.2 subagents to generate 20 trajectories each (100 total) [completed]
4. [~] Implement ClickHouse tokenizer with lookbehind copies for identifiers/strings [in_progress]
5. [ ] Implement GPT-based SLM model (~10MB, CPU-optimized) [pending]
6. [ ] Implement training pipeline [pending]
7. [ ] Train model on generated trajectories [pending]
8. [ ] Implement inference/autocomplete with syntactic correctness sampling [pending]
9. [ ] Implement C++ export for ClickHouse embedding and WASM export [pending]

Output from command in shell a0f653:
---

Exit code: 1

Output from command in shell f41592:


Exit code: 0

Output from command in shell f53d6f:


Exit code: 0

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_04_ddl_dml.jsonl

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl

Output from command in shell 143f10:
Traceback (most recent call last):
  File "<string>", line 12, in <module>
AssertionError: Line 10 has 3 completion_points (need 4-10)
Total lines: 20
Line 1: traj_081 | array_join | completion_points=6 | tables=['user_events']
Line 2: traj_082 | array_map_filter | completion_points=5 | tables=['product_pricing']
Line 3: traj_083 | array_distinct_sort_concat | completion_points=4 | tables=['article_tags']
Line 4: traj_084 | array_enumerate_reduce | completion_points=5 | tables=['web_sessions']
Line 5: traj_085 | map_operations | completion_points=4 | tables=['regional_metrics']
Line 6: traj_086 | json_extract | completion_points=5 | tables=['api_logs']
Line 7: traj_087 | tuples_nested | completion_points=6 | tables=['customer_orders']
Line 8: traj_088 | low_cardinality | completion_points=5 | tables=['support_tickets']
Line 9: traj_089 | materialized_view_to | completion_points=5 | tables=['sales_raw', 'daily_sales_summary']
Line 10: traj_090 | projection | completion_points=3 | tables=['orders_by_region']


Exit code: 1

Output from command in shell e24153:
Line 1: traj_061 - 6 completion_points
Line 2: traj_062 - 5 completion_points
Line 3: traj_063 - 5 completion_points
Line 4: traj_064 - 5 completion_points
Line 5: traj_065 - 5 completion_points
Line 6: traj_066 - 5 completion_points
Line 7: traj_067 - 6 completion_points
Line 8: traj_068 - 4 completion_points
Line 9: traj_069 - 3 completion_points <-- TOO FEW
Line 10: traj_070 - 2 completion_points <-- TOO FEW
Line 11: traj_071 - 5 completion_points
Line 12: traj_072 - 4 completion_points
Line 13: traj_073 - 3 completion_points <-- TOO FEW
Line 14: traj_074 - 4 completion_points
Line 15: traj_075 - 3 completion_points <-- TOO FEW
Line 16: traj_076 - 3 completion_points <-- TOO FEW
Line 17: traj_077 - 3 completion_points <-- TOO FEW
Line 18: traj_078 - 6 completion_points
Line 19: traj_079 - 4 completion_points
Line 20: traj_080 - 2 completion_points <-- TOO FEW


Exit code: 0

The file /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
  7|{"id":"traj_087","category":"advanced","subcategory":"tuples_nested","description":"Tuple and Nested type access with tupleElement for structured column data","query":"SELECT order_id, tupleElement(location, 1) AS city, tupleElement(location, 2) AS country, nested_items.amount AS item_amount FROM customer_orders ARRAY JOIN nested_items WHERE tupleElement(location, 2) = 'USA' ORDER BY item_amount DESC LIMIT 100","tables":["customer_orders"],"columns":{"customer_orders":["order_id","location","nested_items","customer_id","order_date"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"order_id, tupleElement(location, 1) AS city, tupleElement(location, 2) AS country, nested_items.amount AS item_amount ","next_token":"order_id"},{"prefix":"SELECT order_id, tupleElement(location, 1) AS city, tupleElement(location, 2) AS country, nested_items.amount AS item_amount FROM ","completion":"customer_orders ","next_token":"customer_orders"},{"prefix":"SELECT order_id, tupleElement(location, 1) AS city, tupleElement(location, 2) AS country, nested_items.amount AS item_amount FROM customer_orders ARRAY JOIN ","completion":"nested_items ","next_token":"nested_items"},{"prefix":"SELECT order_id, tupleElement(location, 1) AS city, tupleElement(location, 2) AS country, nested_items.amount AS item_amount FROM customer_orders ARRAY JOIN nested_items WHERE ","completion":"tupleElement(location, 2) = 'USA' ","next_token":"tupleElement"},{"prefix":"SELECT order_id, tupleElement(location, 1) AS city, tupleElement(location, 2) AS country, nested_items.amount AS item_amount FROM customer_orders ARRAY JOIN nested_items WHERE tupleElement(location, 2) = 'USA' ORDER BY ","completion":"item_amount DESC ","next_token":"item_amount"},{"prefix":"SELECT order_id, tupleElement(location, 1) AS city, tupleElement(location, 2) AS country, nested_items.amount AS item_amount FROM customer_orders ARRAY JOIN nested_items WHERE tupleElement(location, 2) = 'USA' ORDER BY item_amount DESC LIMIT ","completion":"100","next_token":"100"}]}
  8|{"id":"traj_088","category":"advanced","subcategory":"low_cardinality","description":"LowCardinality column queries with toTypeName and string operations","query":"SELECT toTypeName(status) AS status_type, status, count() AS cnt FROM support_tickets WHERE status IN ('open', 'closed', 'pending') GROUP BY status ORDER BY cnt DESC","tables":["support_tickets"],"columns":{"support_tickets":["ticket_id","status","priority","category","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"toTypeName(status) AS status_type, status, count() AS cnt ","next_token":"toTypeName"},{"prefix":"SELECT toTypeName(status) AS status_type, status, count() AS cnt FROM ","completion":"support_tickets ","next_token":"support_tickets"},{"prefix":"SELECT toTypeName(status) AS status_type, status, count() AS cnt FROM support_tickets WHERE ","completion":"status IN ('open', 'closed', 'pending') ","next_token":"status"},{"prefix":"SELECT toTypeName(status) AS status_type, status, count() AS cnt FROM support_tickets WHERE status IN ('open', 'closed', 'pending') GROUP BY ","completion":"status ","next_token":"status"},{"prefix":"SELECT toTypeName(status) AS status_type, status, count() AS cnt FROM support_tickets WHERE status IN ('open', 'closed', 'pending') GROUP BY status ORDER BY ","completion":"cnt DESC","next_token":"cnt"}]}
  9|{"id":"traj_089","category":"advanced","subcategory":"materialized_view_to","description":"CREATE MATERIALIZED VIEW with TO clause targeting a summary table","query":"CREATE MATERIALIZED VIEW mv_daily_sales TO daily_sales_summary AS SELECT toDate(sale_time) AS sale_date, store_id, sum(amount) AS total_amount, count() AS sale_count FROM sales_raw GROUP BY sale_date, store_id","tables":["sales_raw","daily_sales_summary"],"columns":{"sales_raw":["sale_id","sale_time","store_id","amount","product_id"],"daily_sales_summary":["sale_date","store_id","total_amount","sale_count"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"CREATE MATERIALIZED VIEW ","completion":"mv_daily_sales ","next_token":"mv_daily_sales"},{"prefix":"CREATE MATERIALIZED VIEW mv_daily_sales TO ","completion":"daily_sales_summary ","next_token":"daily_sales_summary"},{"prefix":"CREATE MATERIALIZED VIEW mv_daily_sales TO daily_sales_summary AS ","completion":"SELECT toDate(sale_time) AS sale_date, store_id, sum(amount) AS total_amount, count() AS sale_count ","next_token":"SELECT"},{"prefix":"CREATE MATERIALIZED VIEW mv_daily_sales TO daily_sales_summary AS SELECT toDate(sale_time) AS sale_date, store_id, sum(amount) AS total_amount, count() AS sale_count FROM ","completion":"sales_raw ","next_token":"sales_raw"},{"prefix":"CREATE MATERIALIZED VIEW mv_daily_sales TO daily_sales_summary AS SELECT toDate(sale_time) AS sale_date, store_id, sum(amount) AS total_amount, count() AS sale_count FROM sales_raw GROUP BY ","completion":"sale_date, store_id","next_token":"sale_date"}]}
 10|{"id":"traj_090","category":"advanced","subcategory":"projection","description":"ALTER TABLE ADD PROJECTION and MATERIALIZE PROJECTION for query acceleration","query":"ALTER TABLE orders_by_region ADD PROJECTION proj_region_sum (SELECT region, sum(total) GROUP BY region)","tables":["orders_by_region"],"columns":{"orders_by_region":["order_id","region","total","order_date","customer_id"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"ALTER TABLE ","completion":"orders_by_region ","next_token":"orders_by_region"},{"prefix":"ALTER TABLE orders_by_region ADD PROJECTION ","completion":"proj_region_sum ","next_token":"proj_region_sum"},{"prefix":"ALTER TABLE orders_by_region ADD PROJECTION proj_region_sum (","completion":"SELECT region, sum(total) ","next_token":"SELECT"},{"prefix":"ALTER TABLE orders_by_region ADD PROJECTION proj_region_sum (SELECT region, sum(total) ","completion":"GROUP BY region","next_token":"GROUP"}]}
 11|{"id":"traj_091","category":"advanced","subcategory":"dict_get","description":"dictGet and dictGetOrDefault with dictHas for enriching query results from a dictionary","query":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM users u WHERE dictHas('country_dict', u.country_code) ORDER BY u.user_id LIMIT 1000","tables":["users"],"columns":{"users":["user_id","country_code","city_id","email","signup_date"]},"dictionaries":["country_dict","city_dict"],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name ","next_token":"u"},{"prefix":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM ","completion":"users u ","next_token":"users"},{"prefix":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM users u WHERE ","completion":"dictHas('country_dict', u.country_code) ","next_token":"dictHas"},{"prefix":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM users u WHERE dictHas('country_dict', u.country_code) ORDER BY ","completion":"u.user_id ","next_token":"u"},{"prefix":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM users u WHERE dictHas('country_dict', u.country_code) ORDER BY u.user_id LIMIT ","completion":"1000","next_token":"1000"}]}
 12|{"id":"traj_092","category":"advanced","subcategory":"dict_hierarchy","description":"dictGetChildren and dictGetAll for hierarchical dictionary traversal","query":"SELECT category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names FROM product_categories WHERE length(dictGetChildren('category_hierarchy', category_id)) > 0 ORDER BY category_id","tables":["product_categories"],"columns":{"product_categories":["category_id","parent_id","category_name","created_at"]},"dictionaries":["category_hierarchy"],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names ","next_token":"category_id"},{"prefix":"SELECT category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names FROM ","completion":"product_categories ","next_token":"product_categories"},{"prefix":"SELECT category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names FROM product_categories WHERE ","completion":"length(dictGetChildren('category_hierarchy', category_id)) > 0 ","next_token":"length"},{"prefix":"SELECT category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names FROM product_categories WHERE length(dictGetChildren('category_hierarchy', category_id)) > 0 ORDER BY ","completion":"category_id","next_token":"catego… (9 chars truncated)
… (3 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/dd03cbfd/content.txt
</truncation_notice>

Output from command in shell 105dc0:
Total lines: 20
Line 1: traj_081 | array_join | cp=6 | dicts=[] | settings={}
Traceback (most recent call last):
Line 2: traj_082 | array_map_filter | cp=5 | dicts=[] | settings={}
Line 3: traj_083 | array_distinct_sort_concat | cp=4 | dicts=[] | settings={}
Line 4: traj_084 | array_enumerate_reduce | cp=5 | dicts=[] | settings={}
Line 5: traj_085 | map_operations | cp=4 | dicts=[] | settings={}
Line 6: traj_086 | json_extract | cp=5 | dicts=[] | settings={}
Line 7: traj_087 | tuples_nested | cp=6 | dicts=[] | settings={}
Line 8: traj_088 | low_cardinality | cp=5 | dicts=[] | settings={}
Line 9: traj_089 | materialized_view_to | cp=5 | dicts=[] | settings={}
Line 10: traj_090 | projection | cp=4 | dicts=[] | settings={}
  File "<string>", line 11, in <module>
AssertionError: Line 13 has 2 completion_points (need 4-10)
Line 11: traj_091 | dict_get | cp=5 | dicts=['country_dict', 'city_dict'] | settings={}
Line 12: traj_092 | dict_hierarchy | cp=4 | dicts=['category_hierarchy'] | settings={}


Exit code: 1

The file /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
 10|{"id":"traj_090","category":"advanced","subcategory":"projection","description":"ALTER TABLE ADD PROJECTION and MATERIALIZE PROJECTION for query acceleration","query":"ALTER TABLE orders_by_region ADD PROJECTION proj_region_sum (SELECT region, sum(total) GROUP BY region)","tables":["orders_by_region"],"columns":{"orders_by_region":["order_id","region","total","order_date","customer_id"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"ALTER TABLE ","completion":"orders_by_region ","next_token":"orders_by_region"},{"prefix":"ALTER TABLE orders_by_region ADD PROJECTION ","completion":"proj_region_sum ","next_token":"proj_region_sum"},{"prefix":"ALTER TABLE orders_by_region ADD PROJECTION proj_region_sum (","completion":"SELECT region, sum(total) ","next_token":"SELECT"},{"prefix":"ALTER TABLE orders_by_region ADD PROJECTION proj_region_sum (SELECT region, sum(total) ","completion":"GROUP BY region","next_token":"GROUP"}]}
 11|{"id":"traj_091","category":"advanced","subcategory":"dict_get","description":"dictGet and dictGetOrDefault with dictHas for enriching query results from a dictionary","query":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM users u WHERE dictHas('country_dict', u.country_code) ORDER BY u.user_id LIMIT 1000","tables":["users"],"columns":{"users":["user_id","country_code","city_id","email","signup_date"]},"dictionaries":["country_dict","city_dict"],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name ","next_token":"u"},{"prefix":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM ","completion":"users u ","next_token":"users"},{"prefix":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM users u WHERE ","completion":"dictHas('country_dict', u.country_code) ","next_token":"dictHas"},{"prefix":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM users u WHERE dictHas('country_dict', u.country_code) ORDER BY ","completion":"u.user_id ","next_token":"u"},{"prefix":"SELECT u.user_id, u.country_code, dictGet('country_dict', 'country_name', u.country_code) AS country_name, dictGetOrDefault('city_dict', 'city_name', tuple(u.city_id), 'Unknown') AS city_name FROM users u WHERE dictHas('country_dict', u.country_code) ORDER BY u.user_id LIMIT ","completion":"1000","next_token":"1000"}]}
 12|{"id":"traj_092","category":"advanced","subcategory":"dict_hierarchy","description":"dictGetChildren and dictGetAll for hierarchical dictionary traversal","query":"SELECT category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names FROM product_categories WHERE length(dictGetChildren('category_hierarchy', category_id)) > 0 ORDER BY category_id","tables":["product_categories"],"columns":{"product_categories":["category_id","parent_id","category_name","created_at"]},"dictionaries":["category_hierarchy"],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names ","next_token":"category_id"},{"prefix":"SELECT category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names FROM ","completion":"product_categories ","next_token":"product_categories"},{"prefix":"SELECT category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names FROM product_categories WHERE ","completion":"length(dictGetChildren('category_hierarchy', category_id)) > 0 ","next_token":"length"},{"prefix":"SELECT category_id, dictGetChildren('category_hierarchy', category_id) AS child_ids, dictGetAll('category_hierarchy', 'category_name', category_id) AS all_names FROM product_categories WHERE length(dictGetChildren('category_hierarchy', category_id)) > 0 ORDER BY ","completion":"category_id","next_token":"category_id"}]}
 13|{"id":"traj_093","category":"advanced","subcategory":"ttl_expressions","description":"TTL with DELETE, RECOMPRESS, and MOVE for lifecycle management in ALTER TABLE","query":"ALTER TABLE event_logs MODIFY TTL event_date + INTERVAL 30 DAY DELETE, event_date + INTERVAL 15 DAY RECOMPRESS CODEC(ZSTD(9)), event_date + INTERVAL 7 DAY MOVE TO TABLE archive_logs","tables":["event_logs","archive_logs"],"columns":{"event_logs":["event_id","event_date","event_type","payload","created_at"],"archive_logs":["event_id","event_date","event_type","payload","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"ALTER TABLE ","completion":"event_logs ","next_token":"event_logs"},{"prefix":"ALTER TABLE event_logs MODIFY TTL ","completion":"event_date + INTERVAL 30 DAY DELETE, ","next_token":"event_date"},{"prefix":"ALTER TABLE event_logs MODIFY TTL event_date + INTERVAL 30 DAY DELETE, ","completion":"event_date + INTERVAL 15 DAY RECOMPRESS CODEC(ZSTD(9)), ","next_token":"event_date"},{"prefix":"ALTER TABLE event_logs MODIFY TTL event_date + INTERVAL 30 DAY DELETE, event_date + INTERVAL 15 DAY RECOMPRESS CODEC(ZSTD(9)), ","completion":"event_date + INTERVAL 7 DAY MOVE TO TABLE archive_logs","next_token":"event_date"}]}
 14|{"id":"traj_094","category":"advanced","subcategory":"sampling","description":"SAMPLE clause with fraction, count, and ratio for approximate query results","query":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE 0.1 WHERE sale_date >= today() - INTERVAL 7 DAY GROUP BY region ORDER BY avg_rev DESC","tables":["sales_data"],"columns":{"sales_data":["sale_id","region","revenue","sale_date","store_id"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"region, count() AS sample_cnt, avg(revenue) AS avg_rev ","next_token":"region"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM ","completion":"sales_data ","next_token":"sales_data"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE ","completion":"0.1 ","next_token":"0.1"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE 0.1 WHERE ","completion":"sale_date >= today() - INTERVAL 7 DAY ","next_token":"sale_date"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE 0.1 WHERE sale_date >= today() - INTERVAL 7 DAY GROUP BY ","completion":"region ","next_token":"region"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE 0.1 WHERE sale_date >= today() - INTERVAL 7 DAY GROUP BY region ORDER BY ","completion":"avg_rev DESC","next_token":"avg_rev"}]}
 15|{"id":"traj_095","category":"advanced","subcategory":"cte_expression","description":"CTE with WITH expression AS identifier for reusable scalar calculations","query":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING total > avg_rounded ORDER BY total DESC LIMIT 100","tables":["customer_payments"],"columns":{"customer_payments":["payment_id","customer_id","amount","payment_date","method"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"WITH ","completion":"round(avg_amount, 2) AS avg_rounded ","next_token":"round"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT ","completion":"customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total ","next_token":"customer_id"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM ","completion":"customer_payments ","next_token":"customer_payments"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY ","completion":"customer_id ","next_token":"customer_id"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING ","completion":"total > avg_rounded ","next_token":"total"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING total > avg_rounded ORDER BY ","completion":"total DESC ","next_token":"total"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING total > avg_rounded ORDER BY total DESC LIMIT ","completion":"100","next_token":"100"}]}
 16|{"id":"traj_096","category":"advanced","subcategory":"cte_subquery","description":"CTE with WITH subquery AS identifier for reusable subquery results","query":"WITH top_regions AS (SELECT region, sum(revenue) AS t… (2053 chars truncated)
… (2 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/ce615af7/content.txt
</truncation_notice>

Output from command in shell ac892c:
Total lines: 20
Traceback (most recent call last):
  File "<string>", line 11, in <module>
AssertionError: Line 17 has 3 completion_points (need 4-10)
Line 1: traj_081 | array_join | cp=6
Line 2: traj_082 | array_map_filter | cp=5
Line 3: traj_083 | array_distinct_sort_concat | cp=4
Line 4: traj_084 | array_enumerate_reduce | cp=5
Line 5: traj_085 | map_operations | cp=4
Line 6: traj_086 | json_extract | cp=5
Line 7: traj_087 | tuples_nested | cp=6
Line 8: traj_088 | low_cardinality | cp=5
Line 9: traj_089 | materialized_view_to | cp=5
Line 10: traj_090 | projection | cp=4
Line 11: traj_091 | dict_get | cp=5
Line 12: traj_092 | dict_hierarchy | cp=4
Line 13: traj_093 | ttl_expressions | cp=4
Line 14: traj_094 | sampling | cp=6
Line 15: traj_095 | cte_expression | cp=7
Line 16: traj_096 | cte_subquery | cp=6


Exit code: 1

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/tokenizer/ch_tokenizer.py

The file /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
 14|{"id":"traj_094","category":"advanced","subcategory":"sampling","description":"SAMPLE clause with fraction, count, and ratio for approximate query results","query":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE 0.1 WHERE sale_date >= today() - INTERVAL 7 DAY GROUP BY region ORDER BY avg_rev DESC","tables":["sales_data"],"columns":{"sales_data":["sale_id","region","revenue","sale_date","store_id"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"region, count() AS sample_cnt, avg(revenue) AS avg_rev ","next_token":"region"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM ","completion":"sales_data ","next_token":"sales_data"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE ","completion":"0.1 ","next_token":"0.1"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE 0.1 WHERE ","completion":"sale_date >= today() - INTERVAL 7 DAY ","next_token":"sale_date"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE 0.1 WHERE sale_date >= today() - INTERVAL 7 DAY GROUP BY ","completion":"region ","next_token":"region"},{"prefix":"SELECT region, count() AS sample_cnt, avg(revenue) AS avg_rev FROM sales_data SAMPLE 0.1 WHERE sale_date >= today() - INTERVAL 7 DAY GROUP BY region ORDER BY ","completion":"avg_rev DESC","next_token":"avg_rev"}]}
 15|{"id":"traj_095","category":"advanced","subcategory":"cte_expression","description":"CTE with WITH expression AS identifier for reusable scalar calculations","query":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING total > avg_rounded ORDER BY total DESC LIMIT 100","tables":["customer_payments"],"columns":{"customer_payments":["payment_id","customer_id","amount","payment_date","method"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"WITH ","completion":"round(avg_amount, 2) AS avg_rounded ","next_token":"round"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT ","completion":"customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total ","next_token":"customer_id"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM ","completion":"customer_payments ","next_token":"customer_payments"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY ","completion":"customer_id ","next_token":"customer_id"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING ","completion":"total > avg_rounded ","next_token":"total"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING total > avg_rounded ORDER BY ","completion":"total DESC ","next_token":"total"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING total > avg_rounded ORDER BY total DESC LIMIT ","completion":"100","next_token":"100"}]}
 16|{"id":"traj_096","category":"advanced","subcategory":"cte_subquery","description":"CTE with WITH subquery AS identifier for reusable subquery results","query":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT s.region, s.store_id, sum(s.revenue) AS store_total FROM sales_data s INNER JOIN top_regions t ON s.region = t.region GROUP BY s.region, s.store_id ORDER BY s.region, store_total DESC","tables":["sales_data"],"columns":{"sales_data":["sale_id","region","store_id","revenue","sale_date"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"WITH ","completion":"top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) ","next_token":"top_regions"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT ","completion":"s.region, s.store_id, sum(s.revenue) AS store_total ","next_token":"s"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT s.region, s.store_id, sum(s.revenue) AS store_total FROM ","completion":"sales_data s ","next_token":"sales_data"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT s.region, s.store_id, sum(s.revenue) AS store_total FROM sales_data s INNER JOIN ","completion":"top_regions t ON s.region = t.region ","next_token":"top_regions"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT s.region, s.store_id, sum(s.revenue) AS store_total FROM sales_data s INNER JOIN top_regions t ON s.region = t.region GROUP BY ","completion":"s.region, s.store_id ","next_token":"s"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT s.region, s.store_id, sum(s.revenue) AS store_total FROM sales_data s INNER JOIN top_regions t ON s.region = t.region GROUP BY s.region, s.store_id ORDER BY ","completion":"s.region, store_total DESC","next_token":"s"}]}
 17|{"id":"traj_097","category":"advanced","subcategory":"window_frame","description":"Window function with explicit frame specification ROWS BETWEEN for running totals","query":"SELECT order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, avg(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_avg FROM store_revenue ORDER BY store_id, order_date","tables":["store_revenue"],"columns":{"store_revenue":["store_id","order_date","revenue","region"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, ","next_token":"order_date"},{"prefix":"SELECT order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, ","completion":"avg(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_avg ","next_token":"avg"},{"prefix":"SELECT order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, avg(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_avg FROM ","completion":"store_revenue ","next_token":"store_revenue"},{"prefix":"SELECT order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, avg(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_avg FROM store_revenue ORDER BY ","completion":"store_id, order_date","next_token":"store_id"}]}
 18|{"id":"traj_098","category":"advanced","subcategory":"materialized_alias_columns","description":"Query referencing MATERIALIZED and ALIAS columns with type conversion functions","query":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM purchase_records WHERE order_date >= toDate('2024-01-01') ORDER BY order_date DESC LIMIT 500","tables":["purchase_records"],"columns":{"purchase_records":["order_id","quantity","unit_price","order_date","computed_total","customer_label"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date ","next_token":"order_id"},{"prefix":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM ","completion":"purchase_records ","next_token":"purchase_records"},{"prefix":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM purchase_records WHERE ","completion":"order_date >= toDate('2024-01-01') ","next_token":"order_date"},{"prefix":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM purchase_records WHERE order_date >= toDate('2024-01-01') ORDER BY ","completion":"order_date DESC ","next_token":"order_date"},{"prefix":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM purchase_records WHERE order_date >= toDate('2024-01-01') ORDER BY order_date DESC LIMIT ","completion":"500","next_token":"500"}]}
 19|{"id":"traj_099","category":"ad… (1629 chars truncated)
… (1 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/9d5a657b/content.txt
</truncation_notice>

Output from command in shell 337493:

Query: SELECT count(), avg(price) FROM orders WHERE region = 'US' GROUP BY city ORDER BY count() DESC LIMIT 10
  KW_SELECT
  NEW_IDENT [count]
  LPAREN
  RPAREN
  COMMA
  NEW_IDENT [avg]
  LPAREN
  NEW_IDENT [price]
  RPAREN
  KW_FROM
  NEW_IDENT [orders]
  KW_WHERE
  NEW_IDENT [region]
  EQUALS
  NEW_STRING ['US']
  KW_GROUP
  KW_BY
  NEW_IDENT [city]
  KW_ORDER
  KW_BY
  COPY_BACK_19 [count]
  LPAREN
  RPAREN
  KW_DESC
  KW_LIMIT
  NUM_INT
  IDs: [1, 189, 275, 243, 244, 249, 275, 243, 275, 244, 85, 275, 236, 275, 262, 308, 92, 23, 275, 151, 23, 294, 243, 244, 51, 125, 272, 2]

Query: SELECT * FROM events WHERE event_type = 'click' AND user_id IN (SELECT user_id FROM users WHERE country = 'US') ORDER BY timestamp DESC LIMIT 100
  KW_SELECT
  ASTERISK
  KW_FROM
  KW_EVENTS
  KW_WHERE
  NEW_IDENT [event_type]
  EQUALS
  NEW_STRING ['click']
  KW_AND
  NEW_IDENT [user_id]
  KW_IN
  LPAREN
  KW_SELECT
  COPY_BACK_4 [user_id]
  KW_FROM
  KW_USERS
  KW_WHERE
  NEW_IDENT [country]
  EQUALS
  NEW_STRING ['US']
  RPAREN
  KW_ORDER
  KW_BY
  KW_TIMESTAMP
  KW_DESC
  KW_LIMIT
  NUM_INT
  IDs: [1, 189, 252, 85, 67, 236, 275, 262, 308, 10, 275, 101, 243, 189, 279, 85, 227, 236, 275, 262, 308, 244, 151, 23, 212, 51, 125, 272, 2]

Query: CREATE TABLE IF NOT EXISTS analytics.events (id UUID, event_time DateTime) ENGINE = MergeTree() ORDER BY (event_time)
  KW_CREATE
  KW_TABLE
  KW_IF
  KW_NOT
  KW_EXISTS
  NEW_IDENT [analytics]
  DOT
  KW_EVENTS
  LPAREN
  KW_ID
  KW_UUID
  COMMA
  NEW_IDENT [event_time]
  NEW_IDENT [DateTime]
  RPAREN
  KW_ENGINE
  EQUALS
  NEW_IDENT [MergeTree]
  LPAREN
  RPAREN
  KW_ORDER
  KW_BY
  LPAREN
  COPY_BACK_11 [event_time]
  RPAREN
  IDs: [1, 37, 205, 98, 144, 69, 275, 251, 67, 243, 97, 229, 249, 275, 275, 244, 64, 262, 275, 243, 244, 151, 23, 243, 286, 244, 2]

Vocab size: 341


Exit code: 0

Output from command in shell bf307a:
Total lines: 20
Line 1: traj_081 | array_join | cp=6
Line 2: traj_082 | array_map_filter | cp=5
Line 3: traj_083 | array_distinct_sort_concat | cp=4
Line 4: traj_084 | array_enumerate_reduce | cp=5
Line 5: traj_085 | map_operations | cp=4
Line 6: traj_086 | json_extract | cp=5
Line 7: traj_087 | tuples_nested | cp=6
Line 8: traj_088 | low_cardinality | cp=5
Line 9: traj_089 | materialized_view_to | cp=5
Line 10: traj_090 | projection | cp=4
Line 11: traj_091 | dict_get | cp=5
Line 12: traj_092 | dict_hierarchy | cp=4
Line 13: traj_093 | ttl_expressions | cp=4
Line 14: traj_094 | sampling | cp=6
Line 15: traj_095 | cte_expression | cp=7
Line 16: traj_096 | cte_subquery | cp=6
Line 17: traj_097 | window_frame | cp=4
Line 18: traj_098 | materialized_alias_columns | cp=5
Line 19: traj_099 | with_fill_qualify_groupby_all | cp=6
Line 20: traj_100 | settings_interval_datetime | cp=5
All JSON valid!
ID sequence correct: traj_081 through traj_100


Exit code: 0

Output from command in shell 3f49e7:
Vocab saved


Exit code: 0

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_01_basic_select.jsonl

Output from command in shell 0446ef:
      20 trajectories_05_advanced.jsonl
All fields present, all category=advanced
File size: 34388 bytes


Exit code: 0

Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
Current todo list:
1. [x] Research ClickHouse Lexer token types and syntax elements [completed]
2. [x] Design project structure and trajectory format spec [completed]
3. [x] Launch 5 parallel GLM 5.2 subagents to generate 20 trajectories each (100 total) [completed]
4. [x] Implement ClickHouse tokenizer with lookbehind copies for identifiers/strings [completed]
5. [~] Implement GPT-based SLM model (~10MB, CPU-optimized) [in_progress]
6. [ ] Implement training pipeline [pending]
7. [ ] Train model on generated trajectories [pending]
8. [ ] Implement inference/autocomplete with syntactic correctness sampling [pending]
9. [ ] Implement C++ export for ClickHouse embedding and WASM export [pending]

Output from command in shell 166555:
Total lines: 20
traj_001: simple_select - 4 completion points - OK
traj_002: where_and_orderby - 7 completion points - OK
traj_003: distinct - 6 completion points - OK
traj_004: limit_offset - 7 completion points - OK
traj_005: prewhere - 7 completion points - OK
traj_006: final - 7 completion points - OK
traj_007: sample - 7 completion points - OK
traj_008: union_all - 7 completion points - OK
traj_009: in_subquery - 7 completion points - OK
traj_010: between_like - 7 completion points - OK
traj_011: is_null_not_in - 7 completion points - OK
traj_012: or_conditions - 7 completion points - OK
traj_013: subquery_from - 6 completion points - OK
traj_014: is_not_null_in - 7 completion points - OK
traj_015: sample_final - 8 completion points - OK
traj_016: multi_column_orderby - 6 completion points - OK
traj_017: column_aliases - 6 completion points - OK
traj_018: not_in_subquery - 7 completion points - OK
traj_019: settings_clause - 7 completion points - OK
traj_020: simple_select_count - 4 completion points - OK


Exit code: 0

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_03_joins.jsonl

Output from command in shell a44217:
Total lines: 20
All 20 trajectories valid JSONL with correct IDs, category, completion_points (4-10), and table/column consistency.


Exit code: 0

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_02_aggregation.jsonl

Output from command in shell 7e648f:
Total lines: 20
  traj_021: group_by_single - 5 points - OK
  traj_022: group_by_having - 6 points - OK
  traj_023: with_totals - 6 points - OK
  traj_024: with_cube - 6 points - OK
  traj_025: with_rollup - 6 points - OK
  traj_026: window_partition_order - 5 points - OK
  traj_027: window_ranking - 4 points - OK
  traj_028: window_rows_between - 5 points - OK
  traj_029: window_range_between - 5 points - OK
  traj_030: aggregate_combinators_if - 6 points - OK
  traj_031: group_array - 6 points - OK
  traj_032: uniq_functions - 5 points - OK
  traj_033: any_anylast - 5 points - OK
  traj_034: state_merge_combinators - 5 points - OK
  traj_035: array_aggregate_combinators - 6 points - OK
  traj_036: window_lag_lead - 5 points - OK
  traj_037: group_by_date_functions - 6 points - OK
  traj_038: group_by_having_complex - 6 points - OK
  traj_039: window_dense_rank_percentile - 5 points - OK
  traj_040: multi_feature_aggregation - 8 points - OK
All JSONL valid!


Exit code: 0

The file /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_04_ddl_dml.jsonl has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
  1|{"id":"traj_061","category":"ddl_dml","subcategory":"create_merge_tree","description":"Create MergeTree table with partition by month, TTL, and settings","query":"CREATE TABLE IF NOT EXISTS analytics.events (event_id UUID, user_id UInt64, event_type String, event_data String, event_time DateTime CODEC(Delta, ZSTD), created_at DateTime DEFAULT now() ) ENGINE = MergeTree() PARTITION BY toYYYYMM(event_time) ORDER BY (event_time, user_id) TTL event_time + INTERVAL 30 DAY SETTINGS index_granularity = 8192","tables":["analytics.events"],"columns":{"analytics.events":["event_id","user_id","event_type","event_data","event_time","created_at"]},"dictionaries":[],"settings":{"index_granularity":"8192"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"analytics.events ","next_token":"analytics"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (event_id UUID, user_id UInt64, event_type String, event_data String, event_time DateTime CODEC(Delta, ZSTD), created_at DateTime DEFAULT now() ) ENGINE = ","completion":"MergeTree() ","next_token":"MergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (...) ENGINE = MergeTree() PARTITION BY ","completion":"toYYYYMM(event_time) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (...) ENGINE = MergeTree() PARTITION BY toYYYYMM(event_time) ORDER BY ","completion":"(event_time, user_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (...) ENGINE = MergeTree() PARTITION BY toYYYYMM(event_time) ORDER BY (event_time, user_id) TTL ","completion":"event_time + INTERVAL 30 DAY ","next_token":"event_time"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (...) ENGINE = MergeTree() PARTITION BY toYYYYMM(event_time) ORDER BY (event_time, user_id) TTL event_time + INTERVAL 30 DAY SETTINGS ","completion":"index_granularity = 8192","next_token":"index_granularity"}]}
  2|{"id":"traj_062","category":"ddl_dml","subcategory":"create_replacing_merge_tree","description":"Create ReplacingMergeTree table with version column for deduplication","query":"CREATE TABLE IF NOT EXISTS shop.products (product_id UInt64, sku String, name String, price Decimal(10,2), stock UInt32, updated_at DateTime, version UInt64 ) ENGINE = ReplacingMergeTree(version) ORDER BY (product_id) PARTITION BY tuple() SETTINGS index_granularity = 4096","tables":["shop.products"],"columns":{"shop.products":["product_id","sku","name","price","stock","updated_at","version"]},"dictionaries":[],"settings":{"index_granularity":"4096"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"shop.products ","next_token":"shop"},{"prefix":"CREATE TABLE IF NOT EXISTS shop.products (product_id UInt64, sku String, name String, price Decimal(10,2), stock UInt32, updated_at DateTime, version UInt64 ) ENGINE = ","completion":"ReplacingMergeTree(version) ","next_token":"ReplacingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS shop.products (...) ENGINE = ReplacingMergeTree(version) ORDER BY ","completion":"(product_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS shop.products (...) ENGINE = ReplacingMergeTree(version) ORDER BY (product_id) PARTITION BY ","completion":"tuple() ","next_token":"tuple"},{"prefix":"CREATE TABLE IF NOT EXISTS shop.products (...) ENGINE = ReplacingMergeTree(version) ORDER BY (product_id) PARTITION BY tuple() SETTINGS ","completion":"index_granularity = 4096","next_token":"index_granularity"}]}
  3|{"id":"traj_063","category":"ddl_dml","subcategory":"create_summing_merge_tree","description":"Create SummingMergeTree table for aggregating numeric metrics by key","query":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (stat_date Date, metric_name String, value Float64, count UInt64 ) ENGINE = SummingMergeTree() PARTITION BY toYYYYMM(stat_date) ORDER BY (stat_date, metric_name) SETTINGS index_granularity = 8192","tables":["metrics.daily_stats"],"columns":{"metrics.daily_stats":["stat_date","metric_name","value","count"]},"dictionaries":[],"settings":{"index_granularity":"8192"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"metrics.daily_stats ","next_token":"metrics"},{"prefix":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (stat_date Date, metric_name String, value Float64, count UInt64 ) ENGINE = ","completion":"SummingMergeTree() ","next_token":"SummingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (...) ENGINE = SummingMergeTree() PARTITION BY ","completion":"toYYYYMM(stat_date) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (...) ENGINE = SummingMergeTree() PARTITION BY toYYYYMM(stat_date) ORDER BY ","completion":"(stat_date, metric_name) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (...) ENGINE = SummingMergeTree() PARTITION BY toYYYYMM(stat_date) ORDER BY (stat_date, metric_name) SETTINGS ","completion":"index_granularity = 8192","next_token":"index_granularity"}]}
  4|{"id":"traj_064","category":"ddl_dml","subcategory":"create_aggregating_merge_tree","description":"Create AggregatingMergeTree with AggregateFunction columns for pre-aggregated data","query":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (visit_date Date, site_id UInt32, uniq_users AggregateFunction(uniq, UInt64), total_duration AggregateFunction(sum, UInt32) ) ENGINE = AggregatingMergeTree() PARTITION BY toYYYYMM(visit_date) ORDER BY (visit_date, site_id) SETTINGS index_granularity = 8192","tables":["analytics.agg_visits"],"columns":{"analytics.agg_visits":["visit_date","site_id","uniq_users","total_duration"]},"dictionaries":[],"settings":{"index_granularity":"8192"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"analytics.agg_visits ","next_token":"analytics"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (visit_date Date, site_id UInt32, uniq_users AggregateFunction(uniq, UInt64), total_duration AggregateFunction(sum, UInt32) ) ENGINE = ","completion":"AggregatingMergeTree() ","next_token":"AggregatingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (...) ENGINE = AggregatingMergeTree() PARTITION BY ","completion":"toYYYYMM(visit_date) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (...) ENGINE = AggregatingMergeTree() PARTITION BY toYYYYMM(visit_date) ORDER BY ","completion":"(visit_date, site_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (...) ENGINE = AggregatingMergeTree() PARTITION BY toYYYYMM(visit_date) ORDER BY (visit_date, site_id) SETTINGS ","completion":"index_granularity = 8192","next_token":"index_granularity"}]}
  5|{"id":"traj_065","category":"ddl_dml","subcategory":"create_collapsing_merge_tree","description":"Create CollapsingMergeTree with sign column for upsert semantics","query":"CREATE TABLE IF NOT EXISTS billing.transactions (txn_id UInt64, account_id UInt64, amount Decimal(18,2), status String, sign Int8, created_at DateTime ) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(created_at) ORDER BY (txn_id, account_id) SETTINGS index_granularity = 4096","tables":["billing.transactions"],"columns":{"billing.transactions":["txn_id","account_id","amount","status","sign","created_at"]},"dictionaries":[],"settings":{"index_granularity":"4096"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"billing.transactions ","next_token":"billing"},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (txn_id UInt64, account_id UInt64, amount Decimal(18,2), status String, sign Int8, created_at DateTime ) ENGINE = ","completion":"CollapsingMergeTree(sign) ","next_token":"CollapsingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (...) ENGINE = CollapsingMergeTree(sign) PARTITION BY ","completion":"toYYYYMM(created_at) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (...) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(created_at) ORDER BY ","completion":"(txn_id, account_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (...) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(created_at) ORDER BY (txn_id, account_id) SETTINGS ","completion":"index_granularity = 4096","next_token":"index_granularity"}]}
  6|{"id":"traj_066","category":"ddl_dml","subcategory":"create_versioned_collapsing_merge_tree","description":"Create VersionedCollapsingMergeTree with sign and version columns","query":"CREATE TABLE IF NOT EXISTS inventory.stock_levels (item_id UInt64, warehouse_id UInt32, quantity UInt32, sign Int8, version UInt64, updated_at DateTime ) ENGINE = VersionedCollapsingMergeTree(sign, version) PARTITION BY toYYYYMM(updated_at) ORDER BY (item_id, warehouse_id) SETTINGS index_granularity = 4096","tables":["inventory.stock_levels"],"columns":{"inventory.stock_levels":["item_id","warehouse_id","quantity","sign","version","updated_at"]},"dictionaries":[],"settings":{"index_granularity":"4096"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"inventory.stock_levels ","next_token":"inventory"},{"prefix":"CREATE TABLE IF NOT EXISTS inventory.stock_levels (item_id UInt64, warehouse_id UInt32, quantity UInt32, sign Int8, version UInt64, updated_at DateTime ) ENGINE = ","completion":"VersionedCollapsingMergeTree(sign, version) ","next_token":"VersionedCollapsingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS inventory.stock_levels (...) ENGINE = VersionedCollapsingMergeTree(sign, version) PARTITION BY ","completion":"toYYYYMM(updated_at) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS inventory.stock_levels (...) ENGINE = VersionedCollapsingMergeTree(sign, version) PARTITION BY … (363 chars truncated)
… (14 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/68da2002/content.txt
</truncation_notice>

Output from command in shell 290733:
Total lines: 20
Line 1: traj_061 - 6 pts - create_merge_tree
Line 2: traj_062 - 5 pts - create_replacing_merge_tree
Line 3: traj_063 - 5 pts - create_summing_merge_tree
Line 4: traj_064 - 5 pts - create_aggregating_merge_tree
Line 5: traj_065 - 5 pts - create_collapsing_merge_tree
Line 6: traj_066 - 5 pts - create_versioned_collapsing_merge_tree
Line 7: traj_067 - 6 pts - create_replicated_merge_tree
Line 8: traj_068 - 4 pts - create_distributed
Line 9: traj_069 - 3 pts - create_buffer <-- TOO FEW
Line 10: traj_070 - 4 pts - create_memory
Line 11: traj_071 - 5 pts - create_materialized_view
Line 12: traj_072 - 4 pts - alter_table_columns
Line 13: traj_073 - 4 pts - alter_table_index
Line 14: traj_074 - 4 pts - alter_table_projection
Line 15: traj_075 - 4 pts - alter_table_constraint
Line 16: traj_076 - 4 pts - alter_table_update
Line 17: traj_077 - 4 pts - insert_values
Line 18: traj_078 - 6 pts - insert_select
Line 19: traj_079 - 4 pts - insert_format
Line 20: traj_080 - 7 pts - explain_query
All JSON valid!


Exit code: 0

The file /Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_04_ddl_dml.jsonl has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
  5|{"id":"traj_065","category":"ddl_dml","subcategory":"create_collapsing_merge_tree","description":"Create CollapsingMergeTree with sign column for upsert semantics","query":"CREATE TABLE IF NOT EXISTS billing.transactions (txn_id UInt64, account_id UInt64, amount Decimal(18,2), status String, sign Int8, created_at DateTime ) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(created_at) ORDER BY (txn_id, account_id) SETTINGS index_granularity = 4096","tables":["billing.transactions"],"columns":{"billing.transactions":["txn_id","account_id","amount","status","sign","created_at"]},"dictionaries":[],"settings":{"index_granularity":"4096"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"billing.transactions ","next_token":"billing"},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (txn_id UInt64, account_id UInt64, amount Decimal(18,2), status String, sign Int8, created_at DateTime ) ENGINE = ","completion":"CollapsingMergeTree(sign) ","next_token":"CollapsingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (...) ENGINE = CollapsingMergeTree(sign) PARTITION BY ","completion":"toYYYYMM(created_at) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (...) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(created_at) ORDER BY ","completion":"(txn_id, account_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (...) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(created_at) ORDER BY (txn_id, account_id) SETTINGS ","completion":"index_granularity = 4096","next_token":"index_granularity"}]}
  6|{"id":"traj_066","category":"ddl_dml","subcategory":"create_versioned_collapsing_merge_tree","description":"Create VersionedCollapsingMergeTree with sign and version columns","query":"CREATE TABLE IF NOT EXISTS inventory.stock_levels (item_id UInt64, warehouse_id UInt32, quantity UInt32, sign Int8, version UInt64, updated_at DateTime ) ENGINE = VersionedCollapsingMergeTree(sign, version) PARTITION BY toYYYYMM(updated_at) ORDER BY (item_id, warehouse_id) SETTINGS index_granularity = 4096","tables":["inventory.stock_levels"],"columns":{"inventory.stock_levels":["item_id","warehouse_id","quantity","sign","version","updated_at"]},"dictionaries":[],"settings":{"index_granularity":"4096"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"inventory.stock_levels ","next_token":"inventory"},{"prefix":"CREATE TABLE IF NOT EXISTS inventory.stock_levels (item_id UInt64, warehouse_id UInt32, quantity UInt32, sign Int8, version UInt64, updated_at DateTime ) ENGINE = ","completion":"VersionedCollapsingMergeTree(sign, version) ","next_token":"VersionedCollapsingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS inventory.stock_levels (...) ENGINE = VersionedCollapsingMergeTree(sign, version) PARTITION BY ","completion":"toYYYYMM(updated_at) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS inventory.stock_levels (...) ENGINE = VersionedCollapsingMergeTree(sign, version) PARTITION BY toYYYYMM(updated_at) ORDER BY ","completion":"(item_id, warehouse_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS inventory.stock_levels (...) ENGINE = VersionedCollapsingMergeTree(sign, version) PARTITION BY toYYYYMM(updated_at) ORDER BY (item_id, warehouse_id) SETTINGS ","completion":"index_granularity = 4096","next_token":"index_granularity"}]}
  7|{"id":"traj_067","category":"ddl_dml","subcategory":"create_replicated_merge_tree","description":"Create ReplicatedMergeTree with ZooKeeper path and replica name","query":"CREATE TABLE IF NOT EXISTS logs.access_log (request_id UUID, user_id UInt64, url String, method String, status_code UInt16, response_time_ms UInt32, log_time DateTime ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/logs/access_log', '{replica}') PARTITION BY toYYYYMMDD(log_time) ORDER BY (log_time, user_id) TTL log_time + INTERVAL 7 DAY SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1","tables":["logs.access_log"],"columns":{"logs.access_log":["request_id","user_id","url","method","status_code","response_time_ms","log_time"]},"dictionaries":[],"settings":{"index_granularity":"8192","ttl_only_drop_parts":"1"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"logs.access_log ","next_token":"logs"},{"prefix":"CREATE TABLE IF NOT EXISTS logs.access_log (request_id UUID, user_id UInt64, url String, method String, status_code UInt16, response_time_ms UInt32, log_time DateTime ) ENGINE = ","completion":"ReplicatedMergeTree('/clickhouse/tables/{shard}/logs/access_log', '{replica}') ","next_token":"ReplicatedMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS logs.access_log (...) ENGINE = ReplicatedMergeTree(...) PARTITION BY ","completion":"toYYYYMMDD(log_time) ","next_token":"toYYYYMMDD"},{"prefix":"CREATE TABLE IF NOT EXISTS logs.access_log (...) ENGINE = ReplicatedMergeTree(...) PARTITION BY toYYYYMMDD(log_time) ORDER BY ","completion":"(log_time, user_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS logs.access_log (...) ENGINE = ReplicatedMergeTree(...) PARTITION BY toYYYYMMDD(log_time) ORDER BY (log_time, user_id) TTL ","completion":"log_time + INTERVAL 7 DAY ","next_token":"log_time"},{"prefix":"CREATE TABLE IF NOT EXISTS logs.access_log (...) ENGINE = ReplicatedMergeTree(...) PARTITION BY toYYYYMMDD(log_time) ORDER BY (log_time, user_id) TTL log_time + INTERVAL 7 DAY SETTINGS ","completion":"index_granularity = 8192, ttl_only_drop_parts = 1","next_token":"index_granularity"}]}
  8|{"id":"traj_068","category":"ddl_dml","subcategory":"create_distributed","description":"Create Distributed table sharding across cluster nodes by hash","query":"CREATE TABLE IF NOT EXISTS analytics.events_dist ON CLUSTER analytics_cluster AS analytics.events ENGINE = Distributed(analytics_cluster, analytics, events, rand())","tables":["analytics.events_dist","analytics.events"],"columns":{"analytics.events_dist":["event_id","user_id","event_type","event_data","event_time","created_at"],"analytics.events":["event_id","user_id","event_type","event_data","event_time","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"analytics.events_dist ","next_token":"analytics"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events_dist ON CLUSTER ","completion":"analytics_cluster ","next_token":"analytics_cluster"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events_dist ON CLUSTER analytics_cluster AS ","completion":"analytics.events ","next_token":"analytics"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events_dist ON CLUSTER analytics_cluster AS analytics.events ENGINE = ","completion":"Distributed(analytics_cluster, analytics, events, rand())","next_token":"Distributed"}]}
  9|{"id":"traj_069","category":"ddl_dml","subcategory":"create_buffer","description":"Create Buffer table in front of a MergeTree for write buffering","query":"CREATE TABLE IF NOT EXISTS analytics.events_buffer AS analytics.events ENGINE = Buffer(analytics, events, 16, 10, 100, 10000, 1000000, 2000000, 10000000)","tables":["analytics.events_buffer","analytics.events"],"columns":{"analytics.events_buffer":["event_id","user_id","event_type","event_data","event_time","created_at"],"analytics.events":["event_id","user_id","event_type","event_data","event_time","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"analytics.events_buffer ","next_token":"analytics"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events_buffer ","completion":"AS ","next_token":"AS"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events_buffer AS ","completion":"analytics.events ","next_token":"analytics"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events_buffer AS analytics.events ENGINE = ","completion":"Buffer(analytics, events, 16, 10, 100, 10000, 1000000, 2000000, 10000000)","next_token":"Buffer"}]}
 10|{"id":"traj_070","category":"ddl_dml","subcategory":"create_memory","description":"Create Memory engine table with max_rows setting for fast in-memory cache","query":"CREATE TABLE IF NOT EXISTS cache.session_cache (session_key String, session_data String, expires_at DateTime ) ENGINE = Memory() SETTINGS max_rows = 1000000","tables":["cache.session_cache"],"columns":{"cache.session_cache":["session_key","session_data","expires_at"]},"dictionaries":[],"settings":{"max_rows":"1000000"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"cache.session_cache ","next_token":"cache"},{"prefix":"CREATE TABLE IF NOT EXISTS cache.session_cache ","completion":"(session_key String, session_data String, expires_at DateTime) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS cache.session_cache (...) ENGINE = ","completion":"Memory() ","next_token":"Memory"},{"prefix":"CREATE TABLE IF NOT EXISTS cache.session_cache (...) ENGINE = Memory() SETTINGS ","completion":"max_rows = 1000000","next_token":"max_rows"}]}
 11|{"id":"traj_071","category":"ddl_dml","subcategory":"create_materialized_view","description":"Create MaterializedView with TO target table and populate from source","query":"CREATE MATERIALIZED VIEW IF NOT EXISTS analytics.hourly_visits TO analytics.agg_visits AS SELECT toStartOfHour(visit_time) AS visit_date, site_id, uniqState(user_id) AS uniq_users, sumState(duration) AS total_duration FROM analytics.raw_visits GROUP BY visit_date, site_id","tables":["analytics.hourly_visits","analytics.agg_visits","analytics.raw_visits"],"columns":{"analytics.hourly_visits":["visit_date","site_id","uniq_users","total_duration"],"analytics.agg_visits":["visit_date","site_id","uniq_users","total_duration"],"analytics.raw_visi… (1028 chars truncated)
… (2 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/79beee10/content.txt
</truncation_notice>

Output from command in shell d4cb76:
Total lines: 20
Line 1: traj_061 - 6 pts - create_merge_tree
Line 2: traj_062 - 5 pts - create_replacing_merge_tree
Line 3: traj_063 - 5 pts - create_summing_merge_tree
Line 4: traj_064 - 5 pts - create_aggregating_merge_tree
Line 5: traj_065 - 5 pts - create_collapsing_merge_tree
Line 6: traj_066 - 5 pts - create_versioned_collapsing_merge_tree
Line 7: traj_067 - 6 pts - create_replicated_merge_tree
Line 8: traj_068 - 4 pts - create_distributed
Line 9: traj_069 - 4 pts - create_buffer
Line 10: traj_070 - 4 pts - create_memory
Line 11: traj_071 - 5 pts - create_materialized_view
Line 12: traj_072 - 4 pts - alter_table_columns
Line 13: traj_073 - 4 pts - alter_table_index
Line 14: traj_074 - 4 pts - alter_table_projection
Line 15: traj_075 - 4 pts - alter_table_constraint
Line 16: traj_076 - 4 pts - alter_table_update
Line 17: traj_077 - 4 pts - insert_values
Line 18: traj_078 - 6 pts - insert_select
Line 19: traj_079 - 4 pts - insert_format
Line 20: traj_080 - 7 pts - explain_query

All JSON valid!
IDs match expected: True


Exit code: 0

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/model/config.py

Output from command in shell cf433c:
=== File sizes ===
      20 trajectories_01_basic_select.jsonl
      20 trajectories_02_aggregation.jsonl
      20 trajectories_03_joins.jsonl
      20 trajectories_04_ddl_dml.jsonl
      20 trajectories_05_advanced.jsonl
     100 total
=== Total trajectories ===
     100
=== Validate JSON ===
trajectories_01_basic_select.jsonl: 20 valid
trajectories_02_aggregation.jsonl: 20 valid
trajectories_03_joins.jsonl: 20 valid
trajectories_04_ddl_dml.jsonl: 20 valid
trajectories_05_advanced.jsonl: 20 valid


Exit code: 0

Output from command in shell 2a8549:
Model config: {'vocab_size': 341, 'd_model': 192, 'n_layers': 6, 'n_heads': 6, 'd_ff': 768, 'max_seq_len': 256, 'n_heads_kv': 6, 'dropout': 0.1, 'weight_tying': True, 'pad_token_id': 0, 'bos_token_id': 1, 'eos_token_id': 2}
Parameters: 2,771,328
Model size (float32): 10.57 MB
Model size (float16): 5.29 MB
Model size (int8):    2.64 MB
Estimated params:     2,773,440
Estimated size (f32): 10.58 MB

Tokenized query (28 tokens): [1, 189, 275, 243, 244, 249, 275, 243, 275, 244]...
Logits shape: torch.Size([1, 28, 341])
Loss: 5.6186
Generated 21 tokens


Exit code: 0

Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
Current todo list:
1. [x] Research ClickHouse Lexer token types and syntax elements [completed]
2. [x] Design project structure and trajectory format spec [completed]
3. [x] Launch 5 parallel GLM 5.2 subagents to generate 20 trajectories each (100 total) [completed]
4. [x] Implement ClickHouse tokenizer with lookbehind copies for identifiers/strings [completed]
5. [x] Implement GPT-based SLM model (~10MB, CPU-optimized) [completed]
6. [~] Implement training pipeline [in_progress]
7. [ ] Train model on generated trajectories [pending]
8. [ ] Implement inference/autocomplete with syntactic correctness sampling [pending]
9. [ ] Implement C++ export for ClickHouse embedding and WASM export [pending]

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/model/training.py

Output from command in shell 57b9af:
Loading trajectories from: /Users/root1/Downloads/projects/clickhouse-github/trajectories/data
Loaded 100 trajectories, generated 937 training samples
Dataset size: 937

Sample 0: input_ids shape=torch.Size([7])
  First 20 input IDs: [1, 189, 252, 85, 67, 125, 272]
  First 20 target IDs: [189, 252, 85, 67, 125, 272, 2]

Sample 1: input_ids shape=torch.Size([7])
  First 20 input IDs: [1, 189, 252, 85, 67, 125, 272]
  First 20 target IDs: [0, 252, 85, 67, 125, 272, 2]

Sample 50: input_ids shape=torch.Size([21])
  First 20 input IDs: [1, 189, 97, 249, 275, 249, 275, 249, 275, 85, 275, 78, 236, 275, 262, 308, 151, 23, 97, 125]
  First 20 target IDs: [0, 0, 0, 0, 0, 0, 0, 0, 85, 275, 78, 236, 275, 262, 308, 151, 23, 97, 125, 272]

Sample 100: input_ids shape=torch.Size([19])
  First 20 input IDs: [1, 189, 97, 249, 275, 249, 275, 249, 275, 85, 275, 236, 281, 21, 272, 10, 272, 10, 289]
  First 20 target IDs: [189, 97, 249, 275, 249, 275, 249, 275, 85, 275, 236, 281, 21, 272, 10, 272, 10, 289, 2]

Sample 200: input_ids shape=torch.Size([13])
  First 20 input IDs: [1, 189, 275, 243, 244, 14, 275, 85, 67, 236, 275, 262, 308]
  First 20 target IDs: [189, 275, 243, 244, 14, 275, 85, 67, 236, 275, 262, 308, 2]

Batch: input_ids=torch.Size([4, 71]), target_ids=torch.Size([4, 71])


Exit code: 0

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/train.py

Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
Current todo list:
1. [x] Research ClickHouse Lexer token types and syntax elements [completed]
2. [x] Design project structure and trajectory format spec [completed]
3. [x] Launch 5 parallel GLM 5.2 subagents to generate 20 trajectories each (100 total) [completed]
4. [x] Implement ClickHouse tokenizer with lookbehind copies for identifiers/strings [completed]
5. [x] Implement GPT-based SLM model (~10MB, CPU-optimized) [completed]
6. [x] Implement training pipeline [completed]
7. [~] Train model on generated trajectories [in_progress]
8. [ ] Implement inference/autocomplete with syntactic correctness sampling [pending]
9. [ ] Implement C++ export for ClickHouse embedding and WASM export [pending]

No output produced after 5.0 seconds. Command running in background with ID: e4c0c2. Run get_output with timeout if you need to wait.

Vocabulary saved (341 tokens) to /Users/root1/Downloads/projects/clickhouse-github/tokenizer/vocab.json

Model: ModelConfig(vocab_size=341, d_model=192, n_layers=6, n_heads=6, d_ff=768, max_seq_len=256, n_heads_kv=6, dropout=0.1, weight_tying=True, pad_token_id=0, bos_token_id=1, eos_token_id=2)
Parameters: 2,771,328
Model size (float32): 10.57 MB
Model size (float16): 5.29 MB
Device: cpu

Loading trajectories from: /Users/root1/Downloads/projects/clickhouse-github/trajectories/data
Loaded 80 trajectories, generated 762 training samples
Loaded 20 trajectories, generated 115 training samples
Train samples: 762
Val samples: 115
Steps per epoch: 47, Total steps: 1410

============================================================
Starting training at 06:54:14
============================================================
Starting training: 1410 steps, batch_size=16, device=cpu
Train samples: 762
Traceback (most recent call last):
  File "/Users/root1/Downloads/projects/clickhouse-github/train.py", line 182, in <module>
    main()
  File "/Users/root1/Downloads/projects/clickhouse-github/train.py", line 105, in main
    trainer.train()
  File "/Users/root1/Downloads/projects/clickhouse-github/model/training.py", line 276, in train
    lr = self.get_lr()
         ^^^^^^^^^^^^^
  File "/Users/root1/Downloads/projects/clickhouse-github/model/training.py", line 250, in get_lr
    return self.learning_rate * 0.5 * (1.0 + math.cos(__import__("math").pi * progress))
                                             ^^^^
NameError: name 'math' is not defined. Did you mean: 'Path'? Or did you forget to import 'math'?


Exit code: 1

The file /Users/root1/Downloads/projects/clickhouse-github/model/training.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
  5|ClickHouse tokenizer with lookbehind copies, and creates training samples
  6|from both full queries and completion-point prefixes.
  7|"""
  8|
  9|import json
 10|import math
 11|import os
 12|import random
 13|import torch
 14|from torch.utils.data import Dataset, DataLoader
 15|from pathlib import Path
 16|from typing import Optional

The file /Users/root1/Downloads/projects/clickhouse-github/model/training.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
247|        if self.step < self.warmup_steps:
248|            return self.learning_rate * (self.step + 1) / self.warmup_steps
249|        decay_steps = self.max_steps - self.warmup_steps
250|        progress = (self.step - self.warmup_steps) / max(decay_steps, 1)
251|        return self.learning_rate * 0.5 * (1.0 + math.cos(math.pi * progress))
252|
253|    def train(self):
254|        """Run the training loop."""
255|        import math

The file /Users/root1/Downloads/projects/clickhouse-github/model/training.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
249|        decay_steps = self.max_steps - self.warmup_steps
250|        progress = (self.step - self.warmup_steps) / max(decay_steps, 1)
251|        return self.learning_rate * 0.5 * (1.0 + math.cos(math.pi * progress))
252|
253|    def train(self):
254|        """Run the training loop."""
255|        self.model.to(self.device)
256|        self.model.train()
257|
258|        train_iter = iter(self.train_loader)
259|        running_loss = 0.0

Command running in background with ID: 4b3926

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/inference/autocomplete.py

Vocabulary saved (341 tokens) to /Users/root1/Downloads/projects/clickhouse-github/tokenizer/vocab.json

Model: ModelConfig(vocab_size=341, d_model=192, n_layers=6, n_heads=6, d_ff=768, max_seq_len=256, n_heads_kv=6, dropout=0.1, weight_tying=True, pad_token_id=0, bos_token_id=1, eos_token_id=2)
Parameters: 2,771,328
Model size (float32): 10.57 MB
Model size (float16): 5.29 MB
Device: cpu

Loading trajectories from: /Users/root1/Downloads/projects/clickhouse-github/trajectories/data
Loaded 80 trajectories, generated 762 training samples
Loaded 20 trajectories, generated 115 training samples
Train samples: 762
Val samples: 115
Steps per epoch: 47, Total steps: 1410

============================================================
Starting training at 06:54:38
============================================================
Starting training: 1410 steps, batch_size=16, device=cpu
Train samples: 762
  Step 50/1410 | loss=4.3366 | lr=0.000300
  [Eval] Step 50 | val_loss=3.5611
  Saved checkpoint: /Users/root1/Downloads/projects/clickhouse-github/checkpoints/best.pt
  [Eval] New best val_loss=3.5611, saved best.pt
  Step 100/1410 | loss=2.4945 | lr=0.000299
  [Eval] Step 100 | val_loss=3.2277
  Saved checkpoint: /Users/root1/Downloads/projects/clickhouse-github/checkpoints/best.pt
  [Eval] New best val_loss=3.2277, saved best.pt
  Step 150/1410 | loss=1.8894 | lr=0.000296
  [Eval] Step 150 | val_loss=3.0179
  Saved checkpoint: /Users/root1/Downloads/projects/clickhouse-github/checkpoints/best.pt
  [Eval] New best val_loss=3.0179, saved best.pt
  Step 200/1410 | loss=1.6122 | lr=0.000291
  [Eval] Step 200 | val_loss=3.8316
  Step 250/1410 | loss=1.4060 | lr=0.000284
  [Eval] Step 250 | val_loss=3.0859
  Step 300/1410 | loss=1.3029 | lr=0.000276
  [Eval] Step 300 | val_loss=3.3613
  Step 350/1410 | loss=1.1760 | lr=0.000265
  [Eval] Step 350 | val_loss=3.6656
  Step 400/1410 | loss=1.0764 | lr=0.000254
  [Eval] Step 400 | val_loss=3.4787
  Step 450/1410 | loss=0.9813 | lr=0.000240
  [Eval] Step 450 | val_loss=3.5665
  Step 500/1410 | loss=0.9406 | lr=0.000226
  [Eval] Step 500 | val_loss=3.1280
  Step 550/1410 | loss=0.8862 | lr=0.000211
  [Eval] Step 550 | val_loss=2.9656
  Saved checkpoint: /Users/root1/Downloads/projects/clickhouse-github/checkpoints/best.pt
  [Eval] New best val_loss=2.9656, saved best.pt
  Step 600/1410 | loss=0.8203 | lr=0.000194
  [Eval] Step 600 | val_loss=2.9533
  Saved checkpoint: /Users/root1/Downloads/projects/clickhouse-github/checkpoints/best.pt
  [Eval] New best val_loss=2.9533, saved best.pt
  Step 650/1410 | loss=0.7810 | lr=0.000178
  [Eval] Step 650 | val_loss=3.4847
  Step 700/1410 | loss=0.7300 | lr=0.000160
  [Eval] Step 700 | val_loss=3.8352
  Step 750/1410 | loss=0.7148 | lr=0.000143
  [Eval] Step 750 | val_loss=3.4274
  Step 800/1410 | loss=0.6594 | lr=0.000126
  [Eval] Step 800 | val_loss=3.8388
  Step 850/1410 | loss=0.6362 | lr=0.000109
  [Eval] Step 850 | val_loss=3.8628
  Step 900/1410 | loss=0.6160 | lr=0.000093
  [Eval] Step 900 | val_loss=3.4499
  Step 950/1410 | loss=0.5781 | lr=0.000077
  [Eval] Step 950 | val_loss=3.5202
  Step 1000/1410 | loss=0.5707 | lr=0.000062
  [Eval] Step 1000 | val_loss=3.7562
  Step 1050/1410 | loss=0.5519 | lr=0.000049
  [Eval] Step 1050 | val_loss=3.5994
  Step 1100/1410 | loss=0.5388 | lr=0.000037
  [Eval] Step 1100 | val_loss=3.4979
  Step 1150/1410 | loss=0.5275 | lr=0.000026
  [Eval] Step 1150 | val_loss=4.0606
  Step 1200/1410 | loss=0.5208 | lr=0.000017
  [Eval] Step 1200 | val_loss=3.7245
  Step 1250/1410 | loss=0.5069 | lr=0.000010
  [Eval] Step 1250 | val_loss=4.3563
  Step 1300/1410 | loss=0.5057 | lr=0.000005
  [Eval] Step 1300 | val_loss=3.9750
  Step 1350/1410 | loss=0.5067 | lr=0.000001
  [Eval] Step 1350 | val_loss=3.7100
  Step 1400/1410 | loss=0.4972 | lr=0.000000
  [Eval] Step 1400 | val_loss=3.9493
  Saved checkpoint: /Users/root1/Downloads/projects/clickhouse-github/checkpoints/final.pt

Training complete. Best val_loss=2.9533

Training time: 152.8s (2.5m)

Saving final model...
  ONNX export skipped: No module named 'onnxscript'
  Raw weights saved: /Users/root1/Downloads/projects/clickhouse-github/checkpoints/ch_autocomplete_weights.bin (10.57 MB)

Done! Model saved to /Users/root1/Downloads/projects/clickhouse-github/checkpoints/


Exit code: 0

Output from command in shell eeab70:
=== Autocomplete Test (untrained model) ===

Prefix:  'SELECT '
Completion: "<ident> ASC INTO FOR % 1 <ident> <ident> <ident> % OVER FULL '<str>' >= SYSTEM"
Tokens:  ['COPY_BACK_19', 'KW_ASC', 'KW_INTO', 'KW_FOR', 'PERCENT', 'NUM_HEX', 'COPY_BACK_21', 'COPY_BACK_22', 'COPY_BACK_18', 'PERCENT']
Score:   -33.4645

Prefix:  'SELECT count() FROM '
Completion: 'country <ident> JOIN = = :: DEDUPLICATE ENABLED OVER OVER 1 GROUP BY 1 <ident>'
Tokens:  ['NEW_IDENT', 'COPY_BACK_6', 'KW_JOIN', 'EQUALS', 'EQUALS', 'DOUBLE_COLON', 'KW_DEDUPLICATE', 'KW_ENABLED', 'KW_OVER', 'KW_OVER']
Score:   -30.5779

Prefix:  'SELECT * FROM orders WHERE '
Completion: "'<str>' DESC <ident> <ident> <ident> <ident> <ident> (INTO :: PROCESSLIST TYPE PROCESSLIST ENABLED INSERT"
Tokens:  ['COPY_STR_BACK_21', 'KW_DESC', 'COPY_BACK_13', 'COPY_BACK_24', 'COPY_BACK_13', 'COPY_BACK_13', 'COPY_BACK_24', 'LPAREN', 'KW_INTO', 'DOUBLE_COLON']
Score:   -32.5600

Prefix:  'SELECT city, count() FROM orders GROUP BY '
Completion: "'<str>'.<ident> = PROCESSLIST :: INTO CONSTRAINT :: OVER :: || INTO FULL MATERIALIZED"
Tokens:  ['COPY_STR_BACK_12', 'DOT', 'COPY_BACK_21', 'EQUALS', 'KW_PROCESSLIST', 'DOUBLE_COLON', 'KW_INTO', 'KW_CONSTRAINT', 'DOUBLE_COLON', 'KW_OVER']
Score:   -33.8600

Prefix:  'CREATE TABLE '
Completion: "['<str>' JOIN TOTALS ENABLED ENABLED EVENTS ENABLED user_id <ident> <ident> IN ENABLED ENABLED SUBSTRING"
Tokens:  ['LBRACKET', 'COPY_STR_BACK_29', 'KW_JOIN', 'KW_TOTALS', 'KW_ENABLED', 'KW_ENABLED', 'KW_EVENTS', 'KW_ENABLED', 'NEW_IDENT', 'COPY_BACK_17']
Score:   -33.3886


Exit code: 0

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/test_autocomplete.py

Output from command in shell 7a7965:
Loading model from: /Users/root1/Downloads/projects/clickhouse-github/checkpoints/best.pt
Model loaded: 2,771,328 parameters

=== Autocomplete Test (trained model) ===

Prefix:     'SELECT '
Completion: 'city.duration, <ident>.country, timestamp.name, <ident>.id, timestamp.customers,'
Tokens:     ['NEW_IDENT', 'DOT', 'NEW_IDENT', 'COMMA', 'COPY_BACK_4', 'DOT', 'NEW_IDENT', 'COMMA', 'NEW_IDENT', 'DOT', 'NEW_IDENT', 'COMMA', 'COPY_BACK_4', 'DOT', 'NEW_IDENT']
Score:      -3.5287

Prefix:     'SELECT count() FROM '
Completion: "customer_id WHERE customers (SELECT city (url, order_date, '<value>', url (orders ()))"
Tokens:     ['NEW_IDENT', 'KW_WHERE', 'NEW_IDENT', 'LPAREN', 'KW_SELECT', 'NEW_IDENT', 'LPAREN', 'NEW_IDENT', 'COMMA', 'NEW_IDENT', 'COMMA', 'NEW_STRING', 'COMMA', 'NEW_IDENT', 'LPAREN']
Score:      -5.9018

Prefix:     'SELECT * FROM orders WHERE '
Completion: '<ident> ORDER BY timestamp DESC LIMIT 1 LIMIT 1 (timestamp, orders.user_id, url.email.'
Tokens:     ['COPY_BACK_6', 'KW_ORDER', 'KW_BY', 'NEW_IDENT', 'KW_DESC', 'KW_LIMIT', 'NUM_INT', 'KW_LIMIT', 'NUM_INT', 'LPAREN', 'NEW_IDENT', 'COMMA', 'COPY_BACK_4', 'DOT', 'NEW_IDENT']
Score:      -5.3855

Prefix:     'SELECT city, count() FROM orders GROUP BY '
Completion: "<ident>, <ident>, '<value>' ORDER BY <ident> DESC LIMIT 1 FROM total_amount, orders WHERE total_amount.country ="
Tokens:     ['COPY_BACK_12', 'COMMA', 'COPY_BACK_7', 'COMMA', 'NEW_STRING', 'KW_ORDER', 'KW_BY', 'COPY_BACK_12', 'KW_DESC', 'KW_LIMIT', 'NUM_INT', 'KW_FROM', 'NEW_IDENT', 'COMMA', 'NEW_IDENT']
Score:      -4.5471

Prefix:     'SELECT city, count() FROM orders GROUP BY city ORDER BY '
Completion: 'city DESC LIMIT 1 <ident> DESC LIMIT 1 FROM customers, city.user_id WHERE city = 1 ORDER BY'
Tokens:     ['COPY_BACK_4', 'KW_DESC', 'KW_LIMIT', 'NUM_INT', 'COPY_BACK_8', 'KW_DESC', 'KW_LIMIT', 'NUM_INT', 'KW_FROM', 'NEW_IDENT', 'COMMA', 'COPY_BACK_4', 'DOT', 'NEW_IDENT', 'KW_WHERE']
Score:      -4.5013

Prefix:     'SELECT * FROM orders LEFT JOIN customers '
Completion: 'ORDER BY region DESC LIMIT 1.pageviews, orders.signup_date, region.total_amount, orders.date'
Tokens:     ['KW_ORDER', 'KW_BY', 'NEW_IDENT', 'KW_DESC', 'KW_LIMIT', 'NUM_INT', 'DOT', 'NEW_IDENT', 'COMMA', 'COPY_BACK_4', 'DOT', 'NEW_IDENT', 'COMMA', 'COPY_BACK_4', 'DOT']
Score:      -4.9891

Prefix:     'CREATE TABLE '
Completion: 'event_type >= session_id (timestamp, customer_id (date, 1, customers (id (session_id (event_type,'
Tokens:     ['NEW_IDENT', 'GREATER_OR_EQUALS', 'NEW_IDENT', 'LPAREN', 'NEW_IDENT', 'COMMA', 'NEW_IDENT', 'LPAREN', 'NEW_IDENT', 'COMMA', 'NUM_INT', 'COMMA', 'NEW_IDENT', 'LPAREN', 'NEW_IDENT']
Score:      -9.6146

Prefix:     'SELECT * FROM events WHERE event_type = '
Completion: "'<value>' ORDER BY url, customers DESC LIMIT 1 LIMIT 1 (event_type, events.session_id, customers."
Tokens:     ['NEW_STRING', 'KW_ORDER', 'KW_BY', 'NEW_IDENT', 'COMMA', 'NEW_IDENT', 'KW_DESC', 'KW_LIMIT', 'NUM_INT', 'KW_LIMIT', 'NUM_INT', 'LPAREN', 'NEW_IDENT', 'COMMA', 'NEW_IDENT']
Score:      -5.5758

Prefix:     'SELECT count(), avg(price) FROM orders WHERE '
Completion: "customer_id (total_amount = '<value>', '<value>', '<value>', '<value>', '<value>', '<value>', city (user_id,"
Tokens:     ['NEW_IDENT', 'LPAREN', 'NEW_IDENT', 'EQUALS', 'NEW_STRING', 'COMMA', 'NEW_STRING', 'COMMA', 'NEW_STRING', 'COMMA', 'NEW_STRING', 'COMMA', 'NEW_STRING', 'COMMA', 'NEW_STRING']
Score:      -2.9012

Prefix:     'SELECT user_id, count() FROM events GROUP BY user_id HAVING '
Completion: "<ident> = '<value>' ORDER BY data DESC LIMIT 1 WHERE signup_date ("
Tokens:     ['COPY_BACK_7', 'EQUALS', 'NEW_STRING', 'KW_ORDER', 'KW_BY', 'NEW_IDENT', 'KW_DESC', 'KW_LIMIT', 'NUM_INT', 'KW_WHERE', 'NEW_IDENT', 'LPAREN', '<EOS>']
Score:      -2.6428


=== Top-3 Alternatives for 'SELECT * FROM orders WHERE ' ===

  #1: 'timestamp ORDER BY status DESC LIMIT 1 LIMIT 1.country, orders.user_id, customer_id.customers,'  (score=-3.5114)
  #2: 'url ORDER BY city DESC LIMIT 1 LIMIT 1 (customer_id, duration.status, duration.customer_id.'  (score=-6.0289)
  #3: '<ident> ORDER BY orders DESC LIMIT 1 LIMIT 1 LIMIT 1.signup_date LIMIT 1 LIMIT 1.timestamp,'  (score=-10.7208)


Exit code: 0

Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
Current todo list:
1. [x] Research ClickHouse Lexer token types and syntax elements [completed]
2. [x] Design project structure and trajectory format spec [completed]
3. [x] Launch 5 parallel GLM 5.2 subagents to generate 20 trajectories each (100 total) [completed]
4. [x] Implement ClickHouse tokenizer with lookbehind copies for identifiers/strings [completed]
5. [x] Implement GPT-based SLM model (~10MB, CPU-optimized) [completed]
6. [x] Implement training pipeline [completed]
7. [x] Train model on generated trajectories [completed]
8. [x] Implement inference/autocomplete with syntactic correctness sampling [completed]
9. [~] Implement C++ export for ClickHouse embedding and WASM export [in_progress]

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py

Output from command in shell 4d2141:
Traceback (most recent call last):
  File "/Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py", line 25, in <module>
    from ch_tokenizer import VOCAB, VOCAB_SIZE, KEYWORDS, OPERATOR_TOKENS, NUMBER_TOKENS
ModuleNotFoundError: No module named 'ch_tokenizer'


Exit code: 1

The file /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
 17|
 18|import torch
 19|import numpy as np
 20|
 21|PROJECT_ROOT = Path(__file__).resolve().parent.parent
 22|sys.path.insert(0, str(PROJECT_ROOT / "tokenizer"))
 23|sys.path.insert(0, str(PROJECT_ROOT / "model"))
 24|
 25|from ch_tokenizer import VOCAB, VOCAB_SIZE, KEYWORDS, OPERATOR_TOKENS, NUMBER_TOKENS
 26|from ch_tokenizer import IDENT_COPY_TOKENS, STRING_COPY_TOKENS, MAX_COPY_BACK
 27|from ch_tokenizer import BOS_ID, EOS_ID, PAD_ID, UNK_ID

Output from command in shell 2237f3:
Loading model...

Exporting C++ artifacts:
  Binary weights: /Users/root1/Downloads/projects/clickhouse-github/export/ch_autocomplete.bin (10.57 MB)
  Vocab header: /Users/root1/Downloads/projects/clickhouse-github/export/ch_vocab.h
  Model header: /Users/root1/Downloads/projects/clickhouse-github/export/ch_model.h
  C++ engine: /Users/root1/Downloads/projects/clickhouse-github/export/ch_inference.h
  CMakeLists.txt: /Users/root1/Downloads/projects/clickhouse-github/export/CMakeLists.txt
  C++ test: /Users/root1/Downloads/projects/clickhouse-github/export/test_main.cpp
Traceback (most recent call last):
  File "/Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py", line 959, in <module>
    main()
  File "/Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py", line 939, in main
    export_wasm_build_script(export_dir / "build_wasm.sh")
  File "/Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py", line 907, in export_wasm_build_script
    f.chmod(0o755)
    ^^^^^^^
AttributeError: '_io.TextIOWrapper' object has no attribute 'chmod'


Exit code: 1

The file /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
901|echo "  const autocomplete = mod.cwrap('chai_autocomplete', 'string', ['string', 'number']);"
902|echo "  loadModel('ch_autocomplete.bin');"
903|echo "  const result = autocomplete('SELECT ', 20);"
904|'''
905|    with open(path, "w") as f:
906|        f.write(script)
907|    path.chmod(0o755)
908|    print(f"  WASM build script: {path}")
909|
910|
911|def main():

Output from command in shell 6b87b5:
Loading model...

Exporting C++ artifacts:
  Binary weights: /Users/root1/Downloads/projects/clickhouse-github/export/ch_autocomplete.bin (10.57 MB)
  Vocab header: /Users/root1/Downloads/projects/clickhouse-github/export/ch_vocab.h
  Model header: /Users/root1/Downloads/projects/clickhouse-github/export/ch_model.h
  C++ engine: /Users/root1/Downloads/projects/clickhouse-github/export/ch_inference.h
  CMakeLists.txt: /Users/root1/Downloads/projects/clickhouse-github/export/CMakeLists.txt
  C++ test: /Users/root1/Downloads/projects/clickhouse-github/export/test_main.cpp
  WASM build script: /Users/root1/Downloads/projects/clickhouse-github/export/build_wasm.sh

All C++ artifacts exported to: /Users/root1/Downloads/projects/clickhouse-github/export/

To build natively:
  cd /Users/root1/Downloads/projects/clickhouse-github/export && mkdir build && cd build && cmake .. && make

To build for WebAssembly:
  cd /Users/root1/Downloads/projects/clickhouse-github/export && ./build_wasm.sh


Exit code: 0

Output from command in shell eade40:
bash: cmake: command not found


Exit code: 127

Output from command in shell c3cdae:
/usr/bin/g++
/usr/bin/clang++


Exit code: 0

Output from command in shell 6efba6:


Exit code: 0

Output from command in shell 79d760:
Loading model from: ch_autocomplete.bin
Warning: expected tensor blocks.0.attn.qkv.weight, got blocks.0.ln1.weight
Warning: expected tensor blocks.0.attn.proj.weight, got blocks.0.attn.qkv.weight
Warning: expected tensor blocks.0.ffn.fc1.weight, got blocks.0.attn.proj.weight
Warning: expected tensor blocks.0.ffn.fc2.weight, got blocks.0.ln2.weight
Warning: expected tensor blocks.0.ln1.weight, got blocks.0.ffn.fc1.weight
Warning: expected tensor blocks.0.ln2.weight, got blocks.0.ffn.fc2.weight
Warning: expected tensor blocks.1.attn.qkv.weight, got blocks.1.ln1.weight
Warning: expected tensor blocks.1.attn.proj.weight, got blocks.1.attn.qkv.weight
Warning: expected tensor blocks.1.ffn.fc1.weight, got blocks.1.attn.proj.weight
Warning: expected tensor blocks.1.ffn.fc2.weight, got blocks.1.ln2.weight
Warning: expected tensor blocks.1.ln1.weight, got blocks.1.ffn.fc1.weight
Warning: expected tensor blocks.1.ln2.weight, got blocks.1.ffn.fc2.weight
Warning: expected tensor blocks.2.attn.qkv.weight, got blocks.2.ln1.weight
Warning: expected tensor blocks.2.attn.proj.weight, got blocks.2.attn.qkv.weight
Warning: expected tensor blocks.2.ffn.fc1.weight, got blocks.2.attn.proj.weight
Warning: expected tensor blocks.2.ffn.fc2.weight, got blocks.2.ln2.weight
Warning: expected tensor blocks.2.ln1.weight, got blocks.2.ffn.fc1.weight
Warning: expected tensor blocks.2.ln2.weight, got blocks.2.ffn.fc2.weight
Warning: expected tensor blocks.3.attn.qkv.weight, got blocks.3.ln1.weight
Warning: expected tensor blocks.3.attn.proj.weight, got blocks.3.attn.qkv.weight
Warning: expected tensor blocks.3.ffn.fc1.weight, got blocks.3.attn.proj.weight
Warning: expected tensor blocks.3.ffn.fc2.weight, got blocks.3.ln2.weight
Warning: expected tensor blocks.3.ln1.weight, got blocks.3.ffn.fc1.weight
Warning: expected tensor blocks.3.ln2.weight, got blocks.3.ffn.fc2.weight
Warning: expected tensor blocks.4.attn.qkv.weight, got blocks.4.ln1.weight
Warning: expected tensor blocks.4.attn.proj.weight, got blocks.4.attn.qkv.weight
Warning: expected tensor blocks.4.ffn.fc1.weight, got blocks.4.attn.proj.weight
Warning: expected tensor blocks.4.ffn.fc2.weight, got blocks.4.ln2.weight
Warning: expected tensor blocks.4.ln1.weight, got blocks.4.ffn.fc1.weight
Warning: expected tensor blocks.4.ln2.weight, got blocks.4.ffn.fc2.weight
Warning: expected tensor blocks.5.attn.qkv.weight, got blocks.5.ln1.weight
Warning: expected tensor blocks.5.attn.proj.weight, got blocks.5.attn.qkv.weight
Warning: expected tensor blocks.5.ffn.fc1.weight, got blocks.5.attn.proj.weight
Warning: expected tensor blocks.5.ffn.fc2.weight, got blocks.5.ln2.weight
Warning: expected tensor blocks.5.ln1.weight, got blocks.5.ffn.fc1.weight
Warning: expected tensor blocks.5.ln2.weight, got blocks.5.ffn.fc2.weight
Model loaded: d_model=192, n_layers=6, vocab_size=341
Prefix:     SELECT 
Completion: '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>'
Score:      nan
Time:       434ms

Prefix:     SELECT count() FROM 
Completion: '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>'
Score:      nan
Time:       584ms

Prefix:     SELECT * FROM orders WHERE 
Completion: '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>'
Score:      nan
Time:       584ms

Prefix:     SELECT city, count() FROM orders GROUP BY 
Completion: '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>'
Score:      nan
Time:       778ms

Prefix:     CREATE TABLE 
Completion: '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>' '<value>'
Score:      nan
Time:       473ms



Exit code: 0

The file /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
 27|from ch_tokenizer import BOS_ID, EOS_ID, PAD_ID, UNK_ID
 28|from config import ModelConfig, ClickHouseGPT, create_model
 29|
 30|
 31|def export_model_weights_binary(model: ClickHouseGPT, path: Path):
 32|    """Export model weights as a flat binary file for C++ loading.
 33|
 34|    Format:
 35|      - 4 bytes: magic ("CHAI")
 36|      - 4 bytes: version (uint32)
 37|      - 4 bytes: vocab_size (uint32)
 38|      - 4 bytes: d_model (uint32)
 39|      - 4 bytes: n_layers (uint32)
 40|      - 4 bytes: n_heads (uint32)
 41|      - 4 bytes: d_ff (uint32)
 42|      - 4 bytes: max_seq_len (uint32)
 43|      - 4 bytes: weight_tying (uint32, 0 or 1)
 44|      - 4 bytes: num_tensors (uint32)
 45|      - For each tensor:
 46|        - 4 bytes: name length (uint32)
 47|        - N bytes: name (UTF-8)
 48|        - 4 bytes: ndim (uint32)
 49|        - ndim * 4 bytes: shape (uint32 each)
 50|        - data: float32 array (prod(shape) * 4 bytes)
 51|    """
 52|    config = model.config
 53|    state_dict = model.state_dict()
 54|
 55|    with open(path, "wb") as f:
 56|        # Header
 57|        f.write(b"CHAI")
 58|        f.write(struct.pack("I", 1))  # version
 59|        f.write(struct.pack("I", config.vocab_size))
 60|        f.write(struct.pack("I", config.d_model))
 61|        f.write(struct.pack("I", config.n_layers))
 62|        f.write(struct.pack("I", config.n_heads))
 63|        f.write(struct.pack("I", config.d_ff))
 64|        f.write(struct.pack("I", config.max_seq_len))
 65|        f.write(struct.pack("I", 1 if config.weight_tying else 0))
 66|        f.write(struct.pack("I", len(state_dict)))
 67|
 68|        # Tensors
 69|        for name, tensor in state_dict.items():
 70|            data = tensor.detach().cpu().numpy().astype(np.float32)
 71|            name_bytes = name.encode("utf-8")
 72|            f.write(struct.pack("I", len(name_bytes)))
 73|            f.write(name_bytes)
 74|            f.write(struct.pack("I", data.ndim))
 75|            for dim in data.shape:
 76|                f.write(struct.pack("I", dim))
 77|            f.write(data.tobytes())
 78|
 79|    size_mb = path.stat().st_size / 1024 / 1024
 80|    print(f"  Binary weights: {path} ({size_mb:.2f} MB)")
 81|    return size_mb
 82|
 83|
 84|def export_vocab_header(path: Path):
 85|    """Export vocabulary as a C++ header file."""

The file /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
276|    f.read(reinterpret_cast<char*>(&v), 4);
277|    return v;
278|}
279|
280|inline std::vector<float> read_tensor_raw(std::ifstream& f, std::string& name) {
281|    uint32_t name_len = read_u32(f);
282|    name.resize(name_len);
283|    f.read(&name[0], name_len);
284|    uint32_t ndim = read_u32(f);
285|    std::vector<uint32_t> shape(ndim);
286|    for (uint32_t i = 0; i < ndim; i++) shape[i] = read_u32(f);
287|    size_t total = 1;
288|    for (auto d : shape) total *= d;
289|    std::vector<float> data(total);
290|    f.read(reinterpret_cast<char*>(data.data()), total * sizeof(float));
291|    return data;
292|}
293|
294|inline std::unique_ptr<ModelWeights> load_model(const std::string& path) {
295|    std::ifstream f(path, std::ios::binary);
296|    if (!f) { std::cerr << "Cannot open " << path << std::endl; return nullptr; }
297|
298|    char magic[4];
299|    f.read(magic, 4);
300|    if (std::memcmp(magic, "CHAI", 4) != 0) {
301|        std::cerr << "Invalid model file format" << std::endl; return nullptr;
302|    }
303|
304|    auto w = std::make_unique<ModelWeights>();
305|    uint32_t version = read_u32(f);
306|    w->config.vocab_size = read_u32(f);
307|    w->config.d_model = read_u32(f);
308|    w->config.n_layers = read_u32(f);
309|    w->config.n_heads = read_u32(f);
310|    w->config.d_ff = read_u32(f);
311|    w->config.max_seq_len = read_u32(f);
312|    w->config.weight_tying = read_u32(f) != 0;
313|    uint32_t num_tensors = read_u32(f);
314|
315|    // Read all tensors into a map by name
316|    std::unordered_map<std::string, std::vector<float>> tensors;
317|    for (uint32_t i = 0; i < num_tensors; i++) {
318|        std::string name;
319|        auto data = read_tensor_raw(f, name);
320|        tensors[name] = std::move(data);
321|    }
322|
323|    // Assign by name
324|    w->tok_emb = tensors.at("tok_emb.weight");
325|    w->pos_emb = tensors.at("pos_emb.weight");
326|
327|    w->layers.resize(w->config.n_layers);
328|    for (uint32_t l = 0; l < w->config.n_layers; l++) {
329|        auto& lw = w->layers[l];
330|        std::string prefix = "blocks." + std::to_string(l) + ".";
331|        lw.qkv_weight = tensors.at(prefix + "attn.qkv.weight");
332|        lw.proj_weight = tensors.at(prefix + "attn.proj.weight");
333|        lw.fc1_weight = tensors.at(prefix + "ffn.fc1.weight");
334|        lw.fc2_weight = tensors.at(prefix + "ffn.fc2.weight");
335|        lw.ln1_gamma = tensors.at(prefix + "ln1.weight");
336|        lw.ln2_gamma = tensors.at(prefix + "ln2.weight");
337|    }
338|
339|    w->ln_f_gamma = tensors.at("ln_f.weight");
340|    return w;
341|}
342|
343|// ─── Forward pass ───────────────────────────────────────────────────────────
344|
345|inline std::vector<float> forward(

Output from command in shell 803c8d:
Loading model...

Exporting C++ artifacts:
  Binary weights: /Users/root1/Downloads/projects/clickhouse-github/export/ch_autocomplete.bin (10.57 MB)
  Vocab header: /Users/root1/Downloads/projects/clickhouse-github/export/ch_vocab.h
  Model header: /Users/root1/Downloads/projects/clickhouse-github/export/ch_model.h
  C++ engine: /Users/root1/Downloads/projects/clickhouse-github/export/ch_inference.h
  CMakeLists.txt: /Users/root1/Downloads/projects/clickhouse-github/export/CMakeLists.txt
  C++ test: /Users/root1/Downloads/projects/clickhouse-github/export/test_main.cpp
  WASM build script: /Users/root1/Downloads/projects/clickhouse-github/export/build_wasm.sh

All C++ artifacts exported to: /Users/root1/Downloads/projects/clickhouse-github/export/

To build natively:
  cd /Users/root1/Downloads/projects/clickhouse-github/export && mkdir build && cd build && cmake .. && make

To build for WebAssembly:
  cd /Users/root1/Downloads/projects/clickhouse-github/export && ./build_wasm.sh
Compiled OK


Exit code: 0

Output from command in shell a68a8b:
Loading model from: ch_autocomplete.bin
Model loaded: d_model=192, n_layers=6, vocab_size=341
Prefix:     SELECT 
Completion: != CLUSTERS USER TEST STOP ALIAS STOP STOP OUTER <ident> ARRAY '<value>' '<value>' ADD '<value>' TEST USER ADD ADD USER
Score:      -51.618
Time:       439ms

Prefix:     SELECT count() FROM 
Completion: <= CLUSTERS TOTALS UNBOUNDED OVER CLUSTERS CLUSTERS CLUSTERS CLUSTERS SYNTAX ALIAS UNBOUNDED '<value>' ADD '<value>' TOTALS <ident> CURRENT_USER ALIAS STOP
Score:      -45.5115
Time:       585ms

Prefix:     SELECT * FROM orders WHERE 
Completion: [ AS STEP STOP STOP CLUSTERS STOP STOP FUNCTIONS STOP ALIAS SYNTAX '<value>' ADD '<value>' STOP STOP ALIAS ADD TIES
Score:      -43.5759
Time:       589ms

Prefix:     SELECT city, count() FROM orders GROUP BY 
Completion: <ident> ARRAY) 1 DISK BY END FUNCTIONS CLUSTERS TOTALS CLUSTERS STOP '<value>' ALIAS '<value>' STOP STOP ALIAS ALIAS STOP
Score:      -50.9379
Time:       778ms

Prefix:     CREATE TABLE 
Completion: '<value>' LEFT <ident> SYNTAX STOP OUTER STOP STOP ROWS <ident> ADD '<value>' '<value>' ADD '<value>' USER USER ADD ADD SYNTAX
Score:      -51.2943
Time:       473ms



Exit code: 0

The file /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
200|
201|using Matrix = std::vector<float>;  // row-major
202|using Tensor3D = std::vector<float>; // flattened
203|
204|inline void matmul(const float* A, const float* B, float* C,
205|                   int M, int K, int N) {
206|    // C = A @ B, A: (M,K), B: (K,N), C: (M,N)
207|    for (int i = 0; i < M; i++) {
208|        for (int j = 0; j < N; j++) {
209|            float sum = 0.0f;
210|            for (int k = 0; k < K; k++) {
211|                sum += A[i * K + k] * B[k * N + j];
212|            }
213|            C[i * N + j] = sum;
214|        }
215|    }
216|}
217|
218|// C = A @ B.T  where A: (M,K), B: (N,K) stored row-major, C: (M,N)
219|// This is used for PyTorch nn.Linear weights which are stored as (out_features, in_features)
220|inline void matmul_linear(const float* A, const float* W, float* C,
221|                          int M, int K, int N) {
222|    // A: (M, K), W: (N, K) [PyTorch Linear weight format], C: (M, N)
223|    // C[i,j] = sum_k A[i,k] * W[j,k]
224|    for (int i = 0; i < M; i++) {
225|        for (int j = 0; j < N; j++) {
226|            float sum = 0.0f;
227|            for (int k = 0; k < K; k++) {
228|                sum += A[i * K + k] * W[j * K + k];
229|            }
230|            C[i * N + j] = sum;
231|        }
232|    }
233|}
234|
235|inline void layernorm(const float* x, float* out, const float* gamma,
236|                      int d_model) {
237|    float mean = 0.0f;

The file /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
392|        // Pre-LN1
393|        for (int t = 0; t < T; t++)
394|            layernorm(&x[t * D], &ln_out[t * D], lw.ln1_gamma.data(), D);
395|
396|        // QKV projection: (T, D) @ W.T where W: (3*D, D) -> (T, 3*D)
397|        matmul_linear(ln_out.data(), lw.qkv_weight.data(), qkv.data(), T, D, 3 * D);
398|
399|        // Attention (per head)
400|        for (int h = 0; h < H; h++) {
401|            int offset = h * DH;

The file /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
423|                }
424|            }
425|        }
426|
427|        // Projection + residual: (T, D) @ W.T where W: (D, D) -> (T, D)
428|        matmul_linear(attn_out.data(), lw.proj_weight.data(), proj_out.data(), T, D, D);
429|        for (int i = 0; i < T * D; i++) x[i] += proj_out[i];
430|
431|        // Pre-LN2
432|        for (int t = 0; t < T; t++)

The file /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
431|        // Pre-LN2
432|        for (int t = 0; t < T; t++)
433|            layernorm(&x[t * D], &ln_out[t * D], lw.ln2_gamma.data(), D);
434|
435|        // FFN: fc1 -> GELU -> fc2 (all using PyTorch Linear weight format)
436|        matmul_linear(ln_out.data(), lw.fc1_weight.data(), ffn_hidden.data(), T, D, cfg.d_ff);
437|        for (int i = 0; i < T * (int)cfg.d_ff; i++) ffn_hidden[i] = gelu(ffn_hidden[i]);
438|        matmul_linear(ffn_hidden.data(), lw.fc2_weight.data(), ffn_out.data(), T, cfg.d_ff, D);
439|
440|        // Residual
441|        for (int i = 0; i < T * D; i++) x[i] += ffn_out[i];
442|    }

The file /Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
444|    // Final LayerNorm
445|    for (int t = 0; t < T; t++)
446|        layernorm(&x[t * D], &x[t * D], w.ln_f_gamma.data(), D);
447|
448|    // Output projection (weight-tied with token embedding)
449|    // logits = x @ tok_emb.T  -> (T, vocab_size)
450|    // tok_emb is (vocab_size, D), so we use matmul_linear: (T,D) @ (V,D).T -> (T,V)
451|    std::vector<float> logits(T * cfg.vocab_size);
452|    matmul_linear(x.data(), w.tok_emb.data(), logits.data(), T, D, cfg.vocab_size);
453|
454|    return logits;
455|}

Output from command in shell d54d2c:

To build for WebAssembly:
  cd /Users/root1/Downloads/projects/clickhouse-github/export && ./build_wasm.sh
=== Running C++ test ===
Loading model from: ch_autocomplete.bin
Model loaded: d_model=192, n_layers=6, vocab_size=341
Prefix:     SELECT 
Completion: <ident>, <ident>, <ident> ( <ident>) AS <ident>, <ident> () AS <ident>, MAX ( <ident>
Score:      -8.28494
Time:       213ms

Prefix:     SELECT count() FROM 
Completion: <ident> WHERE <ident> ( SELECT ID, <ident> ()
Score:      -5.42747
Time:       112ms

Prefix:     SELECT * FROM orders WHERE 
Completion: <ident> ORDER BY <ident> DESC LIMIT 1
Score:      -1.73343
Time:       70ms

Prefix:     SELECT city, count() FROM orders GROUP BY 
Completion: <ident>, <ident> ROWS BETWEEN 1 AND CURRENT ROW) AS <ident>,
Score:      -8.87173
Time:       227ms

Prefix:     CREATE TABLE 
Completion: <ident> CODEC, <ident> ( <ident>) AS <ident>, <ident> ( <ident>
Score:      -10.4448
Time:       123ms



Exit code: 0

Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
Current todo list:
1. [x] Research ClickHouse Lexer token types and syntax elements [completed]
2. [x] Design project structure and trajectory format spec [completed]
3. [x] Launch 5 parallel GLM 5.2 subagents to generate 20 trajectories each (100 total) [completed]
4. [x] Implement ClickHouse tokenizer with lookbehind copies for identifiers/strings [completed]
5. [x] Implement GPT-based SLM model (~10MB, CPU-optimized) [completed]
6. [x] Implement training pipeline [completed]
7. [x] Train model on generated trajectories [completed]
8. [x] Implement inference/autocomplete with syntactic correctness sampling [completed]
9. [x] Implement C++ export for ClickHouse embedding and WASM export [completed]
10. [~] Write README and AGENTS.md [in_progress]

File created successfully at: /Users/root1/Downloads/projects/clickhouse-github/README.md

# Web Content from https://github.com/ClickHouse/ClickHouse/issues/87836

[Skip to content][1]

## Navigation Menu

Toggle navigation
[ Sign in ][2]
Appearance settings
* Platform
  * AI CODE CREATION
    * [
      GitHub CopilotWrite better code with AI
      ][3]
    * [
      GitHub Copilot appDirect agents from issue to merge
      ][4]
    * [
      MCP Registry^{New}Integrate external tools
      ][5]
  * DEVELOPER WORKFLOWS
    * [
      ActionsAutomate any workflow
      ][6]
    * [
      CodespacesInstant dev environments
      ][7]
    * [
      IssuesPlan and track work
      ][8]
    * [
      Code ReviewManage code changes
      ][9]
  * APPLICATION SECURITY
    * [
      GitHub Advanced SecurityFind and fix vulnerabilities
      ][10]
    * [
      Code securitySecure your code as you build
      ][11]
    * [
      Secret protectionStop leaks before they start
      ][12]
  * EXPLORE
    * [Why GitHub][13]
    * [Documentation][14]
    * [Blog][15]
    * [Changelog][16]
    * [Marketplace][17]
  [View all features][18]
* Solutions
  * BY COMPANY SIZE
    * [Enterprises][19]
    * [Small and medium teams][20]
    * [Startups][21]
    * [Nonprofits][22]
  * BY USE CASE
    * [App Modernization][23]
    * [DevSecOps][24]
    * [DevOps][25]
    * [CI/CD][26]
    * [View all use cases][27]
  * BY INDUSTRY
    * [Healthcare][28]
    * [Financial services][29]
    * [Manufacturing][30]
    * [Government][31]
    * [View all industries][32]
  [View all solutions][33]
* Resources
  * EXPLORE BY TOPIC
    * [AI][34]
    * [Software Development][35]
    * [DevOps][36]
    * [Security][37]
    * [View all topics][38]
  * EXPLORE BY TYPE
    * [Customer stories][39]
    * [Events & webinars][40]
    * [Ebooks & reports][41]
    * [Business insights][42]
    * [GitHub Skills][43]
  * SUPPORT & SERVICES
    * [Documentation][44]
    * [Customer support][45]
    * [Community forum][46]
    * [Trust center][47]
    * [Partners][48]
  [View all resources][49]
* Open Source
  * COMMUNITY
    * [
      GitHub SponsorsFund open source developers
      ][50]
  * PROGRAMS
    * [Security Lab][51]
    * [Maintainer Community][52]
    * [Accelerator][53]
    * [GitHub Stars][54]
    * [Archive Program][55]
  * REPOSITORIES
    * [Topics][56]
    * [Trending][57]
    * [Collections][58]
* Enterprise
  * ENTERPRISE SOLUTIONS
    * [
      Enterprise platformAI-powered developer platform
      ][59]
  * AVAILABLE ADD-ONS
    * [
      GitHub Advanced SecurityEnterprise-grade security features
      ][60]
    * [
      Copilot for BusinessEnterprise-grade AI features
      ][61]
    * [
      Premium SupportEnterprise-grade 24/7 support
      ][62]
* [Pricing][63]
Search or jump to...

# Search code, repositories, users, issues, pull requests...

Search
Clear
[Search syntax tips][64]

# Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted
Cancel Submit feedback

# Saved searches

## Use saved searches to filter your results more quickly

Name
Query

To see all available qualifiers, see our [documentation][65].

Cancel Create saved search
[ Sign in ][66]
[ Sign up ][67]
Appearance settings
Resetting focus
You signed in with another tab or window. [Reload][68] to refresh your session. You signed out in
another tab or window. [Reload][69] to refresh your session. You switched accounts on another tab or
window. [Reload][70] to refresh your session. Dismiss alert

### Uh oh!


There was an error while loading. [Please reload this page][71].

[ ClickHouse ][72] / ** [ClickHouse][73] ** Public
* [ Notifications ][74] You must be signed in to change notification settings
* [ Fork 8.6k ][75]
* [ Star 48.3k ][76]
* [ Code ][77]
* [ Issues 4.8k ][78]
* [ Pull requests 1.4k ][79]
* [ Discussions ][80]
* [ Actions ][81]
* [ Projects ][82]
* [ Wiki ][83]
* [ Security and quality 7 ][84]
* [ Insights ][85]
Additional navigation options
* [ Code ][86]
* [ Issues ][87]
* [ Pull requests ][88]
* [ Discussions ][89]
* [ Actions ][90]
* [ Projects ][91]
* [ Wiki ][92]
* [ Security and quality ][93]
* [ Insights ][94]

# Intern Tasks 2025/2026 #87836

New issue
Copy link
New issue
Copy link
Open
Open
[Intern Tasks 2025/2026][95]#87836
Copy link
Labels
[feature][96][research][97]

## Description

[[@alexey-milovidov]][98]
[alexey-milovidov][99]
opened [on Sep 29, 2025][100]
Issue body actions

This is the list of proposed tasks. It is to be extended. You can propose more tasks.
You can also find the previous lists here:

2024/2025: [#71175][101]
2023/2024: [#58394][102]
2022/2023: [#42194][103]
2021/2022: [#29601][104]
2020/2021: [#15065][105]
2019/2020: [https://gist.github.com/alexey-milovidov/4251f71275f169d8fd0867e2051715e9][106]
2018/2019: [https://gist.github.com/alexey-milovidov/6735d193762cab1ad3b6e6af643e3a43][107]
2017/2018: [https://gist.github.com/alexey-milovidov/26cc3862eb87e52869b9dac64ab99156][108]

The tasks should be:
* not too hard (doable within about a month) but usually not less than a week;
* not alter some core components of the system;
* mostly isolated, does not require full knowledge of the system;
* somewhat interesting to implement or have some point of research;
* not in the critical path of our roadmap (ok to be thrown away after a year);
* most of them are for C++ developers, but there should also be tasks for frontend developers or
  tools/research that only require Go/Python/whatever;
* some tasks should allow teamwork;
* cover various skills, e.g., system programming, algorithm knowledge, etc...

## SLM-based custom autocomplete

Implement and train a small language model (~10 MB), using a custom tokenizer, for auto-completion
of ClickHouse queries.
The tokenizer should correspond to the ClickHouse Lexer, and support its syntax elements as well as
keywords. However, for identifiers and string literals, it should represent items like lookbehind
copies, instead of coding the values literally. We aim to use a GPT-based model.

The goal is to implement a small and fast model, running on CPU with minimum of resources, that
could be embedded in the ClickHouse binary by default, as well as exported to WebAssembly, and it
could be largerly customizable, so we can do sampling based on the syntaxic correctness of the query
and based on the available dictionaries of tables.

## Porting ClickHouse parser to WASM

Compile a minimal version of ClickHouse to run in the browser. The filesystem and network
interaction should be abstracted so it can request file uploads from the user and interact with
remote HTTP endpoints.

## Negative and fractional LIMIT values

Already done by [@nihalzp][109] and [@0xgouda][110]!

[#28913][111]
[#81892][112]

## Functions for rolling hashes and content-defined chunking

[#81183][113]

## Aggregate functions for Geo types

[#80186][114]

## Aggregation with approximate and spatial clustering (+ strings)

Add a modifier `WITH CLUSTER` to a `GROUP BY` element, defining a maximum distance between groups.
Groups located close to each other will be merged.

Example:

`SELECT user_id, min(event_time), max(event_time) FROM table GROUP BY user_id, event_time WITH
CLUSTER 1800`

This query will group together sessions of events for every user_id, where `session` is a sequence
of events, with the distance between nearest events no longer than 1800 (half an hour).

It can be extended to the multidimensional case:

Example:

`SELECT user_id, count() FROM table GROUP BY user_id, (x, y) WITH CLUSTER 10`

This query will group together points (x, y) with L2-distance less than 10.

A more interesting extension is clustering of arbitrary dimensional metric spaces, such as strings
with Levenstein distance.

## Query Rewrite Rules

[#80084][115]

In-progress, almost finished by [@hinata34][116]

## API for query rewrite and formatting

[#75033][117]

## Key-Value data marts

[#33581][118]

## User authentication with wildcard names

[#78065][119]

## Query cache for partial results

Implement the possibility of caching at different stages of the query pipeline, such as - before
LIMIT, before ORDER BY, or intermediate results of GROUP BY. The most important use case is partial
caching of time-series queries, where the aggregation of the older data should be reused and merged
with the newly coming data.

This requires certain efforts, such as remembering and preserving the split of the work across data
parts for further queries. So that the the results of aggregation can be cached in the form of
virtual projections corresponding to data parts.

## AI functions

Implement scalar and aggregate AI functions in ClickHouse similar to
[https://www.vldb.org/pvldb/vol18/p5415-mhedhbi.pdf][120]

As extensions of this, add AI functions to:
* translate texts,
* summarize texts,
* compute a similarity score between texts
* compute the sentiment (happy, sad, etc.) of a text,
* generate texts based on a prompt (GenAI),
* fix spelling and grammar in a text,
* classify a text in one of N given categories

You can use this PR as a starting point: [#85442][121]

## Support for altering index granularity and primary key

The minimal implementation will allow different data parts to have a different primary key. The
index analysis will run on these sets of data parts differently, and these data parts will never be
merged together. So after the change of the primary key, newer data parts will use the new primary
key, while the old ones remain.

Further implementation will reorder data in the older data parts according to the new primary key.

[#147][122]

## Combinatorics functions

Various functions on arrays to obtain combinations and permutations.

[#43175][123]

## Pipelined SQL syntax

[#69364][124]

## Approximate vector search with LeanNN

If you like data structure, math and low-level programming

[unum-cloud/USearch#662][125]

Can be implemented in the [vector similarity index][126].

## Import and Export of Substrait plans

[Substrait][127… (85 chars truncated)
… (405 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/f81afe78/content.txt
</truncation_notice>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/README.md" start_line="1" end_line="10" total_lines="169">
  1|# ClickHouse SLM Autocomplete
  2|
  3|A small language model (~10 MB) for auto-completion of ClickHouse queries, using a custom tokenizer that mirrors the ClickHouse Lexer.
  4|
  5|## Overview
  6|
  7|This project implements a GPT-based small language model (SLM) designed to:
  8|- Run on CPU with minimal resources (~10 MB model size)
  9|- Be embeddable in the ClickHouse binary by default
 10|- Be exportable to WebAssembly
</file-view>

The file /Users/root1/Downloads/projects/clickhouse-github/README.md has been updated. Here's the result of running `cat -n` on a snippet of the edited file:
  1|# ClickHouse SLM Autocomplete
  2|
  3|A small language model (~10 MB) for auto-completion of ClickHouse queries, using a custom tokenizer that mirrors the ClickHouse Lexer.
  4|
  5|## Task Specification
  6|
  7|> **Source**: [ClickHouse/ClickHouse#87836 — Intern Tasks 2025/2026](https://github.com/ClickHouse/ClickHouse/issues/87836)
  8|> opened by [@alexey-milovidov](https://github.com/alexey-milovidov) on Sep 29, 2025
  9|
 10|### SLM-based custom autocomplete
 11|
 12|Implement and train a small language model (~10 MB), using a custom tokenizer, for auto-completion
 13|of ClickHouse queries.
 14|
 15|The tokenizer should correspond to the ClickHouse Lexer, and support its syntax elements as well as
 16|keywords. However, for identifiers and string literals, it should represent items like lookbehind
 17|copies, instead of coding the values literally. We aim to use a GPT-based model.
 18|
 19|The goal is to implement a small and fast model, running on CPU with minimum of resources, that
 20|could be embedded in the ClickHouse binary by default, as well as exported to WebAssembly, and it
 21|could be largerly customizable, so we can do sampling based on the syntaxic correctness of the query
 22|and based on the available dictionaries of tables.
 23|
 24|---
 25|
 26|## Overview
 27|
 28|This project implements a GPT-based small language model (SLM) designed to:
 29|- Run on CPU with minimal resources (~10 MB model size)
 30|- Be embeddable in the ClickHouse binary by default
 31|- Be exportable to WebAssembly
 32|- Support sampling based on syntactic correctness and available table dictionaries
 33|
 34|## Architecture
 35|
 36|### Tokenizer (`tokenizer/ch_tokenizer.py`)

No skills found in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs matching the search criteria. (searched providers: Devin, Standard, Cursor, Windsurf, Claude, OpenCode, Zed)

No skills found in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs.

No skills found in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs matching the search criteria. (searched providers: Devin, Standard, Cursor, Windsurf, Claude, OpenCode, Zed)

The "devin-for-terminal" skill is running

<skill name="devin-for-terminal" status="running">
Source: /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs
Base directory: /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin
Auto-approved tools (no permission prompt needed): read, grep, glob
Other tools can still be used but will require user permission.

Documentation for Devin CLI is available on disk at:
  /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs

This directory contains `.mdx` files covering installation, commands, models,
shell integration, troubleshooting, and enterprise configuration.

Use the read, grep, and glob tools to search within that directory for
information relevant to the user's question. The tools are auto-approved for
this path while this skill is active.
</skill>

/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/acp/jetbrains.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/acp/zed.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/adaptive.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/changelog/next.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/changelog/stable.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/enterprise/controls.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/enterprise/devin-auth.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/enterprise/team-settings.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/enterprise/windsurf-auth.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/essential-commands.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/configuration.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/hooks/custom-hooks.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/hooks/lifecycle-hooks.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/hooks/overview.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/index.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/mcp/configuration.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/mcp/overview.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/plugins/overview.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/rules.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/skills/creating-skills.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/skills/overview.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/handoff.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/index.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/models.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/reference/commands.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/reference/configuration/config-file.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/reference/configuration/global-vs-local.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/reference/configuration/read-config-from.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/reference/keyboard-shortcuts.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/reference/permissions.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/reference/terminal-compatibility.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/shell-integration.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/subagents.mdx
/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/troubleshooting.mdx

Found 52 match(es) for pattern 'review' in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs:
-- 2 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/reference/commands.mdx
199|[Feature Preview] Shell integration commands. See [Shell Integration](/cli/shell-integration) for full details.
276|| `/loop <prompt>` | Run a prompt then auto-review the diff in a loop |
-- 2 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/shell-integration.mdx
2|title: "Shell Integration [Feature Preview]"
8|  Shell integration is a **Feature Preview**. It is available on macOS, Linux, and WSL with Bash, Zsh, and Fish. Shell integration is not yet supported on Windows (PowerShell or CMD). You can still run Devin CLI on Windows — this feature just isn't available there yet. It is feature complete but may interact poorly with other shell functionality. If you run into something incompatible please let us know!
-- 8 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/subagents.mdx
154|    └── reviewer/
162|    └── reviewer/
170|    └── reviewer/
175|    └── reviewer\
187|name: reviewer
188|description: Reviews code changes for correctness and style
204|You are a code review subagent. Your job is to review code changes
229|Once defined, custom subagent profiles appear alongside the built-in ones. The agent sees a description of each available profile and chooses the most appropriate one when spawning a subagent. You can also ask the agent to use a specific profile by name (e.g., "review this code using the reviewer subagent").
-- 6 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/skills/overview.mdx
15|- Review code according to your team's standards
51|Create a code review skill at `.devin/skills/review/SKILL.md` (or `.windsurf/skills/review/SKILL.md`):
55|name: review
56|description: Review code changes before committing
64|Review the current git diff and provide feedback:
75|Now you can invoke it with `/review` in any session.
-- 1 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/essential-commands.mdx
199|| `/loop <prompt>` | Run a prompt then auto-review the diff in a loop (requires clean git state to start) |
-- 11 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/skills/creating-skills.mdx
144|name: review-pr
145|description: Review the current PR using the reviewer subagent
146|agent: reviewer
149|Review the staged changes for correctness, security, and style issues.
206|4. Review test coverage for recently changed files
280|    name: review-changes
283|    Review these changes:
351|### Code Review Skill
355|name: review
356|description: Review staged changes for issues
368|Review the current changes for quality issues:
-- 1 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/index.mdx
78|Devin CLI can implement new features, fix bugs, review code, answer questions, automate tasks, and more.
-- 3 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/index.mdx
39|You can combine all of these in a single project. For example, you might have an `AGENTS.md` file with coding standards, a `review` skill for code review, an MCP server for your issue tracker, and hooks to block destructive commands.
54|│   │   └── review/
57|│       └── reviewer/
-- 10 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/changelog/stable.mdx
 77|- Revert preview now shows line diff stats (`+N -M`) and a "View diff" button for all action types (restore, delete, recreate).
 89|- Revert preview now shows descriptive warnings for irreversible actions instead of empty placeholders.
129|- Edit tool previews now show correct file line numbers instead of always starting from 1.
166|- File edits from `apply_patch` now display as inline diffs in Windsurf, matching the diff preview already shown for the `edit` tool.
231|- Shell command previews use clearer titles and show commands with a prompt prefix in the preview body.
247|- Normalized tool preview verb tenses: streaming previews now use present progressive ("Editing file.rs") and completed previews use past tense ("Edited file.rs").
372|### Live streaming tool previews
409|- Improved the plan mode exit approval with a dedicated review UI showing the plan summary and contextual button labels.
417|- Plan mode exit approval now has a dedicated review UI showing the plan summary and contextual button labels.
450|- Fixed an issue where the agent would not automatically review and fix lint errors detected after code edits.
-- 8 matches in /Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/extensibility/plugins/overview.mdx
25|    └── review/
41|devin plugins install acme/review-tools
44|devin plugins install https://gitlab.com/acme/review-tools.git
67|devin plugins info review-tools
70|devin plugins update review-tools
74|devin plugins remove review-tools
90|  "name": "review-tools",
92|  "description": "Code-review skills for our team",

<file-view path="/Users/root1/.local/share/devin/cli/_versions/2026.8.18/share/devin/docs/subagents.mdx" start_line="180" end_line="239" total_lines="292">
180|
181|### AGENT.md Format
182|
183|An `AGENT.md` file uses the same YAML frontmatter as skills, followed by the subagent's system prompt:
184|
185|```markdown
186|---
187|name: reviewer
188|description: Reviews code changes for correctness and style
189|model: sonnet
190|allowed-tools:
191|  - read
192|  - grep
193|  - glob
194|  - exec
195|permissions:
196|  allow:
197|    - Exec(git diff)
198|    - Exec(git log)
199|  deny:
200|    - write
201|    - edit
202|---
203|
204|You are a code review subagent. Your job is to review code changes
205|thoroughly and report findings back to the parent agent.
206|
207|Focus on:
208|1. Correctness — logic errors, edge cases, off-by-one mistakes
209|2. Security — potential vulnerabilities
210|3. Style — consistency with the rest of the codebase
211|4. Performance — obvious inefficiencies
212|
213|Always cite specific file paths and line numbers in your findings.
214|```
215|
216|### Frontmatter Fields
217|
218|| Field | Type | Default | Description |
219|| --- | --- | --- | --- |
220|| `name` | string | directory name | Identifier for the profile (must not conflict with built-in profiles) |
221|| `description` | string | none | Shown to the agent when selecting a profile |
222|| `model` | string | default subagent model | Override the model used by this subagent |
223|| `allowed-tools` | list | all tools | Restrict which tools the subagent can use |
224|| `permissions` | object | inherit | Permission overrides (`allow`, `deny`, `ask`) |
225|| `max-nesting` | integer | none | Override the maximum nesting depth, allowing this subagent to spawn its own subagents |
226|
227|### How Custom Subagents Are Used
228|
229|Once defined, custom subagent profiles appear alongside the built-in ones. The agent sees a description of each available profile and chooses the most appropriate one when spawning a subagent. You can also ask the agent to use a specific profile by name (e.g., "review this code using the reviewer subagent").
230|
231|Custom subagent profiles that conflict with a built-in profile name (e.g., `subagent_explore`, `subagent_general`) are skipped with a warning.
232|
233|### Importing From Other Tools
234|
235|Custom subagents are also imported from Claude Code's agent format:
236|
237|| Source | File Pattern |
238|| --- | --- |
239|| `.claude/agents/*.md` | Each `.md` file becomes a subagent profile |
</file-view>

Background subagent started with agent_id=cb8c059d. You can wait for this agent to finish using the read_subagent tool, otherwise you will automatically be notified with a <subagent_completion_notification> when it completes.

Background subagent started with agent_id=9c35fa2d. You can wait for this agent to finish using the read_subagent tool, otherwise you will automatically be notified with a <subagent_completion_notification> when it completes.

Background subagent started with agent_id=31dfd3bd. You can wait for this agent to finish using the read_subagent tool, otherwise you will automatically be notified with a <subagent_completion_notification> when it completes.

Background subagent started with agent_id=95fd219d. You can wait for this agent to finish using the read_subagent tool, otherwise you will automatically be notified with a <subagent_completion_notification> when it completes.

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/tokenizer/ch_tokenizer.py" start_line="1" end_line="514" total_lines="514">
  1|"""
  2|ClickHouse Lexer-based tokenizer with lookbehind copies.
  3|
  4|This tokenizer mirrors the ClickHouse Lexer (src/Parsers/Lexer.h) token types
  5|but replaces identifier and string-literal values with "lookbehind copy" tokens.
  6|
  7|Instead of encoding identifier/string values literally (which would require an
  8|unbounded vocabulary), each identifier or string is represented as either:
  9|  - NEW_IDENT / NEW_STRING  — first occurrence in the sequence
 10|  - COPY_BACK_N / COPY_STR_BACK_N — a repeat of a value seen N tokens ago
 11|
 12|This keeps the vocabulary small (~350 tokens) while allowing the model to
 13|reproduce arbitrary identifiers and strings via a pointer/copy mechanism at
 14|inference time.
 15|
 16|The tokenizer also stores the raw text for each identifier/string so that the
 17|inference engine can resolve copy references.
 18|"""
 19|
 20|import re
 21|import json
 22|from dataclasses import dataclass, field
 23|from typing import Optional
 24|
 25|
 26|# ─── ClickHouse keywords (from utils/antlr/ClickHouseLexer.g4) ───────────────
 27|
 28|KEYWORDS = [
 29|    "ACCESS", "ADD", "AFTER", "ALIAS", "ALL", "ALTER", "AND", "ANTI", "ANY",
 30|    "ARRAY", "AS", "ASC", "ASCENDING", "ASOF", "AST", "ASYNC", "ATTACH",
 31|    "BETWEEN", "BOTH", "BY", "CACHES", "CASE", "CAST", "CHECK", "CLEAR",
 32|    "CLUSTER", "CLUSTERS", "CODEC", "COLLATE", "COLUMN", "COLUMNS", "COMMENT",
 33|    "CONSTRAINT", "CREATE", "CROSS", "CUBE", "CURRENT", "CURRENT_USER",
 34|    "CHANGED", "DATABASE", "DATABASES", "DATE", "DAY", "DEDUPLICATE",
 35|    "DEFAULT", "DELAY", "DELETE", "DESC", "DESCENDING", "DESCRIBE", "DETACH",
 36|    "DICTIONARIES", "DICTIONARY", "DISK", "DISTINCT", "DISTRIBUTED", "DROP",
 37|    "ELSE", "ENABLED", "END", "ENGINE", "ENGINES", "ESTIMATE", "EVENTS",
 38|    "EXCEPT", "EXISTS", "EXPLAIN", "EXPRESSION", "EXTENDED", "EXTRACT",
 39|    "FETCHES", "FIELDS", "FILESYSTEM", "FILL", "FINAL", "FIRST", "FLUSH",
 40|    "FOLLOWING", "FOR", "FORMAT", "FREEZE", "FROM", "FULL", "FUNCTION",
 41|    "FUNCTIONS", "GLOBAL", "GRANULARITY", "GRANTS", "GROUP", "GROUPING",
 42|    "HAVING", "HIERARCHICAL", "HOUR", "ID", "IF", "ILIKE", "IMPLICIT", "IN",
 43|    "INDEX", "INDEXES", "INDICES", "INF", "INFINITY", "INJECTIVE", "INNER",
 44|    "INSERT", "INTERPOLATE", "INTERVAL", "INTO", "IS", "IS_OBJECT_ID", "JOIN",
 45|    "KEY", "KEYS", "KILL", "LAST", "LAYOUT", "LEADING", "LEFT", "LIFETIME",
 46|    "LIKE", "LIMIT", "LIVE", "LOCAL", "LOGS", "MATERIALIZE", "MATERIALIZED",
 47|    "MAX", "MERGES", "MICROSECOND", "MILLISECOND", "MIN", "MINUTE", "MODIFY",
 48|    "MONTH", "MOVE", "MUTATION", "NAN", "NANOSECOND", "NO", "NOT", "NULL",
 49|    "NULLS", "OFFSET", "ON", "OPTIMIZE", "OR", "ORDER", "OUTER", "OUTFILE",
 50|    "OVER", "OVERRIDE", "PARTITION", "PIPELINE", "PLAN", "POLICY", "POLICIES",
 51|    "POPULATE", "PRECEDING", "PREWHERE", "PRIMARY", "PRIVILEGES",
 52|    "PROCESSLIST", "PROFILE", "PROFILES", "PROJECTION", "QUARTER", "QUOTA",
 53|    "QUOTAS", "RANGE", "RECURSIVE", "RELOAD", "REMOVE", "RENAME", "REPLACE",
 54|    "REPLICA", "REPLICATED", "RIGHT", "ROLE", "ROLES", "ROLLUP", "ROW",
 55|    "ROWS", "SAMPLE", "SECOND", "SELECT", "SEMI", "SENDS", "SET", "SETS",
 56|    "SETTING", "SETTINGS", "SHOW", "SOURCE", "START", "STOP", "SUBSTRING",
 57|    "SYNC", "SYNTAX", "SYSTEM", "STEP", "TABLE", "TABLES", "TEMPORARY",
 58|    "TEST", "THEN", "TIES", "TIMEOUT", "TIMESTAMP", "TO", "TOP", "TOTALS",
 59|    "TRAILING", "TREE", "TRIM", "TRUNCATE", "TTL", "TYPE", "UNBOUNDED",
 60|    "UNION", "UPDATE", "USE", "USER", "USERS", "USING", "UUID", "VALUES",
 61|    "VIEW", "VOLUME", "WATCH", "WEEK", "WHEN", "WHERE", "WINDOW", "WITH",
 62|    "YEAR", "YYYY", "FALSE", "TRUE",
 63|]
 64|
 65|# Build a case-insensitive lookup set (ClickHouse keywords are case-insensitive)
 66|KEYWORD_SET = {kw.upper() for kw in KEYWORDS}
 67|
 68|
 69|# ─── Token type enum (mirrors ClickHouse TokenType) ──────────────────────────
 70|
 71|# Significant token types from the ClickHouse lexer that we map to vocab entries
 72|SIGNIFICANT_TOKEN_TYPES = [
 73|    "BareWord", "Number", "StringLiteral", "QuotedIdentifier",
 74|    "OpeningRoundBracket", "ClosingRoundBracket",
 75|    "OpeningSquareBracket", "ClosingSquareBracket",
 76|    "OpeningCurlyBrace", "ClosingCurlyBrace",
 77|    "Comma", "Semicolon", "Dot", "Asterisk",
 78|    "Plus", "Minus", "Slash", "Percent",
 79|    "Arrow", "QuestionMark", "Colon", "Caret", "DoubleColon",
 80|    "Equals", "NotEquals", "Less", "Greater",
 81|    "LessOrEquals", "GreaterOrEquals", "Spaceship",
 82|    "Concatenation", "At", "DoubleAt",
 83|]
 84|
 85|# ─── Vocabulary construction ─────────────────────────────────────────────────
 86|
 87|MAX_COPY_BACK = 32  # maximum lookbehind distance for copy tokens
 88|
 89|SPECIAL_TOKENS = ["<PAD>", "<BOS>", "<EOS>", "<UNK>"]
 90|
 91|# Operator/punctuation tokens with their string representations
 92|OPERATOR_TOKENS = {
 93|    "LPAREN": "(", "RPAREN": ")",
 94|    "LBRACKET": "[", "RBRACKET": "]",
 95|    "LBRACE": "{", "RBRACE": "}",
 96|    "COMMA": ",", "SEMICOLON": ";",
 97|    "DOT": ".", "ASTERISK": "*",
 98|    "PLUS": "+", "MINUS": "-",
 99|    "SLASH": "/", "PERCENT": "%",
100|    "ARROW": "->",
101|    "QUESTION": "?", "COLON": ":",
102|    "CARET": "^", "DOUBLE_COLON": "::",
103|    "EQUALS": "=", "NOT_EQUALS": "!=",
104|    "LESS": "<", "GREATER": ">",
105|    "LESS_OR_EQUALS": "<=", "GREATER_OR_EQUALS": ">=",
106|    "SPACESHIP": "<=>",
107|    "CONCAT": "||",
108|    "AT": "@", "DOUBLE_AT": "@@",
109|}
110|
111|NUMBER_TOKENS = ["NUM_INT", "NUM_FLOAT", "NUM_HEX"]
112|
113|# Identifier copy tokens
114|IDENT_COPY_TOKENS = ["NEW_IDENT"] + [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK + 1)]
115|
116|# String copy tokens
117|STRING_COPY_TOKENS = ["NEW_STRING"] + [f"COPY_STR_BACK_{i}" for i in range(1, MAX_COPY_BACK + 1)]
118|
119|
120|def build_vocab():
121|    """Build the vocabulary mapping token string -> integer id."""
122|    vocab = {}
123|    idx = 0
124|    for tok in SPECIAL_TOKENS:
125|        vocab[tok] = idx
126|        idx += 1
127|    for kw in KEYWORDS:
128|        vocab[f"KW_{kw}"] = idx
129|        idx += 1
130|    for tok in OPERATOR_TOKENS:
131|        vocab[tok] = idx
132|        idx += 1
133|    for tok in NUMBER_TOKENS:
134|        vocab[tok] = idx
135|        idx += 1
136|    for tok in IDENT_COPY_TOKENS:
137|        vocab[tok] = idx
138|        idx += 1
139|    for tok in STRING_COPY_TOKENS:
140|        vocab[tok] = idx
141|        idx += 1
142|    return vocab
143|
144|
145|VOCAB = build_vocab()
146|VOCAB_SIZE = len(VOCAB)
147|ID_TO_TOKEN = {v: k for k, v in VOCAB.items()}
148|
149|# Special token IDs
150|PAD_ID = VOCAB["<PAD>"]
151|BOS_ID = VOCAB["<BOS>"]
152|EOS_ID = VOCAB["<EOS>"]
153|UNK_ID = VOCAB["<UNK>"]
154|
155|
156|# ─── Lexer (Python re-based, mirrors ClickHouse Lexer.cpp) ───────────────────
157|
158|# Token patterns in priority order
159|_TOKEN_PATTERNS = [
160|    ("Whitespace",      r'[ \t\r\n]+'),
161|    ("Comment",         r'--[^\n\r]*|/\*.*?\*/'),
162|    ("HereDoc",         r"\$\$\w*\$\$"),
163|    # String literal: 'text' with '' escape and \' escape
164|    ("StringLiteral",   r"'(?:[^'\\]|\\.|'')*'"),
165|    # Quoted identifier: `text` or "text"
166|    ("QuotedIdentifier",r'`(?:[^`\\]|\\.|``)*`|"(?:[^"\\]|\\.|"")*"'),
167|    # Hex string literal: X'...'
168|    ("HexStringLiteral",r"X'[0-9a-fA-F]+'"),
169|    # Binary string literal: B'...'
170|    ("BinaryStringLit", r"B'[01]+'"),
171|    # Numbers: hex (0x...), binary (0b...), octal (0o...), float, int
172|    ("Number",          r'0[xX][0-9a-fA-F]+(?:[pP][+-]?[0-9]+)?'
173|                        r'|0[bB][01]+'
174|                        r'|0[oO][0-7]+'
175|                        r'|\d+\.\d*(?:[eE][+-]?\d+)?'
176|                        r'|\.\d+(?:[eE][+-]?\d+)?'
177|                        r'|\d+(?:[eE][+-]?\d+)?'),
178|    # BareWord: identifier or keyword
179|    ("BareWord",        r'[a-zA-Z_][a-zA-Z_0-9]*'),
180|    # Multi-char operators (must come before single-char)
181|    ("DoubleColon",     r'::'),
182|    ("Arrow",           r'->'),
183|    ("Spaceship",       r'<=>'),
184|    ("Concatenation",   r'\|\|'),
185|    ("DoubleAt",        r'@@'),
186|    ("NotEquals",       r'!=|<>'),
187|    ("LessOrEquals",    r'<='),
188|    ("GreaterOrEquals", r'>='),
189|    ("Equals",          r'==|='),  # ClickHouse accepts both = and ==
190|    # Single-char operators
191|    ("OpeningRoundBracket", r'\('),
192|    ("ClosingRoundBracket", r'\)'),
193|    ("OpeningSquareBracket", r'\['),
194|    ("ClosingSquareBracket", r'\]'),
195|    ("OpeningCurlyBrace", r'\{'),
196|    ("ClosingCurlyBrace", r'\}'),
197|    ("Comma",           r','),
198|    ("Semicolon",       r';'),
199|    ("Dot",             r'\.'),
200|    ("Asterisk",        r'\*'),
201|    ("Plus",            r'\+'),
202|    ("Minus",           r'-'),
203|    ("Slash",           r'/'),
204|    ("Percent",         r'%'),
205|    ("QuestionMark",    r'\?'),
206|    ("Colon",           r':'),
207|    ("Caret",           r'\^'),
208|    ("At",              r'@'),
209|]
210|
211|_COMPILED_PATTERNS = [(name, re.compile(pat, re.DOTALL)) for name, pat in _TOKEN_PATTERNS]
212|
213|
214|@dataclass
215|class LexToken:
216|    """A raw lexed token (before lookbehind encoding)."""
217|    type: str        # ClickHouse TokenType name
218|    text: str        # raw text
219|    start: int       # character offset in source
220|    end: int         # character offset (exclusive)
221|
222|
223|def lex(query: str) -> list[LexToken]:
224|    """Lex a ClickHouse query string into raw tokens (mirrors ClickHouse Lexer).
225|
226|    Whitespace and comments are skipped (matching isSignificant() in the C++ lexer).
227|    """
228|    tokens: list[LexToken] = []
229|    pos = 0
230|    n = len(query)
231|    while pos < n:
232|        matched = False
233|        for ttype, pattern in _COMPILED_PATTERNS:
234|            m = pattern.match(query, pos)
235|            if m:
236|                text = m.group(0)
237|                if ttype not in ("Whitespace", "Comment"):
238|                    tokens.append(LexToken(ttype, text, pos, m.end()))
239|                pos = m.end()
240|                matched = True
241|                break
242|        if not matched:
243|            # Unrecognized character — skip it (could also raise)
244|            pos += 1
245|    return tokens
246|
247|
248|# ─── Lookbehind copy encoding ────────────────────────────────────────────────
249|
250|@dataclass
251|class EncodedToken:
252|    """A token after lookbehind copy encoding."""
253|    token_id: int           # vocabulary index
254|    token_str: str          # vocabulary token string
255|    raw_text: str           # original text (for identifiers/strings; empty for keywords/ops)
256|    lex_type: str           # original lexer token type
257|    is_ident: bool          # whether this is an identifier (for copy resolution)
258|    is_string: bool         # whether this is a string literal (for copy resolution)
259|
260|
261|def _classify_number(text: str) -> str:
262|    """Classify a number token as NUM_INT, NUM_FLOAT, or NUM_HEX."""
263|    lower = text.lower()
264|    if lower.startswith("0x"):
265|        return "NUM_HEX"
266|    if "." in text or "e" in lower or "p" in lower:
267|        return "NUM_FLOAT"
268|    return "NUM_INT"
269|
270|
271|def encode_tokens(lex_tokens: list[LexToken]) -> list[EncodedToken]:
272|    """Encode raw lexed tokens into vocabulary tokens with lookbehind copies.
273|
274|    For identifiers (BareWord that's not a keyword, QuotedIdentifier):
275|      - First occurrence → NEW_IDENT
276|      - Repeat within MAX_COPY_BACK distance → COPY_BACK_N
277|      - Repeat beyond MAX_COPY_BACK → NEW_IDENT (treated as new)
278|
279|    For string literals:
280|      - First occurrence → NEW_STRING
281|      - Repeat within MAX_COPY_BACK distance → COPY_STR_BACK_N
282|      - Repeat beyond MAX_COPY_BACK → NEW_STRING
283|
284|    The raw_text is preserved so the inference engine can resolve copies.
285|    """
286|    result: list[EncodedToken] = []
287|    # Track recent identifier texts and their positions (token index in result)
288|    ident_history: list[tuple[str, int]] = []  # (text_lower, token_index)
289|    string_history: list[tuple[str, int]] = []  # (text, token_index)
290|
291|    for lt in lex_tokens:
292|        et = EncodedToken(
293|            token_id=UNK_ID, token_str="<UNK>", raw_text="",
294|            lex_type=lt.type, is_ident=False, is_string=False,
295|        )
296|
297|        if lt.type == "BareWord":
298|            upper = lt.text.upper()
299|            if upper in KEYWORD_SET:
300|                kw_tok = f"KW_{upper}"
301|                et.token_id = VOCAB.get(kw_tok, UNK_ID)
302|                et.token_str = kw_tok
303|            else:
304|                # Identifier — lookbehind copy encoding
305|                lower = lt.text.lower()
306|                copy_dist = _find_copy_distance(ident_history, lower, len(result))
307|                if copy_dist is not None and copy_dist <= MAX_COPY_BACK:
308|                    tok_str = f"COPY_BACK_{copy_dist}"
309|                else:
310|                    tok_str = "NEW_IDENT"
311|                et.token_id = VOCAB[tok_str]
312|                et.token_str = tok_str
313|                et.raw_text = lt.text
314|                et.is_ident = True
315|                ident_history.append((lower, len(result)))
316|        elif lt.type == "QuotedIdentifier":
317|            # Quoted identifier — strip quotes for comparison
318|            inner = lt.text
319|            if inner.startswith("`") and inner.endswith("`"):
320|                inner = inner[1:-1].replace("``", "`")
321|            elif inner.startswith('"') and inner.endswith('"'):
322|                inner = inner[1:-1].replace('""', '"')
323|            lower = inner.lower()
324|            copy_dist = _find_copy_distance(ident_history, lower, len(result))
325|            if copy_dist is not None and copy_dist <= MAX_COPY_BACK:
326|                tok_str = f"COPY_BACK_{copy_dist}"
327|            else:
328|                tok_str = "NEW_IDENT"
329|            et.token_id = VOCAB[tok_str]
330|            et.token_str = tok_str
331|            et.raw_text = lt.text
332|            et.is_ident = True
333|            ident_history.append((lower, len(result)))
334|        elif lt.type == "StringLiteral":
335|            # String literal — strip quotes for comparison
336|            inner = lt.text[1:-1].replace("''", "'")
337|            copy_dist = _find_copy_distance(string_history, inner, len(result))
338|            if copy_dist is not None and copy_dist <= MAX_COPY_BACK:
339|                tok_str = f"COPY_STR_BACK_{copy_dist}"
340|            else:
341|                tok_str = "NEW_STRING"
342|            et.token_id = VOCAB[tok_str]
343|            et.token_str = tok_str
344|            et.raw_text = lt.text
345|            et.is_string = True
346|            string_history.append((inner, len(result)))
347|        elif lt.type == "Number":
348|            num_type = _classify_number(lt.text)
349|            et.token_id = VOCAB[num_type]
350|            et.token_str = num_type
351|        elif lt.type in ("HexStringLiteral", "BinaryStringLit"):
352|            et.token_id = VOCAB["NEW_STRING"]
353|            et.token_str = "NEW_STRING"
354|            et.raw_text = lt.text
355|            et.is_string = True
356|        elif lt.type == "HereDoc":
357|            et.token_id = VOCAB["NEW_STRING"]
358|            et.token_str = "NEW_STRING"
359|            et.raw_text = lt.text
360|            et.is_string = True
361|        else:
362|            # Operator/punctuation
363|            op_tok = _LEX_TYPE_TO_OP.get(lt.type)
364|            if op_tok is not None:
365|                et.token_id = VOCAB[op_tok]
366|                et.token_str = op_tok
367|            else:
368|                et.token_id = UNK_ID
369|                et.token_str = "<UNK>"
370|
371|        result.append(et)
372|
373|    return result
374|
375|
376|# Map lexer token types to operator vocabulary tokens
377|_LEX_TYPE_TO_OP = {
378|    "OpeningRoundBracket": "LPAREN",
379|    "ClosingRoundBracket": "RPAREN",
380|    "OpeningSquareBracket": "LBRACKET",
381|    "ClosingSquareBracket": "RBRACKET",
382|    "OpeningCurlyBrace": "LBRACE",
383|    "ClosingCurlyBrace": "RBRACE",
384|    "Comma": "COMMA",
385|    "Semicolon": "SEMICOLON",
386|    "Dot": "DOT",
387|    "Asterisk": "ASTERISK",
388|    "Plus": "PLUS",
389|    "Minus": "MINUS",
390|    "Slash": "SLASH",
391|    "Percent": "PERCENT",
392|    "Arrow": "ARROW",
393|    "QuestionMark": "QUESTION",
394|    "Colon": "COLON",
395|    "Caret": "CARET",
396|    "DoubleColon": "DOUBLE_COLON",
397|    "Equals": "EQUALS",
398|    "NotEquals": "NOT_EQUALS",
399|    "Less": "LESS",
400|    "Greater": "GREATER",
401|    "LessOrEquals": "LESS_OR_EQUALS",
402|    "GreaterOrEquals": "GREATER_OR_EQUALS",
403|    "Spaceship": "SPACESHIP",
404|    "Concatenation": "CONCAT",
405|    "At": "AT",
406|    "DoubleAt": "DOUBLE_AT",
407|}
408|
409|
410|def _find_copy_distance(history: list[tuple[str, int]], text: str, current_idx: int) -> Optional[int]:
411|    """Find the most recent occurrence of text in history and return the distance back."""
412|    # Search from most recent backwards
413|    for i in range(len(history) - 1, -1, -1):
414|        hist_text, hist_idx = history[i]
415|        if hist_text == text:
416|            return current_idx - hist_idx
417|    return None
418|
419|
420|# ─── Public API ──────────────────────────────────────────────────────────────
421|
422|def tokenize(query: str) -> list[EncodedToken]:
423|    """Tokenize a ClickHouse query into encoded tokens with lookbehind copies."""
424|    lex_tokens = lex(query)
425|    return encode_tokens(lex_tokens)
426|
427|
428|def tokenize_to_ids(query: str, add_bos: bool = True, add_eos: bool = True) -> list[int]:
429|    """Tokenize a query and return a list of vocabulary token IDs."""
430|    encoded = tokenize(query)
431|    ids = [et.token_id for et in encoded]
432|    if add_bos:
433|        ids = [BOS_ID] + ids
434|    if add_eos:
435|        ids = ids + [EOS_ID]
436|    return ids
437|
438|
439|def detokenize(encoded_tokens: list[EncodedToken]) -> str:
440|    """Reconstruct the original query text from encoded tokens.
441|
442|    This uses the raw_text stored in each EncodedToken for identifiers and strings.
443|    For copy tokens, the raw_text is the actual text (stored at encode time).
444|    """
445|    parts: list[str] = []
446|    for et in encoded_tokens:
447|        if et.is_ident or et.is_string:
448|            parts.append(et.raw_text)
449|        elif et.token_str.startswith("KW_"):
450|            parts.append(et.token_str[3:])
451|        elif et.token_str in NUMBER_TOKENS:
452|            parts.append(et.raw_text if et.raw_text else "0")
453|        elif et.token_str in OPERATOR_TOKENS:
454|            parts.append(OPERATOR_TOKENS[et.token_str])
455|        elif et.token_str in SPECIAL_TOKENS:
456|            continue
457|        else:
458|            parts.append(et.token_str)
459|    # Simple spacing heuristic
460|    return _join_with_spacing(parts)
461|
462|
463|def _join_with_spacing(parts: list[str]) -> str:
464|    """Join token parts with reasonable spacing."""
465|    result = ""
466|    for i, part in enumerate(parts):
467|        if i == 0:
468|            result = part
469|            continue
470|        prev = parts[i - 1]
471|        # No space after opening brackets or before closing brackets/comma/semicolon/dot
472|        no_space_before = part in (")", "]", "}", ",", ";", ".")
473|        no_space_after = prev in ("(", "[", "{", ".")
474|        if no_space_before or no_space_after:
475|            result += part
476|        else:
477|            result += " " + part
478|    return result
479|
480|
481|def save_vocab(path: str):
482|    """Save the vocabulary to a JSON file."""
483|    with open(path, "w") as f:
484|        json.dump({
485|            "vocab": VOCAB,
486|            "vocab_size": VOCAB_SIZE,
487|            "special_tokens": {
488|                "pad": PAD_ID, "bos": BOS_ID, "eos": EOS_ID, "unk": UNK_ID,
489|            },
490|            "max_copy_back": MAX_COPY_BACK,
491|            "keywords": KEYWORDS,
492|            "operators": OPERATOR_TOKENS,
493|            "number_tokens": NUMBER_TOKENS,
494|            "ident_copy_tokens": IDENT_COPY_TOKENS,
495|            "string_copy_tokens": STRING_COPY_TOKENS,
496|        }, f, indent=2)
497|
498|
499|if __name__ == "__main__":
500|    # Quick self-test
501|    test_queries = [
502|        "SELECT count(), avg(price) FROM orders WHERE region = 'US' GROUP BY city ORDER BY count() DESC LIMIT 10",
503|        "SELECT * FROM events WHERE event_type = 'click' AND user_id IN (SELECT user_id FROM users WHERE country = 'US') ORDER BY timestamp DESC LIMIT 100",
504|        "CREATE TABLE IF NOT EXISTS analytics.events (id UUID, event_time DateTime) ENGINE = MergeTree() ORDER BY (event_time)",
505|    ]
506|    for q in test_queries:
507|        print(f"\nQuery: {q}")
508|        encoded = tokenize(q)
509|        for et in encoded:
510|            extra = f" [{et.raw_text}]" if (et.is_ident or et.is_string) else ""
511|            print(f"  {et.token_str}{extra}")
512|        ids = tokenize_to_ids(q)
513|        print(f"  IDs: {ids}")
514|    print(f"\nVocab size: {VOCAB_SIZE}")
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/model/config.py" start_line="1" end_line="329" total_lines="329">
  1|"""
  2|GPT-based Small Language Model for ClickHouse query autocomplete.
  3|
  4|Design goals:
  5|  - ~10 MB model size (float32) → ~2.7M parameters
  6|  - CPU-optimized: small dimensions, weight tying, pre-LayerNorm
  7|  - Embeddable in C++ (ClickHouse binary) and exportable to WebAssembly
  8|  - Constrained sampling at inference time (syntactic correctness + table dicts)
  9|
 10|Architecture:
 11|  - d_model = 192, n_layers = 6, n_heads = 6, d_ff = 768
 12|  - max_seq_len = 256
 13|  - Weight tying (input embedding == output projection)
 14|  - Pre-LayerNorm transformer blocks
 15|  - Learned positional embeddings
 16|  - GELU activation in FFN
 17|  - ~2.77M params ≈ 10.6 MB (float32), ~5.3 MB (float16), ~2.7 MB (int8)
 18|"""
 19|
 20|import math
 21|import torch
 22|import torch.nn as nn
 23|import torch.nn.functional as F
 24|from dataclasses import dataclass, asdict
 25|from typing import Optional
 26|
 27|import sys
 28|sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent / "tokenizer"))
 29|from ch_tokenizer import VOCAB_SIZE, PAD_ID, BOS_ID, EOS_ID
 30|
 31|
 32|@dataclass
 33|class ModelConfig:
 34|    """Configuration for the ClickHouse autocomplete GPT model."""
 35|    vocab_size: int = VOCAB_SIZE
 36|    d_model: int = 192
 37|    n_layers: int = 6
 38|    n_heads: int = 6
 39|    d_ff: int = 768
 40|    max_seq_len: int = 256
 41|    n_heads_kv: int = 6       # no GQA (d_model is small)
 42|    dropout: float = 0.1
 43|    weight_tying: bool = True
 44|    pad_token_id: int = PAD_ID
 45|    bos_token_id: int = BOS_ID
 46|    eos_token_id: int = EOS_ID
 47|
 48|    def param_count(self) -> int:
 49|        """Estimate total parameter count."""
 50|        embed = self.vocab_size * self.d_model
 51|        pos = self.max_seq_len * self.d_model
 52|        per_layer = 12 * self.d_model * self.d_model + 4 * self.d_model  # approx
 53|        layers = self.n_layers * per_layer
 54|        output = 0 if self.weight_tying else self.vocab_size * self.d_model
 55|        return embed + pos + layers + output
 56|
 57|    def model_size_mb(self, bytes_per_param: int = 4) -> float:
 58|        return self.param_count() * bytes_per_param / (1024 * 1024)
 59|
 60|    def to_dict(self) -> dict:
 61|        return asdict(self)
 62|
 63|    @classmethod
 64|    def from_dict(cls, d: dict) -> "ModelConfig":
 65|        return cls(**d)
 66|
 67|
 68|class CausalSelfAttention(nn.Module):
 69|    """Multi-head causal self-attention with pre-LayerNorm."""
 70|
 71|    def __init__(self, config: ModelConfig):
 72|        super().__init__()
 73|        assert config.d_model % config.n_heads == 0
 74|        self.n_heads = config.n_heads
 75|        self.d_head = config.d_model // config.n_heads
 76|        self.d_model = config.d_model
 77|        self.scale = 1.0 / math.sqrt(self.d_head)
 78|
 79|        # Fused QKV projection
 80|        self.qkv = nn.Linear(config.d_model, 3 * config.d_model, bias=False)
 81|        self.proj = nn.Linear(config.d_model, config.d_model, bias=False)
 82|        self.dropout = config.dropout
 83|
 84|    def forward(self, x: torch.Tensor) -> torch.Tensor:
 85|        B, T, C = x.shape
 86|        qkv = self.qkv(x)  # (B, T, 3*C)
 87|        qkv = qkv.view(B, T, 3, self.n_heads, self.d_head)
 88|        qkv = qkv.permute(2, 0, 3, 1, 4)  # (3, B, n_heads, T, d_head)
 89|        q, k, v = qkv[0], qkv[1], qkv[2]
 90|
 91|        # Scaled dot-product attention with causal mask
 92|        att = torch.matmul(q, k.transpose(-2, -1)) * self.scale  # (B, n_heads, T, T)
 93|        # Causal mask
 94|        mask = torch.triu(torch.ones(T, T, device=x.device, dtype=torch.bool), diagonal=1)
 95|        att = att.masked_fill(mask, float("-inf"))
 96|        att = F.softmax(att, dim=-1)
 97|        if self.dropout > 0 and self.training:
 98|            att = F.dropout(att, p=self.dropout)
 99|
100|        out = torch.matmul(att, v)  # (B, n_heads, T, d_head)
101|        out = out.transpose(1, 2).contiguous().view(B, T, C)
102|        out = self.proj(out)
103|        if self.dropout > 0 and self.training:
104|            out = F.dropout(out, p=self.dropout)
105|        return out
106|
107|
108|class FeedForward(nn.Module):
109|    """Position-wise feed-forward network with GELU."""
110|
111|    def __init__(self, config: ModelConfig):
112|        super().__init__()
113|        self.fc1 = nn.Linear(config.d_model, config.d_ff, bias=False)
114|        self.fc2 = nn.Linear(config.d_ff, config.d_model, bias=False)
115|        self.dropout = config.dropout
116|
117|    def forward(self, x: torch.Tensor) -> torch.Tensor:
118|        x = self.fc1(x)
119|        x = F.gelu(x)
120|        if self.dropout > 0 and self.training:
121|            x = F.dropout(x, p=self.dropout)
122|        x = self.fc2(x)
123|        if self.dropout > 0 and self.training:
124|            x = F.dropout(x, p=self.dropout)
125|        return x
126|
127|
128|class TransformerBlock(nn.Module):
129|    """Pre-LayerNorm transformer block."""
130|
131|    def __init__(self, config: ModelConfig):
132|        super().__init__()
133|        self.ln1 = nn.LayerNorm(config.d_model, bias=False)
134|        self.attn = CausalSelfAttention(config)
135|        self.ln2 = nn.LayerNorm(config.d_model, bias=False)
136|        self.ffn = FeedForward(config)
137|
138|    def forward(self, x: torch.Tensor) -> torch.Tensor:
139|        x = x + self.attn(self.ln1(x))
140|        x = x + self.ffn(self.ln2(x))
141|        return x
142|
143|
144|class ClickHouseGPT(nn.Module):
145|    """GPT model for ClickHouse query autocomplete.
146|
147|    Architecture:
148|      - Token embedding + learned positional embedding
149|      - N pre-LayerNorm transformer blocks
150|      - Final LayerNorm
151|      - Output projection (tied with token embedding if weight_tying=True)
152|    """
153|
154|    def __init__(self, config: ModelConfig):
155|        super().__init__()
156|        self.config = config
157|        self.tok_emb = nn.Embedding(config.vocab_size, config.d_model, padding_idx=config.pad_token_id)
158|        self.pos_emb = nn.Embedding(config.max_seq_len, config.d_model)
159|        self.blocks = nn.ModuleList([TransformerBlock(config) for _ in range(config.n_layers)])
160|        self.ln_f = nn.LayerNorm(config.d_model, bias=False)
161|        self.dropout = config.dropout
162|
163|        if config.weight_tying:
164|            # Tie output projection weights to token embedding
165|            self.lm_head = None  # use tok_emb.weight in forward
166|        else:
167|            self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)
168|
169|        # Initialize weights
170|        self.apply(self._init_weights)
171|
172|    def _init_weights(self, module: nn.Module):
173|        if isinstance(module, nn.Linear):
174|            nn.init.normal_(module.weight, mean=0.0, std=0.02)
175|            if module.bias is not None:
176|                nn.init.zeros_(module.bias)
177|        elif isinstance(module, nn.Embedding):
178|            nn.init.normal_(module.weight, mean=0.0, std=0.02)
179|            if module.padding_idx is not None:
180|                with torch.no_grad():
181|                    module.weight[module.padding_idx].fill_(0)
182|
183|    def forward(self, idx: torch.Tensor, targets: Optional[torch.Tensor] = None) -> dict:
184|        """
185|        Args:
186|            idx: (B, T) token IDs
187|            targets: (B, T) target token IDs for loss computation
188|
189|        Returns:
190|            dict with 'logits' (B, T, V) and optionally 'loss'
191|        """
192|        B, T = idx.shape
193|        assert T <= self.config.max_seq_len, f"Sequence length {T} exceeds max {self.config.max_seq_len}"
194|
195|        # Embeddings
196|        tok = self.tok_emb(idx)  # (B, T, d_model)
197|        pos = self.pos_emb(torch.arange(T, device=idx.device).unsqueeze(0))  # (1, T, d_model)
198|        x = tok + pos
199|        if self.dropout > 0 and self.training:
200|            x = F.dropout(x, p=self.dropout)
201|
202|        # Transformer blocks
203|        for block in self.blocks:
204|            x = block(x)
205|
206|        x = self.ln_f(x)  # (B, T, d_model)
207|
208|        # Output projection
209|        if self.config.weight_tying:
210|            logits = x @ self.tok_emb.weight.T  # (B, T, vocab_size)
211|        else:
212|            logits = self.lm_head(x)
213|
214|        result = {"logits": logits}
215|
216|        if targets is not None:
217|            # Cross-entropy loss, ignoring padding
218|            loss = F.cross_entropy(
219|                logits.view(-1, logits.size(-1)),
220|                targets.view(-1),
221|                ignore_index=self.config.pad_token_id,
222|            )
223|            result["loss"] = loss
224|
225|        return result
226|
227|    @torch.no_grad()
228|    def generate(
229|        self,
230|        idx: torch.Tensor,
231|        max_new_tokens: int,
232|        temperature: float = 1.0,
233|        top_k: int = 0,
234|        top_p: float = 0.0,
235|        eos_token_id: Optional[int] = None,
236|    ) -> torch.Tensor:
237|        """Autoregressive generation with optional top-k / top-p sampling.
238|
239|        Args:
240|            idx: (1, T) starting token IDs (the prefix)
241|            max_new_tokens: maximum number of tokens to generate
242|            temperature: sampling temperature (1.0 = no change, <1 = sharper)
243|            top_k: if > 0, only sample from top-k logits
244|            top_p: if > 0, use nucleus sampling
245|            eos_token_id: if set, stop generation when this token is produced
246|
247|        Returns:
248|            (1, T + max_new_tokens) token IDs
249|        """
250|        self.eval()
251|        eos = eos_token_id if eos_token_id is not None else self.config.eos_token_id
252|
253|        for _ in range(max_new_tokens):
254|            # Crop context if too long
255|            idx_cond = idx if idx.size(1) <= self.config.max_seq_len else idx[:, -self.config.max_seq_len:]
256|
257|            logits = self(idx_cond)["logits"]  # (1, T, V)
258|            logits = logits[:, -1, :] / max(temperature, 1e-8)  # (1, V)
259|
260|            # Apply top-k filtering
261|            if top_k > 0:
262|                v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
263|                logits[logits < v[:, [-1]]] = float("-inf")
264|
265|            # Apply top-p (nucleus) filtering
266|            if top_p > 0.0:
267|                sorted_logits, sorted_indices = torch.sort(logits, descending=True)
268|                cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
269|                sorted_indices_to_remove = cumulative_probs > top_p
270|                sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
271|                sorted_indices_to_remove[..., 0] = False
272|                indices_to_remove = sorted_indices_to_remove.scatter(
273|                    1, sorted_indices, sorted_indices_to_remove
274|                )
275|                logits = logits.masked_fill(indices_to_remove, float("-inf"))
276|
277|            probs = F.softmax(logits, dim=-1)
278|            next_token = torch.multinomial(probs, num_samples=1)  # (1, 1)
279|            idx = torch.cat([idx, next_token], dim=1)
280|
281|            if eos is not None and next_token.item() == eos:
282|                break
283|
284|        return idx
285|
286|    def num_parameters(self) -> int:
287|        return sum(p.numel() for p in self.parameters())
288|
289|    def model_size_mb(self, bytes_per_param: int = 4) -> float:
290|        return self.num_parameters() * bytes_per_param / (1024 * 1024)
291|
292|
293|def create_model(config: Optional[ModelConfig] = None) -> ClickHouseGPT:
294|    """Create a ClickHouseGPT model with the given or default config."""
295|    if config is None:
296|        config = ModelConfig()
297|    model = ClickHouseGPT(config)
298|    return model
299|
300|
301|if __name__ == "__main__":
302|    config = ModelConfig()
303|    model = create_model(config)
304|    print(f"Model config: {config.to_dict()}")
305|    print(f"Parameters: {model.num_parameters():,}")
306|    print(f"Model size (float32): {model.model_size_mb(4):.2f} MB")
307|    print(f"Model size (float16): {model.model_size_mb(2):.2f} MB")
308|    print(f"Model size (int8):    {model.model_size_mb(1):.2f} MB")
309|    print(f"Estimated params:     {config.param_count():,}")
310|    print(f"Estimated size (f32): {config.model_size_mb(4):.2f} MB")
311|
312|    # Quick forward pass test
313|    import sys
314|    sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent / "tokenizer"))
315|    from ch_tokenizer import tokenize_to_ids
316|
317|    query = "SELECT count(), avg(price) FROM orders WHERE region = 'US' GROUP BY city ORDER BY count() DESC LIMIT 10"
318|    ids = tokenize_to_ids(query)
319|    print(f"\nTokenized query ({len(ids)} tokens): {ids[:10]}...")
320|
321|    x = torch.tensor([ids])
322|    out = model(x, targets=x)
323|    print(f"Logits shape: {out['logits'].shape}")
324|    print(f"Loss: {out['loss'].item():.4f}")
325|
326|    # Generation test
327|    prefix = torch.tensor([[BOS_ID]])
328|    generated = model.generate(prefix, max_new_tokens=20, temperature=0.8, top_k=10)
329|    print(f"Generated {generated.size(1)} tokens")
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/model/training.py" start_line="1" end_line="417" total_lines="417">
  1|"""
  2|Dataset and training pipeline for ClickHouse autocomplete GPT model.
  3|
  4|The dataset loads trajectory JSONL files, tokenizes queries using the
  5|ClickHouse tokenizer with lookbehind copies, and creates training samples
  6|from both full queries and completion-point prefixes.
  7|"""
  8|
  9|import json
 10|import math
 11|import os
 12|import random
 13|import torch
 14|from torch.utils.data import Dataset, DataLoader
 15|from pathlib import Path
 16|from typing import Optional
 17|
 18|import sys
 19|sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tokenizer"))
 20|sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "model"))
 21|
 22|from ch_tokenizer import (
 23|    tokenize, tokenize_to_ids, VOCAB, VOCAB_SIZE,
 24|    BOS_ID, EOS_ID, PAD_ID, UNK_ID,
 25|    EncodedToken,
 26|)
 27|from config import ModelConfig, ClickHouseGPT, create_model
 28|
 29|
 30|class TrajectoryDataset(Dataset):
 31|    """Dataset of ClickHouse query trajectories for training the autocomplete model.
 32|
 33|    Two types of training samples are generated from each trajectory:
 34|      1. Full-query samples: the complete query tokenized with BOS/EOS
 35|      2. Completion-point samples: prefix → completion pairs at natural boundaries
 36|
 37|    For completion-point samples, the model sees the prefix and learns to
 38|    predict the completion tokens, with an EOS after the completion.
 39|    """
 40|
 41|    def __init__(
 42|        self,
 43|        trajectory_dir: str,
 44|        max_seq_len: int = 256,
 45|        augment: bool = True,
 46|        augment_factor: int = 3,
 47|    ):
 48|        self.max_seq_len = max_seq_len
 49|        self.augment = augment
 50|        self.augment_factor = augment_factor
 51|        self.samples: list[dict] = []
 52|        self._load_trajectories(trajectory_dir)
 53|
 54|    def _load_trajectories(self, trajectory_dir: str):
 55|        """Load all JSONL files from the trajectory directory."""
 56|        traj_dir = Path(trajectory_dir)
 57|        jsonl_files = sorted(traj_dir.glob("*.jsonl"))
 58|        total_trajectories = 0
 59|
 60|        for fpath in jsonl_files:
 61|            with open(fpath) as f:
 62|                for line in f:
 63|                    line = line.strip()
 64|                    if not line:
 65|                        continue
 66|                    traj = json.loads(line)
 67|                    self._process_trajectory(traj)
 68|                    total_trajectories += 1
 69|
 70|        print(f"Loaded {total_trajectories} trajectories, generated {len(self.samples)} training samples")
 71|
 72|    def _process_trajectory(self, traj: dict):
 73|        """Generate training samples from a single trajectory."""
 74|        query = traj["query"]
 75|
 76|        # Sample 1: Full query (causal language modeling)
 77|        full_ids = tokenize_to_ids(query, add_bos=True, add_eos=True)
 78|        if len(full_ids) <= self.max_seq_len:
 79|            self.samples.append({
 80|                "type": "full",
 81|                "input_ids": full_ids,
 82|                "traj_id": traj.get("id", ""),
 83|            })
 84|
 85|        # Sample 2+: Completion points (prefix → completion)
 86|        for cp in traj.get("completion_points", []):
 87|            prefix = cp["prefix"]
 88|            completion = cp["completion"]
 89|
 90|            # Tokenize prefix and completion separately, then combine
 91|            prefix_ids = tokenize_to_ids(prefix, add_bos=True, add_eos=False)
 92|            completion_ids = tokenize_to_ids(completion, add_bos=False, add_eos=True)
 93|
 94|            combined = prefix_ids + completion_ids
 95|            if len(combined) <= self.max_seq_len:
 96|                # The input is the full sequence, targets are shifted
 97|                # But we only compute loss on the completion part
 98|                self.samples.append({
 99|                    "type": "completion",
100|                    "input_ids": combined,
101|                    "prefix_len": len(prefix_ids),
102|                    "traj_id": traj.get("id", ""),
103|                })
104|
105|        # Augmentation: generate partial-prefix samples
106|        if self.augment:
107|            full_ids_no_special = tokenize_to_ids(query, add_bos=True, add_eos=False)
108|            # Create random truncation points
109|            n_tokens = len(full_ids_no_special)
110|            if n_tokens > 10:
111|                for _ in range(self.augment_factor):
112|                    # Random truncation point (between 30% and 90% of the query)
113|                    cut = random.randint(int(n_tokens * 0.3), int(n_tokens * 0.9))
114|                    truncated = full_ids_no_special[:cut] + [EOS_ID]
115|                    if len(truncated) <= self.max_seq_len:
116|                        self.samples.append({
117|                            "type": "truncated",
118|                            "input_ids": truncated,
119|                            "traj_id": traj.get("id", ""),
120|                        })
121|
122|    def __len__(self) -> int:
123|        return len(self.samples)
124|
125|    def __getitem__(self, idx: int) -> dict:
126|        sample = self.samples[idx]
127|        input_ids = sample["input_ids"]
128|
129|        # Create targets (shifted right by 1)
130|        # input: [BOS, t1, t2, ..., tn, EOS]
131|        # target: [t1, t2, ..., tn, EOS, PAD]
132|        x = input_ids[:-1]
133|        y = input_ids[1:]
134|
135|        # For completion samples, mask the loss on prefix tokens with PAD
136|        if sample["type"] == "completion":
137|            prefix_len = sample["prefix_len"]
138|            # The prefix in x is input_ids[:prefix_len-1] (since x = input_ids[:-1])
139|            # We want to mask everything before the completion starts
140|            y_masked = list(y)
141|            for i in range(max(0, prefix_len - 1)):
142|                y_masked[i] = PAD_ID
143|            y = y_masked
144|
145|        return {
146|            "input_ids": torch.tensor(x, dtype=torch.long),
147|            "target_ids": torch.tensor(y, dtype=torch.long),
148|        }
149|
150|
151|def collate_fn(batch: list[dict], pad_id: int = PAD_ID) -> dict:
152|    """Collate variable-length sequences into a padded batch."""
153|    max_len = max(len(item["input_ids"]) for item in batch)
154|    # Cap at max_seq_len
155|    max_len = min(max_len, 256)
156|
157|    input_ids = []
158|    target_ids = []
159|    attention_mask = []
160|
161|    for item in batch:
162|        x = item["input_ids"][:max_len]
163|        y = item["target_ids"][:max_len]
164|        attn = [1] * len(x)
165|
166|        # Pad to max_len
167|        pad_count = max_len - len(x)
168|        x = torch.cat([x, torch.full((pad_count,), pad_id, dtype=torch.long)])
169|        y = torch.cat([y, torch.full((pad_count,), pad_id, dtype=torch.long)])
170|        attn = attn + [0] * pad_count
171|
172|        input_ids.append(x)
173|        target_ids.append(y)
174|        attention_mask.append(torch.tensor(attn, dtype=torch.long))
175|
176|    return {
177|        "input_ids": torch.stack(input_ids),
178|        "target_ids": torch.stack(target_ids),
179|        "attention_mask": torch.stack(attention_mask),
180|    }
181|
182|
183|class Trainer:
184|    """Training loop for the ClickHouse autocomplete GPT model."""
185|
186|    def __init__(
187|        self,
188|        model: ClickHouseGPT,
189|        train_dataset: TrajectoryDataset,
190|        val_dataset: Optional[TrajectoryDataset] = None,
191|        batch_size: int = 16,
192|        learning_rate: float = 3e-4,
193|        weight_decay: float = 0.01,
194|        warmup_steps: int = 50,
195|        max_steps: int = 2000,
196|        eval_interval: int = 200,
197|        eval_iters: int = 50,
198|        save_dir: str = "checkpoints",
199|        device: str = "cpu",
200|    ):
201|        self.model = model
202|        self.train_dataset = train_dataset
203|        self.val_dataset = val_dataset
204|        self.batch_size = batch_size
205|        self.learning_rate = learning_rate
206|        self.weight_decay = weight_decay
207|        self.warmup_steps = warmup_steps
208|        self.max_steps = max_steps
209|        self.eval_interval = eval_interval
210|        self.eval_iters = eval_iters
211|        self.save_dir = Path(save_dir)
212|        self.save_dir.mkdir(parents=True, exist_ok=True)
213|        self.device = device
214|
215|        self.train_loader = DataLoader(
216|            train_dataset,
217|            batch_size=batch_size,
218|            shuffle=True,
219|            collate_fn=collate_fn,
220|            drop_last=False,
221|        )
222|
223|        # Optimizer with weight decay (no decay on biases and layernorms)
224|        decay_params = []
225|        nodecay_params = []
226|        for name, param in model.named_parameters():
227|            if param.requires_grad:
228|                if param.ndim >= 2:
229|                    decay_params.append(param)
230|                else:
231|                    nodecay_params.append(param)
232|
233|        self.optimizer = torch.optim.AdamW(
234|            [
235|                {"params": decay_params, "weight_decay": weight_decay},
236|                {"params": nodecay_params, "weight_decay": 0.0},
237|            ],
238|            lr=learning_rate,
239|            betas=(0.9, 0.95),
240|        )
241|
242|        self.step = 0
243|        self.best_val_loss = float("inf")
244|
245|    def get_lr(self) -> float:
246|        """Learning rate schedule: linear warmup → cosine decay."""
247|        if self.step < self.warmup_steps:
248|            return self.learning_rate * (self.step + 1) / self.warmup_steps
249|        decay_steps = self.max_steps - self.warmup_steps
250|        progress = (self.step - self.warmup_steps) / max(decay_steps, 1)
251|        return self.learning_rate * 0.5 * (1.0 + math.cos(math.pi * progress))
252|
253|    def train(self):
254|        """Run the training loop."""
255|        self.model.to(self.device)
256|        self.model.train()
257|
258|        train_iter = iter(self.train_loader)
259|        running_loss = 0.0
260|        running_count = 0
261|
262|        print(f"Starting training: {self.max_steps} steps, batch_size={self.batch_size}, device={self.device}")
263|        print(f"Train samples: {len(self.train_dataset)}")
264|
265|        for step in range(1, self.max_steps + 1):
266|            self.step = step
267|
268|            # Get batch
269|            try:
270|                batch = next(train_iter)
271|            except StopIteration:
272|                train_iter = iter(self.train_loader)
273|                batch = next(train_iter)
274|
275|            # Update learning rate
276|            lr = self.get_lr()
277|            for pg in self.optimizer.param_groups:
278|                pg["lr"] = lr
279|
280|            # Forward pass
281|            input_ids = batch["input_ids"].to(self.device)
282|            target_ids = batch["target_ids"].to(self.device)
283|
284|            out = self.model(input_ids, targets=target_ids)
285|            loss = out["loss"]
286|
287|            # Backward pass
288|            self.optimizer.zero_grad()
289|            loss.backward()
290|            # Gradient clipping
291|            torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
292|            self.optimizer.step()
293|
294|            running_loss += loss.item()
295|            running_count += 1
296|
297|            if step % 50 == 0:
298|                avg_loss = running_loss / running_count
299|                print(f"  Step {step}/{self.max_steps} | loss={avg_loss:.4f} | lr={lr:.6f}")
300|                running_loss = 0.0
301|                running_count = 0
302|
303|            # Evaluation
304|            if step % self.eval_interval == 0:
305|                val_loss = self.evaluate()
306|                print(f"  [Eval] Step {step} | val_loss={val_loss:.4f}")
307|                if val_loss < self.best_val_loss:
308|                    self.best_val_loss = val_loss
309|                    self.save_checkpoint("best.pt")
310|                    print(f"  [Eval] New best val_loss={val_loss:.4f}, saved best.pt")
311|
312|        # Final save
313|        self.save_checkpoint("final.pt")
314|        print(f"\nTraining complete. Best val_loss={self.best_val_loss:.4f}")
315|
316|    def evaluate(self) -> float:
317|        """Evaluate on the validation dataset (or a subset of train)."""
318|        self.model.eval()
319|        total_loss = 0.0
320|        count = 0
321|
322|        dataset = self.val_dataset if self.val_dataset else self.train_dataset
323|        n = min(self.eval_iters, len(dataset))
324|
325|        # Random subset
326|        indices = random.sample(range(len(dataset)), n)
327|        subset = torch.utils.data.Subset(dataset, indices)
328|        loader = DataLoader(subset, batch_size=self.batch_size, collate_fn=collate_fn)
329|
330|        with torch.no_grad():
331|            for batch in loader:
332|                input_ids = batch["input_ids"].to(self.device)
333|                target_ids = batch["target_ids"].to(self.device)
334|                out = self.model(input_ids, targets=target_ids)
335|                total_loss += out["loss"].item()
336|                count += 1
337|
338|        self.model.train()
339|        return total_loss / max(count, 1)
340|
341|    def save_checkpoint(self, filename: str):
342|        """Save model checkpoint."""
343|        path = self.save_dir / filename
344|        torch.save({
345|            "model_state_dict": self.model.state_dict(),
346|            "model_config": self.model.config.to_dict(),
347|            "step": self.step,
348|            "best_val_loss": self.best_val_loss,
349|        }, path)
350|        print(f"  Saved checkpoint: {path}")
351|
352|    def load_checkpoint(self, path: str):
353|        """Load model checkpoint."""
354|        checkpoint = torch.load(path, map_location=self.device, weights_only=False)
355|        self.model.load_state_dict(checkpoint["model_state_dict"])
356|        self.step = checkpoint.get("step", 0)
357|        self.best_val_loss = checkpoint.get("best_val_loss", float("inf"))
358|        print(f"  Loaded checkpoint: {path} (step={self.step}, val_loss={self.best_val_loss:.4f})")
359|
360|
361|def create_data_loaders(
362|    trajectory_dir: str,
363|    batch_size: int = 16,
364|    val_ratio: float = 0.15,
365|    max_seq_len: int = 256,
366|) -> tuple[TrajectoryDataset, TrajectoryDataset]:
367|    """Create train and validation datasets from trajectory files.
368|
369|    Splits by trajectory files (not individual samples) to avoid data leakage.
370|    """
371|    traj_dir = Path(trajectory_dir)
372|    jsonl_files = sorted(traj_dir.glob("*.jsonl"))
373|
374|    # Split files into train/val
375|    random.seed(42)
376|    random.shuffle(jsonl_files)
377|    n_val = max(1, int(len(jsonl_files) * val_ratio))
378|    val_files = jsonl_files[:n_val]
379|    train_files = jsonl_files[n_val:]
380|
381|    # Create temp dirs for split
382|    import tempfile
383|    train_dir = Path(tempfile.mkdtemp(prefix="ch_train_"))
384|    val_dir = Path(tempfile.mkdtemp(prefix="ch_val_"))
385|
386|    for f in train_files:
387|        (train_dir / f.name).symlink_to(f.resolve())
388|    for f in val_files:
389|        (val_dir / f.name).symlink_to(f.resolve())
390|
391|    train_dataset = TrajectoryDataset(str(train_dir), max_seq_len=max_seq_len)
392|    val_dataset = TrajectoryDataset(str(val_dir), max_seq_len=max_seq_len, augment=False)
393|
394|    return train_dataset, val_dataset
395|
396|
397|if __name__ == "__main__":
398|    # Quick dataset test
399|    traj_dir = str(Path(__file__).resolve().parent.parent / "trajectories" / "data")
400|    print(f"Loading trajectories from: {traj_dir}")
401|
402|    dataset = TrajectoryDataset(traj_dir, max_seq_len=256)
403|    print(f"Dataset size: {len(dataset)}")
404|
405|    # Check a few samples
406|    for i in [0, 1, 50, 100, 200]:
407|        if i < len(dataset):
408|            sample = dataset[i]
409|            print(f"\nSample {i}: input_ids shape={sample['input_ids'].shape}")
410|            print(f"  First 20 input IDs: {sample['input_ids'][:20].tolist()}")
411|            print(f"  First 20 target IDs: {sample['target_ids'][:20].tolist()}")
412|
413|    # Test collate
414|    from torch.utils.data import DataLoader
415|    loader = DataLoader(dataset, batch_size=4, shuffle=True, collate_fn=collate_fn)
416|    batch = next(iter(loader))
417|    print(f"\nBatch: input_ids={batch['input_ids'].shape}, target_ids={batch['target_ids'].shape}")
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/train.py" start_line="1" end_line="182" total_lines="182">
  1|#!/usr/bin/env python3
  2|"""
  3|Main training script for ClickHouse autocomplete GPT model.
  4|
  5|Usage:
  6|    python3 train.py [--epochs N] [--batch_size N] [--lr LR] [--device cpu|mps|cuda]
  7|"""
  8|
  9|import argparse
 10|import sys
 11|import time
 12|from pathlib import Path
 13|
 14|# Add project paths
 15|PROJECT_ROOT = Path(__file__).resolve().parent
 16|sys.path.insert(0, str(PROJECT_ROOT / "tokenizer"))
 17|sys.path.insert(0, str(PROJECT_ROOT / "model"))
 18|
 19|from ch_tokenizer import save_vocab, VOCAB_SIZE
 20|from config import ModelConfig, ClickHouseGPT, create_model
 21|from training import TrajectoryDataset, Trainer, create_data_loaders, collate_fn
 22|
 23|
 24|def main():
 25|    parser = argparse.ArgumentParser(description="Train ClickHouse autocomplete GPT model")
 26|    parser.add_argument("--epochs", type=int, default=30, help="Number of epochs")
 27|    parser.add_argument("--batch_size", type=int, default=16, help="Batch size")
 28|    parser.add_argument("--lr", type=float, default=3e-4, help="Learning rate")
 29|    parser.add_argument("--device", type=str, default="cpu", help="Device: cpu, mps, or cuda")
 30|    parser.add_argument("--max_seq_len", type=int, default=256, help="Max sequence length")
 31|    parser.add_argument("--d_model", type=int, default=192, help="Model dimension")
 32|    parser.add_argument("--n_layers", type=int, default=6, help="Number of transformer layers")
 33|    parser.add_argument("--n_heads", type=int, default=6, help="Number of attention heads")
 34|    parser.add_argument("--d_ff", type=int, default=768, help="FFN dimension")
 35|    parser.add_argument("--save_dir", type=str, default="checkpoints", help="Checkpoint directory")
 36|    parser.add_argument("--trajectory_dir", type=str, default="trajectories/data",
 37|                        help="Directory with trajectory JSONL files")
 38|    args = parser.parse_args()
 39|
 40|    # Save vocabulary
 41|    vocab_path = PROJECT_ROOT / "tokenizer" / "vocab.json"
 42|    save_vocab(str(vocab_path))
 43|    print(f"Vocabulary saved ({VOCAB_SIZE} tokens) to {vocab_path}")
 44|
 45|    # Create model config
 46|    config = ModelConfig(
 47|        vocab_size=VOCAB_SIZE,
 48|        d_model=args.d_model,
 49|        n_layers=args.n_layers,
 50|        n_heads=args.n_heads,
 51|        d_ff=args.d_ff,
 52|        max_seq_len=args.max_seq_len,
 53|    )
 54|
 55|    model = create_model(config)
 56|    print(f"\nModel: {config}")
 57|    print(f"Parameters: {model.num_parameters():,}")
 58|    print(f"Model size (float32): {model.model_size_mb(4):.2f} MB")
 59|    print(f"Model size (float16): {model.model_size_mb(2):.2f} MB")
 60|
 61|    # Determine device
 62|    device = args.device
 63|    if device == "mps" and not hasattr(torch.backends, "mps"):
 64|        device = "cpu"
 65|    if device == "cuda" and not torch.cuda.is_available():
 66|        device = "cpu"
 67|    print(f"Device: {device}")
 68|
 69|    # Create datasets
 70|    traj_dir = str(PROJECT_ROOT / args.trajectory_dir)
 71|    print(f"\nLoading trajectories from: {traj_dir}")
 72|    train_dataset, val_dataset = create_data_loaders(
 73|        traj_dir,
 74|        batch_size=args.batch_size,
 75|        max_seq_len=args.max_seq_len,
 76|    )
 77|    print(f"Train samples: {len(train_dataset)}")
 78|    print(f"Val samples: {len(val_dataset)}")
 79|
 80|    # Calculate steps
 81|    steps_per_epoch = len(train_dataset) // args.batch_size
 82|    total_steps = steps_per_epoch * args.epochs
 83|    print(f"Steps per epoch: {steps_per_epoch}, Total steps: {total_steps}")
 84|
 85|    # Create trainer
 86|    save_dir = PROJECT_ROOT / args.save_dir
 87|    trainer = Trainer(
 88|        model=model,
 89|        train_dataset=train_dataset,
 90|        val_dataset=val_dataset,
 91|        batch_size=args.batch_size,
 92|        learning_rate=args.lr,
 93|        max_steps=total_steps,
 94|        eval_interval=max(steps_per_epoch, 50),
 95|        eval_iters=min(50, len(val_dataset)),
 96|        save_dir=str(save_dir),
 97|        device=device,
 98|    )
 99|
100|    # Train
101|    print(f"\n{'='*60}")
102|    print(f"Starting training at {time.strftime('%H:%M:%S')}")
103|    print(f"{'='*60}")
104|    start_time = time.time()
105|    trainer.train()
106|    elapsed = time.time() - start_time
107|    print(f"\nTraining time: {elapsed:.1f}s ({elapsed/60:.1f}m)")
108|
109|    # Save final model in multiple formats
110|    print(f"\nSaving final model...")
111|    save_dir.mkdir(parents=True, exist_ok=True)
112|
113|    # PyTorch checkpoint
114|    torch.save({
115|        "model_state_dict": model.state_dict(),
116|        "model_config": config.to_dict(),
117|    }, save_dir / "ch_autocomplete.pt")
118|
119|    # Export to ONNX for C++/WASM embedding
120|    try:
121|        export_onnx(model, config, save_dir / "ch_autocomplete.onnx")
122|    except Exception as e:
123|        print(f"  ONNX export skipped: {e}")
124|
125|    # Export to raw binary weights (for C++ embedding)
126|    export_raw_weights(model, config, save_dir / "ch_autocomplete_weights.bin")
127|
128|    print(f"\nDone! Model saved to {save_dir}/")
129|
130|
131|def export_onnx(model, config, path):
132|    """Export model to ONNX format for cross-platform inference."""
133|    import torch
134|    model.eval()
135|    dummy_input = torch.randint(0, config.vocab_size, (1, 32), dtype=torch.long)
136|    torch.onnx.export(
137|        model,
138|        dummy_input,
139|        str(path),
140|        export_params=True,
141|        opset_version=14,
142|        do_constant_folding=True,
143|        input_names=["input_ids"],
144|        output_names=["logits"],
145|        dynamic_axes={
146|            "input_ids": {0: "batch", 1: "sequence"},
147|            "logits": {0: "batch", 1: "sequence"},
148|        },
149|    )
150|    print(f"  ONNX model saved: {path} ({path.stat().st_size / 1024 / 1024:.2f} MB)")
151|
152|
153|def export_raw_weights(model, config, path):
154|    """Export model weights as raw binary for C++ embedding."""
155|    import struct
156|    state_dict = model.state_dict()
157|    with open(path, "wb") as f:
158|        # Write config header
159|        config_bytes = str(config.to_dict()).encode("utf-8")
160|        f.write(struct.pack("I", len(config_bytes)))
161|        f.write(config_bytes)
162|
163|        # Write each tensor
164|        for name, tensor in state_dict.items():
165|            data = tensor.detach().cpu().numpy()
166|            name_bytes = name.encode("utf-8")
167|            f.write(struct.pack("I", len(name_bytes)))
168|            f.write(name_bytes)
169|            # Write shape
170|            shape = data.shape
171|            f.write(struct.pack("I", len(shape)))
172|            for dim in shape:
173|                f.write(struct.pack("I", dim))
174|            # Write data as float32
175|            f.write(data.astype("float32").tobytes())
176|
177|    print(f"  Raw weights saved: {path} ({path.stat().st_size / 1024 / 1024:.2f} MB)")
178|
179|
180|if __name__ == "__main__":
181|    import torch
182|    main()
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/inference/autocomplete.py" start_line="1" end_line="493" total_lines="497">
  1|"""
  2|Inference engine for ClickHouse query autocomplete.
  3|
  4|Features:
  5|  - Constrained sampling based on syntactic correctness
  6|  - Table/column dictionary awareness for identifier prediction
  7|  - Copy-back resolution for identifiers and strings
  8|  - Top-k / top-p sampling with temperature
  9|  - Beam search option
 10|
 11|The constrained sampler uses a transition table that defines which token types
 12|are valid after the current token, based on ClickHouse SQL grammar rules.
 13|"""
 14|
 15|import sys
 16|import torch
 17|import torch.nn.functional as F
 18|from pathlib import Path
 19|from typing import Optional
 20|from dataclasses import dataclass, field
 21|
 22|sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tokenizer"))
 23|sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "model"))
 24|
 25|from ch_tokenizer import (
 26|    tokenize, lex, encode_tokens, EncodedToken,
 27|    VOCAB, VOCAB_SIZE, ID_TO_TOKEN,
 28|    BOS_ID, EOS_ID, PAD_ID, UNK_ID,
 29|    KEYWORDS, KEYWORD_SET, OPERATOR_TOKENS,
 30|    IDENT_COPY_TOKENS, STRING_COPY_TOKENS, NUMBER_TOKENS,
 31|    MAX_COPY_BACK,
 32|)
 33|from config import ModelConfig, ClickHouseGPT, create_model
 34|
 35|
 36|# ─── Grammar-based valid next token sets ─────────────────────────────────────
 37|
 38|# Token categories for constraint checking
 39|KW_CLAUSES = {"KW_SELECT", "KW_FROM", "KW_WHERE", "KW_GROUP", "KW_BY",
 40|              "KW_HAVING", "KW_ORDER", "KW_LIMIT", "KW_OFFSET", "KW_UNION",
 41|              "KW_JOIN", "KW_LEFT", "KW_RIGHT", "KW_INNER", "KW_FULL",
 42|              "KW_CROSS", "KW_ON", "KW_USING", "KW_AS", "KW_AND", "KW_OR",
 43|              "KW_NOT", "KW_IN", "KW_BETWEEN", "KW_LIKE", "KW_IS",
 44|              "KW_NULL", "KW_DISTINCT", "KW_ALL", "KW_ASC", "KW_DESC",
 45|              "KW_INSERT", "KW_INTO", "KW_VALUES", "KW_CREATE", "KW_TABLE",
 46|              "KW_IF", "KW_EXISTS", "KW_ENGINE", "KW_PARTITION", "KW_TTL",
 47|              "KW_SETTINGS", "KW_ALTER", "KW_ADD", "KW_DROP", "KW_MODIFY",
 48|              "KW_COLUMN", "KW_INDEX", "KW_WITH", "KW_CASE", "KW_WHEN",
 49|              "KW_THEN", "KW_ELSE", "KW_END", "KW_OVER", "KW_PARTITION",
 50|              "KW_SAMPLE", "KW_FINAL", "KW_PREWHERE", "KW_ARRAY", "KW_SAMPLE"}
 51|
 52|KW_START = {"KW_SELECT", "KW_CREATE", "KW_INSERT", "KW_ALTER", "KW_DROP",
 53|            "KW_WITH", "KW_SHOW", "KW_DESCRIBE", "KW_EXPLAIN", "KW_SYSTEM",
 54|            "KW_OPTIMIZE", "KW_TRUNCATE", "KW_USE", "KW_SET", "KW_KILL",
 55|            "KW_ATTACH", "KW_DETACH", "KW_RENAME", "KW_GRANT", "KW_REVOKE"}
 56|
 57|# Valid tokens after BOS or start of query
 58|VALID_AFTER_START = set()
 59|for tok_str in KW_START:
 60|    if tok_str in VOCAB:
 61|        VALID_AFTER_START.add(VOCAB[tok_str])
 62|
 63|# Valid tokens after SELECT (column list start)
 64|VALID_AFTER_SELECT = set()
 65|for tok_str in ["NEW_IDENT", "ASTERISK", "KW_DISTINCT", "KW_ALL"] + \
 66|                 [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
 67|    if tok_str in VOCAB:
 68|        VALID_AFTER_SELECT.add(VOCAB[tok_str])
 69|
 70|# Valid tokens after FROM (table name)
 71|VALID_AFTER_FROM = set()
 72|for tok_str in ["NEW_IDENT"] + [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
 73|    if tok_str in VOCAB:
 74|        VALID_AFTER_FROM.add(VOCAB[tok_str])
 75|
 76|# Valid tokens after a clause keyword (WHERE, HAVING, ON, etc.)
 77|VALID_AFTER_CONDITION = set()
 78|for tok_str in ["NEW_IDENT", "KW_NOT", "KW_CASE"] + \
 79|                 [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)] + \
 80|                 ["NUM_INT", "NUM_FLOAT", "NUM_HEX"] + \
 81|                 ["NEW_STRING"] + [f"COPY_STR_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
 82|    if tok_str in VOCAB:
 83|        VALID_AFTER_CONDITION.add(VOCAB[tok_str])
 84|
 85|# Valid tokens after an identifier (operators, AS, comma, etc.)
 86|VALID_AFTER_IDENT = set()
 87|for tok_str in ["DOT", "COMMA", "LPAREN", "EQUALS", "NOT_EQUALS", "LESS", "GREATER",
 88|                "LESS_OR_EQUALS", "GREATER_OR_EQUALS", "KW_AS", "KW_AND", "KW_OR",
 89|                "KW_IN", "KW_NOT", "KW_BETWEEN", "KW_LIKE", "KW_IS", "KW_FROM",
 90|                "KW_WHERE", "KW_GROUP", "KW_ORDER", "KW_HAVING", "KW_LIMIT",
 91|                "KW_JOIN", "KW_LEFT", "KW_RIGHT", "KW_INNER", "KW_FULL",
 92|                "KW_UNION", "KW_ON", "KW_USING", "KW_ASC", "KW_DESC",
 93|                "PLUS", "MINUS", "ASTERISK", "SLASH", "PERCENT",
 94|                "SEMICOLON", "EOS_ID"] + \
 95|                [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
 96|    if tok_str in VOCAB:
 97|        VALID_AFTER_IDENT.add(VOCAB[tok_str])
 98|
 99|# Valid tokens after a string literal
100|VALID_AFTER_STRING = VALID_AFTER_IDENT
101|
102|# Valid tokens after a number
103|VALID_AFTER_NUMBER = VALID_AFTER_IDENT
104|
105|# Valid tokens after closing parenthesis
106|VALID_AFTER_RPAREN = set()
107|for tok_str in ["COMMA", "DOT", "EQUALS", "NOT_EQUALS", "LESS", "GREATER",
108|                "LESS_OR_EQUALS", "GREATER_OR_EQUALS", "KW_AS", "KW_AND", "KW_OR",
109|                "KW_FROM", "KW_WHERE", "KW_GROUP", "KW_ORDER", "KW_HAVING",
110|                "KW_LIMIT", "KW_JOIN", "KW_LEFT", "KW_RIGHT", "KW_INNER",
111|                "KW_UNION", "KW_ON", "KW_USING", "KW_ASC", "KW_DESC",
112|                "PLUS", "MINUS", "ASTERISK", "SLASH", "PERCENT",
113|                "SEMICOLON", "RPAREN", "RBRACKET"]:
114|    if tok_str in VOCAB:
115|        VALID_AFTER_RPAREN.add(VOCAB[tok_str])
116|
117|# Valid tokens after comma
118|VALID_AFTER_COMMA = set()
119|for tok_str in ["NEW_IDENT", "NEW_STRING", "NUM_INT", "NUM_FLOAT", "NUM_HEX",
120|                "KW_SELECT", "LPAREN", "ASTERISK"] + \
121|                [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)] + \
122|                [f"COPY_STR_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
123|    if tok_str in VOCAB:
124|        VALID_AFTER_COMMA.add(VOCAB[tok_str])
125|
126|# Valid tokens after dot (property access)
127|VALID_AFTER_DOT = set()
128|for tok_str in ["NEW_IDENT", "NUM_INT"] + \
129|                [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
130|    if tok_str in VOCAB:
131|        VALID_AFTER_DOT.add(VOCAB[tok_str])
132|
133|
134|def _get_valid_next_tokens(last_token_str: str, context: dict) -> set[int]:
135|    """Get the set of valid next token IDs based on the last token and context.
136|
137|    This is a simplified grammar-based constraint. In a full implementation,
138|    this would use the ClickHouse parser's state machine.
139|    """
140|    # Default: allow everything (no constraint)
141|    all_valid = None
142|
143|    # Determine context based on last token
144|    if last_token_str in ("<BOS>", "<PAD>"):
145|        return VALID_AFTER_START
146|    elif last_token_str == "KW_SELECT":
147|        return VALID_AFTER_SELECT
148|    elif last_token_str == "KW_FROM":
149|        return VALID_AFTER_FROM
150|    elif last_token_str in ("KW_WHERE", "KW_HAVING", "KW_ON", "KW_AND", "KW_OR"):
151|        return VALID_AFTER_CONDITION
152|    elif last_token_str in ("NEW_IDENT",) or last_token_str.startswith("COPY_BACK_"):
153|        return VALID_AFTER_IDENT
154|    elif last_token_str in ("NEW_STRING",) or last_token_str.startswith("COPY_STR_BACK_"):
155|        return VALID_AFTER_STRING
156|    elif last_token_str in NUMBER_TOKENS:
157|        return VALID_AFTER_NUMBER
158|    elif last_token_str == "RPAREN":
159|        return VALID_AFTER_RPAREN
160|    elif last_token_str == "COMMA":
161|        return VALID_AFTER_COMMA
162|    elif last_token_str == "DOT":
163|        return VALID_AFTER_DOT
164|    elif last_token_str == "KW_GROUP":
165|        return {VOCAB["KW_BY"]}
166|    elif last_token_str == "KW_ORDER":
167|        return {VOCAB["KW_BY"]}
168|    elif last_token_str == "KW_BY":
169|        return VALID_AFTER_COMMA  # similar to after comma
170|
171|    return all_valid  # None = no constraint
172|
173|
174|# ─── Autocomplete engine ────────────────────────────────────────────────────
175|
176|@dataclass
177|class AutocompleteResult:
178|    """Result of an autocomplete request."""
179|    completion: str          # the completed text
180|    tokens: list[int]        # token IDs of the completion
181|    token_strs: list[str]    # token strings of the completion
182|    score: float             # log probability score
183|    raw_texts: list[str]     # raw text for each token (for copy resolution)
184|
185|
186|class ClickHouseAutocomplete:
187|    """Autocomplete engine using the trained GPT model with constrained sampling.
188|
189|    Features:
190|      - Syntactic correctness: filters invalid next tokens using grammar rules
191|      - Table dictionary: biases identifier tokens toward known table/column names
192|      - Copy resolution: resolves COPY_BACK_N tokens to actual text from the prefix
193|      - Temperature / top-k / top-p sampling
194|    """
195|
196|    def __init__(
197|        self,
198|        model: ClickHouseGPT,
199|        device: str = "cpu",
200|        table_dict: Optional[dict[str, list[str]]] = None,
201|        max_copy_back: int = MAX_COPY_BACK,
202|    ):
203|        self.model = model
204|        self.device = device
205|        self.model.to(device)
206|        self.model.eval()
207|        self.table_dict = table_dict or {}
208|        self.max_copy_back = max_copy_back
209|
210|        # Build identifier vocabulary from table dictionary
211|        self._ident_bias = self._build_ident_bias()
212|
213|    def _build_ident_bias(self) -> dict[int, float]:
214|        """Build a bias map for identifier tokens based on the table dictionary.
215|
216|        When we know the available tables and columns, we can bias the model
217|        toward NEW_IDENT (which will be resolved from the dictionary) over
218|        COPY_BACK tokens that don't match any known name.
219|        """
220|        bias = {}
221|        if self.table_dict:
222|            # Slightly boost NEW_IDENT when we have a dictionary
223|            new_ident_id = VOCAB.get("NEW_IDENT")
224|            if new_ident_id is not None:
225|                bias[new_ident_id] = 0.5
226|        return bias
227|
228|    def autocomplete(
229|        self,
230|        prefix: str,
231|        max_new_tokens: int = 50,
232|        temperature: float = 0.7,
233|        top_k: int = 20,
234|        top_p: float = 0.9,
235|        use_grammar_constraints: bool = True,
236|        use_table_dict: bool = True,
237|        stop_on_eos: bool = True,
238|    ) -> AutocompleteResult:
239|        """Autocomplete a ClickHouse query prefix.
240|
241|        Args:
242|            prefix: The partial query text to autocomplete
243|            max_new_tokens: Maximum tokens to generate
244|            temperature: Sampling temperature
245|            top_k: Top-k sampling cutoff
246|            top_p: Nucleus sampling cutoff
247|            use_grammar_constraints: Filter invalid tokens using grammar rules
248|            use_table_dict: Bias identifier tokens using table dictionary
249|            stop_on_eos: Stop generation at EOS token
250|
251|        Returns:
252|            AutocompleteResult with the completion text and metadata
253|        """
254|        # Tokenize the prefix
255|        prefix_encoded = tokenize(prefix)
256|        prefix_ids = [BOS_ID] + [et.token_id for et in prefix_encoded]
257|
258|        # Track identifier and string history for copy resolution
259|        ident_history: list[str] = []
260|        string_history: list[str] = []
261|        for et in prefix_encoded:
262|            if et.is_ident:
263|                ident_history.append(et.raw_text)
264|            elif et.is_string:
265|                string_history.append(et.raw_text)
266|
267|        # Generate tokens autoregressively
268|        generated_ids: list[int] = []
269|        generated_strs: list[str] = []
270|        generated_raw: list[str] = []
271|        total_log_prob = 0.0
272|
273|        idx = torch.tensor([prefix_ids], dtype=torch.long, device=self.device)
274|
275|        with torch.no_grad():
276|            for _ in range(max_new_tokens):
277|                # Crop context if too long
278|                if idx.size(1) > self.model.config.max_seq_len:
279|                    idx = idx[:, -self.model.config.max_seq_len:]
280|
281|                logits = self.model(idx)["logits"]  # (1, T, V)
282|                next_logits = logits[0, -1, :] / max(temperature, 1e-8)  # (V,)
283|
284|                # Apply grammar-based constraints
285|                if use_grammar_constraints:
286|                    last_token_str = generated_strs[-1] if generated_strs else \
287|                        (prefix_encoded[-1].token_str if prefix_encoded else "<BOS>")
288|                    valid_ids = _get_valid_next_tokens(last_token_str, {})
289|                    if valid_ids is not None and len(valid_ids) > 0:
290|                        # Set invalid tokens to -inf
291|                        mask = torch.full_like(next_logits, float("-inf"))
292|                        mask[list(valid_ids)] = 0.0
293|                        next_logits = next_logits + mask
294|
295|                # Apply table dictionary bias
296|                if use_table_dict and self._ident_bias:
297|                    for tok_id, bias_val in self._ident_bias.items():
298|                        next_logits[tok_id] += bias_val
299|
300|                # Apply top-k filtering
301|                if top_k > 0:
302|                    v, _ = torch.topk(next_logits, min(top_k, next_logits.size(-1)))
303|                    next_logits[next_logits < v[-1]] = float("-inf")
304|
305|                # Apply top-p (nucleus) filtering
306|                if top_p > 0.0 and top_p < 1.0:
307|                    sorted_logits, sorted_indices = torch.sort(next_logits, descending=True)
308|                    cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
309|                    sorted_indices_to_remove = cumulative_probs > top_p
310|                    sorted_indices_to_remove[1:] = sorted_indices_to_remove[:-1].clone()
311|                    sorted_indices_to_remove[0] = False
312|                    indices_to_remove = sorted_indices_to_remove.scatter(
313|                        0, sorted_indices, sorted_indices_to_remove
314|                    )
315|                    next_logits = next_logits.masked_fill(indices_to_remove, float("-inf"))
316|
317|                # Sample
318|                probs = F.softmax(next_logits, dim=-1)
319|                next_token_id = torch.multinomial(probs, num_samples=1).item()
320|                next_token_str = ID_TO_TOKEN.get(next_token_id, "<UNK>")
321|
322|                # Compute log probability
323|                log_prob = torch.log(probs[next_token_id] + 1e-10).item()
324|                total_log_prob += log_prob
325|
326|                # Resolve copy tokens to raw text
327|                raw_text = self._resolve_token_text(
328|                    next_token_id, next_token_str, ident_history, string_history
329|                )
330|
331|                generated_ids.append(next_token_id)
332|                generated_strs.append(next_token_str)
333|                generated_raw.append(raw_text)
334|
335|                # Update histories
336|                if next_token_str in ("NEW_IDENT",) or next_token_str.startswith("COPY_BACK_"):
337|                    ident_history.append(raw_text)
338|                elif next_token_str in ("NEW_STRING",) or next_token_str.startswith("COPY_STR_BACK_"):
339|                    string_history.append(raw_text)
340|
341|                # Append to input for next step
342|                idx = torch.cat([idx, torch.tensor([[next_token_id]], device=self.device)], dim=1)
343|
344|                # Stop conditions
345|                if stop_on_eos and next_token_id == EOS_ID:
346|                    break
347|                if next_token_str == "SEMICOLON":
348|                    break
349|
350|        # Build completion text from generated raw texts
351|        completion = self._build_completion_text(generated_strs, generated_raw)
352|
353|        return AutocompleteResult(
354|            completion=completion,
355|            tokens=generated_ids,
356|            token_strs=generated_strs,
357|            score=total_log_prob,
358|            raw_texts=generated_raw,
359|        )
360|
361|    def _resolve_token_text(
362|        self,
363|        token_id: int,
364|        token_str: str,
365|        ident_history: list[str],
366|        string_history: list[str],
367|    ) -> str:
368|        """Resolve a generated token to its raw text representation.
369|
370|        For COPY_BACK_N tokens, look up the identifier N positions back in history.
371|        For NEW_IDENT, pick from the table dictionary or use a placeholder.
372|        For NEW_STRING, use a placeholder string.
373|        For keywords and operators, return their canonical text.
374|        """
375|        if token_str.startswith("COPY_BACK_"):
376|            n = int(token_str.split("_")[-1])
377|            if n <= len(ident_history):
378|                return ident_history[-n]
379|            return "<ident>"
380|        elif token_str.startswith("COPY_STR_BACK_"):
381|            n = int(token_str.split("_")[-1])
382|            if n <= len(string_history):
383|                return string_history[-n]
384|            return "'<str>'"
385|        elif token_str == "NEW_IDENT":
386|            # Try to pick from table dictionary
387|            if self.table_dict:
388|                # Collect all known identifiers
389|                all_idents = list(self.table_dict.keys())
390|                for cols in self.table_dict.values():
391|                    all_idents.extend(cols)
392|                if all_idents:
393|                    import random
394|                    return random.choice(all_idents)
395|            return "<ident>"
396|        elif token_str == "NEW_STRING":
397|            return "'<value>'"
398|        elif token_str.startswith("KW_"):
399|            return token_str[3:]
400|        elif token_str in NUMBER_TOKENS:
401|            return "1"
402|        elif token_str in OPERATOR_TOKENS:
403|            return OPERATOR_TOKENS[token_str]
404|        elif token_str in ("<BOS>", "<EOS>", "<PAD>", "<UNK>"):
405|            return ""
406|        else:
407|            return token_str
408|
409|    def _build_completion_text(self, token_strs: list[str], raw_texts: list[str]) -> str:
410|        """Build readable completion text from generated tokens."""
411|        parts = []
412|        for ts, rt in zip(token_strs, raw_texts):
413|            if ts in ("<BOS>", "<EOS>", "<PAD>", "<UNK>"):
414|                continue
415|            parts.append(rt)
416|
417|        # Join with spacing heuristics
418|        result = ""
419|        for i, part in enumerate(parts):
420|            if i == 0:
421|                result = part
422|                continue
423|            prev = parts[i - 1]
424|            no_space_before = part in (")", "]", "}", ",", ";", ".")
425|            no_space_after = prev in ("(", "[", "{", ".")
426|            if no_space_before or no_space_after:
427|                result += part
428|            else:
429|                result += " " + part
430|        return result
431|
432|    def autocomplete_top_k(
433|        self,
434|        prefix: str,
435|        k: int = 5,
436|        max_new_tokens: int = 50,
437|        temperature: float = 0.7,
438|        use_grammar_constraints: bool = True,
439|    ) -> list[AutocompleteResult]:
440|        """Generate k alternative completions and return them sorted by score."""
441|        results = []
442|        for _ in range(k):
443|            result = self.autocomplete(
444|                prefix,
445|                max_new_tokens=max_new_tokens,
446|                temperature=temperature,
447|                top_k=20,
448|                top_p=0.9,
449|                use_grammar_constraints=use_grammar_constraints,
450|            )
451|            results.append(result)
452|        # Sort by score (higher = better)
453|        results.sort(key=lambda r: r.score, reverse=True)
454|        return results
455|
456|
457|def load_model(checkpoint_path: str, device: str = "cpu") -> ClickHouseGPT:
458|    """Load a trained model from a checkpoint."""
459|    checkpoint = torch.load(checkpoint_path, map_location=device, weights_only=False)
460|    config = ModelConfig.from_dict(checkpoint["model_config"])
461|    model = create_model(config)
462|    model.load_state_dict(checkpoint["model_state_dict"])
463|    model.to(device)
464|    model.eval()
465|    return model
466|
467|
468|if __name__ == "__main__":
469|    # Test with untrained model (just to verify the pipeline works)
470|    config = ModelConfig()
471|    model = create_model(config)
472|
473|    # Table dictionary for constrained sampling
474|    table_dict = {
475|        "orders": ["id", "customer_id", "order_date", "status", "total_amount", "city"],
476|        "customers": ["customer_id", "name", "email", "country", "signup_date"],
477|        "events": ["id", "user_id", "event_type", "timestamp", "data"],
478|    }
479|
480|    engine = ClickHouseAutocomplete(model, device="cpu", table_dict=table_dict)
481|
482|    test_prefixes = [
483|        "SELECT ",
484|        "SELECT count() FROM ",
485|        "SELECT * FROM orders WHERE ",
486|        "SELECT city, count() FROM orders GROUP BY ",
487|        "CREATE TABLE ",
488|    ]
489|
490|    print("=== Autocomplete Test (untrained model) ===\n")
491|    for prefix in test_prefixes:
492|        result = engine.autocomplete(prefix, max_new_tokens=15, temperature=0.8, top_k=10)
493|        print(f"Prefix:  {prefix!r}")
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py" start_line="1" end_line="551" total_lines="984">
  1|#!/usr/bin/env python3
  2|"""
  3|Export the trained model to a C++ header + binary weights for embedding
  4|in the ClickHouse binary, and to WebAssembly.
  5|
  6|This script:
  7|  1. Converts model weights to a flat binary format
  8|  2. Generates a C++ header with the model config and vocabulary
  9|  3. Generates the C++ inference engine source code
 10|  4. Provides build instructions for WASM (emscripten)
 11|"""
 12|
 13|import json
 14|import struct
 15|import sys
 16|from pathlib import Path
 17|
 18|import torch
 19|import numpy as np
 20|
 21|PROJECT_ROOT = Path(__file__).resolve().parent.parent
 22|sys.path.insert(0, str(PROJECT_ROOT / "tokenizer"))
 23|sys.path.insert(0, str(PROJECT_ROOT / "model"))
 24|
 25|from ch_tokenizer import VOCAB, VOCAB_SIZE, KEYWORDS, OPERATOR_TOKENS, NUMBER_TOKENS
 26|from ch_tokenizer import IDENT_COPY_TOKENS, STRING_COPY_TOKENS, MAX_COPY_BACK
 27|from ch_tokenizer import BOS_ID, EOS_ID, PAD_ID, UNK_ID
 28|from config import ModelConfig, ClickHouseGPT, create_model
 29|
 30|
 31|def export_model_weights_binary(model: ClickHouseGPT, path: Path):
 32|    """Export model weights as a flat binary file for C++ loading.
 33|
 34|    Format:
 35|      - 4 bytes: magic ("CHAI")
 36|      - 4 bytes: version (uint32)
 37|      - 4 bytes: vocab_size (uint32)
 38|      - 4 bytes: d_model (uint32)
 39|      - 4 bytes: n_layers (uint32)
 40|      - 4 bytes: n_heads (uint32)
 41|      - 4 bytes: d_ff (uint32)
 42|      - 4 bytes: max_seq_len (uint32)
 43|      - 4 bytes: weight_tying (uint32, 0 or 1)
 44|      - 4 bytes: num_tensors (uint32)
 45|      - For each tensor:
 46|        - 4 bytes: name length (uint32)
 47|        - N bytes: name (UTF-8)
 48|        - 4 bytes: ndim (uint32)
 49|        - ndim * 4 bytes: shape (uint32 each)
 50|        - data: float32 array (prod(shape) * 4 bytes)
 51|    """
 52|    config = model.config
 53|    state_dict = model.state_dict()
 54|
 55|    with open(path, "wb") as f:
 56|        # Header
 57|        f.write(b"CHAI")
 58|        f.write(struct.pack("I", 1))  # version
 59|        f.write(struct.pack("I", config.vocab_size))
 60|        f.write(struct.pack("I", config.d_model))
 61|        f.write(struct.pack("I", config.n_layers))
 62|        f.write(struct.pack("I", config.n_heads))
 63|        f.write(struct.pack("I", config.d_ff))
 64|        f.write(struct.pack("I", config.max_seq_len))
 65|        f.write(struct.pack("I", 1 if config.weight_tying else 0))
 66|        f.write(struct.pack("I", len(state_dict)))
 67|
 68|        # Tensors
 69|        for name, tensor in state_dict.items():
 70|            data = tensor.detach().cpu().numpy().astype(np.float32)
 71|            name_bytes = name.encode("utf-8")
 72|            f.write(struct.pack("I", len(name_bytes)))
 73|            f.write(name_bytes)
 74|            f.write(struct.pack("I", data.ndim))
 75|            for dim in data.shape:
 76|                f.write(struct.pack("I", dim))
 77|            f.write(data.tobytes())
 78|
 79|    size_mb = path.stat().st_size / 1024 / 1024
 80|    print(f"  Binary weights: {path} ({size_mb:.2f} MB)")
 81|    return size_mb
 82|
 83|
 84|def export_vocab_header(path: Path):
 85|    """Export vocabulary as a C++ header file."""
 86|    with open(path, "w") as f:
 87|        f.write("// Auto-generated by export_cpp.py — ClickHouse autocomplete vocabulary\n")
 88|        f.write("#pragma once\n\n")
 89|        f.write("#include <cstdint>\n#include <string>\n#include <unordered_map>\n\n")
 90|        f.write("namespace CHAI {\n\n")
 91|        f.write(f"constexpr uint32_t VOCAB_SIZE = {VOCAB_SIZE};\n")
 92|        f.write(f"constexpr uint32_t MAX_COPY_BACK = {MAX_COPY_BACK};\n\n")
 93|
 94|        # Special token IDs
 95|        f.write(f"constexpr uint32_t PAD_ID = {PAD_ID};\n")
 96|        f.write(f"constexpr uint32_t BOS_ID = {BOS_ID};\n")
 97|        f.write(f"constexpr uint32_t EOS_ID = {EOS_ID};\n")
 98|        f.write(f"constexpr uint32_t UNK_ID = {UNK_ID};\n\n")
 99|
100|        # Keyword list
101|        f.write(f"constexpr size_t NUM_KEYWORDS = {len(KEYWORDS)};\n")
102|        f.write("const char* const KEYWORDS[] = {\n")
103|        for i, kw in enumerate(KEYWORDS):
104|            comma = "," if i < len(KEYWORDS) - 1 else ""
105|            f.write(f'    "{kw}"{comma}\n')
106|        f.write("};\n\n")
107|
108|        # Operator tokens
109|        f.write(f"constexpr size_t NUM_OPERATORS = {len(OPERATOR_TOKENS)};\n")
110|        f.write("struct OperatorToken { const char* name; const char* symbol; };\n")
111|        f.write("const OperatorToken OPERATORS[] = {\n")
112|        for i, (name, symbol) in enumerate(OPERATOR_TOKENS.items()):
113|            comma = "," if i < len(OPERATOR_TOKENS) - 1 else ""
114|            f.write(f'    {{"{name}", "{symbol}"}}{comma}\n')
115|        f.write("};\n\n")
116|
117|        # Vocabulary mapping (token string -> ID)
118|        f.write("inline std::unordered_map<std::string, uint32_t> buildVocab() {\n")
119|        f.write("    std::unordered_map<std::string, uint32_t> v;\n")
120|        for token_str, token_id in sorted(VOCAB.items(), key=lambda x: x[1]):
121|            f.write(f'    v["{token_str}"] = {token_id};\n')
122|        f.write("    return v;\n")
123|        f.write("}\n\n")
124|
125|        # Reverse vocab (ID -> token string)
126|        f.write("inline std::unordered_map<uint32_t, std::string> buildReverseVocab() {\n")
127|        f.write("    std::unordered_map<uint32_t, std::string> v;\n")
128|        for token_str, token_id in sorted(VOCAB.items(), key=lambda x: x[1]):
129|            f.write(f'    v[{token_id}] = "{token_str}";\n')
130|        f.write("    return v;\n")
131|        f.write("}\n\n")
132|
133|        f.write("} // namespace CHAI\n")
134|
135|    print(f"  Vocab header: {path}")
136|
137|
138|def export_model_header(path: Path):
139|    """Export model architecture as a C++ header."""
140|    with open(path, "w") as f:
141|        f.write("// Auto-generated by export_cpp.py — ClickHouse autocomplete model architecture\n")
142|        f.write("#pragma once\n\n")
143|        f.write("#include <cstdint>\n#include <vector>\n#include <string>\n\n")
144|        f.write("namespace CHAI {\n\n")
145|        f.write("// Model configuration (must match the Python ModelConfig)\n")
146|        f.write("struct ModelConfig {\n")
147|        f.write("    uint32_t vocab_size;\n")
148|        f.write("    uint32_t d_model;\n")
149|        f.write("    uint32_t n_layers;\n")
150|        f.write("    uint32_t n_heads;\n")
151|        f.write("    uint32_t d_ff;\n")
152|        f.write("    uint32_t max_seq_len;\n")
153|        f.write("    uint32_t n_heads_kv;\n")
154|        f.write("    bool weight_tying;\n")
155|        f.write("};\n\n")
156|        f.write("// Default config — update after training\n")
157|        f.write("constexpr ModelConfig DEFAULT_CONFIG = {\n")
158|        f.write(f"    {VOCAB_SIZE},  // vocab_size\n")
159|        f.write(f"    192,           // d_model\n")
160|        f.write(f"    6,             // n_layers\n")
161|        f.write(f"    6,             // n_heads\n")
162|        f.write(f"    768,           // d_ff\n")
163|        f.write(f"    256,           // max_seq_len\n")
164|        f.write(f"    6,             // n_heads_kv\n")
165|        f.write(f"    true,          // weight_tying\n")
166|        f.write("};\n\n")
167|        f.write("} // namespace CHAI\n")
168|
169|    print(f"  Model header: {path}")
170|
171|
172|def export_cpp_inference_engine(path: Path):
173|    """Export the C++ inference engine source code."""
174|    code = r'''// Auto-generated by export_cpp.py — ClickHouse autocomplete C++ inference engine
175|// This file can be embedded in the ClickHouse binary or compiled to WebAssembly.
176|//
177|// Dependencies: None (pure C++17, no external libs)
178|// For matrix operations, a minimal implementation is included.
179|
180|#pragma once
181|
182|#include <cmath>
183|#include <cstdint>
184|#include <cstring>
185|#include <fstream>
186|#include <iostream>
187|#include <memory>
188|#include <string>
189|#include <unordered_map>
190|#include <vector>
191|#include <algorithm>
192|#include <random>
193|
194|#include "ch_vocab.h"
195|#include "ch_model.h"
196|
197|namespace CHAI {
198|
199|// ─── Minimal matrix/tensor operations (CPU, no BLAS) ────────────────────────
200|
201|using Matrix = std::vector<float>;  // row-major
202|using Tensor3D = std::vector<float>; // flattened
203|
204|inline void matmul(const float* A, const float* B, float* C,
205|                   int M, int K, int N) {
206|    // C = A @ B, A: (M,K), B: (K,N), C: (M,N)
207|    for (int i = 0; i < M; i++) {
208|        for (int j = 0; j < N; j++) {
209|            float sum = 0.0f;
210|            for (int k = 0; k < K; k++) {
211|                sum += A[i * K + k] * B[k * N + j];
212|            }
213|            C[i * N + j] = sum;
214|        }
215|    }
216|}
217|
218|// C = A @ B.T  where A: (M,K), B: (N,K) stored row-major, C: (M,N)
219|// This is used for PyTorch nn.Linear weights which are stored as (out_features, in_features)
220|inline void matmul_linear(const float* A, const float* W, float* C,
221|                          int M, int K, int N) {
222|    // A: (M, K), W: (N, K) [PyTorch Linear weight format], C: (M, N)
223|    // C[i,j] = sum_k A[i,k] * W[j,k]
224|    for (int i = 0; i < M; i++) {
225|        for (int j = 0; j < N; j++) {
226|            float sum = 0.0f;
227|            for (int k = 0; k < K; k++) {
228|                sum += A[i * K + k] * W[j * K + k];
229|            }
230|            C[i * N + j] = sum;
231|        }
232|    }
233|}
234|
235|inline void layernorm(const float* x, float* out, const float* gamma,
236|                      int d_model) {
237|    float mean = 0.0f;
238|    for (int i = 0; i < d_model; i++) mean += x[i];
239|    mean /= d_model;
240|    float var = 0.0f;
241|    for (int i = 0; i < d_model; i++) {
242|        float diff = x[i] - mean;
243|        var += diff * diff;
244|    }
245|    var /= d_model;
246|    float inv_std = 1.0f / std::sqrt(var + 1e-5f);
247|    for (int i = 0; i < d_model; i++) {
248|        out[i] = (x[i] - mean) * inv_std * gamma[i];
249|    }
250|}
251|
252|inline float gelu(float x) {
253|    return 0.5f * x * (1.0f + std::tanh(0.7978845608f * (x + 0.044715f * x * x * x)));
254|}
255|
256|inline void softmax(float* x, int n) {
257|    float max_val = x[0];
258|    for (int i = 1; i < n; i++) max_val = std::max(max_val, x[i]);
259|    float sum = 0.0f;
260|    for (int i = 0; i < n; i++) {
261|        x[i] = std::exp(x[i] - max_val);
262|        sum += x[i];
263|    }
264|    for (int i = 0; i < n; i++) x[i] /= sum;
265|}
266|
267|// ─── Model weights container ────────────────────────────────────────────────
268|
269|struct LayerWeights {
270|    // Attention
271|    std::vector<float> qkv_weight;  // (3*d_model, d_model)
272|    std::vector<float> proj_weight; // (d_model, d_model)
273|    // FFN
274|    std::vector<float> fc1_weight;  // (d_ff, d_model)
275|    std::vector<float> fc2_weight;  // (d_model, d_ff)
276|    // LayerNorm
277|    std::vector<float> ln1_gamma;   // (d_model,)
278|    std::vector<float> ln2_gamma;   // (d_model,)
279|};
280|
281|struct ModelWeights {
282|    ModelConfig config;
283|    std::vector<float> tok_emb;     // (vocab_size, d_model)
284|    std::vector<float> pos_emb;     // (max_seq_len, d_model)
285|    std::vector<LayerWeights> layers;
286|    std::vector<float> ln_f_gamma;  // (d_model,)
287|};
288|
289|// ─── Binary weight loader ──────────────────────────────────────────────────
290|
291|inline uint32_t read_u32(std::ifstream& f) {
292|    uint32_t v;
293|    f.read(reinterpret_cast<char*>(&v), 4);
294|    return v;
295|}
296|
297|inline std::vector<float> read_tensor_raw(std::ifstream& f, std::string& name) {
298|    uint32_t name_len = read_u32(f);
299|    name.resize(name_len);
300|    f.read(&name[0], name_len);
301|    uint32_t ndim = read_u32(f);
302|    std::vector<uint32_t> shape(ndim);
303|    for (uint32_t i = 0; i < ndim; i++) shape[i] = read_u32(f);
304|    size_t total = 1;
305|    for (auto d : shape) total *= d;
306|    std::vector<float> data(total);
307|    f.read(reinterpret_cast<char*>(data.data()), total * sizeof(float));
308|    return data;
309|}
310|
311|inline std::unique_ptr<ModelWeights> load_model(const std::string& path) {
312|    std::ifstream f(path, std::ios::binary);
313|    if (!f) { std::cerr << "Cannot open " << path << std::endl; return nullptr; }
314|
315|    char magic[4];
316|    f.read(magic, 4);
317|    if (std::memcmp(magic, "CHAI", 4) != 0) {
318|        std::cerr << "Invalid model file format" << std::endl; return nullptr;
319|    }
320|
321|    auto w = std::make_unique<ModelWeights>();
322|    uint32_t version = read_u32(f);
323|    w->config.vocab_size = read_u32(f);
324|    w->config.d_model = read_u32(f);
325|    w->config.n_layers = read_u32(f);
326|    w->config.n_heads = read_u32(f);
327|    w->config.d_ff = read_u32(f);
328|    w->config.max_seq_len = read_u32(f);
329|    w->config.weight_tying = read_u32(f) != 0;
330|    uint32_t num_tensors = read_u32(f);
331|
332|    // Read all tensors into a map by name
333|    std::unordered_map<std::string, std::vector<float>> tensors;
334|    for (uint32_t i = 0; i < num_tensors; i++) {
335|        std::string name;
336|        auto data = read_tensor_raw(f, name);
337|        tensors[name] = std::move(data);
338|    }
339|
340|    // Assign by name
341|    w->tok_emb = tensors.at("tok_emb.weight");
342|    w->pos_emb = tensors.at("pos_emb.weight");
343|
344|    w->layers.resize(w->config.n_layers);
345|    for (uint32_t l = 0; l < w->config.n_layers; l++) {
346|        auto& lw = w->layers[l];
347|        std::string prefix = "blocks." + std::to_string(l) + ".";
348|        lw.qkv_weight = tensors.at(prefix + "attn.qkv.weight");
349|        lw.proj_weight = tensors.at(prefix + "attn.proj.weight");
350|        lw.fc1_weight = tensors.at(prefix + "ffn.fc1.weight");
351|        lw.fc2_weight = tensors.at(prefix + "ffn.fc2.weight");
352|        lw.ln1_gamma = tensors.at(prefix + "ln1.weight");
353|        lw.ln2_gamma = tensors.at(prefix + "ln2.weight");
354|    }
355|
356|    w->ln_f_gamma = tensors.at("ln_f.weight");
357|    return w;
358|}
359|
360|// ─── Forward pass ───────────────────────────────────────────────────────────
361|
362|inline std::vector<float> forward(
363|    const ModelWeights& w,
364|    const std::vector<uint32_t>& token_ids
365|) {
366|    const auto& cfg = w.config;
367|    int T = (int)token_ids.size();
368|    int D = (int)cfg.d_model;
369|    int H = (int)cfg.n_heads;
370|    int DH = D / H;
371|
372|    // Embedding: token + position
373|    std::vector<float> x(T * D);
374|    for (int t = 0; t < T; t++) {
375|        uint32_t tok = token_ids[t];
376|        for (int d = 0; d < D; d++) {
377|            x[t * D + d] = w.tok_emb[tok * D + d] + w.pos_emb[t * D + d];
378|        }
379|    }
380|
381|    // Transformer layers
382|    std::vector<float> ln_out(T * D);
383|    std::vector<float> qkv(T * 3 * D);
384|    std::vector<float> attn_out(T * D);
385|    std::vector<float> proj_out(T * D);
386|    std::vector<float> ffn_hidden(T * cfg.d_ff);
387|    std::vector<float> ffn_out(T * D);
388|
389|    for (uint32_t layer = 0; layer < cfg.n_layers; layer++) {
390|        const auto& lw = w.layers[layer];
391|
392|        // Pre-LN1
393|        for (int t = 0; t < T; t++)
394|            layernorm(&x[t * D], &ln_out[t * D], lw.ln1_gamma.data(), D);
395|
396|        // QKV projection: (T, D) @ W.T where W: (3*D, D) -> (T, 3*D)
397|        matmul_linear(ln_out.data(), lw.qkv_weight.data(), qkv.data(), T, D, 3 * D);
398|
399|        // Attention (per head)
400|        for (int h = 0; h < H; h++) {
401|            int offset = h * DH;
402|            // For each query position
403|            for (int qi = 0; qi < T; qi++) {
404|                float* q = &qkv[qi * 3 * D + offset];           // Q
405|                std::vector<float> scores(qi + 1);
406|                float scale = 1.0f / std::sqrt((float)DH);
407|
408|                for (int ki = 0; ki <= qi; ki++) {
409|                    float* k = &qkv[ki * 3 * D + D + offset];   // K
410|                    float s = 0.0f;
411|                    for (int d = 0; d < DH; d++) s += q[d] * k[d];
412|                    scores[ki] = s * scale;
413|                }
414|                softmax(scores.data(), qi + 1);
415|
416|                // Weighted sum of V
417|                float* out = &attn_out[qi * D + offset];
418|                std::fill(out, out + DH, 0.0f);
419|                for (int vi = 0; vi <= qi; vi++) {
420|                    float* v = &qkv[vi * 3 * D + 2 * D + offset]; // V
421|                    float weight = scores[vi];
422|                    for (int d = 0; d < DH; d++) out[d] += weight * v[d];
423|                }
424|            }
425|        }
426|
427|        // Projection + residual: (T, D) @ W.T where W: (D, D) -> (T, D)
428|        matmul_linear(attn_out.data(), lw.proj_weight.data(), proj_out.data(), T, D, D);
429|        for (int i = 0; i < T * D; i++) x[i] += proj_out[i];
430|
431|        // Pre-LN2
432|        for (int t = 0; t < T; t++)
433|            layernorm(&x[t * D], &ln_out[t * D], lw.ln2_gamma.data(), D);
434|
435|        // FFN: fc1 -> GELU -> fc2 (all using PyTorch Linear weight format)
436|        matmul_linear(ln_out.data(), lw.fc1_weight.data(), ffn_hidden.data(), T, D, cfg.d_ff);
437|        for (int i = 0; i < T * (int)cfg.d_ff; i++) ffn_hidden[i] = gelu(ffn_hidden[i]);
438|        matmul_linear(ffn_hidden.data(), lw.fc2_weight.data(), ffn_out.data(), T, cfg.d_ff, D);
439|
440|        // Residual
441|        for (int i = 0; i < T * D; i++) x[i] += ffn_out[i];
442|    }
443|
444|    // Final LayerNorm
445|    for (int t = 0; t < T; t++)
446|        layernorm(&x[t * D], &x[t * D], w.ln_f_gamma.data(), D);
447|
448|    // Output projection (weight-tied with token embedding)
449|    // logits = x @ tok_emb.T  -> (T, vocab_size)
450|    // tok_emb is (vocab_size, D), so we use matmul_linear: (T,D) @ (V,D).T -> (T,V)
451|    std::vector<float> logits(T * cfg.vocab_size);
452|    matmul_linear(x.data(), w.tok_emb.data(), logits.data(), T, D, cfg.vocab_size);
453|
454|    return logits;
455|}
456|
457|// ─── Tokenizer (simplified C++ port) ────────────────────────────────────────
458|
459|struct Token {
460|    std::string text;
461|    std::string type;  // token type string
462|    bool is_ident;
463|    bool is_string;
464|};
465|
466|inline bool is_keyword(const std::string& word) {
467|    std::string upper = word;
468|    std::transform(upper.begin(), upper.end(), upper.begin(), ::toupper);
469|    for (const auto& kw : KEYWORDS) {
470|        if (upper == kw) return true;
471|    }
472|    return false;
473|}
474|
475|inline std::vector<Token> tokenize_cpp(const std::string& query) {
476|    // Simplified tokenizer — matches the Python version's logic
477|    std::vector<Token> tokens;
478|    size_t pos = 0;
479|    size_t n = query.size();
480|
481|    while (pos < n) {
482|        // Skip whitespace
483|        while (pos < n && isspace(query[pos])) pos++;
484|        if (pos >= n) break;
485|
486|        // Comments
487|        if (pos + 1 < n && query[pos] == '-' && query[pos+1] == '-') {
488|            while (pos < n && query[pos] != '\n') pos++;
489|            continue;
490|        }
491|        if (pos + 1 < n && query[pos] == '/' && query[pos+1] == '*') {
492|            pos += 2;
493|            while (pos + 1 < n && !(query[pos] == '*' && query[pos+1] == '/')) pos++;
494|            pos += 2;
495|            continue;
496|        }
497|
498|        // String literal
499|        if (query[pos] == '\'') {
500|            size_t start = pos;
501|            pos++;
502|            while (pos < n && query[pos] != '\'') {
503|                if (query[pos] == '\\' && pos + 1 < n) pos++;
504|                pos++;
505|            }
506|            if (pos < n) pos++; // closing quote
507|            tokens.push_back({query.substr(start, pos - start), "StringLiteral", false, true});
508|            continue;
509|        }
510|
511|        // Quoted identifier
512|        if (query[pos] == '`' || query[pos] == '"') {
513|            char quote = query[pos];
514|            size_t start = pos;
515|            pos++;
516|            while (pos < n && query[pos] != quote) {
517|                if (query[pos] == '\\' && pos + 1 < n) pos++;
518|                pos++;
519|            }
520|            if (pos < n) pos++;
521|            tokens.push_back({query.substr(start, pos - start), "QuotedIdentifier", false, false});
522|            tokens.back().is_ident = true;
523|            continue;
524|        }
525|
526|        // Numbers
527|        if (isdigit(query[pos]) || (query[pos] == '.' && pos + 1 < n && isdigit(query[pos+1]))) {
528|            size_t start = pos;
529|            while (pos < n && (isalnum(query[pos]) || query[pos] == '.')) pos++;
530|            tokens.push_back({query.substr(start, pos - start), "Number", false, false});
531|            continue;
532|        }
533|
534|        // BareWord (identifier or keyword)
535|        if (isalpha(query[pos]) || query[pos] == '_') {
536|            size_t start = pos;
537|            while (pos < n && (isalnum(query[pos]) || query[pos] == '_')) pos++;
538|            std::string word = query.substr(start, pos - start);
539|            if (is_keyword(word)) {
540|                tokens.push_back({word, "Keyword", false, false});
541|            } else {
542|                tokens.push_back({word, "BareWord", true, false});
543|            }
544|            continue;
545|        }
546|
547|        // Multi-char operators
548|        if (pos + 1 < n) {
549|            std::string two = query.substr(pos, 2);
550|            if (two == "::" || two == "->" || two == "<=" || two == ">=" ||
551|                two == "!=" || two == "||" || two == "@@" || two == "<>") {
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/trajectory_format_spec.md" start_line="1" end_line="100" total_lines="100">
  1|# ClickHouse Autocomplete Trajectory Format Specification
  2|
  3|## Overview
  4|
  5|Each trajectory is a JSON object (one per line in JSONL format) representing a
  6|rich ClickHouse query autocomplete example. Trajectories are used to train a
  7|small GPT-based language model (~10 MB) that autocompletes ClickHouse queries.
  8|
  9|## Trajectory JSON Schema
 10|
 11|```json
 12|{
 13|  "id": "traj_001",
 14|  "category": "aggregation",
 15|  "subcategory": "group_by_having",
 16|  "description": "Aggregation with GROUP BY, HAVING, and ORDER BY",
 17|  "query": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY avg_price DESC LIMIT 20",
 18|  "tables": ["orders"],
 19|  "columns": {
 20|    "orders": ["id", "user_id", "price", "status", "city", "region", "created_at"]
 21|  },
 22|  "dictionaries": [],
 23|  "settings": {},
 24|  "completion_points": [
 25|    {
 26|      "prefix": "SELECT ",
 27|      "completion": "city, count() AS cnt, avg(price) AS avg_price ",
 28|      "next_token": "FROM"
 29|    },
 30|    {
 31|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM ",
 32|      "completion": "orders ",
 33|      "next_token": "WHERE"
 34|    },
 35|    {
 36|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE ",
 37|      "completion": "status = 'completed' ",
 38|      "next_token": "GROUP"
 39|    },
 40|    {
 41|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY ",
 42|      "completion": "city ",
 43|      "next_token": "HAVING"
 44|    },
 45|    {
 46|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING ",
 47|      "completion": "cnt > 10 ",
 48|      "next_token": "ORDER"
 49|    },
 50|    {
 51|      "prefix": "SELECT city, count() AS cnt, avg(price) AS avg_price FROM orders WHERE status = 'completed' GROUP BY city HAVING cnt > 10 ORDER BY ",
 52|      "completion": "avg_price DESC ",
 53|      "next_token": "LIMIT"
 54|    }
 55|  ]
 56|}
 57|```
 58|
 59|## Fields
 60|
 61|| Field | Type | Description |
 62||---|---|---|
 63|| `id` | string | Unique trajectory ID, format: `traj_NNN` |
 64|| `category` | string | Broad category (see Categories below) |
 65|| `subcategory` | string | More specific subcategory |
 66|| `description` | string | Human-readable description of what the query demonstrates |
 67|| `query` | string | The complete, syntactically valid ClickHouse query |
 68|| `tables` | string[] | Table names referenced in the query (for dictionary-based sampling) |
 69|| `columns` | object | Map of table name -> column names available on that table |
 70|| `dictionaries` | string[] | ClickHouse dictionary names available (if any) |
 71|| `settings` | object | ClickHouse SETTINGS clauses if relevant (e.g. `{"max_threads": "8"}`) |
 72|| `completion_points` | array | Array of prefix/completion pairs for training autocomplete |
 73|
 74|## Completion Points
 75|
 76|Each completion point represents a natural autocomplete boundary — a place
 77|where a user would pause and the model should predict the next segment.
 78|
 79|- `prefix`: The partial query up to this point (ends with a space or token boundary)
 80|- `completion`: The text that should be predicted next (up to the next natural boundary)
 81|- `next_token`: The first keyword/token of the completion (for constrained sampling)
 82|
 83|## Categories
 84|
 85|1. **basic_select** — Simple SELECT, WHERE, ORDER BY, LIMIT
 86|2. **aggregation** — GROUP BY, HAVING, aggregate functions, window functions
 87|3. **joins** — INNER/LEFT/RIGHT/FULL JOIN, ASOF JOIN, ARRAY JOIN, subqueries
 88|4. **ddl_dml** — CREATE TABLE, ALTER, DROP, INSERT, TRUNCATE, system queries
 89|5. **advanced** — Arrays, maps, JSON, materialized views, projections, dictionaries, TTL, CTEs, sampling
 90|
 91|## Rules
 92|
 93|1. All queries MUST be syntactically valid ClickHouse SQL
 94|2. Use realistic table and column names (not foo/bar)
 95|3. Include diverse ClickHouse-specific features (array functions, dictGet, etc.)
 96|4. Each trajectory should have 4-10 completion points
 97|5. Completion points should be at natural syntactic boundaries
 98|6. Table/column definitions must be consistent with the query
 99|7. Vary query complexity — some simple, some complex
100|8. Use ClickHouse-specific functions: count(), avg(), sum(), dictGet(), arrayJoin(), etc.
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/README.md" start_line="1" end_line="190" total_lines="190">
  1|# ClickHouse SLM Autocomplete
  2|
  3|A small language model (~10 MB) for auto-completion of ClickHouse queries, using a custom tokenizer that mirrors the ClickHouse Lexer.
  4|
  5|## Task Specification
  6|
  7|> **Source**: [ClickHouse/ClickHouse#87836 — Intern Tasks 2025/2026](https://github.com/ClickHouse/ClickHouse/issues/87836)
  8|> opened by [@alexey-milovidov](https://github.com/alexey-milovidov) on Sep 29, 2025
  9|
 10|### SLM-based custom autocomplete
 11|
 12|Implement and train a small language model (~10 MB), using a custom tokenizer, for auto-completion
 13|of ClickHouse queries.
 14|
 15|The tokenizer should correspond to the ClickHouse Lexer, and support its syntax elements as well as
 16|keywords. However, for identifiers and string literals, it should represent items like lookbehind
 17|copies, instead of coding the values literally. We aim to use a GPT-based model.
 18|
 19|The goal is to implement a small and fast model, running on CPU with minimum of resources, that
 20|could be embedded in the ClickHouse binary by default, as well as exported to WebAssembly, and it
 21|could be largerly customizable, so we can do sampling based on the syntaxic correctness of the query
 22|and based on the available dictionaries of tables.
 23|
 24|---
 25|
 26|## Overview
 27|
 28|This project implements a GPT-based small language model (SLM) designed to:
 29|- Run on CPU with minimal resources (~10 MB model size)
 30|- Be embeddable in the ClickHouse binary by default
 31|- Be exportable to WebAssembly
 32|- Support sampling based on syntactic correctness and available table dictionaries
 33|
 34|## Architecture
 35|
 36|### Tokenizer (`tokenizer/ch_tokenizer.py`)
 37|
 38|The tokenizer mirrors the ClickHouse Lexer (`src/Parsers/Lexer.h`) token types:
 39|- **Keywords**: All 200+ ClickHouse keywords (from `utils/antlr/ClickHouseLexer.g4`)
 40|- **Operators**: All punctuation and operator tokens
 41|- **Numbers**: `NUM_INT`, `NUM_FLOAT`, `NUM_HEX`
 42|- **Identifiers**: Encoded as **lookbehind copies** instead of literal values
 43|- **String literals**: Encoded as **lookbehind copies** instead of literal values
 44|
 45|#### Lookbehind Copy Encoding
 46|
 47|Instead of encoding identifier/string values literally (which would require an unbounded vocabulary), each identifier or string is represented as:
 48|- `NEW_IDENT` / `NEW_STRING` — first occurrence in the sequence
 49|- `COPY_BACK_N` / `COPY_STR_BACK_N` — a repeat of a value seen N tokens ago
 50|
 51|This keeps the vocabulary at **341 tokens** while allowing the model to reproduce arbitrary identifiers and strings via a pointer/copy mechanism at inference time.
 52|
 53|### Model (`model/config.py`)
 54|
 55|- **Architecture**: GPT (decoder-only transformer)
 56|- **Parameters**: 2,771,328 (~10.57 MB float32, ~5.29 MB float16, ~2.64 MB int8)
 57|- **Config**: d_model=192, n_layers=6, n_heads=6, d_ff=768, max_seq_len=256
 58|- **Features**: Pre-LayerNorm, weight tying (input embedding == output projection), GELU activation, learned positional embeddings
 59|
 60|### Training (`model/training.py`)
 61|
 62|- **Dataset**: 100 trajectories → 937 training samples (full queries + completion points + augmented truncations)
 63|- **Training**: AdamW with cosine LR schedule, warmup, gradient clipping
 64|- **Loss**: Cross-entropy on completion tokens (prefix tokens masked)
 65|
 66|### Inference (`inference/autocomplete.py`)
 67|
 68|- **Constrained sampling**: Grammar-based filtering of invalid next tokens
 69|- **Table dictionary awareness**: Biases identifier tokens toward known table/column names
 70|- **Copy resolution**: Resolves `COPY_BACK_N` tokens to actual text from the prefix
 71|- **Sampling**: Temperature, top-k, top-p (nucleus) sampling
 72|- **Top-k alternatives**: Generate multiple completions ranked by score
 73|
 74|### C++ Export (`export/`)
 75|
 76|- **Binary weights**: `ch_autocomplete.bin` (10.57 MB, flat binary format)
 77|- **C++ headers**: `ch_vocab.h`, `ch_model.h`, `ch_inference.h` (header-only library)
 78|- **C++ test**: `test_main.cpp` (standalone test program)
 79|- **WASM build**: `build_wasm.sh` (emscripten build script)
 80|- **C API**: `chai_load_model()`, `chai_autocomplete()`, `chai_free_model()` for WASM export
 81|
 82|## Project Structure
 83|
 84|```
 85|clickhouse-github/
 86|├── tokenizer/
 87|│   ├── ch_tokenizer.py      # ClickHouse lexer + lookbehind copy tokenizer
 88|│   └── vocab.json           # Saved vocabulary (341 tokens)
 89|├── model/
 90|│   ├── config.py            # GPT model architecture + config
 91|│   └── training.py          # Dataset, Trainer, data loaders
 92|├── trajectories/
 93|│   ├── trajectory_format_spec.md  # Trajectory JSON schema
 94|│   └── data/
 95|│       ├── trajectories_01_basic_select.jsonl  (20 trajectories)
 96|│       ├── trajectories_02_aggregation.jsonl   (20 trajectories)
 97|│       ├── trajectories_03_joins.jsonl         (20 trajectories)
 98|│       ├── trajectories_04_ddl_dml.jsonl       (20 trajectories)
 99|│       └── trajectories_05_advanced.jsonl      (20 trajectories)
100|├── inference/
101|│   └── autocomplete.py      # Constrained autocomplete engine
102|├── export/
103|│   ├── export_cpp.py        # Python → C++ export script
104|│   ├── ch_autocomplete.bin  # Binary model weights (10.57 MB)
105|│   ├── ch_vocab.h           # C++ vocabulary header
106|│   ├── ch_model.h           # C++ model config header
107|│   ├── ch_inference.h       # C++ inference engine (header-only)
108|│   ├── test_main.cpp        # C++ test program
109|│   ├── CMakeLists.txt       # CMake build file
110|│   └── build_wasm.sh        # WebAssembly build script
111|├── train.py                 # Main training script
112|├── test_autocomplete.py     # Python inference test
113|└── checkpoints/
114|    ├── best.pt              # Best model checkpoint
115|    └── final.pt             # Final model checkpoint
116|```
117|
118|## Quick Start
119|
120|### Train the model
121|
122|```bash
123|python3 train.py --epochs 30 --batch_size 16 --lr 3e-4
124|```
125|
126|### Test autocomplete (Python)
127|
128|```bash
129|python3 test_autocomplete.py
130|```
131|
132|### Export to C++ and test
133|
134|```bash
135|python3 export/export_cpp.py
136|cd export
137|g++ -std=c++17 -O2 -I. -o build/chai_test test_main.cpp
138|./build/chai_test ch_autocomplete.bin
139|```
140|
141|### Build for WebAssembly
142|
143|```bash
144|cd export
145|./build_wasm.sh
146|```
147|
148|## Trajectory Generation
149|
150|100 rich trajectories were generated by 5 parallel GLM 5.2 subagents (20 each):
151|
152|| Category | File | Trajectories | Features |
153||---|---|---|---|
154|| basic_select | trajectories_01 | traj_001–020 | SELECT, WHERE, ORDER BY, LIMIT, DISTINCT, PREWHERE, FINAL, SAMPLE, UNION ALL |
155|| aggregation | trajectories_02 | traj_021–040 | GROUP BY, HAVING, WITH TOTALS/CUBE/ROLLUP, window functions, aggregate combinators |
156|| joins | trajectories_03 | traj_041–060 | INNER/LEFT/RIGHT/FULL JOIN, ASOF JOIN, ARRAY JOIN, CTEs, subqueries |
157|| ddl_dml | trajectories_04 | traj_061–080 | CREATE TABLE (all engines), ALTER, INSERT, SYSTEM, SHOW, EXPLAIN |
158|| advanced | trajectories_05 | traj_081–100 | Arrays, maps, JSON, dictionaries, materialized views, projections, TTL, sampling |
159|
160|## Customization
161|
162|### Syntactic Correctness Sampling
163|
164|The inference engine filters invalid next tokens using grammar rules:
165|```python
166|engine.autocomplete(prefix, use_grammar_constraints=True)
167|```
168|
169|### Table Dictionary Sampling
170|
171|Provide known tables and columns to bias identifier prediction:
172|```python
173|table_dict = {
174|    "orders": ["id", "customer_id", "status", "city"],
175|    "customers": ["customer_id", "name", "email"],
176|}
177|engine = ClickHouseAutocomplete(model, table_dict=table_dict)
178|```
179|
180|### Sampling Parameters
181|
182|```python
183|result = engine.autocomplete(
184|    prefix,
185|    max_new_tokens=50,
186|    temperature=0.7,
187|    top_k=20,
188|    top_p=0.9,
189|)
190|```
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/test_autocomplete.py" start_line="1" end_line="70" total_lines="70">
  1|#!/usr/bin/env python3
  2|"""Test autocomplete with the trained model."""
  3|
  4|import sys
  5|from pathlib import Path
  6|
  7|PROJECT_ROOT = Path(__file__).resolve().parent
  8|sys.path.insert(0, str(PROJECT_ROOT / "tokenizer"))
  9|sys.path.insert(0, str(PROJECT_ROOT / "model"))
 10|sys.path.insert(0, str(PROJECT_ROOT / "inference"))
 11|
 12|from autocomplete import ClickHouseAutocomplete, load_model
 13|
 14|# Load trained model
 15|checkpoint = str(PROJECT_ROOT / "checkpoints" / "best.pt")
 16|print(f"Loading model from: {checkpoint}")
 17|model = load_model(checkpoint, device="cpu")
 18|print(f"Model loaded: {model.num_parameters():,} parameters\n")
 19|
 20|# Table dictionary for constrained sampling
 21|table_dict = {
 22|    "orders": ["id", "customer_id", "order_date", "status", "total_amount", "city", "region"],
 23|    "customers": ["customer_id", "name", "email", "country", "signup_date"],
 24|    "events": ["id", "user_id", "event_type", "timestamp", "data", "session_id"],
 25|    "pageviews": ["id", "user_id", "url", "country", "city", "duration", "date"],
 26|}
 27|
 28|engine = ClickHouseAutocomplete(model, device="cpu", table_dict=table_dict)
 29|
 30|test_prefixes = [
 31|    "SELECT ",
 32|    "SELECT count() FROM ",
 33|    "SELECT * FROM orders WHERE ",
 34|    "SELECT city, count() FROM orders GROUP BY ",
 35|    "SELECT city, count() FROM orders GROUP BY city ORDER BY ",
 36|    "SELECT * FROM orders LEFT JOIN customers ",
 37|    "CREATE TABLE ",
 38|    "SELECT * FROM events WHERE event_type = ",
 39|    "SELECT count(), avg(price) FROM orders WHERE ",
 40|    "SELECT user_id, count() FROM events GROUP BY user_id HAVING ",
 41|]
 42|
 43|print("=== Autocomplete Test (trained model) ===\n")
 44|for prefix in test_prefixes:
 45|    result = engine.autocomplete(
 46|        prefix,
 47|        max_new_tokens=20,
 48|        temperature=0.5,
 49|        top_k=15,
 50|        top_p=0.9,
 51|        use_grammar_constraints=True,
 52|        use_table_dict=True,
 53|    )
 54|    print(f"Prefix:     {prefix!r}")
 55|    print(f"Completion: {result.completion!r}")
 56|    print(f"Tokens:     {result.token_strs[:15]}")
 57|    print(f"Score:      {result.score:.4f}")
 58|    print()
 59|
 60|# Test top-k alternatives
 61|print("\n=== Top-3 Alternatives for 'SELECT * FROM orders WHERE ' ===\n")
 62|results = engine.autocomplete_top_k(
 63|    "SELECT * FROM orders WHERE ",
 64|    k=3,
 65|    max_new_tokens=20,
 66|    temperature=0.7,
 67|    use_grammar_constraints=True,
 68|)
 69|for i, result in enumerate(results):
 70|    print(f"  #{i+1}: {result.completion!r}  (score={result.score:.4f})")
</file-view>

/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_01_basic_select.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_02_aggregation.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_03_joins.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_04_ddl_dml.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py" start_line="551" end_line="984" total_lines="984">
551|                two == "!=" || two == "||" || two == "@@" || two == "<>") {
552|                tokens.push_back({two, "Operator", false, false});
553|                pos += 2;
554|                continue;
555|            }
556|            if (pos + 2 < n && query.substr(pos, 3) == "<=>") {
557|                tokens.push_back({"<=>", "Operator", false, false});
558|                pos += 3;
559|                continue;
560|            }
561|        }
562|
563|        // Single-char operators
564|        char c = query[pos];
565|        if (strchr("()[]{},.;*+-/%?:^=@<>", c)) {
566|            tokens.push_back({std::string(1, c), "Operator", false, false});
567|            pos++;
568|            continue;
569|        }
570|
571|        // Unknown char — skip
572|        pos++;
573|    }
574|
575|    return tokens;
576|}
577|
578|// ─── Autocomplete API ───────────────────────────────────────────────────────
579|
580|struct AutocompleteResult {
581|    std::string completion;
582|    std::vector<uint32_t> token_ids;
583|    float score;
584|};
585|
586|inline AutocompleteResult autocomplete(
587|    const ModelWeights& w,
588|    const std::string& prefix,
589|    int max_new_tokens = 20,
590|    float temperature = 0.7f,
591|    int top_k = 15,
592|    uint32_t eos_id = EOS_ID
593|) {
594|    AutocompleteResult result;
595|
596|    // Tokenize prefix
597|    auto tokens = tokenize_cpp(prefix);
598|    std::vector<uint32_t> input_ids = {BOS_ID};
599|
600|    // Track identifier history for copy resolution
601|    std::vector<std::string> ident_history;
602|    std::vector<std::string> string_history;
603|    auto vocab = buildVocab();
604|
605|    for (const auto& tok : tokens) {
606|        if (tok.is_ident) {
607|            ident_history.push_back(tok.text);
608|            input_ids.push_back(vocab["NEW_IDENT"]);
609|        } else if (tok.is_string) {
610|            string_history.push_back(tok.text);
611|            input_ids.push_back(vocab["NEW_STRING"]);
612|        } else if (tok.type == "Keyword") {
613|            std::string upper = tok.text;
614|            std::transform(upper.begin(), upper.end(), upper.begin(), ::toupper);
615|            std::string kw_tok = "KW_" + upper;
616|            if (vocab.count(kw_tok)) input_ids.push_back(vocab[kw_tok]);
617|        } else if (tok.type == "Number") {
618|            input_ids.push_back(vocab["NUM_INT"]);
619|        } else if (tok.type == "Operator") {
620|            // Map operator to vocab token
621|            // (simplified — full mapping would match the Python version)
622|            if (tok.text == "(") input_ids.push_back(vocab["LPAREN"]);
623|            else if (tok.text == ")") input_ids.push_back(vocab["RPAREN"]);
624|            else if (tok.text == "[") input_ids.push_back(vocab["LBRACKET"]);
625|            else if (tok.text == "]") input_ids.push_back(vocab["RBRACKET"]);
626|            else if (tok.text == "{") input_ids.push_back(vocab["LBRACE"]);
627|            else if (tok.text == "}") input_ids.push_back(vocab["RBRACE"]);
628|            else if (tok.text == ",") input_ids.push_back(vocab["COMMA"]);
629|            else if (tok.text == ";") input_ids.push_back(vocab["SEMICOLON"]);
630|            else if (tok.text == ".") input_ids.push_back(vocab["DOT"]);
631|            else if (tok.text == "*") input_ids.push_back(vocab["ASTERISK"]);
632|            else if (tok.text == "+") input_ids.push_back(vocab["PLUS"]);
633|            else if (tok.text == "-") input_ids.push_back(vocab["MINUS"]);
634|            else if (tok.text == "/") input_ids.push_back(vocab["SLASH"]);
635|            else if (tok.text == "%") input_ids.push_back(vocab["PERCENT"]);
636|            else if (tok.text == "=") input_ids.push_back(vocab["EQUALS"]);
637|            else if (tok.text == "!=" || tok.text == "<>") input_ids.push_back(vocab["NOT_EQUALS"]);
638|            else if (tok.text == "<") input_ids.push_back(vocab["LESS"]);
639|            else if (tok.text == ">") input_ids.push_back(vocab["GREATER"]);
640|            else if (tok.text == "<=") input_ids.push_back(vocab["LESS_OR_EQUALS"]);
641|            else if (tok.text == ">=") input_ids.push_back(vocab["GREATER_OR_EQUALS"]);
642|            else if (tok.text == "->") input_ids.push_back(vocab["ARROW"]);
643|            else if (tok.text == "::") input_ids.push_back(vocab["DOUBLE_COLON"]);
644|            else if (tok.text == "||") input_ids.push_back(vocab["CONCAT"]);
645|            else if (tok.text == "?") input_ids.push_back(vocab["QUESTION"]);
646|            else if (tok.text == ":") input_ids.push_back(vocab["COLON"]);
647|            else if (tok.text == "@") input_ids.push_back(vocab["AT"]);
648|            else input_ids.push_back(UNK_ID);
649|        }
650|    }
651|
652|    // Generate
653|    std::mt19937 rng(42);
654|    auto reverse_vocab = buildReverseVocab();
655|    result.score = 0.0f;
656|
657|    for (int step = 0; step < max_new_tokens; step++) {
658|        // Crop context
659|        std::vector<uint32_t> context = input_ids;
660|        if ((int)context.size() > (int)w.config.max_seq_len) {
661|            context = std::vector<uint32_t>(context.end() - w.config.max_seq_len, context.end());
662|        }
663|
664|        auto logits = forward(w, context);
665|        int T = (int)context.size();
666|        int V = (int)w.config.vocab_size;
667|
668|        // Get last token logits
669|        std::vector<float> next_logits(V);
670|        for (int v = 0; v < V; v++) {
671|            next_logits[v] = logits[(T-1) * V + v] / std::max(temperature, 1e-8f);
672|        }
673|
674|        // Top-k filtering
675|        if (top_k > 0 && top_k < V) {
676|            std::vector<float> sorted = next_logits;
677|            std::nth_element(sorted.begin(), sorted.begin() + V - top_k, sorted.end());
678|            float threshold = sorted[V - top_k];
679|            for (int v = 0; v < V; v++) {
680|                if (next_logits[v] < threshold) next_logits[v] = -INFINITY;
681|            }
682|        }
683|
684|        // Softmax
685|        softmax(next_logits.data(), V);
686|
687|        // Sample
688|        std::discrete_distribution<int> dist(next_logits.begin(), next_logits.end());
689|        int next_id = dist(rng);
690|        float log_prob = std::log(next_logits[next_id] + 1e-10f);
691|        result.score += log_prob;
692|
693|        input_ids.push_back((uint32_t)next_id);
694|        result.token_ids.push_back((uint32_t)next_id);
695|
696|        // Resolve token to text
697|        std::string token_str = reverse_vocab.count(next_id) ? reverse_vocab[next_id] : "<UNK>";
698|        std::string raw_text;
699|        if (token_str.substr(0, 10) == "COPY_BACK_") {
700|            int back = std::stoi(token_str.substr(10));
701|            if (back <= (int)ident_history.size()) raw_text = ident_history[ident_history.size() - back];
702|            else raw_text = "<ident>";
703|            ident_history.push_back(raw_text);
704|        } else if (token_str.substr(0, 14) == "COPY_STR_BACK_") {
705|            int back = std::stoi(token_str.substr(14));
706|            if (back <= (int)string_history.size()) raw_text = string_history[string_history.size() - back];
707|            else raw_text = "'<value>'";
708|            string_history.push_back(raw_text);
709|        } else if (token_str == "NEW_IDENT") {
710|            raw_text = "<ident>";
711|            ident_history.push_back(raw_text);
712|        } else if (token_str == "NEW_STRING") {
713|            raw_text = "'<value>'";
714|            string_history.push_back(raw_text);
715|        } else if (token_str.substr(0, 3) == "KW_") {
716|            raw_text = token_str.substr(3);
717|        } else if (token_str == "NUM_INT" || token_str == "NUM_FLOAT" || token_str == "NUM_HEX") {
718|            raw_text = "1";
719|        } else if (token_str == "LPAREN") raw_text = "(";
720|        else if (token_str == "RPAREN") raw_text = ")";
721|        else if (token_str == "LBRACKET") raw_text = "[";
722|        else if (token_str == "RBRACKET") raw_text = "]";
723|        else if (token_str == "LBRACE") raw_text = "{";
724|        else if (token_str == "RBRACE") raw_text = "}";
725|        else if (token_str == "COMMA") raw_text = ",";
726|        else if (token_str == "SEMICOLON") raw_text = ";";
727|        else if (token_str == "DOT") raw_text = ".";
728|        else if (token_str == "ASTERISK") raw_text = "*";
729|        else if (token_str == "PLUS") raw_text = "+";
730|        else if (token_str == "MINUS") raw_text = "-";
731|        else if (token_str == "SLASH") raw_text = "/";
732|        else if (token_str == "PERCENT") raw_text = "%";
733|        else if (token_str == "EQUALS") raw_text = "=";
734|        else if (token_str == "NOT_EQUALS") raw_text = "!=";
735|        else if (token_str == "LESS") raw_text = "<";
736|        else if (token_str == "GREATER") raw_text = ">";
737|        else if (token_str == "LESS_OR_EQUALS") raw_text = "<=";
738|        else if (token_str == "GREATER_OR_EQUALS") raw_text = ">=";
739|        else if (token_str == "ARROW") raw_text = "->";
740|        else if (token_str == "DOUBLE_COLON") raw_text = "::";
741|        else if (token_str == "CONCAT") raw_text = "||";
742|        else if (token_str == "QUESTION") raw_text = "?";
743|        else if (token_str == "COLON") raw_text = ":";
744|        else if (token_str == "CARET") raw_text = "^";
745|        else if (token_str == "AT") raw_text = "@";
746|        else raw_text = "";
747|
748|        // Build completion text with spacing
749|        if (!raw_text.empty()) {
750|            if (result.completion.empty()) {
751|                result.completion = raw_text;
752|            } else {
753|                if (raw_text == ")" || raw_text == "]" || raw_text == "}" ||
754|                    raw_text == "," || raw_text == ";" || raw_text == ".") {
755|                    result.completion += raw_text;
756|                } else {
757|                    result.completion += " " + raw_text;
758|                }
759|            }
760|        }
761|
762|        if ((uint32_t)next_id == eos_id) break;
763|        if (token_str == "SEMICOLON") break;
764|    }
765|
766|    return result;
767|}
768|
769|} // namespace CHAI
770|
771|// ─── C API for WebAssembly export ───────────────────────────────────────────
772|
773|#ifdef __EMSCRIPTEN__
774|#include <emscripten/emscripten.h>
775|#define EXPORT EMSCRIPTEN_KEEPALIVE
776|#else
777|#define EXPORT
778|#endif
779|
780|static CHAI::ModelWeights* g_model = nullptr;
781|
782|extern "C" {
783|
784|EXPORT int chai_load_model(const char* path) {
785|    g_model = CHAI::load_model(path).release();
786|    return g_model ? 1 : 0;
787|}
788|
789|EXPORT const char* chai_autocomplete(const char* prefix, int max_tokens) {
790|    static std::string result_str;
791|    if (!g_model) return "";
792|    auto result = CHAI::autocomplete(*g_model, std::string(prefix), max_tokens);
793|    result_str = result.completion;
794|    return result_str.c_str();
795|}
796|
797|EXPORT void chai_free_model() {
798|    delete g_model;
799|    g_model = nullptr;
800|}
801|
802|} // extern "C"
803|'''
804|    with open(path, "w") as f:
805|        f.write(code)
806|    print(f"  C++ engine: {path}")
807|
808|
809|def export_cmakefile(path: Path):
810|    """Export CMakeLists.txt for building the C++ library and WASM target."""
811|    cmake = r'''cmake_minimum_required(VERSION 3.14)
812|project(chai_clickhouse_autocomplete CXX)
813|
814|set(CMAKE_CXX_STANDARD 17)
815|set(CMAKE_CXX_STANDARD_REQUIRED ON)
816|
817|# Native library (for ClickHouse embedding)
818|add_library(chai_native STATIC
819|    ch_inference.h  # header-only library
820|)
821|target_include_directories(chai_native PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
822|
823|# Test binary
824|add_executable(chai_test test_main.cpp)
825|target_link_libraries(chai_test PRIVATE chai_native)
826|
827|# WebAssembly target (requires emscripten)
828|if(EMSCRIPTEN)
829|    add_executable(chai_wasm test_main.cpp)
830|    target_link_libraries(chai_wasm PRIVATE chai_native)
831|    set_target_properties(chai_wasm PROPERTIES
832|        LINK_FLAGS "-s EXPORTED_FUNCTIONS=['_chai_load_model','_chai_autocomplete','_chai_free_model'] -s EXPORTED_RUNTIME_METHODS=['ccall','cwrap']"
833|    )
834|endif()
835|'''
836|    with open(path, "w") as f:
837|        f.write(cmake)
838|    print(f"  CMakeLists.txt: {path}")
839|
840|
841|def export_test_main(path: Path):
842|    """Export a C++ test program."""
843|    code = r'''// Test program for CHAI (ClickHouse Autocomplete Inference)
844|#include "ch_inference.h"
845|#include <iostream>
846|#include <chrono>
847|
848|int main(int argc, char* argv[]) {
849|    std::string model_path = "ch_autocomplete.bin";
850|    if (argc > 1) model_path = argv[1];
851|
852|    std::cerr << "Loading model from: " << model_path << std::endl;
853|    auto model = CHAI::load_model(model_path);
854|    if (!model) {
855|        std::cerr << "Failed to load model!" << std::endl;
856|        return 1;
857|    }
858|
859|    std::cerr << "Model loaded: "
860|              << "d_model=" << model->config.d_model
861|              << ", n_layers=" << model->config.n_layers
862|              << ", vocab_size=" << model->config.vocab_size
863|              << std::endl;
864|
865|    std::vector<std::string> test_prefixes = {
866|        "SELECT ",
867|        "SELECT count() FROM ",
868|        "SELECT * FROM orders WHERE ",
869|        "SELECT city, count() FROM orders GROUP BY ",
870|        "CREATE TABLE ",
871|    };
872|
873|    for (const auto& prefix : test_prefixes) {
874|        auto start = std::chrono::high_resolution_clock::now();
875|        auto result = CHAI::autocomplete(*model, prefix, 20, 0.5f, 15);
876|        auto end = std::chrono::high_resolution_clock::now();
877|        auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
878|
879|        std::cout << "Prefix:     " << prefix << std::endl;
880|        std::cout << "Completion: " << result.completion << std::endl;
881|        std::cout << "Score:      " << result.score << std::endl;
882|        std::cout << "Time:       " << ms.count() << "ms" << std::endl;
883|        std::cout << std::endl;
884|    }
885|
886|    return 0;
887|}
888|'''
889|    with open(path, "w") as f:
890|        f.write(code)
891|    print(f"  C++ test: {path}")
892|
893|
894|def export_wasm_build_script(path: Path):
895|    """Export a shell script for building WebAssembly with emscripten."""
896|    script = r'''#!/bin/bash
897|# Build CHAI (ClickHouse Autocomplete Inference) as WebAssembly
898|# Requires emscripten (emcc) to be installed.
899|#
900|# Install emscripten:
901|#   git clone https://github.com/emscripten-core/emsdk.git
902|#   cd emsdk && ./emsdk install latest && ./emsdk activate latest
903|#   source ./emsdk_env.sh
904|
905|set -e
906|
907|echo "Building CHAI WebAssembly..."
908|
909|# Compile to WASM
910|emcc -std=c++17 -O2 \
911|    -s WASM=1 \
912|    -s EXPORTED_FUNCTIONS="['_chai_load_model','_chai_autocomplete','_chai_free_model']" \
913|    -s EXPORTED_RUNTIME_METHODS="['ccall','cwrap']" \
914|    -s ALLOW_MEMORY_GROWTH=1 \
915|    -s MODULARIZE=1 \
916|    -s EXPORT_ES6=1 \
917|    -o chai_wasm.js \
918|    test_main.cpp
919|
920|echo "WASM build complete: chai_wasm.js + chai_wasm.wasm"
921|echo ""
922|echo "Usage in JavaScript:"
923|echo "  const Module = require('./chai_wasm.js');"
924|echo "  const mod = await Module();"
925|echo "  const loadModel = mod.cwrap('chai_load_model', 'number', ['string']);"
926|echo "  const autocomplete = mod.cwrap('chai_autocomplete', 'string', ['string', 'number']);"
927|echo "  loadModel('ch_autocomplete.bin');"
928|echo "  const result = autocomplete('SELECT ', 20);"
929|'''
930|    with open(path, "w") as f:
931|        f.write(script)
932|    path.chmod(0o755)
933|    print(f"  WASM build script: {path}")
934|
935|
936|def main():
937|    export_dir = PROJECT_ROOT / "export"
938|    export_dir.mkdir(parents=True, exist_ok=True)
939|
940|    # Load model
941|    checkpoint_path = PROJECT_ROOT / "checkpoints" / "best.pt"
942|    if not checkpoint_path.exists():
943|        print(f"Checkpoint not found: {checkpoint_path}")
944|        print("Please run train.py first.")
945|        sys.exit(1)
946|
947|    print("Loading model...")
948|    model = load_model(str(checkpoint_path))
949|
950|    print("\nExporting C++ artifacts:")
951|    # Export binary weights
952|    export_model_weights_binary(model, export_dir / "ch_autocomplete.bin")
953|
954|    # Export headers
955|    export_vocab_header(export_dir / "ch_vocab.h")
956|    export_model_header(export_dir / "ch_model.h")
957|
958|    # Export C++ inference engine
959|    export_cpp_inference_engine(export_dir / "ch_inference.h")
960|
961|    # Export build files
962|    export_cmakefile(export_dir / "CMakeLists.txt")
963|    export_test_main(export_dir / "test_main.cpp")
964|    export_wasm_build_script(export_dir / "build_wasm.sh")
965|
966|    print(f"\nAll C++ artifacts exported to: {export_dir}/")
967|    print("\nTo build natively:")
968|    print(f"  cd {export_dir} && mkdir build && cd build && cmake .. && make")
969|    print("\nTo build for WebAssembly:")
970|    print(f"  cd {export_dir} && ./build_wasm.sh")
971|
972|
973|def load_model(path: str) -> ClickHouseGPT:
974|    """Load a trained model from a checkpoint."""
975|    checkpoint = torch.load(path, map_location="cpu", weights_only=False)
976|    config = ModelConfig.from_dict(checkpoint["model_config"])
977|    model = create_model(config)
978|    model.load_state_dict(checkpoint["model_state_dict"])
979|    model.eval()
980|    return model
981|
982|
983|if __name__ == "__main__":
984|    main()
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_01_basic_select.jsonl" start_line="1" end_line="5" total_lines="20">
  1|{"id":"traj_001","category":"basic_select","subcategory":"simple_select","description":"Simple select all with limit","query":"SELECT * FROM events LIMIT 10","tables":["events"],"columns":{"events":["id","event_type","user_id","timestamp","data"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"* FROM events LIMIT 10","next_token":"*"},{"prefix":"SELECT * ","completion":"FROM events LIMIT 10","next_token":"FROM"},{"prefix":"SELECT * FROM ","completion":"events LIMIT 10","next_token":"events"},{"prefix":"SELECT * FROM events ","completion":"LIMIT 10","next_token":"LIMIT"}]}
  2|{"id":"traj_002","category":"basic_select","subcategory":"where_and_orderby","description":"Select with WHERE multiple conditions and ORDER BY","query":"SELECT user_id, event_type, timestamp FROM events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","tables":["events"],"columns":{"events":["id","event_type","user_id","timestamp","data"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"user_id, event_type, timestamp FROM events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"user_id"},{"prefix":"SELECT user_id, event_type, timestamp ","completion":"FROM events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"FROM"},{"prefix":"SELECT user_id, event_type, timestamp FROM ","completion":"events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"events"},{"prefix":"SELECT user_id, event_type, timestamp FROM events ","completion":"WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"WHERE"},{"prefix":"SELECT user_id, event_type, timestamp FROM events WHERE ","completion":"event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"event_type"},{"prefix":"SELECT user_id, event_type, timestamp FROM events WHERE event_type = 'click' AND user_id = 1001 ","completion":"ORDER BY timestamp DESC LIMIT 50","next_token":"ORDER"},{"prefix":"SELECT user_id, event_type, timestamp FROM events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC ","completion":"LIMIT 50","next_token":"LIMIT"}]}
  3|{"id":"traj_003","category":"basic_select","subcategory":"distinct","description":"Select distinct values with limit","query":"SELECT DISTINCT country, region FROM users ORDER BY country ASC, region ASC LIMIT 100","tables":["users"],"columns":{"users":["id","username","email","country","region","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"DISTINCT country, region FROM users ORDER BY country ASC, region ASC LIMIT 100","next_token":"DISTINCT"},{"prefix":"SELECT DISTINCT ","completion":"country, region FROM users ORDER BY country ASC, region ASC LIMIT 100","next_token":"country"},{"prefix":"SELECT DISTINCT country, region ","completion":"FROM users ORDER BY country ASC, region ASC LIMIT 100","next_token":"FROM"},{"prefix":"SELECT DISTINCT country, region FROM ","completion":"users ORDER BY country ASC, region ASC LIMIT 100","next_token":"users"},{"prefix":"SELECT DISTINCT country, region FROM users ","completion":"ORDER BY country ASC, region ASC LIMIT 100","next_token":"ORDER"},{"prefix":"SELECT DISTINCT country, region FROM users ORDER BY country ASC, region ASC ","completion":"LIMIT 100","next_token":"LIMIT"}]}
  4|{"id":"traj_004","category":"basic_select","subcategory":"limit_offset","description":"Select with LIMIT and OFFSET for pagination","query":"SELECT id, title, price FROM products WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","tables":["products"],"columns":{"products":["id","title","price","category","stock","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, title, price FROM products WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"id"},{"prefix":"SELECT id, title, price ","completion":"FROM products WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"FROM"},{"prefix":"SELECT id, title, price FROM ","completion":"products WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"products"},{"prefix":"SELECT id, title, price FROM products ","completion":"WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"WHERE"},{"prefix":"SELECT id, title, price FROM products WHERE price > 50 ","completion":"ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"ORDER"},{"prefix":"SELECT id, title, price FROM products WHERE price > 50 ORDER BY price DESC ","completion":"LIMIT 20 OFFSET 40","next_token":"LIMIT"},{"prefix":"SELECT id, title, price FROM products WHERE price > 50 ORDER BY price DESC LIMIT 20 ","completion":"OFFSET 40","next_token":"OFFSET"}]}
  5|{"id":"traj_005","category":"basic_select","subcategory":"prewhere","description":"Select with PREWHERE optimization for MergeTree table","query":"SELECT user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","tables":["logs"],"columns":{"logs":["user_id","event_name","event_time","event_date","ip_address","user_agent"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"user_id"},{"prefix":"SELECT user_id, event_name, event_time ","completion":"FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"FROM"},{"prefix":"SELECT user_id, event_name, event_time FROM ","completion":"logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"logs"},{"prefix":"SELECT user_id, event_name, event_time FROM logs ","completion":"PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"PREWHERE"},{"prefix":"SELECT user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' ","completion":"WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"WHERE"},{"prefix":"SELECT user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ","completion":"ORDER BY event_time DESC LIMIT 100","next_token":"ORDER"},{"prefix":"SELECT user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC ","completion":"LIMIT 100","next_token":"LIMIT"}]}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_02_aggregation.jsonl" start_line="1" end_line="5" total_lines="20">
  1|{"id":"traj_021","category":"aggregation","subcategory":"group_by_single","description":"Simple GROUP BY single column with count and sum aggregates","query":"SELECT product_category, count() AS order_count, sum(revenue) AS total_revenue, avg(revenue) AS avg_revenue FROM sales WHERE sale_date >= '2024-01-01' GROUP BY product_category ORDER BY total_revenue DESC LIMIT 100","tables":["sales"],"columns":{"sales":["id","product_category","revenue","sale_date","region","customer_id","quantity","discount"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"product_category, count() AS order_count, sum(revenue) AS total_revenue, avg(revenue) AS avg_revenue ","next_token":"product_category"},{"prefix":"SELECT product_category, count() AS order_count, sum(revenue) AS total_revenue, avg(revenue) AS avg_revenue FROM ","completion":"sales ","next_token":"sales"},{"prefix":"SELECT product_category, count() AS order_count, sum(revenue) AS total_revenue, avg(revenue) AS avg_revenue FROM sales WHERE ","completion":"sale_date >= '2024-01-01' ","next_token":"sale_date"},{"prefix":"SELECT product_category, count() AS order_count, sum(revenue) AS total_revenue, avg(revenue) AS avg_revenue FROM sales WHERE sale_date >= '2024-01-01' GROUP BY ","completion":"product_category ","next_token":"product_category"},{"prefix":"SELECT product_category, count() AS order_count, sum(revenue) AS total_revenue, avg(revenue) AS avg_revenue FROM sales WHERE sale_date >= '2024-01-01' GROUP BY product_category ORDER BY ","completion":"total_revenue DESC ","next_token":"total_revenue"}]}
  2|{"id":"traj_022","category":"aggregation","subcategory":"group_by_having","description":"GROUP BY multiple columns with HAVING clause and multiple aggregates","query":"SELECT country, city, count() AS visits, uniq(user_id) AS unique_users, avg(session_duration) AS avg_duration FROM pageviews WHERE date >= '2024-01-01' GROUP BY country, city HAVING visits > 100 AND unique_users > 50 ORDER BY unique_users DESC LIMIT 50","tables":["pageviews"],"columns":{"pageviews":["id","user_id","country","city","session_duration","date","url","referrer"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"country, city, count() AS visits, uniq(user_id) AS unique_users, avg(session_duration) AS avg_duration ","next_token":"country"},{"prefix":"SELECT country, city, count() AS visits, uniq(user_id) AS unique_users, avg(session_duration) AS avg_duration FROM ","completion":"pageviews ","next_token":"pageviews"},{"prefix":"SELECT country, city, count() AS visits, uniq(user_id) AS unique_users, avg(session_duration) AS avg_duration FROM pageviews WHERE ","completion":"date >= '2024-01-01' ","next_token":"date"},{"prefix":"SELECT country, city, count() AS visits, uniq(user_id) AS unique_users, avg(session_duration) AS avg_duration FROM pageviews WHERE date >= '2024-01-01' GROUP BY ","completion":"country, city ","next_token":"country"},{"prefix":"SELECT country, city, count() AS visits, uniq(user_id) AS unique_users, avg(session_duration) AS avg_duration FROM pageviews WHERE date >= '2024-01-01' GROUP BY country, city HAVING ","completion":"visits > 100 AND unique_users > 50 ","next_token":"visits"},{"prefix":"SELECT country, city, count() AS visits, uniq(user_id) AS unique_users, avg(session_duration) AS avg_duration FROM pageviews WHERE date >= '2024-01-01' GROUP BY country, city HAVING visits > 100 AND unique_users > 50 ORDER BY ","completion":"unique_users DESC ","next_token":"unique_users"}]}
  3|{"id":"traj_023","category":"aggregation","subcategory":"with_totals","description":"GROUP BY with WITH TOTALS to produce a summary row","query":"SELECT region, product_line, sum(amount) AS total_amount, count() AS txn_count FROM orders WHERE order_date BETWEEN '2024-01-01' AND '2024-03-31' GROUP BY region, product_line WITH TOTALS ORDER BY region, total_amount DESC","tables":["orders"],"columns":{"orders":["id","region","product_line","amount","order_date","customer_id","status"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"region, product_line, sum(amount) AS total_amount, count() AS txn_count ","next_token":"region"},{"prefix":"SELECT region, product_line, sum(amount) AS total_amount, count() AS txn_count FROM ","completion":"orders ","next_token":"orders"},{"prefix":"SELECT region, product_line, sum(amount) AS total_amount, count() AS txn_count FROM orders WHERE ","completion":"order_date BETWEEN '2024-01-01' AND '2024-03-31' ","next_token":"order_date"},{"prefix":"SELECT region, product_line, sum(amount) AS total_amount, count() AS txn_count FROM orders WHERE order_date BETWEEN '2024-01-01' AND '2024-03-31' GROUP BY ","completion":"region, product_line ","next_token":"region"},{"prefix":"SELECT region, product_line, sum(amount) AS total_amount, count() AS txn_count FROM orders WHERE order_date BETWEEN '2024-01-01' AND '2024-03-31' GROUP BY region, product_line WITH TOTALS ","completion":"","next_token":"WITH"},{"prefix":"SELECT region, product_line, sum(amount) AS total_amount, count() AS txn_count FROM orders WHERE order_date BETWEEN '2024-01-01' AND '2024-03-31' GROUP BY region, product_line WITH TOTALS ORDER BY ","completion":"region, total_amount DESC ","next_token":"region"}]}
  4|{"id":"traj_024","category":"aggregation","subcategory":"with_cube","description":"GROUP BY with WITH CUBE for multi-dimensional aggregation across all column combinations","query":"SELECT year, quarter, region, sum(sales_amount) AS total_sales, avg(sales_amount) AS avg_sales, count() AS record_count FROM sales_summary WHERE year >= 2023 GROUP BY year, quarter, region WITH CUBE ORDER BY year, quarter, region","tables":["sales_summary"],"columns":{"sales_summary":["year","quarter","region","sales_amount","product_id","channel","cost"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"year, quarter, region, sum(sales_amount) AS total_sales, avg(sales_amount) AS avg_sales, count() AS record_count ","next_token":"year"},{"prefix":"SELECT year, quarter, region, sum(sales_amount) AS total_sales, avg(sales_amount) AS avg_sales, count() AS record_count FROM ","completion":"sales_summary ","next_token":"sales_summary"},{"prefix":"SELECT year, quarter, region, sum(sales_amount) AS total_sales, avg(sales_amount) AS avg_sales, count() AS record_count FROM sales_summary WHERE ","completion":"year >= 2023 ","next_token":"year"},{"prefix":"SELECT year, quarter, region, sum(sales_amount) AS total_sales, avg(sales_amount) AS avg_sales, count() AS record_count FROM sales_summary WHERE year >= 2023 GROUP BY ","completion":"year, quarter, region ","next_token":"year"},{"prefix":"SELECT year, quarter, region, sum(sales_amount) AS total_sales, avg(sales_amount) AS avg_sales, count() AS record_count FROM sales_summary WHERE year >= 2023 GROUP BY year, quarter, region WITH CUBE ","completion":"","next_token":"WITH"},{"prefix":"SELECT year, quarter, region, sum(sales_amount) AS total_sales, avg(sales_amount) AS avg_sales, count() AS record_count FROM sales_summary WHERE year >= 2023 GROUP BY year, quarter, region WITH CUBE ORDER BY ","completion":"year, quarter, region ","next_token":"year"}]}
  5|{"id":"traj_025","category":"aggregation","subcategory":"with_rollup","description":"GROUP BY with WITH ROLLUP for hierarchical subtotals","query":"SELECT country, state, city, count() AS user_count, sum(lifetime_value) AS total_ltv FROM users WHERE created_at >= '2023-01-01' GROUP BY country, state, city WITH ROLLUP ORDER BY country, state, city","tables":["users"],"columns":{"users":["id","country","state","city","lifetime_value","created_at","email","status"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"country, state, city, count() AS user_count, sum(lifetime_value) AS total_ltv ","next_token":"country"},{"prefix":"SELECT country, state, city, count() AS user_count, sum(lifetime_value) AS total_ltv FROM ","completion":"users ","next_token":"users"},{"prefix":"SELECT country, state, city, count() AS user_count, sum(lifetime_value) AS total_ltv FROM users WHERE ","completion":"created_at >= '2023-01-01' ","next_token":"created_at"},{"prefix":"SELECT country, state, city, count() AS user_count, sum(lifetime_value) AS total_ltv FROM users WHERE created_at >= '2023-01-01' GROUP BY ","completion":"country, state, city ","next_token":"country"},{"prefix":"SELECT country, state, city, count() AS user_count, sum(lifetime_value) AS total_ltv FROM users WHERE created_at >= '2023-01-01' GROUP BY country, state, city WITH ROLLUP ","completion":"","next_token":"WITH"},{"prefix":"SELECT country, state, city, count() AS user_count, sum(lifetime_value) AS total_ltv FROM users WHERE created_at >= '2023-01-01' GROUP BY country, state, city WITH ROLLUP ORDER BY ","completion":"country, state, city ","next_token":"country"}]}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_03_joins.jsonl" start_line="1" end_line="5" total_lines="20">
  1|{"id":"traj_041","category":"joins","subcategory":"inner_join_using","description":"Inner join with USING clause between orders and customers","query":"SELECT o.id, o.order_date, c.name AS customer_name, c.email FROM orders AS o INNER JOIN customers AS c USING (customer_id) WHERE o.status = 'shipped' ORDER BY o.order_date DESC LIMIT 100","tables":["orders","customers"],"columns":{"orders":["id","customer_id","order_date","status","total_amount"],"customers":["customer_id","name","email","country","signup_date"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"o.id, o.order_date, c.name AS customer_name, c.email ","next_token":"o"},{"prefix":"SELECT o.id, o.order_date, c.name AS customer_name, c.email FROM ","completion":"orders AS o ","next_token":"orders"},{"prefix":"SELECT o.id, o.order_date, c.name AS customer_name, c.email FROM orders AS o INNER JOIN ","completion":"customers AS c USING (customer_id) ","next_token":"customers"},{"prefix":"SELECT o.id, o.order_date, c.name AS customer_name, c.email FROM orders AS o INNER JOIN customers AS c USING (customer_id) WHERE ","completion":"o.status = 'shipped' ","next_token":"o"},{"prefix":"SELECT o.id, o.order_date, c.name AS customer_name, c.email FROM orders AS o INNER JOIN customers AS c USING (customer_id) WHERE o.status = 'shipped' ORDER BY ","completion":"o.order_date DESC ","next_token":"o"}]}
  2|{"id":"traj_042","category":"joins","subcategory":"left_join_on","description":"Left join with ON clause between users and sessions","query":"SELECT u.user_id, u.username, s.session_id, s.started_at, s.duration_seconds FROM users AS u LEFT JOIN sessions AS s ON u.user_id = s.user_id WHERE u.is_active = 1 ORDER BY s.started_at DESC LIMIT 50","tables":["users","sessions"],"columns":{"users":["user_id","username","email","is_active","created_at"],"sessions":["session_id","user_id","started_at","duration_seconds","ip_address"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"u.user_id, u.username, s.session_id, s.started_at, s.duration_seconds ","next_token":"u"},{"prefix":"SELECT u.user_id, u.username, s.session_id, s.started_at, s.duration_seconds FROM ","completion":"users AS u ","next_token":"users"},{"prefix":"SELECT u.user_id, u.username, s.session_id, s.started_at, s.duration_seconds FROM users AS u LEFT JOIN ","completion":"sessions AS s ON u.user_id = s.user_id ","next_token":"sessions"},{"prefix":"SELECT u.user_id, u.username, s.session_id, s.started_at, s.duration_seconds FROM users AS u LEFT JOIN sessions AS s ON u.user_id = s.user_id WHERE ","completion":"u.is_active = 1 ","next_token":"u"},{"prefix":"SELECT u.user_id, u.username, s.session_id, s.started_at, s.duration_seconds FROM users AS u LEFT JOIN sessions AS s ON u.user_id = s.user_id WHERE u.is_active = 1 ORDER BY ","completion":"s.started_at DESC ","next_token":"s"}]}
  3|{"id":"traj_043","category":"joins","subcategory":"multiple_joins","description":"Three-table join: orders, customers, and products","query":"SELECT o.id, o.order_date, c.name AS customer_name, p.name AS product_name, o.total_amount FROM orders AS o INNER JOIN customers AS c USING (customer_id) INNER JOIN products AS p USING (product_id) WHERE o.order_date >= '2024-01-01' ORDER BY o.total_amount DESC LIMIT 200","tables":["orders","customers","products"],"columns":{"orders":["id","customer_id","product_id","order_date","status","total_amount"],"customers":["customer_id","name","email","country","signup_date"],"products":["product_id","name","category","price","stock"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"o.id, o.order_date, c.name AS customer_name, p.name AS product_name, o.total_amount ","next_token":"o"},{"prefix":"SELECT o.id, o.order_date, c.name AS customer_name, p.name AS product_name, o.total_amount FROM ","completion":"orders AS o ","next_token":"orders"},{"prefix":"SELECT o.id, o.order_date, c.name AS customer_name, p.name AS product_name, o.total_amount FROM orders AS o INNER JOIN ","completion":"customers AS c USING (customer_id) ","next_token":"customers"},{"prefix":"SELECT o.id, o.order_date, c.name AS customer_name, p.name AS product_name, o.total_amount FROM orders AS o INNER JOIN customers AS c USING (customer_id) INNER JOIN ","completion":"products AS p USING (product_id) ","next_token":"products"},{"prefix":"SELECT o.id, o.order_date, c.name AS customer_name, p.name AS product_name, o.total_amount FROM orders AS o INNER JOIN customers AS c USING (customer_id) INNER JOIN products AS p USING (product_id) WHERE ","completion":"o.order_date >= '2024-01-01' ","next_token":"o"},{"prefix":"SELECT o.id, o.order_date, c.name AS customer_name, p.name AS product_name, o.total_amount FROM orders AS o INNER JOIN customers AS c USING (customer_id) INNER JOIN products AS p USING (product_id) WHERE o.order_date >= ... [truncated 79 chars]
  4|{"id":"traj_044","category":"joins","subcategory":"asof_join","description":"ASOF join for time-series: match events to nearest preceding session","query":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.started_at FROM events AS e ASOF LEFT JOIN sessions AS s ON e.user_id = s.user_id AND e.event_time > s.started_at WHERE e.event_date = '2024-06-01' ORDER BY e.event_time LIMIT 500","tables":["events","sessions"],"columns":{"events":["event_id","user_id","event_time","event_date","event_type","properties"],"sessions":["session_id","user_id","started_at","ended_at","device_type"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"e.event_id, e.event_time, e.event_type, s.session_id, s.started_at ","next_token":"e"},{"prefix":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.started_at FROM ","completion":"events AS e ","next_token":"events"},{"prefix":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.started_at FROM events AS e ASOF LEFT JOIN ","completion":"sessions AS s ON e.user_id = s.user_id AND e.event_time > s.started_at ","next_token":"sessions"},{"prefix":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.started_at FROM events AS e ASOF LEFT JOIN sessions AS s ON e.user_id = s.user_id AND e.event_time > s.started_at WHERE ","completion":"e.event_date = '2024-06-01' ","next_token":"e"},{"prefix":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.started_at FROM events AS e ASOF LEFT JOIN sessions AS s ON e.user_id = s.user_id AND e.event_time > s.started_at WHERE e.event_date = '2024-06-01' ORDER BY ","completion":"e.event_time ","next_token":"e"}]}
  5|{"id":"traj_045","category":"joins","subcategory":"array_join","description":"ARRAY JOIN to expand array column of tags per order","query":"SELECT o.id, o.order_date, tag FROM orders AS o ARRAY JOIN o.tags AS tag WHERE o.status = 'completed' ORDER BY o.id, tag LIMIT 1000","tables":["orders"],"columns":{"orders":["id","customer_id","order_date","status","total_amount","tags"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"o.id, o.order_date, tag ","next_token":"o"},{"prefix":"SELECT o.id, o.order_date, tag FROM ","completion":"orders AS o ","next_token":"orders"},{"prefix":"SELECT o.id, o.order_date, tag FROM orders AS o ARRAY JOIN ","completion":"o.tags AS tag ","next_token":"o"},{"prefix":"SELECT o.id, o.order_date, tag FROM orders AS o ARRAY JOIN o.tags AS tag WHERE ","completion":"o.status = 'completed' ","next_token":"o"},{"prefix":"SELECT o.id, o.order_date, tag FROM orders AS o ARRAY JOIN o.tags AS tag WHERE o.status = 'completed' ORDER BY ","completion":"o.id, tag ","next_token":"o"}]}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_04_ddl_dml.jsonl" start_line="1" end_line="5" total_lines="20">
  1|{"id":"traj_061","category":"ddl_dml","subcategory":"create_merge_tree","description":"Create MergeTree table with partition by month, TTL, and settings","query":"CREATE TABLE IF NOT EXISTS analytics.events (event_id UUID, user_id UInt64, event_type String, event_data String, event_time DateTime CODEC(Delta, ZSTD), created_at DateTime DEFAULT now() ) ENGINE = MergeTree() PARTITION BY toYYYYMM(event_time) ORDER BY (event_time, user_id) TTL event_time + INTERVAL 30 DAY SETTINGS index_granularity = 8192","tables":["analytics.events"],"columns":{"analytics.events":["event_id","user_id","event_type","event_data","event_time","created_at"]},"dictionaries":[],"settings":{"index_granularity":"8192"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"analytics.events ","next_token":"analytics"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (event_id UUID, user_id UInt64, event_type String, event_data String, event_time DateTime CODEC(Delta, ZSTD), created_at DateTime DEFAULT now() ) ENGINE = ","completion":"MergeTree() ","next_token":"MergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (...) ENGINE = MergeTree() PARTITION BY ","completion":"toYYYYMM(event_time) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (...) ENGINE = MergeTree() PARTITION BY toYYYYMM(event_time) ORDER BY ","completion":"(event_time, user_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (...) ENGINE = MergeTree() PARTITION BY toYYYYMM(event_time) ORDER BY (event_time, user_id) TTL ","completion":"event_time + INTERVAL 30 DAY ","next_token":"event_time"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.events (...) ENGINE = MergeTree() PARTITION BY toYYYYMM(event_time) ORDER BY (event_time, user_id) TTL event_time + INTERVAL 30 DAY SETTINGS ","completion":"index_granularity = 8192","next_token":"index_granularity"}]}
  2|{"id":"traj_062","category":"ddl_dml","subcategory":"create_replacing_merge_tree","description":"Create ReplacingMergeTree table with version column for deduplication","query":"CREATE TABLE IF NOT EXISTS shop.products (product_id UInt64, sku String, name String, price Decimal(10,2), stock UInt32, updated_at DateTime, version UInt64 ) ENGINE = ReplacingMergeTree(version) ORDER BY (product_id) PARTITION BY tuple() SETTINGS index_granularity = 4096","tables":["shop.products"],"columns":{"shop.products":["product_id","sku","name","price","stock","updated_at","version"]},"dictionaries":[],"settings":{"index_granularity":"4096"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"shop.products ","next_token":"shop"},{"prefix":"CREATE TABLE IF NOT EXISTS shop.products (product_id UInt64, sku String, name String, price Decimal(10,2), stock UInt32, updated_at DateTime, version UInt64 ) ENGINE = ","completion":"ReplacingMergeTree(version) ","next_token":"ReplacingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS shop.products (...) ENGINE = ReplacingMergeTree(version) ORDER BY ","completion":"(product_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS shop.products (...) ENGINE = ReplacingMergeTree(version) ORDER BY (product_id) PARTITION BY ","completion":"tuple() ","next_token":"tuple"},{"prefix":"CREATE TABLE IF NOT EXISTS shop.products (...) ENGINE = ReplacingMergeTree(version) ORDER BY (product_id) PARTITION BY tuple() SETTINGS ","completion":"index_granularity = 4096","next_token":"index_granularity"}]}
  3|{"id":"traj_063","category":"ddl_dml","subcategory":"create_summing_merge_tree","description":"Create SummingMergeTree table for aggregating numeric metrics by key","query":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (stat_date Date, metric_name String, value Float64, count UInt64 ) ENGINE = SummingMergeTree() PARTITION BY toYYYYMM(stat_date) ORDER BY (stat_date, metric_name) SETTINGS index_granularity = 8192","tables":["metrics.daily_stats"],"columns":{"metrics.daily_stats":["stat_date","metric_name","value","count"]},"dictionaries":[],"settings":{"index_granularity":"8192"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"metrics.daily_stats ","next_token":"metrics"},{"prefix":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (stat_date Date, metric_name String, value Float64, count UInt64 ) ENGINE = ","completion":"SummingMergeTree() ","next_token":"SummingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (...) ENGINE = SummingMergeTree() PARTITION BY ","completion":"toYYYYMM(stat_date) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (...) ENGINE = SummingMergeTree() PARTITION BY toYYYYMM(stat_date) ORDER BY ","completion":"(stat_date, metric_name) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS metrics.daily_stats (...) ENGINE = SummingMergeTree() PARTITION BY toYYYYMM(stat_date) ORDER BY (stat_date, metric_name) SETTINGS ","completion":"index_granularity = 8192","next_token":"index_granularity"}]}
  4|{"id":"traj_064","category":"ddl_dml","subcategory":"create_aggregating_merge_tree","description":"Create AggregatingMergeTree with AggregateFunction columns for pre-aggregated data","query":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (visit_date Date, site_id UInt32, uniq_users AggregateFunction(uniq, UInt64), total_duration AggregateFunction(sum, UInt32) ) ENGINE = AggregatingMergeTree() PARTITION BY toYYYYMM(visit_date) ORDER BY (visit_date, site_id) SETTINGS index_granularity = 8192","tables":["analytics.agg_visits"],"columns":{"analytics.agg_visits":["visit_date","site_id","uniq_users","total_duration"]},"dictionaries":[],"settings":{"index_granularity":"8192"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"analytics.agg_visits ","next_token":"analytics"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (visit_date Date, site_id UInt32, uniq_users AggregateFunction(uniq, UInt64), total_duration AggregateFunction(sum, UInt32) ) ENGINE = ","completion":"AggregatingMergeTree() ","next_token":"AggregatingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (...) ENGINE = AggregatingMergeTree() PARTITION BY ","completion":"toYYYYMM(visit_date) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (...) ENGINE = AggregatingMergeTree() PARTITION BY toYYYYMM(visit_date) ORDER BY ","completion":"(visit_date, site_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS analytics.agg_visits (...) ENGINE = AggregatingMergeTree() PARTITION BY toYYYYMM(visit_date) ORDER BY (visit_date, site_id) SETTINGS ","completion":"index_granularity = 8192","next_token":"index_granularity"}]}
  5|{"id":"traj_065","category":"ddl_dml","subcategory":"create_collapsing_merge_tree","description":"Create CollapsingMergeTree with sign column for upsert semantics","query":"CREATE TABLE IF NOT EXISTS billing.transactions (txn_id UInt64, account_id UInt64, amount Decimal(18,2), status String, sign Int8, created_at DateTime ) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(created_at) ORDER BY (txn_id, account_id) SETTINGS index_granularity = 4096","tables":["billing.transactions"],"columns":{"billing.transactions":["txn_id","account_id","amount","status","sign","created_at"]},"dictionaries":[],"settings":{"index_granularity":"4096"},"completion_points":[{"prefix":"CREATE TABLE IF NOT EXISTS ","completion":"billing.transactions ","next_token":"billing"},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (txn_id UInt64, account_id UInt64, amount Decimal(18,2), status String, sign Int8, created_at DateTime ) ENGINE = ","completion":"CollapsingMergeTree(sign) ","next_token":"CollapsingMergeTree"},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (...) ENGINE = CollapsingMergeTree(sign) PARTITION BY ","completion":"toYYYYMM(created_at) ","next_token":"toYYYYMM"},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (...) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(created_at) ORDER BY ","completion":"(txn_id, account_id) ","next_token":"("},{"prefix":"CREATE TABLE IF NOT EXISTS billing.transactions (...) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(created_at) ORDER BY (txn_id, account_id) SETTINGS ","completion":"index_granularity = 4096","next_token":"index_granularity"}]}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl" start_line="1" end_line="5" total_lines="20">
  1|{"id":"traj_081","category":"advanced","subcategory":"array_join","description":"Array join with arrayJoin to explode event arrays for per-event analysis","query":"SELECT user_id, event_name, event_time FROM user_events ARRAY JOIN arrayJoin(events) AS event_name, event_time WHERE event_time >= now() - INTERVAL 1 DAY ORDER BY event_time DESC LIMIT 100","tables":["user_events"],"columns":{"user_events":["user_id","events","session_id","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"user_id, event_name, event_time ","next_token":"user_id"},{"prefix":"SELECT user_id, event_name, event_time FROM ","completion":"user_events ","next_token":"user_events"},{"prefix":"SELECT user_id, event_name, event_time FROM user_events ARRAY JOIN ","completion":"arrayJoin(events) AS event_name, event_time ","next_token":"arrayJoin"},{"prefix":"SELECT user_id, event_name, event_time FROM user_events ARRAY JOIN arrayJoin(events) AS event_name, event_time WHERE ","completion":"event_time >= now() - INTERVAL 1 DAY ","next_token":"event_time"},{"prefix":"SELECT user_id, event_name, event_time FROM user_events ARRAY JOIN arrayJoin(events) AS event_name, event_time WHERE event_time >= now() - INTERVAL 1 DAY ORDER BY ","completion":"event_time DESC ","next_token":"event_time"},{"prefix":"SELECT user_id, event_name, event_time FROM user_events ARRAY JOIN arrayJoin(events) AS event_name, event_time WHERE event_time >= now() - INTERVAL 1 DAY ORDER BY event_time DESC LIMIT ","completion":"100","next_token":"100"}]}
  2|{"id":"traj_082","category":"advanced","subcategory":"array_map_filter","description":"arrayMap and arrayFilter to transform and filter numeric arrays in-place","query":"SELECT product_id, arrayFilter(x -> x > 0, arrayMap(x -> x * 1.1, prices)) AS adjusted_prices FROM product_pricing WHERE length(prices) > 0 ORDER BY product_id LIMIT 50","tables":["product_pricing"],"columns":{"product_pricing":["product_id","prices","currency","updated_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"product_id, arrayFilter(x -> x > 0, arrayMap(x -> x * 1.1, prices)) AS adjusted_prices ","next_token":"product_id"},{"prefix":"SELECT product_id, arrayFilter(x -> x > 0, arrayMap(x -> x * 1.1, prices)) AS adjusted_prices FROM ","completion":"product_pricing ","next_token":"product_pricing"},{"prefix":"SELECT product_id, arrayFilter(x -> x > 0, arrayMap(x -> x * 1.1, prices)) AS adjusted_prices FROM product_pricing WHERE ","completion":"length(prices) > 0 ","next_token":"length"},{"prefix":"SELECT product_id, arrayFilter(x -> x > 0, arrayMap(x -> x * 1.1, prices)) AS adjusted_prices FROM product_pricing WHERE length(prices) > 0 ORDER BY ","completion":"product_id ","next_token":"product_id"},{"prefix":"SELECT product_id, arrayFilter(x -> x > 0, arrayMap(x -> x * 1.1, prices)) AS adjusted_prices FROM product_pricing WHERE length(prices) > 0 ORDER BY product_id LIMIT ","completion":"50","next_token":"50"}]}
  3|{"id":"traj_083","category":"advanced","subcategory":"array_distinct_sort_concat","description":"arrayDistinct, arraySort, and arrayConcat to merge and deduplicate tag arrays","query":"SELECT article_id, arraySort(arrayDistinct(arrayConcat(tags, extra_tags))) AS merged_tags FROM article_tags WHERE has(tags, 'clickhouse') OR has(extra_tags, 'database') ORDER BY article_id","tables":["article_tags"],"columns":{"article_tags":["article_id","tags","extra_tags","author_id","published_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"article_id, arraySort(arrayDistinct(arrayConcat(tags, extra_tags))) AS merged_tags ","next_token":"article_id"},{"prefix":"SELECT article_id, arraySort(arrayDistinct(arrayConcat(tags, extra_tags))) AS merged_tags FROM ","completion":"article_tags ","next_token":"article_tags"},{"prefix":"SELECT article_id, arraySort(arrayDistinct(arrayConcat(tags, extra_tags))) AS merged_tags FROM article_tags WHERE ","completion":"has(tags, 'clickhouse') OR has(extra_tags, 'database') ","next_token":"has"},{"prefix":"SELECT article_id, arraySort(arrayDistinct(arrayConcat(tags, extra_tags))) AS merged_tags FROM article_tags WHERE has(tags, 'clickhouse') OR has(extra_tags, 'database') ORDER BY ","completion":"article_id","next_token":"article_id"}]}
  4|{"id":"traj_084","category":"advanced","subcategory":"array_enumerate_reduce","description":"arrayEnumerate, arrayReduce, has, and indexOf for array indexing and aggregation","query":"SELECT session_id, arrayReduce('sum', page_views) AS total_views, indexOf(page_urls, '/checkout') AS checkout_pos, has(page_urls, '/home') AS visited_home FROM web_sessions WHERE arrayReduce('max', page_views) > 5 ORDER BY total_views DESC LIMIT 200","tables":["web_sessions"],"columns":{"web_sessions":["session_id","page_urls","page_views","user_id","started_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"session_id, arrayReduce('sum', page_views) AS total_views, indexOf(page_urls, '/checkout') AS checkout_pos, has(page_urls, '/home') AS visited_home ","next_token":"session_id"},{"prefix":"SELECT session_id, arrayReduce('sum', page_views) AS total_views, indexOf(page_urls, '/checkout') AS checkout_pos, has(page_urls, '/home') AS visited_home FROM ","completion":"web_sessions ","next_token":"web_sessions"},{"prefix":"SELECT session_id, arrayReduce('sum', page_views) AS total_views, indexOf(page_urls, '/checkout') AS checkout_pos, has(page_urls, '/home') AS visited_home FROM web_sessions WHERE ","completion":"arrayReduce('max', page_views) > 5 ","next_token":"arrayReduce"},{"prefix":"SELECT session_id, arrayReduce('sum', page_views) AS total_views, indexOf(page_urls, '/checkout') AS checkout_pos, has(page_urls, '/home') AS visited_home FROM web_sessions WHERE arrayReduce('max', page_views) > 5 ORDER BY ","completion":"total_views DESC ","next_token":"total_views"},{"prefix":"SELECT session_id, arrayReduce('sum', page_views) AS total_views, indexOf(page_urls, '/checkout') AS checkout_pos, has(page_urls, '/home') AS visited_home FROM web_sessions WHERE arrayReduce('max', page_views) > 5 ORDER BY total_views DESC LIMIT ","completion":"200","next_token":"200"}]}
  5|{"id":"traj_085","category":"advanced","subcategory":"map_operations","description":"map, mapKeys, mapValues, and mapAdd for aggregating key-value metric maps","query":"SELECT region, mapKeys(mapAdd(metric_map)) AS metric_names, mapValues(mapAdd(metric_map)) AS metric_totals FROM regional_metrics WHERE mapAdd(metric_map)['revenue'] > 1000 ORDER BY region","tables":["regional_metrics"],"columns":{"regional_metrics":["region","metric_map","recorded_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"region, mapKeys(mapAdd(metric_map)) AS metric_names, mapValues(mapAdd(metric_map)) AS metric_totals ","next_token":"region"},{"prefix":"SELECT region, mapKeys(mapAdd(metric_map)) AS metric_names, mapValues(mapAdd(metric_map)) AS metric_totals FROM ","completion":"regional_metrics ","next_token":"regional_metrics"},{"prefix":"SELECT region, mapKeys(mapAdd(metric_map)) AS metric_names, mapValues(mapAdd(metric_map)) AS metric_totals FROM regional_metrics WHERE ","completion":"mapAdd(metric_map)['revenue'] > 1000 ","next_token":"mapAdd"},{"prefix":"SELECT region, mapKeys(mapAdd(metric_map)) AS metric_names, mapValues(mapAdd(metric_map)) AS metric_totals FROM regional_metrics WHERE mapAdd(metric_map)['revenue'] > 1000 ORDER BY ","completion":"region","next_token":"region"}]}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_01_basic_select.jsonl" start_line="15" end_line="19" total_lines="20">
 15|{"id":"traj_015","category":"basic_select","subcategory":"sample_final","description":"Select with SAMPLE and FINAL together","query":"SELECT id, user_id, amount, status FROM transactions FINAL SAMPLE 1000 WHERE amount > 100 ORDER BY amount DESC LIMIT 10","tables":["transactions"],"columns":{"transactions":["id","user_id","amount","status","currency","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, user_id, amount, status FROM transactions FINAL SAMPLE 1000 WHERE amount > 100 ORDER BY amount DESC LIMIT 10","next_token":"id"},{"prefix":"SELECT id, user_id, amount, status ","completion":"FROM transactions FINAL SAMPLE 1000 WHERE amount > 100 ORDER BY amount DESC LIMIT 10","next_token":"FROM"},{"prefix":"SELECT id, user_id, amount, status FROM ","completion":"transactions FINAL SAMPLE 1000 WHERE amount > 100 ORDER BY amount DESC LIMIT 10","next_token":"transactions"},{"prefix":"SELECT id, user_id, amount, status FROM transactions ","completion":"FINAL SAMPLE 1000 WHERE amount > 100 ORDER BY amount DESC LIMIT 10","next_token":"FINAL"},{"prefix":"SELECT id, user_id, amount, status FROM transactions FINAL ","completion":"SAMPLE 1000 WHERE amount > 100 ORDER BY amount DESC LIMIT 10","next_token":"SAMPLE"},{"prefix":"SELECT id, user_id, amount, status FROM transactions FINAL SAMPLE 1000 ","completion":"WHERE amount > 100 ORDER BY amount DESC LIMIT 10","next_token":"WHERE"},{"prefix":"SELECT id, user_id, amount, status FROM transactions FINAL SAMPLE 1000 WHERE amount > 100 ","completion":"ORDER BY amount DESC LIMIT 10","next_token":"ORDER"},{"prefix":"SELECT id, user_id, amount, status FROM transactions FINAL SAMPLE 1000 WHERE amount > 100 ORDER BY amount DESC ","completion":"LIMIT 10","next_token":"LIMIT"}]}
 16|{"id":"traj_016","category":"basic_select","subcategory":"multi_column_orderby","description":"Select with multi-column ORDER BY with mixed sort directions","query":"SELECT id, user_id, status, amount, created_at FROM orders WHERE status = 'pending' ORDER BY user_id ASC, amount DESC, created_at ASC LIMIT 200","tables":["orders"],"columns":{"orders":["id","user_id","amount","status","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, user_id, status, amount, created_at FROM orders WHERE status = 'pending' ORDER BY user_id ASC, amount DESC, created_at ASC LIMIT 200","next_token":"id"},{"prefix":"SELECT id, user_id, status, amount, created_at ","completion":"FROM orders WHERE status = 'pending' ORDER BY user_id ASC, amount DESC, created_at ASC LIMIT 200","next_token":"FROM"},{"prefix":"SELECT id, user_id, status, amount, created_at FROM ","completion":"orders WHERE status = 'pending' ORDER BY user_id ASC, amount DESC, created_at ASC LIMIT 200","next_token":"orders"},{"prefix":"SELECT id, user_id, status, amount, created_at FROM orders ","completion":"WHERE status = 'pending' ORDER BY user_id ASC, amount DESC, created_at ASC LIMIT 200","next_token":"WHERE"},{"prefix":"SELECT id, user_id, status, amount, created_at FROM orders WHERE status = 'pending' ","completion":"ORDER BY user_id ASC, amount DESC, created_at ASC LIMIT 200","next_token":"ORDER"},{"prefix":"SELECT id, user_id, status, amount, created_at FROM orders WHERE status = 'pending' ORDER BY user_id ASC, amount DESC, created_at ASC ","completion":"LIMIT 200","next_token":"LIMIT"}]}
 17|{"id":"traj_017","category":"basic_select","subcategory":"column_aliases","description":"Select with column aliases and simple WHERE","query":"SELECT id AS order_id, user_id AS customer, amount AS total FROM orders WHERE amount > 500 ORDER BY total DESC LIMIT 30","tables":["orders"],"columns":{"orders":["id","user_id","amount","status","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id AS order_id, user_id AS customer, amount AS total FROM orders WHERE amount > 500 ORDER BY total DESC LIMIT 30","next_token":"id"},{"prefix":"SELECT id AS order_id, user_id AS customer, amount AS total ","completion":"FROM orders WHERE amount > 500 ORDER BY total DESC LIMIT 30","next_token":"FROM"},{"prefix":"SELECT id AS order_id, user_id AS customer, amount AS total FROM ","completion":"orders WHERE amount > 500 ORDER BY total DESC LIMIT 30","next_token":"orders"},{"prefix":"SELECT id AS order_id, user_id AS customer, amount AS total FROM orders ","completion":"WHERE amount > 500 ORDER BY total DESC LIMIT 30","next_token":"WHERE"},{"prefix":"SELECT id AS order_id, user_id AS customer, amount AS total FROM orders WHERE amount > 500 ","completion":"ORDER BY total DESC LIMIT 30","next_token":"ORDER"},{"prefix":"SELECT id AS order_id, user_id AS customer, amount AS total FROM orders WHERE amount > 500 ORDER BY total DESC ","completion":"LIMIT 30","next_token":"LIMIT"}]}
 18|{"id":"traj_018","category":"basic_select","subcategory":"not_in_subquery","description":"Select with NOT IN subquery in WHERE clause","query":"SELECT id, user_id, url, timestamp FROM pageviews WHERE user_id NOT IN (SELECT id FROM users WHERE status = 'banned') ORDER BY timestamp DESC LIMIT 100","tables":["pageviews","users"],"columns":{"pageviews":["id","user_id","url","timestamp","duration"],"users":["id","username","email","status","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, user_id, url, timestamp FROM pageviews WHERE user_id NOT IN (SELECT id FROM users WHERE status = 'banned') ORDER BY timestamp DESC LIMIT 100","next_token":"id"},{"prefix":"SELECT id, user_id, url, timestamp ","completion":"FROM pageviews WHERE user_id NOT IN (SELECT id FROM users WHERE status = 'banned') ORDER BY timestamp DESC LIMIT 100","next_token":"FROM"},{"prefix":"SELECT id, user_id, url, timestamp FROM ","completion":"pageviews WHERE user_id NOT IN (SELECT id FROM users WHERE status = 'banned') ORDER BY timestamp DESC LIMIT 100","next_token":"pageviews"},{"prefix":"SELECT id, user_id, url, timestamp FROM pageviews ","completion":"WHERE user_id NOT IN (SELECT id FROM users WHERE status = 'banned') ORDER BY timestamp DESC LIMIT 100","next_token":"WHERE"},{"prefix":"SELECT id, user_id, url, timestamp FROM pageviews WHERE ","completion":"user_id NOT IN (SELECT id FROM users WHERE status = 'banned') ORDER BY timestamp DESC LIMIT 100","next_token":"user_id"},{"prefix":"SELECT id, user_id, url, timestamp FROM pageviews WHERE user_id NOT IN (SELECT id FROM users WHERE status = 'banned') ","completion":"ORDER BY timestamp DESC LIMIT 100","next_token":"ORDER"},{"prefix":"SELECT id, user_id, url, timestamp FROM pageviews WHERE user_id NOT IN (SELECT id FROM users WHERE status = 'banned') ORDER BY timestamp DESC ","completion":"LIMIT 100","next_token":"LIMIT"}]}
 19|{"id":"traj_019","category":"basic_select","subcategory":"settings_clause","description":"Select with SETTINGS clause for query tuning","query":"SELECT id, event_type, user_id, timestamp FROM events WHERE event_type = 'purchase' ORDER BY timestamp DESC LIMIT 500 SETTINGS max_threads = 4","tables":["events"],"columns":{"events":["id","event_type","user_id","timestamp","data"]},"dictionaries":[],"settings":{"max_threads":"4"},"completion_points":[{"prefix":"SELECT ","completion":"id, event_type, user_id, timestamp FROM events WHERE event_type = 'purchase' ORDER BY timestamp DESC LIMIT 500 SETTINGS max_threads = 4","next_token":"id"},{"prefix":"SELECT id, event_type, user_id, timestamp ","completion":"FROM events WHERE event_type = 'purchase' ORDER BY timestamp DESC LIMIT 500 SETTINGS max_threads = 4","next_token":"FROM"},{"prefix":"SELECT id, event_type, user_id, timestamp FROM ","completion":"events WHERE event_type = 'purchase' ORDER BY timestamp DESC LIMIT 500 SETTINGS max_threads = 4","next_token":"events"},{"prefix":"SELECT id, event_type, user_id, timestamp FROM events ","completion":"WHERE event_type = 'purchase' ORDER BY timestamp DESC LIMIT 500 SETTINGS max_threads = 4","next_token":"WHERE"},{"prefix":"SELECT id, event_type, user_id, timestamp FROM events WHERE event_type = 'purchase' ","completion":"ORDER BY timestamp DESC LIMIT 500 SETTINGS max_threads = 4","next_token":"ORDER"},{"prefix":"SELECT id, event_type, user_id, timestamp FROM events WHERE event_type = 'purchase' ORDER BY timestamp DESC ","completion":"LIMIT 500 SETTINGS max_threads = 4","next_token":"LIMIT"},{"prefix":"SELECT id, event_type, user_id, timestamp FROM events WHERE event_type = 'purchase' ORDER BY timestamp DESC LIMIT 500 ","completion":"SETTINGS max_threads = 4","next_token":"SETTINGS"}]}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_02_aggregation.jsonl" start_line="15" end_line="19" total_lines="20">
 15|{"id":"traj_035","category":"aggregation","subcategory":"array_aggregate_combinators","description":"Array aggregate combinators sumArrayIf and groupArrayIf with arrayJoin","query":"SELECT product_id, sumArrayIf(quantity_array, is_active = 1) AS total_active_qty, avgArrayIf(price_array, is_active = 1) AS avg_active_price, groupArrayIf(tag, is_featured = 1) AS featured_tags, countArrayIf(quantity_array, is_active = 1) AS active_count FROM products WHERE created_at >= '2024-01-01' GROUP BY product_id HAVING active_count > 0 ORDER BY total_active_qty DESC LIMIT 100","tables":["products"],"columns":{"products":["product_id","quantity_array","price_array","tag","is_active","is_featured","created_at","category"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"product_id, sumArrayIf(quantity_array, is_active = 1) AS total_active_qty, avgArrayIf(price_array, is_active = 1) AS avg_active_price, groupArrayIf(tag, is_featured = 1) AS featured_tags, countArrayIf(quantity_array, is_active = 1) AS active_count ","next_token":"product_id"},{"prefix":"SELECT product_id, sumArrayIf(quantity_array, is_active = 1) AS total_active_qty, avgArrayIf(price_array, is_active = 1) AS avg_active_price, groupArrayIf(tag, is_featured = 1) AS featured_tags, countArrayIf(quantity_array, is_active = 1) AS active_count FROM ","completion":"products ","next_token":"products"},{"prefix":"SELECT product_id, sumArrayIf(quantity_array, is_active = 1) AS total_active_qty, avgArrayIf(price_array, is_active = 1) AS avg_active_price, groupArrayIf(tag, is_featured = 1) AS featured_tags, countArrayIf(quantity_array, is_active = 1) AS active_count FROM products WHERE ","completion":"created_at >= '2024-01-01' ","next_token":"created_at"},{"prefix":"SELECT product_id, sumArrayIf(quantity_array, is_active = 1) AS total_active_qty, avgArrayIf(price_array, is_active = 1) AS avg_active_price, groupArrayIf(tag, is_featured = 1) AS featured_tags, countArrayIf(quantity_array, is_... [truncated 984 chars]
 16|{"id":"traj_036","category":"aggregation","subcategory":"window_lag_lead","description":"Window functions with lag and lead for period-over-period comparison","query":"SELECT month, product_category, sum(revenue) AS monthly_revenue, lag(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS prev_month_revenue, lead(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS next_month_revenue, sum(revenue) - lag(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS mom_change FROM sales WHERE sale_date >= '2023-01-01' GROUP BY month, product_category ORDER BY product_category, month","tables":["sales"],"columns":{"sales":["id","sale_date","revenue","product_category","region","customer_id","quantity","cost"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"month, product_category, sum(revenue) AS monthly_revenue, lag(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS prev_month_revenue, lead(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS next_month_revenue, sum(revenue) - lag(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS mom_change ","next_token":"month"},{"prefix":"SELECT month, product_category, sum(revenue) AS monthly_revenue, lag(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS prev_month_revenue, lead(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS next_month_revenue, sum(revenue) - lag(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS mom_change FROM ","completion":"sales ","next_token":"sales"},{"prefix":"SELECT month, product_category, sum(revenue) AS monthly_revenue, lag(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS prev_month_revenue, lead(sum(revenue), 1) OVER (PARTITION BY product_category ORDER BY month) AS next_month_revenue, sum(revenue) - lag(sum(revenue), 1) OVER (PARTITION BY product_category ORDER B... [truncated 1129 chars]
 17|{"id":"traj_037","category":"aggregation","subcategory":"group_by_date_functions","description":"GROUP BY with date functions and time bucketing for hourly metrics","query":"SELECT toStartOfHour(timestamp) AS hour_bucket, service_name, count() AS request_count, avg(latency_ms) AS avg_latency, quantile(0.95)(latency_ms) AS p95_latency, quantile(0.99)(latency_ms) AS p99_latency, max(latency_ms) AS max_latency FROM metrics WHERE timestamp >= '2024-07-20 00:00:00' AND timestamp < '2024-07-21 00:00:00' GROUP BY hour_bucket, service_name HAVING avg_latency > 100 ORDER BY hour_bucket, service_name","tables":["metrics"],"columns":{"metrics":["id","timestamp","service_name","latency_ms","status_code","endpoint","host"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"toStartOfHour(timestamp) AS hour_bucket, service_name, count() AS request_count, avg(latency_ms) AS avg_latency, quantile(0.95)(latency_ms) AS p95_latency, quantile(0.99)(latency_ms) AS p99_latency, max(latency_ms) AS max_latency ","next_token":"toStartOfHour"},{"prefix":"SELECT toStartOfHour(timestamp) AS hour_bucket, service_name, count() AS request_count, avg(latency_ms) AS avg_latency, quantile(0.95)(latency_ms) AS p95_latency, quantile(0.99)(latency_ms) AS p99_latency, max(latency_ms) AS max_latency FROM ","completion":"metrics ","next_token":"metrics"},{"prefix":"SELECT toStartOfHour(timestamp) AS hour_bucket, service_name, count() AS request_count, avg(latency_ms) AS avg_latency, quantile(0.95)(latency_ms) AS p95_latency, quantile(0.99)(latency_ms) AS p99_latency, max(latency_ms) AS max_latency FROM metrics WHERE ","completion":"timestamp >= '2024-07-20 00:00:00' AND timestamp < '2024-07-21 00:00:00' ","next_token":"timestamp"},{"prefix":"SELECT toStartOfHour(timestamp) AS hour_bucket, service_name, count() AS request_count, avg(latency_ms) AS avg_latency, quantile(0.95)(latency_ms) AS p95_latency, quantile(0.99)(latency_ms) AS p99_latency, max(latency_ms) AS max_lat... [truncated 1106 chars]
 18|{"id":"traj_038","category":"aggregation","subcategory":"group_by_having_complex","description":"Complex HAVING with aggregate function conditions and OR logic","query":"SELECT store_id, product_category, count() AS sales_count, sum(amount) AS total_amount, avg(amount) AS avg_amount, min(amount) AS min_amount, max(amount) AS max_amount FROM transactions WHERE txn_date >= '2024-01-01' GROUP BY store_id, product_category HAVING (sales_count > 50 AND avg_amount > 25) OR total_amount > 10000 ORDER BY total_amount DESC, store_id LIMIT 200","tables":["transactions"],"columns":{"transactions":["id","store_id","product_category","amount","txn_date","payment_method","customer_id"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"store_id, product_category, count() AS sales_count, sum(amount) AS total_amount, avg(amount) AS avg_amount, min(amount) AS min_amount, max(amount) AS max_amount ","next_token":"store_id"},{"prefix":"SELECT store_id, product_category, count() AS sales_count, sum(amount) AS total_amount, avg(amount) AS avg_amount, min(amount) AS min_amount, max(amount) AS max_amount FROM ","completion":"transactions ","next_token":"transactions"},{"prefix":"SELECT store_id, product_category, count() AS sales_count, sum(amount) AS total_amount, avg(amount) AS avg_amount, min(amount) AS min_amount, max(amount) AS max_amount FROM transactions WHERE ","completion":"txn_date >= '2024-01-01' ","next_token":"txn_date"},{"prefix":"SELECT store_id, product_category, count() AS sales_count, sum(amount) AS total_amount, avg(amount) AS avg_amount, min(amount) AS min_amount, max(amount) AS max_amount FROM transactions WHERE txn_date >= '2024-01-01' GROUP BY ","completion":"store_id, product_category ","next_token":"store_id"},{"prefix":"SELECT store_id, product_category, count() AS sales_count, sum(amount) AS total_amount, avg(amount) AS avg_amount, min(amount) AS min_amount, max(amount) AS max_amount FROM transactions WHERE txn_date >= '2024-... [truncated 568 chars]
 19|{"id":"traj_039","category":"aggregation","subcategory":"window_dense_rank_percentile","description":"Window functions with DENSE_RANK and percentile ranking for sales performance","query":"SELECT sales_rep_id, region, quarter, sum(deal_amount) AS total_deals, DENSE_RANK() OVER (PARTITION BY region ORDER BY sum(deal_amount) DESC) AS regional_rank, RANK() OVER (PARTITION BY quarter ORDER BY sum(deal_amount) DESC) AS quarterly_rank, sum(deal_amount) / sum(sum(deal_amount)) OVER (PARTITION BY region) AS region_share FROM deals WHERE deal_date >= '2024-01-01' AND deal_status = 'won' GROUP BY sales_rep_id, region, quarter ORDER BY region, regional_rank","tables":["deals"],"columns":{"deals":["id","sales_rep_id","region","quarter","deal_amount","deal_date","deal_status","customer_id"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"sales_rep_id, region, quarter, sum(deal_amount) AS total_deals, DENSE_RANK() OVER (PARTITION BY region ORDER BY sum(deal_amount) DESC) AS regional_rank, RANK() OVER (PARTITION BY quarter ORDER BY sum(deal_amount) DESC) AS quarterly_rank, sum(deal_amount) / sum(sum(deal_amount)) OVER (PARTITION BY region) AS region_share ","next_token":"sales_rep_id"},{"prefix":"SELECT sales_rep_id, region, quarter, sum(deal_amount) AS total_deals, DENSE_RANK() OVER (PARTITION BY region ORDER BY sum(deal_amount) DESC) AS regional_rank, RANK() OVER (PARTITION BY quarter ORDER BY sum(deal_amount) DESC) AS quarterly_rank, sum(deal_amount) / sum(sum(deal_amount)) OVER (PARTITION BY region) AS region_share FROM ","completion":"deals ","next_token":"deals"},{"prefix":"SELECT sales_rep_id, region, quarter, sum(deal_amount) AS total_deals, DENSE_RANK() OVER (PARTITION BY region ORDER BY sum(deal_amount) DESC) AS regional_rank, RANK() OVER (PARTITION BY quarter ORDER BY sum(deal_amount) DESC) AS quarterly_rank, sum(deal_amount) / sum(sum(deal_amount)) OVER (PARTITION BY region) AS region_share FROM deals WHERE ","completion":"deal_... [truncated 1085 chars]
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_03_joins.jsonl" start_line="15" end_line="19" total_lines="20">
 15|{"id":"traj_055","category":"joins","subcategory":"global_join","description":"GLOBAL JOIN across distributed tables for users and their pageviews","query":"SELECT u.user_id, u.username, count() AS pv_count FROM users AS u GLOBAL INNER JOIN pageviews AS pv ON u.user_id = pv.user_id WHERE pv.view_date = '2024-06-01' GROUP BY u.user_id, u.username ORDER BY pv_count DESC LIMIT 100","tables":["users","pageviews"],"columns":{"users":["user_id","username","email","is_active","created_at"],"pageviews":["view_id","user_id","view_date","url","duration_seconds"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"u.user_id, u.username, count() AS pv_count ","next_token":"u"},{"prefix":"SELECT u.user_id, u.username, count() AS pv_count FROM ","completion":"users AS u ","next_token":"users"},{"prefix":"SELECT u.user_id, u.username, count() AS pv_count FROM users AS u GLOBAL INNER JOIN ","completion":"pageviews AS pv ON u.user_id = pv.user_id ","next_token":"pageviews"},{"prefix":"SELECT u.user_id, u.username, count() AS pv_count FROM users AS u GLOBAL INNER JOIN pageviews AS pv ON u.user_id = pv.user_id WHERE ","completion":"pv.view_date = '2024-06-01' ","next_token":"pv"},{"prefix":"SELECT u.user_id, u.username, count() AS pv_count FROM users AS u GLOBAL INNER JOIN pageviews AS pv ON u.user_id = pv.user_id WHERE pv.view_date = '2024-06-01' GROUP BY ","completion":"u.user_id, u.username ","next_token":"u"},{"prefix":"SELECT u.user_id, u.username, count() AS pv_count FROM users AS u GLOBAL INNER JOIN pageviews AS pv ON u.user_id = pv.user_id WHERE pv.view_date = '2024-06-01' GROUP BY u.user_id, u.username ORDER BY ","completion":"pv_count DESC ","next_token":"pv_count"}]}
 16|{"id":"traj_056","category":"joins","subcategory":"multiple_cte_joins","description":"Multiple CTEs with joins: active users, their orders, and product details","query":"WITH active_users AS (SELECT user_id, username FROM users WHERE is_active = 1), user_orders AS (SELECT o.id, o.customer_id AS user_id, o.total_amount, o.order_date FROM orders AS o WHERE o.status = 'completed') SELECT au.username, uo.id AS order_id, uo.total_amount, uo.order_date, p.name AS product_name FROM active_users AS au INNER JOIN user_orders AS uo USING (user_id) INNER JOIN products AS p ON uo.id = p.product_id ORDER BY uo.order_date DESC LIMIT 100","tables":["users","orders","products"],"columns":{"users":["user_id","username","email","is_active","created_at"],"orders":["id","customer_id","order_date","status","total_amount","product_id"],"products":["product_id","name","category","price","stock"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"WITH ","completion":"active_users AS (SELECT user_id, username FROM users WHERE is_active = 1), user_orders AS (SELECT o.id, o.customer_id AS user_id, o.total_amount, o.order_date FROM orders AS o WHERE o.status = 'completed') ","next_token":"active_users"},{"prefix":"WITH active_users AS (SELECT user_id, username FROM users WHERE is_active = 1), user_orders AS (SELECT o.id, o.customer_id AS user_id, o.total_amount, o.order_date FROM orders AS o WHERE o.status = 'completed') SELECT ","completion":"au.username, uo.id AS order_id, uo.total_amount, uo.order_date, p.name AS product_name ","next_token":"au"},{"prefix":"WITH active_users AS (SELECT user_id, username FROM users WHERE is_active = 1), user_orders AS (SELECT o.id, o.customer_id AS user_id, o.total_amount, o.order_date FROM orders AS o WHERE o.status = 'completed') SELECT au.username, uo.id AS order_id, uo.total_amount, uo.order_date, p.name AS product_name FROM ","completion":"active_users AS au ","next_token":"active_users"},{"prefix":"WITH active_users AS (SELECT user_id, us... [truncated 1357 chars]
 17|{"id":"traj_057","category":"joins","subcategory":"array_join_with_left_join","description":"ARRAY JOIN combined with LEFT JOIN to expand event properties and join user info","query":"SELECT e.event_id, e.event_time, prop_key, u.username FROM events AS e ARRAY JOIN mapKeys(e.properties) AS prop_key LEFT JOIN users AS u ON e.user_id = u.user_id WHERE e.event_date = '2024-06-01' ORDER BY e.event_time, prop_key LIMIT 500","tables":["events","users"],"columns":{"events":["event_id","user_id","event_time","event_date","event_type","properties"],"users":["user_id","username","email","is_active","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"e.event_id, e.event_time, prop_key, u.username ","next_token":"e"},{"prefix":"SELECT e.event_id, e.event_time, prop_key, u.username FROM ","completion":"events AS e ","next_token":"events"},{"prefix":"SELECT e.event_id, e.event_time, prop_key, u.username FROM events AS e ARRAY JOIN ","completion":"mapKeys(e.properties) AS prop_key ","next_token":"mapKeys"},{"prefix":"SELECT e.event_id, e.event_time, prop_key, u.username FROM events AS e ARRAY JOIN mapKeys(e.properties) AS prop_key LEFT JOIN ","completion":"users AS u ON e.user_id = u.user_id ","next_token":"users"},{"prefix":"SELECT e.event_id, e.event_time, prop_key, u.username FROM events AS e ARRAY JOIN mapKeys(e.properties) AS prop_key LEFT JOIN users AS u ON e.user_id = u.user_id WHERE ","completion":"e.event_date = '2024-06-01' ","next_token":"e"},{"prefix":"SELECT e.event_id, e.event_time, prop_key, u.username FROM events AS e ARRAY JOIN mapKeys(e.properties) AS prop_key LEFT JOIN users AS u ON e.user_id = u.user_id WHERE e.event_date = '2024-06-01' ORDER BY ","completion":"e.event_time, prop_key ","next_token":"e"}]}
 18|{"id":"traj_058","category":"joins","subcategory":"subquery_in_join","description":"Subquery as the right side of a join with aggregated pageview counts","query":"SELECT u.user_id, u.username, pv.pv_count, pv.last_view FROM users AS u LEFT JOIN (SELECT user_id, count() AS pv_count, max(view_date) AS last_view FROM pageviews GROUP BY user_id) AS pv ON u.user_id = pv.user_id WHERE u.is_active = 1 ORDER BY pv.pv_count DESC NULLS LAST LIMIT 100","tables":["users","pageviews"],"columns":{"users":["user_id","username","email","is_active","created_at"],"pageviews":["view_id","user_id","view_date","url","duration_seconds"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"u.user_id, u.username, pv.pv_count, pv.last_view ","next_token":"u"},{"prefix":"SELECT u.user_id, u.username, pv.pv_count, pv.last_view FROM ","completion":"users AS u ","next_token":"users"},{"prefix":"SELECT u.user_id, u.username, pv.pv_count, pv.last_view FROM users AS u LEFT JOIN ","completion":"(SELECT user_id, count() AS pv_count, max(view_date) AS last_view FROM pageviews GROUP BY user_id) AS pv ON u.user_id = pv.user_id ","next_token":"(SELECT"},{"prefix":"SELECT u.user_id, u.username, pv.pv_count, pv.last_view FROM users AS u LEFT JOIN (SELECT user_id, count() AS pv_count, max(view_date) AS last_view FROM pageviews GROUP BY user_id) AS pv ON u.user_id = pv.user_id WHERE ","completion":"u.is_active = 1 ","next_token":"u"},{"prefix":"SELECT u.user_id, u.username, pv.pv_count, pv.last_view FROM users AS u LEFT JOIN (SELECT user_id, count() AS pv_count, max(view_date) AS last_view FROM pageviews GROUP BY user_id) AS pv ON u.user_id = pv.user_id WHERE u.is_active = 1 ORDER BY ","completion":"pv.pv_count DESC NULLS LAST ","next_token":"pv"}]}
 19|{"id":"traj_059","category":"joins","subcategory":"correlated_subquery_join","description":"Correlated subquery in SELECT with a join between customers and orders","query":"SELECT c.customer_id, c.name, (SELECT sum(o.total_amount) FROM orders AS o WHERE o.customer_id = c.customer_id AND o.status = 'completed') AS lifetime_value FROM customers AS c INNER JOIN transactions AS t ON c.customer_id = t.customer_id WHERE t.transaction_date >= '2024-01-01' ORDER BY lifetime_value DESC LIMIT 100","tables":["customers","orders","transactions"],"columns":{"customers":["customer_id","name","email","country","signup_date"],"orders":["id","customer_id","order_date","status","total_amount"],"transactions":["transaction_id","customer_id","amount","currency","transaction_date","status"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"c.customer_id, c.name, (SELECT sum(o.total_amount) FROM orders AS o WHERE o.customer_id = c.customer_id AND o.status = 'completed') AS lifetime_value ","next_token":"c"},{"prefix":"SELECT c.customer_id, c.name, (SELECT sum(o.total_amount) FROM orders AS o WHERE o.customer_id = c.customer_id AND o.status = 'completed') AS lifetime_value FROM ","completion":"customers AS c ","next_token":"customers"},{"prefix":"SELECT c.customer_id, c.name, (SELECT sum(o.total_amount) FROM orders AS o WHERE o.customer_id = c.customer_id AND o.status = 'completed') AS lifetime_value FROM customers AS c INNER JOIN ","completion":"transactions AS t ON c.customer_id = t.customer_id ","next_token":"transactions"},{"prefix":"SELECT c.customer_id, c.name, (SELECT sum(o.total_amount) FROM orders AS o WHERE o.customer_id = c.customer_id AND o.status = 'completed') AS lifetime_value FROM customers AS c INNER JOIN transactions AS t ON c.customer_id = t.customer_id WHERE ","completion":"t.transaction_date >= '2024-01-01' ","next_token":"t"},{"prefix":"SELECT c.customer_id, c.name, (SELECT sum(o.total_amount) FROM orders AS o WHERE o.customer_id... [truncated 265 chars]
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_04_ddl_dml.jsonl" start_line="15" end_line="19" total_lines="20">
 15|{"id":"traj_075","category":"ddl_dml","subcategory":"alter_table_constraint","description":"ALTER TABLE add check constraint and drop an existing constraint","query":"ALTER TABLE shop.products ADD CONSTRAINT chk_price CHECK price >= 0, DROP CONSTRAINT IF EXISTS chk_old_price","tables":["shop.products"],"columns":{"shop.products":["product_id","sku","name","price","stock","updated_at","version"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"ALTER TABLE ","completion":"shop.products ","next_token":"shop"},{"prefix":"ALTER TABLE shop.products ADD CONSTRAINT ","completion":"chk_price ","next_token":"chk_price"},{"prefix":"ALTER TABLE shop.products ADD CONSTRAINT chk_price ","completion":"CHECK price >= 0, ","next_token":"CHECK"},{"prefix":"ALTER TABLE shop.products ADD CONSTRAINT ..., DROP CONSTRAINT IF EXISTS ","completion":"chk_old_price","next_token":"chk_old_price"}]}
 16|{"id":"traj_076","category":"ddl_dml","subcategory":"alter_table_update","description":"ALTER TABLE lightweight UPDATE with multiple columns and WHERE filter","query":"ALTER TABLE shop.products UPDATE stock = stock + 100, version = version + 1 WHERE product_id = 42","tables":["shop.products"],"columns":{"shop.products":["product_id","sku","name","price","stock","updated_at","version"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"ALTER TABLE ","completion":"shop.products ","next_token":"shop"},{"prefix":"ALTER TABLE shop.products UPDATE ","completion":"stock = stock + 100, ","next_token":"stock"},{"prefix":"ALTER TABLE shop.products UPDATE stock = stock + 100, ","completion":"version = version + 1 ","next_token":"version"},{"prefix":"ALTER TABLE shop.products UPDATE stock = stock + 100, version = version + 1 WHERE ","completion":"product_id = 42","next_token":"product_id"}]}
 17|{"id":"traj_077","category":"ddl_dml","subcategory":"insert_values","description":"INSERT INTO table with explicit column list and multiple VALUES rows","query":"INSERT INTO shop.products (product_id, sku, name, price, stock, updated_at, version) VALUES (1, 'SKU-001', 'Widget', 19.99, 500, '2024-01-15 10:30:00', 1), (2, 'SKU-002', 'Gadget', 29.99, 250, '2024-01-15 10:31:00', 1)","tables":["shop.products"],"columns":{"shop.products":["product_id","sku","name","price","stock","updated_at","version"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"INSERT INTO ","completion":"shop.products ","next_token":"shop"},{"prefix":"INSERT INTO shop.products ","completion":"(product_id, sku, name, price, stock, updated_at, version) ","next_token":"("},{"prefix":"INSERT INTO shop.products (...) VALUES ","completion":"(1, 'SKU-001', 'Widget', 19.99, 500, '2024-01-15 10:30:00', 1), ","next_token":"("},{"prefix":"INSERT INTO shop.products (...) VALUES (1, ...), ","completion":"(2, 'SKU-002', 'Gadget', 29.99, 250, '2024-01-15 10:31:00', 1)","next_token":"("}]}
 18|{"id":"traj_078","category":"ddl_dml","subcategory":"insert_select","description":"INSERT INTO target table from SELECT with filtering and transformation","query":"INSERT INTO analytics.agg_visits (visit_date, site_id, uniq_users, total_duration) SELECT toStartOfMonth(visit_time) AS visit_date, site_id, uniqState(user_id) AS uniq_users, sumState(duration) AS total_duration FROM analytics.raw_visits WHERE visit_time >= '2024-01-01 00:00:00' GROUP BY visit_date, site_id","tables":["analytics.agg_visits","analytics.raw_visits"],"columns":{"analytics.agg_visits":["visit_date","site_id","uniq_users","total_duration"],"analytics.raw_visits":["visit_time","site_id","user_id","duration"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"INSERT INTO ","completion":"analytics.agg_visits ","next_token":"analytics"},{"prefix":"INSERT INTO analytics.agg_visits ","completion":"(visit_date, site_id, uniq_users, total_duration) ","next_token":"("},{"prefix":"INSERT INTO analytics.agg_visits (...) SELECT ","completion":"toStartOfMonth(visit_time) AS visit_date, site_id, uniqState(user_id) AS uniq_users, sumState(duration) AS total_duration ","next_token":"toStartOfMonth"},{"prefix":"INSERT INTO analytics.agg_visits (...) SELECT ... FROM ","completion":"analytics.raw_visits ","next_token":"analytics"},{"prefix":"INSERT INTO analytics.agg_visits (...) SELECT ... FROM analytics.raw_visits WHERE ","completion":"visit_time >= '2024-01-01 00:00:00' ","next_token":"visit_time"},{"prefix":"INSERT INTO analytics.agg_visits (...) SELECT ... FROM analytics.raw_visits WHERE visit_time >= '2024-01-01 00:00:00' GROUP BY ","completion":"visit_date, site_id","next_token":"visit_date"}]}
 19|{"id":"traj_079","category":"ddl_dml","subcategory":"insert_format","description":"INSERT INTO table using JSONEachRow format with inline JSON data","query":"INSERT INTO analytics.events FORMAT JSONEachRow {\"event_id\":\"550e8400-e29b-41d4-a716-446655440000\",\"user_id\":12345,\"event_type\":\"click\",\"event_data\":\"{\\\"button\\\":\\\"signup\\\"}\",\"event_time\":\"2024-01-15 10:30:00\"} {\"event_id\":\"550e8400-e29b-41d4-a716-446655440001\",\"user_id\":12346,\"event_type\":\"view\",\"event_data\":\"{\\\"page\\\":\\\"home\\\"}\",\"event_time\":\"2024-01-15 10:31:00\"}","tables":["analytics.events"],"columns":{"analytics.events":["event_id","user_id","event_type","event_data","event_time","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"INSERT INTO ","completion":"analytics.events ","next_token":"analytics"},{"prefix":"INSERT INTO analytics.events FORMAT ","completion":"JSONEachRow ","next_token":"JSONEachRow"},{"prefix":"INSERT INTO analytics.events FORMAT JSONEachRow ","completion":"{\"event_id\":\"550e8400-e29b-41d4-a716-446655440000\",\"user_id\":12345,\"event_type\":\"click\",\"event_data\":\"{\\\"button\\\":\\\"signup\\\"}\",\"event_time\":\"2024-01-15 10:30:00\"} ","next_token":"{"},{"prefix":"INSERT INTO analytics.events FORMAT JSONEachRow {\"event_id\":...} ","completion":"{\"event_id\":\"550e8400-e29b-41d4-a716-446655440001\",\"user_id\":12346,\"event_type\":\"view\",\"event_data\":\"{\\\"page\\\":\\\"home\\\"}\",\"event_time\":\"2024-01-15 10:31:00\"}","next_token":"{"}]}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl" start_line="15" end_line="19" total_lines="20">
 15|{"id":"traj_095","category":"advanced","subcategory":"cte_expression","description":"CTE with WITH expression AS identifier for reusable scalar calculations","query":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING total > avg_rounded ORDER BY total DESC LIMIT 100","tables":["customer_payments"],"columns":{"customer_payments":["payment_id","customer_id","amount","payment_date","method"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"WITH ","completion":"round(avg_amount, 2) AS avg_rounded ","next_token":"round"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT ","completion":"customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total ","next_token":"customer_id"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM ","completion":"customer_payments ","next_token":"customer_payments"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY ","completion":"customer_id ","next_token":"customer_id"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING ","completion":"total > avg_rounded ","next_token":"total"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING total > avg_rounded ORDER BY ","completion":"total DESC ","next_token":"total"},{"prefix":"WITH round(avg_amount, 2) AS avg_rounded SELECT customer_id, avg(amount) AS avg_amount, avg_rounded, sum(amount) AS total FROM customer_payments GROUP BY customer_id HAVING total > avg_rounded ORDER BY total DE... [truncated 51 chars]
 16|{"id":"traj_096","category":"advanced","subcategory":"cte_subquery","description":"CTE with WITH subquery AS identifier for reusable subquery results","query":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT s.region, s.store_id, sum(s.revenue) AS store_total FROM sales_data s INNER JOIN top_regions t ON s.region = t.region GROUP BY s.region, s.store_id ORDER BY s.region, store_total DESC","tables":["sales_data"],"columns":{"sales_data":["sale_id","region","store_id","revenue","sale_date"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"WITH ","completion":"top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) ","next_token":"top_regions"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT ","completion":"s.region, s.store_id, sum(s.revenue) AS store_total ","next_token":"s"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT s.region, s.store_id, sum(s.revenue) AS store_total FROM ","completion":"sales_data s ","next_token":"sales_data"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT s.region, s.store_id, sum(s.revenue) AS store_total FROM sales_data s INNER JOIN ","completion":"top_regions t ON s.region = t.region ","next_token":"top_regions"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP BY region ORDER BY total DESC LIMIT 10) SELECT s.region, s.store_id, sum(s.revenue) AS store_total FROM sales_data s INNER JOIN top_regions t ON s.region = t.region GROUP BY ","completion":"s.region, s.store_id ","next_token":"s"},{"prefix":"WITH top_regions AS (SELECT region, sum(revenue) AS total FROM sales_data GROUP B... [truncated 266 chars]
 17|{"id":"traj_097","category":"advanced","subcategory":"window_frame","description":"Window function with explicit frame specification ROWS BETWEEN for running totals","query":"SELECT order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, avg(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_avg FROM store_revenue ORDER BY store_id, order_date","tables":["store_revenue"],"columns":{"store_revenue":["store_id","order_date","revenue","region"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, ","next_token":"order_date"},{"prefix":"SELECT order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, ","completion":"avg(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_avg ","next_token":"avg"},{"prefix":"SELECT order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, avg(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_avg FROM ","completion":"store_revenue ","next_token":"store_revenue"},{"prefix":"SELECT order_date, store_id, revenue, sum(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, avg(revenue) OVER (PARTITION BY store_id ORDER BY order_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_avg FROM store_revenue ORDER BY ","completion":"store_id, order_date","next_token":"store_id"}]}
 18|{"id":"traj_098","category":"advanced","subcategory":"materialized_alias_columns","description":"Query referencing MATERIALIZED and ALIAS columns with type conversion functions","query":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM purchase_records WHERE order_date >= toDate('2024-01-01') ORDER BY order_date DESC LIMIT 500","tables":["purchase_records"],"columns":{"purchase_records":["order_id","quantity","unit_price","order_date","computed_total","customer_label"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date ","next_token":"order_id"},{"prefix":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM ","completion":"purchase_records ","next_token":"purchase_records"},{"prefix":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM purchase_records WHERE ","completion":"order_date >= toDate('2024-01-01') ","next_token":"order_date"},{"prefix":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM purchase_records WHERE order_date >= toDate('2024-01-01') ORDER BY ","completion":"order_date DESC ","next_token":"order_date"},{"prefix":"SELECT order_id, toUInt32(quantity * unit_price) AS total_uint, toString(order_date) AS date_str, formatDateTime(order_date, '%Y-%m-%d') AS formatted_date FROM purchase_records WHERE order_date >= toDate('2024-01-01') ORDER BY order_date DESC LIMIT ","completion":"500","next_token":"500"}]}
 19|{"id":"traj_099","category":"advanced","subcategory":"with_fill_qualify_groupby_all","description":"WITH FILL, GROUP BY ALL, and QUALIFY clause combined for gap-filling and ranked filtering","query":"SELECT sale_date, store_id, sum(revenue) AS total FROM store_revenue WHERE sale_date BETWEEN '2024-01-01' AND '2024-01-31' GROUP BY ALL ORDER BY store_id, sale_date WITH FILL STEP INTERVAL 1 DAY LIMIT 1000","tables":["store_revenue"],"columns":{"store_revenue":["store_id","sale_date","revenue","region"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"sale_date, store_id, sum(revenue) AS total ","next_token":"sale_date"},{"prefix":"SELECT sale_date, store_id, sum(revenue) AS total FROM ","completion":"store_revenue ","next_token":"store_revenue"},{"prefix":"SELECT sale_date, store_id, sum(revenue) AS total FROM store_revenue WHERE ","completion":"sale_date BETWEEN '2024-01-01' AND '2024-01-31' ","next_token":"sale_date"},{"prefix":"SELECT sale_date, store_id, sum(revenue) AS total FROM store_revenue WHERE sale_date BETWEEN '2024-01-01' AND '2024-01-31' GROUP BY ","completion":"ALL ","next_token":"ALL"},{"prefix":"SELECT sale_date, store_id, sum(revenue) AS total FROM store_revenue WHERE sale_date BETWEEN '2024-01-01' AND '2024-01-31' GROUP BY ALL ORDER BY ","completion":"store_id, sale_date WITH FILL STEP INTERVAL 1 DAY ","next_token":"store_id"},{"prefix":"SELECT sale_date, store_id, sum(revenue) AS total FROM store_revenue WHERE sale_date BETWEEN '2024-01-01' AND '2024-01-31' GROUP BY ALL ORDER BY store_id, sale_date WITH FILL STEP INTERVAL 1 DAY LIMIT ","completion":"1000","next_token":"1000"}]}
</file-view>

/Users/root1/Downloads/projects/clickhouse-github/README.md
/Users/root1/Downloads/projects/clickhouse-github/checkpoints/best.pt
/Users/root1/Downloads/projects/clickhouse-github/checkpoints/ch_autocomplete.pt
/Users/root1/Downloads/projects/clickhouse-github/checkpoints/ch_autocomplete_weights.bin
/Users/root1/Downloads/projects/clickhouse-github/checkpoints/final.pt
/Users/root1/Downloads/projects/clickhouse-github/export/CMakeLists.txt
/Users/root1/Downloads/projects/clickhouse-github/export/build/chai_test
/Users/root1/Downloads/projects/clickhouse-github/export/build_wasm.sh
/Users/root1/Downloads/projects/clickhouse-github/export/ch_autocomplete.bin
/Users/root1/Downloads/projects/clickhouse-github/export/ch_inference.h
/Users/root1/Downloads/projects/clickhouse-github/export/ch_model.h
/Users/root1/Downloads/projects/clickhouse-github/export/ch_vocab.h
/Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py
/Users/root1/Downloads/projects/clickhouse-github/export/test_main.cpp
/Users/root1/Downloads/projects/clickhouse-github/inference/__pycache__/autocomplete.cpython-312.pyc
/Users/root1/Downloads/projects/clickhouse-github/inference/autocomplete.py
/Users/root1/Downloads/projects/clickhouse-github/model/__pycache__/config.cpython-312.pyc
/Users/root1/Downloads/projects/clickhouse-github/model/__pycache__/training.cpython-312.pyc
/Users/root1/Downloads/projects/clickhouse-github/model/config.py
/Users/root1/Downloads/projects/clickhouse-github/model/training.py
/Users/root1/Downloads/projects/clickhouse-github/test_autocomplete.py
/Users/root1/Downloads/projects/clickhouse-github/tokenizer/__pycache__/ch_tokenizer.cpython-312.pyc
/Users/root1/Downloads/projects/clickhouse-github/tokenizer/ch_tokenizer.py
/Users/root1/Downloads/projects/clickhouse-github/tokenizer/vocab.json
/Users/root1/Downloads/projects/clickhouse-github/train.py
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_01_basic_select.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_02_aggregation.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_03_joins.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_04_ddl_dml.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/trajectory_format_spec.md

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/model/config.py" start_line="1" end_line="329" total_lines="329">
  1|"""
  2|GPT-based Small Language Model for ClickHouse query autocomplete.
  3|
  4|Design goals:
  5|  - ~10 MB model size (float32) → ~2.7M parameters
  6|  - CPU-optimized: small dimensions, weight tying, pre-LayerNorm
  7|  - Embeddable in C++ (ClickHouse binary) and exportable to WebAssembly
  8|  - Constrained sampling at inference time (syntactic correctness + table dicts)
  9|
 10|Architecture:
 11|  - d_model = 192, n_layers = 6, n_heads = 6, d_ff = 768
 12|  - max_seq_len = 256
 13|  - Weight tying (input embedding == output projection)
 14|  - Pre-LayerNorm transformer blocks
 15|  - Learned positional embeddings
 16|  - GELU activation in FFN
 17|  - ~2.77M params ≈ 10.6 MB (float32), ~5.3 MB (float16), ~2.7 MB (int8)
 18|"""
 19|
 20|import math
 21|import torch
 22|import torch.nn as nn
 23|import torch.nn.functional as F
 24|from dataclasses import dataclass, asdict
 25|from typing import Optional
 26|
 27|import sys
 28|sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent / "tokenizer"))
 29|from ch_tokenizer import VOCAB_SIZE, PAD_ID, BOS_ID, EOS_ID
 30|
 31|
 32|@dataclass
 33|class ModelConfig:
 34|    """Configuration for the ClickHouse autocomplete GPT model."""
 35|    vocab_size: int = VOCAB_SIZE
 36|    d_model: int = 192
 37|    n_layers: int = 6
 38|    n_heads: int = 6
 39|    d_ff: int = 768
 40|    max_seq_len: int = 256
 41|    n_heads_kv: int = 6       # no GQA (d_model is small)
 42|    dropout: float = 0.1
 43|    weight_tying: bool = True
 44|    pad_token_id: int = PAD_ID
 45|    bos_token_id: int = BOS_ID
 46|    eos_token_id: int = EOS_ID
 47|
 48|    def param_count(self) -> int:
 49|        """Estimate total parameter count."""
 50|        embed = self.vocab_size * self.d_model
 51|        pos = self.max_seq_len * self.d_model
 52|        per_layer = 12 * self.d_model * self.d_model + 4 * self.d_model  # approx
 53|        layers = self.n_layers * per_layer
 54|        output = 0 if self.weight_tying else self.vocab_size * self.d_model
 55|        return embed + pos + layers + output
 56|
 57|    def model_size_mb(self, bytes_per_param: int = 4) -> float:
 58|        return self.param_count() * bytes_per_param / (1024 * 1024)
 59|
 60|    def to_dict(self) -> dict:
 61|        return asdict(self)
 62|
 63|    @classmethod
 64|    def from_dict(cls, d: dict) -> "ModelConfig":
 65|        return cls(**d)
 66|
 67|
 68|class CausalSelfAttention(nn.Module):
 69|    """Multi-head causal self-attention with pre-LayerNorm."""
 70|
 71|    def __init__(self, config: ModelConfig):
 72|        super().__init__()
 73|        assert config.d_model % config.n_heads == 0
 74|        self.n_heads = config.n_heads
 75|        self.d_head = config.d_model // config.n_heads
 76|        self.d_model = config.d_model
 77|        self.scale = 1.0 / math.sqrt(self.d_head)
 78|
 79|        # Fused QKV projection
 80|        self.qkv = nn.Linear(config.d_model, 3 * config.d_model, bias=False)
 81|        self.proj = nn.Linear(config.d_model, config.d_model, bias=False)
 82|        self.dropout = config.dropout
 83|
 84|    def forward(self, x: torch.Tensor) -> torch.Tensor:
 85|        B, T, C = x.shape
 86|        qkv = self.qkv(x)  # (B, T, 3*C)
 87|        qkv = qkv.view(B, T, 3, self.n_heads, self.d_head)
 88|        qkv = qkv.permute(2, 0, 3, 1, 4)  # (3, B, n_heads, T, d_head)
 89|        q, k, v = qkv[0], qkv[1], qkv[2]
 90|
 91|        # Scaled dot-product attention with causal mask
 92|        att = torch.matmul(q, k.transpose(-2, -1)) * self.scale  # (B, n_heads, T, T)
 93|        # Causal mask
 94|        mask = torch.triu(torch.ones(T, T, device=x.device, dtype=torch.bool), diagonal=1)
 95|        att = att.masked_fill(mask, float("-inf"))
 96|        att = F.softmax(att, dim=-1)
 97|        if self.dropout > 0 and self.training:
 98|            att = F.dropout(att, p=self.dropout)
 99|
100|        out = torch.matmul(att, v)  # (B, n_heads, T, d_head)
101|        out = out.transpose(1, 2).contiguous().view(B, T, C)
102|        out = self.proj(out)
103|        if self.dropout > 0 and self.training:
104|            out = F.dropout(out, p=self.dropout)
105|        return out
106|
107|
108|class FeedForward(nn.Module):
109|    """Position-wise feed-forward network with GELU."""
110|
111|    def __init__(self, config: ModelConfig):
112|        super().__init__()
113|        self.fc1 = nn.Linear(config.d_model, config.d_ff, bias=False)
114|        self.fc2 = nn.Linear(config.d_ff, config.d_model, bias=False)
115|        self.dropout = config.dropout
116|
117|    def forward(self, x: torch.Tensor) -> torch.Tensor:
118|        x = self.fc1(x)
119|        x = F.gelu(x)
120|        if self.dropout > 0 and self.training:
121|            x = F.dropout(x, p=self.dropout)
122|        x = self.fc2(x)
123|        if self.dropout > 0 and self.training:
124|            x = F.dropout(x, p=self.dropout)
125|        return x
126|
127|
128|class TransformerBlock(nn.Module):
129|    """Pre-LayerNorm transformer block."""
130|
131|    def __init__(self, config: ModelConfig):
132|        super().__init__()
133|        self.ln1 = nn.LayerNorm(config.d_model, bias=False)
134|        self.attn = CausalSelfAttention(config)
135|        self.ln2 = nn.LayerNorm(config.d_model, bias=False)
136|        self.ffn = FeedForward(config)
137|
138|    def forward(self, x: torch.Tensor) -> torch.Tensor:
139|        x = x + self.attn(self.ln1(x))
140|        x = x + self.ffn(self.ln2(x))
141|        return x
142|
143|
144|class ClickHouseGPT(nn.Module):
145|    """GPT model for ClickHouse query autocomplete.
146|
147|    Architecture:
148|      - Token embedding + learned positional embedding
149|      - N pre-LayerNorm transformer blocks
150|      - Final LayerNorm
151|      - Output projection (tied with token embedding if weight_tying=True)
152|    """
153|
154|    def __init__(self, config: ModelConfig):
155|        super().__init__()
156|        self.config = config
157|        self.tok_emb = nn.Embedding(config.vocab_size, config.d_model, padding_idx=config.pad_token_id)
158|        self.pos_emb = nn.Embedding(config.max_seq_len, config.d_model)
159|        self.blocks = nn.ModuleList([TransformerBlock(config) for _ in range(config.n_layers)])
160|        self.ln_f = nn.LayerNorm(config.d_model, bias=False)
161|        self.dropout = config.dropout
162|
163|        if config.weight_tying:
164|            # Tie output projection weights to token embedding
165|            self.lm_head = None  # use tok_emb.weight in forward
166|        else:
167|            self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)
168|
169|        # Initialize weights
170|        self.apply(self._init_weights)
171|
172|    def _init_weights(self, module: nn.Module):
173|        if isinstance(module, nn.Linear):
174|            nn.init.normal_(module.weight, mean=0.0, std=0.02)
175|            if module.bias is not None:
176|                nn.init.zeros_(module.bias)
177|        elif isinstance(module, nn.Embedding):
178|            nn.init.normal_(module.weight, mean=0.0, std=0.02)
179|            if module.padding_idx is not None:
180|                with torch.no_grad():
181|                    module.weight[module.padding_idx].fill_(0)
182|
183|    def forward(self, idx: torch.Tensor, targets: Optional[torch.Tensor] = None) -> dict:
184|        """
185|        Args:
186|            idx: (B, T) token IDs
187|            targets: (B, T) target token IDs for loss computation
188|
189|        Returns:
190|            dict with 'logits' (B, T, V) and optionally 'loss'
191|        """
192|        B, T = idx.shape
193|        assert T <= self.config.max_seq_len, f"Sequence length {T} exceeds max {self.config.max_seq_len}"
194|
195|        # Embeddings
196|        tok = self.tok_emb(idx)  # (B, T, d_model)
197|        pos = self.pos_emb(torch.arange(T, device=idx.device).unsqueeze(0))  # (1, T, d_model)
198|        x = tok + pos
199|        if self.dropout > 0 and self.training:
200|            x = F.dropout(x, p=self.dropout)
201|
202|        # Transformer blocks
203|        for block in self.blocks:
204|            x = block(x)
205|
206|        x = self.ln_f(x)  # (B, T, d_model)
207|
208|        # Output projection
209|        if self.config.weight_tying:
210|            logits = x @ self.tok_emb.weight.T  # (B, T, vocab_size)
211|        else:
212|            logits = self.lm_head(x)
213|
214|        result = {"logits": logits}
215|
216|        if targets is not None:
217|            # Cross-entropy loss, ignoring padding
218|            loss = F.cross_entropy(
219|                logits.view(-1, logits.size(-1)),
220|                targets.view(-1),
221|                ignore_index=self.config.pad_token_id,
222|            )
223|            result["loss"] = loss
224|
225|        return result
226|
227|    @torch.no_grad()
228|    def generate(
229|        self,
230|        idx: torch.Tensor,
231|        max_new_tokens: int,
232|        temperature: float = 1.0,
233|        top_k: int = 0,
234|        top_p: float = 0.0,
235|        eos_token_id: Optional[int] = None,
236|    ) -> torch.Tensor:
237|        """Autoregressive generation with optional top-k / top-p sampling.
238|
239|        Args:
240|            idx: (1, T) starting token IDs (the prefix)
241|            max_new_tokens: maximum number of tokens to generate
242|            temperature: sampling temperature (1.0 = no change, <1 = sharper)
243|            top_k: if > 0, only sample from top-k logits
244|            top_p: if > 0, use nucleus sampling
245|            eos_token_id: if set, stop generation when this token is produced
246|
247|        Returns:
248|            (1, T + max_new_tokens) token IDs
249|        """
250|        self.eval()
251|        eos = eos_token_id if eos_token_id is not None else self.config.eos_token_id
252|
253|        for _ in range(max_new_tokens):
254|            # Crop context if too long
255|            idx_cond = idx if idx.size(1) <= self.config.max_seq_len else idx[:, -self.config.max_seq_len:]
256|
257|            logits = self(idx_cond)["logits"]  # (1, T, V)
258|            logits = logits[:, -1, :] / max(temperature, 1e-8)  # (1, V)
259|
260|            # Apply top-k filtering
261|            if top_k > 0:
262|                v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
263|                logits[logits < v[:, [-1]]] = float("-inf")
264|
265|            # Apply top-p (nucleus) filtering
266|            if top_p > 0.0:
267|                sorted_logits, sorted_indices = torch.sort(logits, descending=True)
268|                cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
269|                sorted_indices_to_remove = cumulative_probs > top_p
270|                sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
271|                sorted_indices_to_remove[..., 0] = False
272|                indices_to_remove = sorted_indices_to_remove.scatter(
273|                    1, sorted_indices, sorted_indices_to_remove
274|                )
275|                logits = logits.masked_fill(indices_to_remove, float("-inf"))
276|
277|            probs = F.softmax(logits, dim=-1)
278|            next_token = torch.multinomial(probs, num_samples=1)  # (1, 1)
279|            idx = torch.cat([idx, next_token], dim=1)
280|
281|            if eos is not None and next_token.item() == eos:
282|                break
283|
284|        return idx
285|
286|    def num_parameters(self) -> int:
287|        return sum(p.numel() for p in self.parameters())
288|
289|    def model_size_mb(self, bytes_per_param: int = 4) -> float:
290|        return self.num_parameters() * bytes_per_param / (1024 * 1024)
291|
292|
293|def create_model(config: Optional[ModelConfig] = None) -> ClickHouseGPT:
294|    """Create a ClickHouseGPT model with the given or default config."""
295|    if config is None:
296|        config = ModelConfig()
297|    model = ClickHouseGPT(config)
298|    return model
299|
300|
301|if __name__ == "__main__":
302|    config = ModelConfig()
303|    model = create_model(config)
304|    print(f"Model config: {config.to_dict()}")
305|    print(f"Parameters: {model.num_parameters():,}")
306|    print(f"Model size (float32): {model.model_size_mb(4):.2f} MB")
307|    print(f"Model size (float16): {model.model_size_mb(2):.2f} MB")
308|    print(f"Model size (int8):    {model.model_size_mb(1):.2f} MB")
309|    print(f"Estimated params:     {config.param_count():,}")
310|    print(f"Estimated size (f32): {config.model_size_mb(4):.2f} MB")
311|
312|    # Quick forward pass test
313|    import sys
314|    sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent / "tokenizer"))
315|    from ch_tokenizer import tokenize_to_ids
316|
317|    query = "SELECT count(), avg(price) FROM orders WHERE region = 'US' GROUP BY city ORDER BY count() DESC LIMIT 10"
318|    ids = tokenize_to_ids(query)
319|    print(f"\nTokenized query ({len(ids)} tokens): {ids[:10]}...")
320|
321|    x = torch.tensor([ids])
322|    out = model(x, targets=x)
323|    print(f"Logits shape: {out['logits'].shape}")
324|    print(f"Loss: {out['loss'].item():.4f}")
325|
326|    # Generation test
327|    prefix = torch.tensor([[BOS_ID]])
328|    generated = model.generate(prefix, max_new_tokens=20, temperature=0.8, top_k=10)
329|    print(f"Generated {generated.size(1)} tokens")
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/tokenizer/ch_tokenizer.py" start_line="1" end_line="514" total_lines="514">
  1|"""
  2|ClickHouse Lexer-based tokenizer with lookbehind copies.
  3|
  4|This tokenizer mirrors the ClickHouse Lexer (src/Parsers/Lexer.h) token types
  5|but replaces identifier and string-literal values with "lookbehind copy" tokens.
  6|
  7|Instead of encoding identifier/string values literally (which would require an
  8|unbounded vocabulary), each identifier or string is represented as either:
  9|  - NEW_IDENT / NEW_STRING  — first occurrence in the sequence
 10|  - COPY_BACK_N / COPY_STR_BACK_N — a repeat of a value seen N tokens ago
 11|
 12|This keeps the vocabulary small (~350 tokens) while allowing the model to
 13|reproduce arbitrary identifiers and strings via a pointer/copy mechanism at
 14|inference time.
 15|
 16|The tokenizer also stores the raw text for each identifier/string so that the
 17|inference engine can resolve copy references.
 18|"""
 19|
 20|import re
 21|import json
 22|from dataclasses import dataclass, field
 23|from typing import Optional
 24|
 25|
 26|# ─── ClickHouse keywords (from utils/antlr/ClickHouseLexer.g4) ───────────────
 27|
 28|KEYWORDS = [
 29|    "ACCESS", "ADD", "AFTER", "ALIAS", "ALL", "ALTER", "AND", "ANTI", "ANY",
 30|    "ARRAY", "AS", "ASC", "ASCENDING", "ASOF", "AST", "ASYNC", "ATTACH",
 31|    "BETWEEN", "BOTH", "BY", "CACHES", "CASE", "CAST", "CHECK", "CLEAR",
 32|    "CLUSTER", "CLUSTERS", "CODEC", "COLLATE", "COLUMN", "COLUMNS", "COMMENT",
 33|    "CONSTRAINT", "CREATE", "CROSS", "CUBE", "CURRENT", "CURRENT_USER",
 34|    "CHANGED", "DATABASE", "DATABASES", "DATE", "DAY", "DEDUPLICATE",
 35|    "DEFAULT", "DELAY", "DELETE", "DESC", "DESCENDING", "DESCRIBE", "DETACH",
 36|    "DICTIONARIES", "DICTIONARY", "DISK", "DISTINCT", "DISTRIBUTED", "DROP",
 37|    "ELSE", "ENABLED", "END", "ENGINE", "ENGINES", "ESTIMATE", "EVENTS",
 38|    "EXCEPT", "EXISTS", "EXPLAIN", "EXPRESSION", "EXTENDED", "EXTRACT",
 39|    "FETCHES", "FIELDS", "FILESYSTEM", "FILL", "FINAL", "FIRST", "FLUSH",
 40|    "FOLLOWING", "FOR", "FORMAT", "FREEZE", "FROM", "FULL", "FUNCTION",
 41|    "FUNCTIONS", "GLOBAL", "GRANULARITY", "GRANTS", "GROUP", "GROUPING",
 42|    "HAVING", "HIERARCHICAL", "HOUR", "ID", "IF", "ILIKE", "IMPLICIT", "IN",
 43|    "INDEX", "INDEXES", "INDICES", "INF", "INFINITY", "INJECTIVE", "INNER",
 44|    "INSERT", "INTERPOLATE", "INTERVAL", "INTO", "IS", "IS_OBJECT_ID", "JOIN",
 45|    "KEY", "KEYS", "KILL", "LAST", "LAYOUT", "LEADING", "LEFT", "LIFETIME",
 46|    "LIKE", "LIMIT", "LIVE", "LOCAL", "LOGS", "MATERIALIZE", "MATERIALIZED",
 47|    "MAX", "MERGES", "MICROSECOND", "MILLISECOND", "MIN", "MINUTE", "MODIFY",
 48|    "MONTH", "MOVE", "MUTATION", "NAN", "NANOSECOND", "NO", "NOT", "NULL",
 49|    "NULLS", "OFFSET", "ON", "OPTIMIZE", "OR", "ORDER", "OUTER", "OUTFILE",
 50|    "OVER", "OVERRIDE", "PARTITION", "PIPELINE", "PLAN", "POLICY", "POLICIES",
 51|    "POPULATE", "PRECEDING", "PREWHERE", "PRIMARY", "PRIVILEGES",
 52|    "PROCESSLIST", "PROFILE", "PROFILES", "PROJECTION", "QUARTER", "QUOTA",
 53|    "QUOTAS", "RANGE", "RECURSIVE", "RELOAD", "REMOVE", "RENAME", "REPLACE",
 54|    "REPLICA", "REPLICATED", "RIGHT", "ROLE", "ROLES", "ROLLUP", "ROW",
 55|    "ROWS", "SAMPLE", "SECOND", "SELECT", "SEMI", "SENDS", "SET", "SETS",
 56|    "SETTING", "SETTINGS", "SHOW", "SOURCE", "START", "STOP", "SUBSTRING",
 57|    "SYNC", "SYNTAX", "SYSTEM", "STEP", "TABLE", "TABLES", "TEMPORARY",
 58|    "TEST", "THEN", "TIES", "TIMEOUT", "TIMESTAMP", "TO", "TOP", "TOTALS",
 59|    "TRAILING", "TREE", "TRIM", "TRUNCATE", "TTL", "TYPE", "UNBOUNDED",
 60|    "UNION", "UPDATE", "USE", "USER", "USERS", "USING", "UUID", "VALUES",
 61|    "VIEW", "VOLUME", "WATCH", "WEEK", "WHEN", "WHERE", "WINDOW", "WITH",
 62|    "YEAR", "YYYY", "FALSE", "TRUE",
 63|]
 64|
 65|# Build a case-insensitive lookup set (ClickHouse keywords are case-insensitive)
 66|KEYWORD_SET = {kw.upper() for kw in KEYWORDS}
 67|
 68|
 69|# ─── Token type enum (mirrors ClickHouse TokenType) ──────────────────────────
 70|
 71|# Significant token types from the ClickHouse lexer that we map to vocab entries
 72|SIGNIFICANT_TOKEN_TYPES = [
 73|    "BareWord", "Number", "StringLiteral", "QuotedIdentifier",
 74|    "OpeningRoundBracket", "ClosingRoundBracket",
 75|    "OpeningSquareBracket", "ClosingSquareBracket",
 76|    "OpeningCurlyBrace", "ClosingCurlyBrace",
 77|    "Comma", "Semicolon", "Dot", "Asterisk",
 78|    "Plus", "Minus", "Slash", "Percent",
 79|    "Arrow", "QuestionMark", "Colon", "Caret", "DoubleColon",
 80|    "Equals", "NotEquals", "Less", "Greater",
 81|    "LessOrEquals", "GreaterOrEquals", "Spaceship",
 82|    "Concatenation", "At", "DoubleAt",
 83|]
 84|
 85|# ─── Vocabulary construction ─────────────────────────────────────────────────
 86|
 87|MAX_COPY_BACK = 32  # maximum lookbehind distance for copy tokens
 88|
 89|SPECIAL_TOKENS = ["<PAD>", "<BOS>", "<EOS>", "<UNK>"]
 90|
 91|# Operator/punctuation tokens with their string representations
 92|OPERATOR_TOKENS = {
 93|    "LPAREN": "(", "RPAREN": ")",
 94|    "LBRACKET": "[", "RBRACKET": "]",
 95|    "LBRACE": "{", "RBRACE": "}",
 96|    "COMMA": ",", "SEMICOLON": ";",
 97|    "DOT": ".", "ASTERISK": "*",
 98|    "PLUS": "+", "MINUS": "-",
 99|    "SLASH": "/", "PERCENT": "%",
100|    "ARROW": "->",
101|    "QUESTION": "?", "COLON": ":",
102|    "CARET": "^", "DOUBLE_COLON": "::",
103|    "EQUALS": "=", "NOT_EQUALS": "!=",
104|    "LESS": "<", "GREATER": ">",
105|    "LESS_OR_EQUALS": "<=", "GREATER_OR_EQUALS": ">=",
106|    "SPACESHIP": "<=>",
107|    "CONCAT": "||",
108|    "AT": "@", "DOUBLE_AT": "@@",
109|}
110|
111|NUMBER_TOKENS = ["NUM_INT", "NUM_FLOAT", "NUM_HEX"]
112|
113|# Identifier copy tokens
114|IDENT_COPY_TOKENS = ["NEW_IDENT"] + [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK + 1)]
115|
116|# String copy tokens
117|STRING_COPY_TOKENS = ["NEW_STRING"] + [f"COPY_STR_BACK_{i}" for i in range(1, MAX_COPY_BACK + 1)]
118|
119|
120|def build_vocab():
121|    """Build the vocabulary mapping token string -> integer id."""
122|    vocab = {}
123|    idx = 0
124|    for tok in SPECIAL_TOKENS:
125|        vocab[tok] = idx
126|        idx += 1
127|    for kw in KEYWORDS:
128|        vocab[f"KW_{kw}"] = idx
129|        idx += 1
130|    for tok in OPERATOR_TOKENS:
131|        vocab[tok] = idx
132|        idx += 1
133|    for tok in NUMBER_TOKENS:
134|        vocab[tok] = idx
135|        idx += 1
136|    for tok in IDENT_COPY_TOKENS:
137|        vocab[tok] = idx
138|        idx += 1
139|    for tok in STRING_COPY_TOKENS:
140|        vocab[tok] = idx
141|        idx += 1
142|    return vocab
143|
144|
145|VOCAB = build_vocab()
146|VOCAB_SIZE = len(VOCAB)
147|ID_TO_TOKEN = {v: k for k, v in VOCAB.items()}
148|
149|# Special token IDs
150|PAD_ID = VOCAB["<PAD>"]
151|BOS_ID = VOCAB["<BOS>"]
152|EOS_ID = VOCAB["<EOS>"]
153|UNK_ID = VOCAB["<UNK>"]
154|
155|
156|# ─── Lexer (Python re-based, mirrors ClickHouse Lexer.cpp) ───────────────────
157|
158|# Token patterns in priority order
159|_TOKEN_PATTERNS = [
160|    ("Whitespace",      r'[ \t\r\n]+'),
161|    ("Comment",         r'--[^\n\r]*|/\*.*?\*/'),
162|    ("HereDoc",         r"\$\$\w*\$\$"),
163|    # String literal: 'text' with '' escape and \' escape
164|    ("StringLiteral",   r"'(?:[^'\\]|\\.|'')*'"),
165|    # Quoted identifier: `text` or "text"
166|    ("QuotedIdentifier",r'`(?:[^`\\]|\\.|``)*`|"(?:[^"\\]|\\.|"")*"'),
167|    # Hex string literal: X'...'
168|    ("HexStringLiteral",r"X'[0-9a-fA-F]+'"),
169|    # Binary string literal: B'...'
170|    ("BinaryStringLit", r"B'[01]+'"),
171|    # Numbers: hex (0x...), binary (0b...), octal (0o...), float, int
172|    ("Number",          r'0[xX][0-9a-fA-F]+(?:[pP][+-]?[0-9]+)?'
173|                        r'|0[bB][01]+'
174|                        r'|0[oO][0-7]+'
175|                        r'|\d+\.\d*(?:[eE][+-]?\d+)?'
176|                        r'|\.\d+(?:[eE][+-]?\d+)?'
177|                        r'|\d+(?:[eE][+-]?\d+)?'),
178|    # BareWord: identifier or keyword
179|    ("BareWord",        r'[a-zA-Z_][a-zA-Z_0-9]*'),
180|    # Multi-char operators (must come before single-char)
181|    ("DoubleColon",     r'::'),
182|    ("Arrow",           r'->'),
183|    ("Spaceship",       r'<=>'),
184|    ("Concatenation",   r'\|\|'),
185|    ("DoubleAt",        r'@@'),
186|    ("NotEquals",       r'!=|<>'),
187|    ("LessOrEquals",    r'<='),
188|    ("GreaterOrEquals", r'>='),
189|    ("Equals",          r'==|='),  # ClickHouse accepts both = and ==
190|    # Single-char operators
191|    ("OpeningRoundBracket", r'\('),
192|    ("ClosingRoundBracket", r'\)'),
193|    ("OpeningSquareBracket", r'\['),
194|    ("ClosingSquareBracket", r'\]'),
195|    ("OpeningCurlyBrace", r'\{'),
196|    ("ClosingCurlyBrace", r'\}'),
197|    ("Comma",           r','),
198|    ("Semicolon",       r';'),
199|    ("Dot",             r'\.'),
200|    ("Asterisk",        r'\*'),
201|    ("Plus",            r'\+'),
202|    ("Minus",           r'-'),
203|    ("Slash",           r'/'),
204|    ("Percent",         r'%'),
205|    ("QuestionMark",    r'\?'),
206|    ("Colon",           r':'),
207|    ("Caret",           r'\^'),
208|    ("At",              r'@'),
209|]
210|
211|_COMPILED_PATTERNS = [(name, re.compile(pat, re.DOTALL)) for name, pat in _TOKEN_PATTERNS]
212|
213|
214|@dataclass
215|class LexToken:
216|    """A raw lexed token (before lookbehind encoding)."""
217|    type: str        # ClickHouse TokenType name
218|    text: str        # raw text
219|    start: int       # character offset in source
220|    end: int         # character offset (exclusive)
221|
222|
223|def lex(query: str) -> list[LexToken]:
224|    """Lex a ClickHouse query string into raw tokens (mirrors ClickHouse Lexer).
225|
226|    Whitespace and comments are skipped (matching isSignificant() in the C++ lexer).
227|    """
228|    tokens: list[LexToken] = []
229|    pos = 0
230|    n = len(query)
231|    while pos < n:
232|        matched = False
233|        for ttype, pattern in _COMPILED_PATTERNS:
234|            m = pattern.match(query, pos)
235|            if m:
236|                text = m.group(0)
237|                if ttype not in ("Whitespace", "Comment"):
238|                    tokens.append(LexToken(ttype, text, pos, m.end()))
239|                pos = m.end()
240|                matched = True
241|                break
242|        if not matched:
243|            # Unrecognized character — skip it (could also raise)
244|            pos += 1
245|    return tokens
246|
247|
248|# ─── Lookbehind copy encoding ────────────────────────────────────────────────
249|
250|@dataclass
251|class EncodedToken:
252|    """A token after lookbehind copy encoding."""
253|    token_id: int           # vocabulary index
254|    token_str: str          # vocabulary token string
255|    raw_text: str           # original text (for identifiers/strings; empty for keywords/ops)
256|    lex_type: str           # original lexer token type
257|    is_ident: bool          # whether this is an identifier (for copy resolution)
258|    is_string: bool         # whether this is a string literal (for copy resolution)
259|
260|
261|def _classify_number(text: str) -> str:
262|    """Classify a number token as NUM_INT, NUM_FLOAT, or NUM_HEX."""
263|    lower = text.lower()
264|    if lower.startswith("0x"):
265|        return "NUM_HEX"
266|    if "." in text or "e" in lower or "p" in lower:
267|        return "NUM_FLOAT"
268|    return "NUM_INT"
269|
270|
271|def encode_tokens(lex_tokens: list[LexToken]) -> list[EncodedToken]:
272|    """Encode raw lexed tokens into vocabulary tokens with lookbehind copies.
273|
274|    For identifiers (BareWord that's not a keyword, QuotedIdentifier):
275|      - First occurrence → NEW_IDENT
276|      - Repeat within MAX_COPY_BACK distance → COPY_BACK_N
277|      - Repeat beyond MAX_COPY_BACK → NEW_IDENT (treated as new)
278|
279|    For string literals:
280|      - First occurrence → NEW_STRING
281|      - Repeat within MAX_COPY_BACK distance → COPY_STR_BACK_N
282|      - Repeat beyond MAX_COPY_BACK → NEW_STRING
283|
284|    The raw_text is preserved so the inference engine can resolve copies.
285|    """
286|    result: list[EncodedToken] = []
287|    # Track recent identifier texts and their positions (token index in result)
288|    ident_history: list[tuple[str, int]] = []  # (text_lower, token_index)
289|    string_history: list[tuple[str, int]] = []  # (text, token_index)
290|
291|    for lt in lex_tokens:
292|        et = EncodedToken(
293|            token_id=UNK_ID, token_str="<UNK>", raw_text="",
294|            lex_type=lt.type, is_ident=False, is_string=False,
295|        )
296|
297|        if lt.type == "BareWord":
298|            upper = lt.text.upper()
299|            if upper in KEYWORD_SET:
300|                kw_tok = f"KW_{upper}"
301|                et.token_id = VOCAB.get(kw_tok, UNK_ID)
302|                et.token_str = kw_tok
303|            else:
304|                # Identifier — lookbehind copy encoding
305|                lower = lt.text.lower()
306|                copy_dist = _find_copy_distance(ident_history, lower, len(result))
307|                if copy_dist is not None and copy_dist <= MAX_COPY_BACK:
308|                    tok_str = f"COPY_BACK_{copy_dist}"
309|                else:
310|                    tok_str = "NEW_IDENT"
311|                et.token_id = VOCAB[tok_str]
312|                et.token_str = tok_str
313|                et.raw_text = lt.text
314|                et.is_ident = True
315|                ident_history.append((lower, len(result)))
316|        elif lt.type == "QuotedIdentifier":
317|            # Quoted identifier — strip quotes for comparison
318|            inner = lt.text
319|            if inner.startswith("`") and inner.endswith("`"):
320|                inner = inner[1:-1].replace("``", "`")
321|            elif inner.startswith('"') and inner.endswith('"'):
322|                inner = inner[1:-1].replace('""', '"')
323|            lower = inner.lower()
324|            copy_dist = _find_copy_distance(ident_history, lower, len(result))
325|            if copy_dist is not None and copy_dist <= MAX_COPY_BACK:
326|                tok_str = f"COPY_BACK_{copy_dist}"
327|            else:
328|                tok_str = "NEW_IDENT"
329|            et.token_id = VOCAB[tok_str]
330|            et.token_str = tok_str
331|            et.raw_text = lt.text
332|            et.is_ident = True
333|            ident_history.append((lower, len(result)))
334|        elif lt.type == "StringLiteral":
335|            # String literal — strip quotes for comparison
336|            inner = lt.text[1:-1].replace("''", "'")
337|            copy_dist = _find_copy_distance(string_history, inner, len(result))
338|            if copy_dist is not None and copy_dist <= MAX_COPY_BACK:
339|                tok_str = f"COPY_STR_BACK_{copy_dist}"
340|            else:
341|                tok_str = "NEW_STRING"
342|            et.token_id = VOCAB[tok_str]
343|            et.token_str = tok_str
344|            et.raw_text = lt.text
345|            et.is_string = True
346|            string_history.append((inner, len(result)))
347|        elif lt.type == "Number":
348|            num_type = _classify_number(lt.text)
349|            et.token_id = VOCAB[num_type]
350|            et.token_str = num_type
351|        elif lt.type in ("HexStringLiteral", "BinaryStringLit"):
352|            et.token_id = VOCAB["NEW_STRING"]
353|            et.token_str = "NEW_STRING"
354|            et.raw_text = lt.text
355|            et.is_string = True
356|        elif lt.type == "HereDoc":
357|            et.token_id = VOCAB["NEW_STRING"]
358|            et.token_str = "NEW_STRING"
359|            et.raw_text = lt.text
360|            et.is_string = True
361|        else:
362|            # Operator/punctuation
363|            op_tok = _LEX_TYPE_TO_OP.get(lt.type)
364|            if op_tok is not None:
365|                et.token_id = VOCAB[op_tok]
366|                et.token_str = op_tok
367|            else:
368|                et.token_id = UNK_ID
369|                et.token_str = "<UNK>"
370|
371|        result.append(et)
372|
373|    return result
374|
375|
376|# Map lexer token types to operator vocabulary tokens
377|_LEX_TYPE_TO_OP = {
378|    "OpeningRoundBracket": "LPAREN",
379|    "ClosingRoundBracket": "RPAREN",
380|    "OpeningSquareBracket": "LBRACKET",
381|    "ClosingSquareBracket": "RBRACKET",
382|    "OpeningCurlyBrace": "LBRACE",
383|    "ClosingCurlyBrace": "RBRACE",
384|    "Comma": "COMMA",
385|    "Semicolon": "SEMICOLON",
386|    "Dot": "DOT",
387|    "Asterisk": "ASTERISK",
388|    "Plus": "PLUS",
389|    "Minus": "MINUS",
390|    "Slash": "SLASH",
391|    "Percent": "PERCENT",
392|    "Arrow": "ARROW",
393|    "QuestionMark": "QUESTION",
394|    "Colon": "COLON",
395|    "Caret": "CARET",
396|    "DoubleColon": "DOUBLE_COLON",
397|    "Equals": "EQUALS",
398|    "NotEquals": "NOT_EQUALS",
399|    "Less": "LESS",
400|    "Greater": "GREATER",
401|    "LessOrEquals": "LESS_OR_EQUALS",
402|    "GreaterOrEquals": "GREATER_OR_EQUALS",
403|    "Spaceship": "SPACESHIP",
404|    "Concatenation": "CONCAT",
405|    "At": "AT",
406|    "DoubleAt": "DOUBLE_AT",
407|}
408|
409|
410|def _find_copy_distance(history: list[tuple[str, int]], text: str, current_idx: int) -> Optional[int]:
411|    """Find the most recent occurrence of text in history and return the distance back."""
412|    # Search from most recent backwards
413|    for i in range(len(history) - 1, -1, -1):
414|        hist_text, hist_idx = history[i]
415|        if hist_text == text:
416|            return current_idx - hist_idx
417|    return None
418|
419|
420|# ─── Public API ──────────────────────────────────────────────────────────────
421|
422|def tokenize(query: str) -> list[EncodedToken]:
423|    """Tokenize a ClickHouse query into encoded tokens with lookbehind copies."""
424|    lex_tokens = lex(query)
425|    return encode_tokens(lex_tokens)
426|
427|
428|def tokenize_to_ids(query: str, add_bos: bool = True, add_eos: bool = True) -> list[int]:
429|    """Tokenize a query and return a list of vocabulary token IDs."""
430|    encoded = tokenize(query)
431|    ids = [et.token_id for et in encoded]
432|    if add_bos:
433|        ids = [BOS_ID] + ids
434|    if add_eos:
435|        ids = ids + [EOS_ID]
436|    return ids
437|
438|
439|def detokenize(encoded_tokens: list[EncodedToken]) -> str:
440|    """Reconstruct the original query text from encoded tokens.
441|
442|    This uses the raw_text stored in each EncodedToken for identifiers and strings.
443|    For copy tokens, the raw_text is the actual text (stored at encode time).
444|    """
445|    parts: list[str] = []
446|    for et in encoded_tokens:
447|        if et.is_ident or et.is_string:
448|            parts.append(et.raw_text)
449|        elif et.token_str.startswith("KW_"):
450|            parts.append(et.token_str[3:])
451|        elif et.token_str in NUMBER_TOKENS:
452|            parts.append(et.raw_text if et.raw_text else "0")
453|        elif et.token_str in OPERATOR_TOKENS:
454|            parts.append(OPERATOR_TOKENS[et.token_str])
455|        elif et.token_str in SPECIAL_TOKENS:
456|            continue
457|        else:
458|            parts.append(et.token_str)
459|    # Simple spacing heuristic
460|    return _join_with_spacing(parts)
461|
462|
463|def _join_with_spacing(parts: list[str]) -> str:
464|    """Join token parts with reasonable spacing."""
465|    result = ""
466|    for i, part in enumerate(parts):
467|        if i == 0:
468|            result = part
469|            continue
470|        prev = parts[i - 1]
471|        # No space after opening brackets or before closing brackets/comma/semicolon/dot
472|        no_space_before = part in (")", "]", "}", ",", ";", ".")
473|        no_space_after = prev in ("(", "[", "{", ".")
474|        if no_space_before or no_space_after:
475|            result += part
476|        else:
477|            result += " " + part
478|    return result
479|
480|
481|def save_vocab(path: str):
482|    """Save the vocabulary to a JSON file."""
483|    with open(path, "w") as f:
484|        json.dump({
485|            "vocab": VOCAB,
486|            "vocab_size": VOCAB_SIZE,
487|            "special_tokens": {
488|                "pad": PAD_ID, "bos": BOS_ID, "eos": EOS_ID, "unk": UNK_ID,
489|            },
490|            "max_copy_back": MAX_COPY_BACK,
491|            "keywords": KEYWORDS,
492|            "operators": OPERATOR_TOKENS,
493|            "number_tokens": NUMBER_TOKENS,
494|            "ident_copy_tokens": IDENT_COPY_TOKENS,
495|            "string_copy_tokens": STRING_COPY_TOKENS,
496|        }, f, indent=2)
497|
498|
499|if __name__ == "__main__":
500|    # Quick self-test
501|    test_queries = [
502|        "SELECT count(), avg(price) FROM orders WHERE region = 'US' GROUP BY city ORDER BY count() DESC LIMIT 10",
503|        "SELECT * FROM events WHERE event_type = 'click' AND user_id IN (SELECT user_id FROM users WHERE country = 'US') ORDER BY timestamp DESC LIMIT 100",
504|        "CREATE TABLE IF NOT EXISTS analytics.events (id UUID, event_time DateTime) ENGINE = MergeTree() ORDER BY (event_time)",
505|    ]
506|    for q in test_queries:
507|        print(f"\nQuery: {q}")
508|        encoded = tokenize(q)
509|        for et in encoded:
510|            extra = f" [{et.raw_text}]" if (et.is_ident or et.is_string) else ""
511|            print(f"  {et.token_str}{extra}")
512|        ids = tokenize_to_ids(q)
513|        print(f"  IDs: {ids}")
514|    print(f"\nVocab size: {VOCAB_SIZE}")
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/train.py" start_line="1" end_line="182" total_lines="182">
  1|#!/usr/bin/env python3
  2|"""
  3|Main training script for ClickHouse autocomplete GPT model.
  4|
  5|Usage:
  6|    python3 train.py [--epochs N] [--batch_size N] [--lr LR] [--device cpu|mps|cuda]
  7|"""
  8|
  9|import argparse
 10|import sys
 11|import time
 12|from pathlib import Path
 13|
 14|# Add project paths
 15|PROJECT_ROOT = Path(__file__).resolve().parent
 16|sys.path.insert(0, str(PROJECT_ROOT / "tokenizer"))
 17|sys.path.insert(0, str(PROJECT_ROOT / "model"))
 18|
 19|from ch_tokenizer import save_vocab, VOCAB_SIZE
 20|from config import ModelConfig, ClickHouseGPT, create_model
 21|from training import TrajectoryDataset, Trainer, create_data_loaders, collate_fn
 22|
 23|
 24|def main():
 25|    parser = argparse.ArgumentParser(description="Train ClickHouse autocomplete GPT model")
 26|    parser.add_argument("--epochs", type=int, default=30, help="Number of epochs")
 27|    parser.add_argument("--batch_size", type=int, default=16, help="Batch size")
 28|    parser.add_argument("--lr", type=float, default=3e-4, help="Learning rate")
 29|    parser.add_argument("--device", type=str, default="cpu", help="Device: cpu, mps, or cuda")
 30|    parser.add_argument("--max_seq_len", type=int, default=256, help="Max sequence length")
 31|    parser.add_argument("--d_model", type=int, default=192, help="Model dimension")
 32|    parser.add_argument("--n_layers", type=int, default=6, help="Number of transformer layers")
 33|    parser.add_argument("--n_heads", type=int, default=6, help="Number of attention heads")
 34|    parser.add_argument("--d_ff", type=int, default=768, help="FFN dimension")
 35|    parser.add_argument("--save_dir", type=str, default="checkpoints", help="Checkpoint directory")
 36|    parser.add_argument("--trajectory_dir", type=str, default="trajectories/data",
 37|                        help="Directory with trajectory JSONL files")
 38|    args = parser.parse_args()
 39|
 40|    # Save vocabulary
 41|    vocab_path = PROJECT_ROOT / "tokenizer" / "vocab.json"
 42|    save_vocab(str(vocab_path))
 43|    print(f"Vocabulary saved ({VOCAB_SIZE} tokens) to {vocab_path}")
 44|
 45|    # Create model config
 46|    config = ModelConfig(
 47|        vocab_size=VOCAB_SIZE,
 48|        d_model=args.d_model,
 49|        n_layers=args.n_layers,
 50|        n_heads=args.n_heads,
 51|        d_ff=args.d_ff,
 52|        max_seq_len=args.max_seq_len,
 53|    )
 54|
 55|    model = create_model(config)
 56|    print(f"\nModel: {config}")
 57|    print(f"Parameters: {model.num_parameters():,}")
 58|    print(f"Model size (float32): {model.model_size_mb(4):.2f} MB")
 59|    print(f"Model size (float16): {model.model_size_mb(2):.2f} MB")
 60|
 61|    # Determine device
 62|    device = args.device
 63|    if device == "mps" and not hasattr(torch.backends, "mps"):
 64|        device = "cpu"
 65|    if device == "cuda" and not torch.cuda.is_available():
 66|        device = "cpu"
 67|    print(f"Device: {device}")
 68|
 69|    # Create datasets
 70|    traj_dir = str(PROJECT_ROOT / args.trajectory_dir)
 71|    print(f"\nLoading trajectories from: {traj_dir}")
 72|    train_dataset, val_dataset = create_data_loaders(
 73|        traj_dir,
 74|        batch_size=args.batch_size,
 75|        max_seq_len=args.max_seq_len,
 76|    )
 77|    print(f"Train samples: {len(train_dataset)}")
 78|    print(f"Val samples: {len(val_dataset)}")
 79|
 80|    # Calculate steps
 81|    steps_per_epoch = len(train_dataset) // args.batch_size
 82|    total_steps = steps_per_epoch * args.epochs
 83|    print(f"Steps per epoch: {steps_per_epoch}, Total steps: {total_steps}")
 84|
 85|    # Create trainer
 86|    save_dir = PROJECT_ROOT / args.save_dir
 87|    trainer = Trainer(
 88|        model=model,
 89|        train_dataset=train_dataset,
 90|        val_dataset=val_dataset,
 91|        batch_size=args.batch_size,
 92|        learning_rate=args.lr,
 93|        max_steps=total_steps,
 94|        eval_interval=max(steps_per_epoch, 50),
 95|        eval_iters=min(50, len(val_dataset)),
 96|        save_dir=str(save_dir),
 97|        device=device,
 98|    )
 99|
100|    # Train
101|    print(f"\n{'='*60}")
102|    print(f"Starting training at {time.strftime('%H:%M:%S')}")
103|    print(f"{'='*60}")
104|    start_time = time.time()
105|    trainer.train()
106|    elapsed = time.time() - start_time
107|    print(f"\nTraining time: {elapsed:.1f}s ({elapsed/60:.1f}m)")
108|
109|    # Save final model in multiple formats
110|    print(f"\nSaving final model...")
111|    save_dir.mkdir(parents=True, exist_ok=True)
112|
113|    # PyTorch checkpoint
114|    torch.save({
115|        "model_state_dict": model.state_dict(),
116|        "model_config": config.to_dict(),
117|    }, save_dir / "ch_autocomplete.pt")
118|
119|    # Export to ONNX for C++/WASM embedding
120|    try:
121|        export_onnx(model, config, save_dir / "ch_autocomplete.onnx")
122|    except Exception as e:
123|        print(f"  ONNX export skipped: {e}")
124|
125|    # Export to raw binary weights (for C++ embedding)
126|    export_raw_weights(model, config, save_dir / "ch_autocomplete_weights.bin")
127|
128|    print(f"\nDone! Model saved to {save_dir}/")
129|
130|
131|def export_onnx(model, config, path):
132|    """Export model to ONNX format for cross-platform inference."""
133|    import torch
134|    model.eval()
135|    dummy_input = torch.randint(0, config.vocab_size, (1, 32), dtype=torch.long)
136|    torch.onnx.export(
137|        model,
138|        dummy_input,
139|        str(path),
140|        export_params=True,
141|        opset_version=14,
142|        do_constant_folding=True,
143|        input_names=["input_ids"],
144|        output_names=["logits"],
145|        dynamic_axes={
146|            "input_ids": {0: "batch", 1: "sequence"},
147|            "logits": {0: "batch", 1: "sequence"},
148|        },
149|    )
150|    print(f"  ONNX model saved: {path} ({path.stat().st_size / 1024 / 1024:.2f} MB)")
151|
152|
153|def export_raw_weights(model, config, path):
154|    """Export model weights as raw binary for C++ embedding."""
155|    import struct
156|    state_dict = model.state_dict()
157|    with open(path, "wb") as f:
158|        # Write config header
159|        config_bytes = str(config.to_dict()).encode("utf-8")
160|        f.write(struct.pack("I", len(config_bytes)))
161|        f.write(config_bytes)
162|
163|        # Write each tensor
164|        for name, tensor in state_dict.items():
165|            data = tensor.detach().cpu().numpy()
166|            name_bytes = name.encode("utf-8")
167|            f.write(struct.pack("I", len(name_bytes)))
168|            f.write(name_bytes)
169|            # Write shape
170|            shape = data.shape
171|            f.write(struct.pack("I", len(shape)))
172|            for dim in shape:
173|                f.write(struct.pack("I", dim))
174|            # Write data as float32
175|            f.write(data.astype("float32").tobytes())
176|
177|    print(f"  Raw weights saved: {path} ({path.stat().st_size / 1024 / 1024:.2f} MB)")
178|
179|
180|if __name__ == "__main__":
181|    import torch
182|    main()
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py" start_line="1" end_line="551" total_lines="984">
  1|#!/usr/bin/env python3
  2|"""
  3|Export the trained model to a C++ header + binary weights for embedding
  4|in the ClickHouse binary, and to WebAssembly.
  5|
  6|This script:
  7|  1. Converts model weights to a flat binary format
  8|  2. Generates a C++ header with the model config and vocabulary
  9|  3. Generates the C++ inference engine source code
 10|  4. Provides build instructions for WASM (emscripten)
 11|"""
 12|
 13|import json
 14|import struct
 15|import sys
 16|from pathlib import Path
 17|
 18|import torch
 19|import numpy as np
 20|
 21|PROJECT_ROOT = Path(__file__).resolve().parent.parent
 22|sys.path.insert(0, str(PROJECT_ROOT / "tokenizer"))
 23|sys.path.insert(0, str(PROJECT_ROOT / "model"))
 24|
 25|from ch_tokenizer import VOCAB, VOCAB_SIZE, KEYWORDS, OPERATOR_TOKENS, NUMBER_TOKENS
 26|from ch_tokenizer import IDENT_COPY_TOKENS, STRING_COPY_TOKENS, MAX_COPY_BACK
 27|from ch_tokenizer import BOS_ID, EOS_ID, PAD_ID, UNK_ID
 28|from config import ModelConfig, ClickHouseGPT, create_model
 29|
 30|
 31|def export_model_weights_binary(model: ClickHouseGPT, path: Path):
 32|    """Export model weights as a flat binary file for C++ loading.
 33|
 34|    Format:
 35|      - 4 bytes: magic ("CHAI")
 36|      - 4 bytes: version (uint32)
 37|      - 4 bytes: vocab_size (uint32)
 38|      - 4 bytes: d_model (uint32)
 39|      - 4 bytes: n_layers (uint32)
 40|      - 4 bytes: n_heads (uint32)
 41|      - 4 bytes: d_ff (uint32)
 42|      - 4 bytes: max_seq_len (uint32)
 43|      - 4 bytes: weight_tying (uint32, 0 or 1)
 44|      - 4 bytes: num_tensors (uint32)
 45|      - For each tensor:
 46|        - 4 bytes: name length (uint32)
 47|        - N bytes: name (UTF-8)
 48|        - 4 bytes: ndim (uint32)
 49|        - ndim * 4 bytes: shape (uint32 each)
 50|        - data: float32 array (prod(shape) * 4 bytes)
 51|    """
 52|    config = model.config
 53|    state_dict = model.state_dict()
 54|
 55|    with open(path, "wb") as f:
 56|        # Header
 57|        f.write(b"CHAI")
 58|        f.write(struct.pack("I", 1))  # version
 59|        f.write(struct.pack("I", config.vocab_size))
 60|        f.write(struct.pack("I", config.d_model))
 61|        f.write(struct.pack("I", config.n_layers))
 62|        f.write(struct.pack("I", config.n_heads))
 63|        f.write(struct.pack("I", config.d_ff))
 64|        f.write(struct.pack("I", config.max_seq_len))
 65|        f.write(struct.pack("I", 1 if config.weight_tying else 0))
 66|        f.write(struct.pack("I", len(state_dict)))
 67|
 68|        # Tensors
 69|        for name, tensor in state_dict.items():
 70|            data = tensor.detach().cpu().numpy().astype(np.float32)
 71|            name_bytes = name.encode("utf-8")
 72|            f.write(struct.pack("I", len(name_bytes)))
 73|            f.write(name_bytes)
 74|            f.write(struct.pack("I", data.ndim))
 75|            for dim in data.shape:
 76|                f.write(struct.pack("I", dim))
 77|            f.write(data.tobytes())
 78|
 79|    size_mb = path.stat().st_size / 1024 / 1024
 80|    print(f"  Binary weights: {path} ({size_mb:.2f} MB)")
 81|    return size_mb
 82|
 83|
 84|def export_vocab_header(path: Path):
 85|    """Export vocabulary as a C++ header file."""
 86|    with open(path, "w") as f:
 87|        f.write("// Auto-generated by export_cpp.py — ClickHouse autocomplete vocabulary\n")
 88|        f.write("#pragma once\n\n")
 89|        f.write("#include <cstdint>\n#include <string>\n#include <unordered_map>\n\n")
 90|        f.write("namespace CHAI {\n\n")
 91|        f.write(f"constexpr uint32_t VOCAB_SIZE = {VOCAB_SIZE};\n")
 92|        f.write(f"constexpr uint32_t MAX_COPY_BACK = {MAX_COPY_BACK};\n\n")
 93|
 94|        # Special token IDs
 95|        f.write(f"constexpr uint32_t PAD_ID = {PAD_ID};\n")
 96|        f.write(f"constexpr uint32_t BOS_ID = {BOS_ID};\n")
 97|        f.write(f"constexpr uint32_t EOS_ID = {EOS_ID};\n")
 98|        f.write(f"constexpr uint32_t UNK_ID = {UNK_ID};\n\n")
 99|
100|        # Keyword list
101|        f.write(f"constexpr size_t NUM_KEYWORDS = {len(KEYWORDS)};\n")
102|        f.write("const char* const KEYWORDS[] = {\n")
103|        for i, kw in enumerate(KEYWORDS):
104|            comma = "," if i < len(KEYWORDS) - 1 else ""
105|            f.write(f'    "{kw}"{comma}\n')
106|        f.write("};\n\n")
107|
108|        # Operator tokens
109|        f.write(f"constexpr size_t NUM_OPERATORS = {len(OPERATOR_TOKENS)};\n")
110|        f.write("struct OperatorToken { const char* name; const char* symbol; };\n")
111|        f.write("const OperatorToken OPERATORS[] = {\n")
112|        for i, (name, symbol) in enumerate(OPERATOR_TOKENS.items()):
113|            comma = "," if i < len(OPERATOR_TOKENS) - 1 else ""
114|            f.write(f'    {{"{name}", "{symbol}"}}{comma}\n')
115|        f.write("};\n\n")
116|
117|        # Vocabulary mapping (token string -> ID)
118|        f.write("inline std::unordered_map<std::string, uint32_t> buildVocab() {\n")
119|        f.write("    std::unordered_map<std::string, uint32_t> v;\n")
120|        for token_str, token_id in sorted(VOCAB.items(), key=lambda x: x[1]):
121|            f.write(f'    v["{token_str}"] = {token_id};\n')
122|        f.write("    return v;\n")
123|        f.write("}\n\n")
124|
125|        # Reverse vocab (ID -> token string)
126|        f.write("inline std::unordered_map<uint32_t, std::string> buildReverseVocab() {\n")
127|        f.write("    std::unordered_map<uint32_t, std::string> v;\n")
128|        for token_str, token_id in sorted(VOCAB.items(), key=lambda x: x[1]):
129|            f.write(f'    v[{token_id}] = "{token_str}";\n')
130|        f.write("    return v;\n")
131|        f.write("}\n\n")
132|
133|        f.write("} // namespace CHAI\n")
134|
135|    print(f"  Vocab header: {path}")
136|
137|
138|def export_model_header(path: Path):
139|    """Export model architecture as a C++ header."""
140|    with open(path, "w") as f:
141|        f.write("// Auto-generated by export_cpp.py — ClickHouse autocomplete model architecture\n")
142|        f.write("#pragma once\n\n")
143|        f.write("#include <cstdint>\n#include <vector>\n#include <string>\n\n")
144|        f.write("namespace CHAI {\n\n")
145|        f.write("// Model configuration (must match the Python ModelConfig)\n")
146|        f.write("struct ModelConfig {\n")
147|        f.write("    uint32_t vocab_size;\n")
148|        f.write("    uint32_t d_model;\n")
149|        f.write("    uint32_t n_layers;\n")
150|        f.write("    uint32_t n_heads;\n")
151|        f.write("    uint32_t d_ff;\n")
152|        f.write("    uint32_t max_seq_len;\n")
153|        f.write("    uint32_t n_heads_kv;\n")
154|        f.write("    bool weight_tying;\n")
155|        f.write("};\n\n")
156|        f.write("// Default config — update after training\n")
157|        f.write("constexpr ModelConfig DEFAULT_CONFIG = {\n")
158|        f.write(f"    {VOCAB_SIZE},  // vocab_size\n")
159|        f.write(f"    192,           // d_model\n")
160|        f.write(f"    6,             // n_layers\n")
161|        f.write(f"    6,             // n_heads\n")
162|        f.write(f"    768,           // d_ff\n")
163|        f.write(f"    256,           // max_seq_len\n")
164|        f.write(f"    6,             // n_heads_kv\n")
165|        f.write(f"    true,          // weight_tying\n")
166|        f.write("};\n\n")
167|        f.write("} // namespace CHAI\n")
168|
169|    print(f"  Model header: {path}")
170|
171|
172|def export_cpp_inference_engine(path: Path):
173|    """Export the C++ inference engine source code."""
174|    code = r'''// Auto-generated by export_cpp.py — ClickHouse autocomplete C++ inference engine
175|// This file can be embedded in the ClickHouse binary or compiled to WebAssembly.
176|//
177|// Dependencies: None (pure C++17, no external libs)
178|// For matrix operations, a minimal implementation is included.
179|
180|#pragma once
181|
182|#include <cmath>
183|#include <cstdint>
184|#include <cstring>
185|#include <fstream>
186|#include <iostream>
187|#include <memory>
188|#include <string>
189|#include <unordered_map>
190|#include <vector>
191|#include <algorithm>
192|#include <random>
193|
194|#include "ch_vocab.h"
195|#include "ch_model.h"
196|
197|namespace CHAI {
198|
199|// ─── Minimal matrix/tensor operations (CPU, no BLAS) ────────────────────────
200|
201|using Matrix = std::vector<float>;  // row-major
202|using Tensor3D = std::vector<float>; // flattened
203|
204|inline void matmul(const float* A, const float* B, float* C,
205|                   int M, int K, int N) {
206|    // C = A @ B, A: (M,K), B: (K,N), C: (M,N)
207|    for (int i = 0; i < M; i++) {
208|        for (int j = 0; j < N; j++) {
209|            float sum = 0.0f;
210|            for (int k = 0; k < K; k++) {
211|                sum += A[i * K + k] * B[k * N + j];
212|            }
213|            C[i * N + j] = sum;
214|        }
215|    }
216|}
217|
218|// C = A @ B.T  where A: (M,K), B: (N,K) stored row-major, C: (M,N)
219|// This is used for PyTorch nn.Linear weights which are stored as (out_features, in_features)
220|inline void matmul_linear(const float* A, const float* W, float* C,
221|                          int M, int K, int N) {
222|    // A: (M, K), W: (N, K) [PyTorch Linear weight format], C: (M, N)
223|    // C[i,j] = sum_k A[i,k] * W[j,k]
224|    for (int i = 0; i < M; i++) {
225|        for (int j = 0; j < N; j++) {
226|            float sum = 0.0f;
227|            for (int k = 0; k < K; k++) {
228|                sum += A[i * K + k] * W[j * K + k];
229|            }
230|            C[i * N + j] = sum;
231|        }
232|    }
233|}
234|
235|inline void layernorm(const float* x, float* out, const float* gamma,
236|                      int d_model) {
237|    float mean = 0.0f;
238|    for (int i = 0; i < d_model; i++) mean += x[i];
239|    mean /= d_model;
240|    float var = 0.0f;
241|    for (int i = 0; i < d_model; i++) {
242|        float diff = x[i] - mean;
243|        var += diff * diff;
244|    }
245|    var /= d_model;
246|    float inv_std = 1.0f / std::sqrt(var + 1e-5f);
247|    for (int i = 0; i < d_model; i++) {
248|        out[i] = (x[i] - mean) * inv_std * gamma[i];
249|    }
250|}
251|
252|inline float gelu(float x) {
253|    return 0.5f * x * (1.0f + std::tanh(0.7978845608f * (x + 0.044715f * x * x * x)));
254|}
255|
256|inline void softmax(float* x, int n) {
257|    float max_val = x[0];
258|    for (int i = 1; i < n; i++) max_val = std::max(max_val, x[i]);
259|    float sum = 0.0f;
260|    for (int i = 0; i < n; i++) {
261|        x[i] = std::exp(x[i] - max_val);
262|        sum += x[i];
263|    }
264|    for (int i = 0; i < n; i++) x[i] /= sum;
265|}
266|
267|// ─── Model weights container ────────────────────────────────────────────────
268|
269|struct LayerWeights {
270|    // Attention
271|    std::vector<float> qkv_weight;  // (3*d_model, d_model)
272|    std::vector<float> proj_weight; // (d_model, d_model)
273|    // FFN
274|    std::vector<float> fc1_weight;  // (d_ff, d_model)
275|    std::vector<float> fc2_weight;  // (d_model, d_ff)
276|    // LayerNorm
277|    std::vector<float> ln1_gamma;   // (d_model,)
278|    std::vector<float> ln2_gamma;   // (d_model,)
279|};
280|
281|struct ModelWeights {
282|    ModelConfig config;
283|    std::vector<float> tok_emb;     // (vocab_size, d_model)
284|    std::vector<float> pos_emb;     // (max_seq_len, d_model)
285|    std::vector<LayerWeights> layers;
286|    std::vector<float> ln_f_gamma;  // (d_model,)
287|};
288|
289|// ─── Binary weight loader ──────────────────────────────────────────────────
290|
291|inline uint32_t read_u32(std::ifstream& f) {
292|    uint32_t v;
293|    f.read(reinterpret_cast<char*>(&v), 4);
294|    return v;
295|}
296|
297|inline std::vector<float> read_tensor_raw(std::ifstream& f, std::string& name) {
298|    uint32_t name_len = read_u32(f);
299|    name.resize(name_len);
300|    f.read(&name[0], name_len);
301|    uint32_t ndim = read_u32(f);
302|    std::vector<uint32_t> shape(ndim);
303|    for (uint32_t i = 0; i < ndim; i++) shape[i] = read_u32(f);
304|    size_t total = 1;
305|    for (auto d : shape) total *= d;
306|    std::vector<float> data(total);
307|    f.read(reinterpret_cast<char*>(data.data()), total * sizeof(float));
308|    return data;
309|}
310|
311|inline std::unique_ptr<ModelWeights> load_model(const std::string& path) {
312|    std::ifstream f(path, std::ios::binary);
313|    if (!f) { std::cerr << "Cannot open " << path << std::endl; return nullptr; }
314|
315|    char magic[4];
316|    f.read(magic, 4);
317|    if (std::memcmp(magic, "CHAI", 4) != 0) {
318|        std::cerr << "Invalid model file format" << std::endl; return nullptr;
319|    }
320|
321|    auto w = std::make_unique<ModelWeights>();
322|    uint32_t version = read_u32(f);
323|    w->config.vocab_size = read_u32(f);
324|    w->config.d_model = read_u32(f);
325|    w->config.n_layers = read_u32(f);
326|    w->config.n_heads = read_u32(f);
327|    w->config.d_ff = read_u32(f);
328|    w->config.max_seq_len = read_u32(f);
329|    w->config.weight_tying = read_u32(f) != 0;
330|    uint32_t num_tensors = read_u32(f);
331|
332|    // Read all tensors into a map by name
333|    std::unordered_map<std::string, std::vector<float>> tensors;
334|    for (uint32_t i = 0; i < num_tensors; i++) {
335|        std::string name;
336|        auto data = read_tensor_raw(f, name);
337|        tensors[name] = std::move(data);
338|    }
339|
340|    // Assign by name
341|    w->tok_emb = tensors.at("tok_emb.weight");
342|    w->pos_emb = tensors.at("pos_emb.weight");
343|
344|    w->layers.resize(w->config.n_layers);
345|    for (uint32_t l = 0; l < w->config.n_layers; l++) {
346|        auto& lw = w->layers[l];
347|        std::string prefix = "blocks." + std::to_string(l) + ".";
348|        lw.qkv_weight = tensors.at(prefix + "attn.qkv.weight");
349|        lw.proj_weight = tensors.at(prefix + "attn.proj.weight");
350|        lw.fc1_weight = tensors.at(prefix + "ffn.fc1.weight");
351|        lw.fc2_weight = tensors.at(prefix + "ffn.fc2.weight");
352|        lw.ln1_gamma = tensors.at(prefix + "ln1.weight");
353|        lw.ln2_gamma = tensors.at(prefix + "ln2.weight");
354|    }
355|
356|    w->ln_f_gamma = tensors.at("ln_f.weight");
357|    return w;
358|}
359|
360|// ─── Forward pass ───────────────────────────────────────────────────────────
361|
362|inline std::vector<float> forward(
363|    const ModelWeights& w,
364|    const std::vector<uint32_t>& token_ids
365|) {
366|    const auto& cfg = w.config;
367|    int T = (int)token_ids.size();
368|    int D = (int)cfg.d_model;
369|    int H = (int)cfg.n_heads;
370|    int DH = D / H;
371|
372|    // Embedding: token + position
373|    std::vector<float> x(T * D);
374|    for (int t = 0; t < T; t++) {
375|        uint32_t tok = token_ids[t];
376|        for (int d = 0; d < D; d++) {
377|            x[t * D + d] = w.tok_emb[tok * D + d] + w.pos_emb[t * D + d];
378|        }
379|    }
380|
381|    // Transformer layers
382|    std::vector<float> ln_out(T * D);
383|    std::vector<float> qkv(T * 3 * D);
384|    std::vector<float> attn_out(T * D);
385|    std::vector<float> proj_out(T * D);
386|    std::vector<float> ffn_hidden(T * cfg.d_ff);
387|    std::vector<float> ffn_out(T * D);
388|
389|    for (uint32_t layer = 0; layer < cfg.n_layers; layer++) {
390|        const auto& lw = w.layers[layer];
391|
392|        // Pre-LN1
393|        for (int t = 0; t < T; t++)
394|            layernorm(&x[t * D], &ln_out[t * D], lw.ln1_gamma.data(), D);
395|
396|        // QKV projection: (T, D) @ W.T where W: (3*D, D) -> (T, 3*D)
397|        matmul_linear(ln_out.data(), lw.qkv_weight.data(), qkv.data(), T, D, 3 * D);
398|
399|        // Attention (per head)
400|        for (int h = 0; h < H; h++) {
401|            int offset = h * DH;
402|            // For each query position
403|            for (int qi = 0; qi < T; qi++) {
404|                float* q = &qkv[qi * 3 * D + offset];           // Q
405|                std::vector<float> scores(qi + 1);
406|                float scale = 1.0f / std::sqrt((float)DH);
407|
408|                for (int ki = 0; ki <= qi; ki++) {
409|                    float* k = &qkv[ki * 3 * D + D + offset];   // K
410|                    float s = 0.0f;
411|                    for (int d = 0; d < DH; d++) s += q[d] * k[d];
412|                    scores[ki] = s * scale;
413|                }
414|                softmax(scores.data(), qi + 1);
415|
416|                // Weighted sum of V
417|                float* out = &attn_out[qi * D + offset];
418|                std::fill(out, out + DH, 0.0f);
419|                for (int vi = 0; vi <= qi; vi++) {
420|                    float* v = &qkv[vi * 3 * D + 2 * D + offset]; // V
421|                    float weight = scores[vi];
422|                    for (int d = 0; d < DH; d++) out[d] += weight * v[d];
423|                }
424|            }
425|        }
426|
427|        // Projection + residual: (T, D) @ W.T where W: (D, D) -> (T, D)
428|        matmul_linear(attn_out.data(), lw.proj_weight.data(), proj_out.data(), T, D, D);
429|        for (int i = 0; i < T * D; i++) x[i] += proj_out[i];
430|
431|        // Pre-LN2
432|        for (int t = 0; t < T; t++)
433|            layernorm(&x[t * D], &ln_out[t * D], lw.ln2_gamma.data(), D);
434|
435|        // FFN: fc1 -> GELU -> fc2 (all using PyTorch Linear weight format)
436|        matmul_linear(ln_out.data(), lw.fc1_weight.data(), ffn_hidden.data(), T, D, cfg.d_ff);
437|        for (int i = 0; i < T * (int)cfg.d_ff; i++) ffn_hidden[i] = gelu(ffn_hidden[i]);
438|        matmul_linear(ffn_hidden.data(), lw.fc2_weight.data(), ffn_out.data(), T, cfg.d_ff, D);
439|
440|        // Residual
441|        for (int i = 0; i < T * D; i++) x[i] += ffn_out[i];
442|    }
443|
444|    // Final LayerNorm
445|    for (int t = 0; t < T; t++)
446|        layernorm(&x[t * D], &x[t * D], w.ln_f_gamma.data(), D);
447|
448|    // Output projection (weight-tied with token embedding)
449|    // logits = x @ tok_emb.T  -> (T, vocab_size)
450|    // tok_emb is (vocab_size, D), so we use matmul_linear: (T,D) @ (V,D).T -> (T,V)
451|    std::vector<float> logits(T * cfg.vocab_size);
452|    matmul_linear(x.data(), w.tok_emb.data(), logits.data(), T, D, cfg.vocab_size);
453|
454|    return logits;
455|}
456|
457|// ─── Tokenizer (simplified C++ port) ────────────────────────────────────────
458|
459|struct Token {
460|    std::string text;
461|    std::string type;  // token type string
462|    bool is_ident;
463|    bool is_string;
464|};
465|
466|inline bool is_keyword(const std::string& word) {
467|    std::string upper = word;
468|    std::transform(upper.begin(), upper.end(), upper.begin(), ::toupper);
469|    for (const auto& kw : KEYWORDS) {
470|        if (upper == kw) return true;
471|    }
472|    return false;
473|}
474|
475|inline std::vector<Token> tokenize_cpp(const std::string& query) {
476|    // Simplified tokenizer — matches the Python version's logic
477|    std::vector<Token> tokens;
478|    size_t pos = 0;
479|    size_t n = query.size();
480|
481|    while (pos < n) {
482|        // Skip whitespace
483|        while (pos < n && isspace(query[pos])) pos++;
484|        if (pos >= n) break;
485|
486|        // Comments
487|        if (pos + 1 < n && query[pos] == '-' && query[pos+1] == '-') {
488|            while (pos < n && query[pos] != '\n') pos++;
489|            continue;
490|        }
491|        if (pos + 1 < n && query[pos] == '/' && query[pos+1] == '*') {
492|            pos += 2;
493|            while (pos + 1 < n && !(query[pos] == '*' && query[pos+1] == '/')) pos++;
494|            pos += 2;
495|            continue;
496|        }
497|
498|        // String literal
499|        if (query[pos] == '\'') {
500|            size_t start = pos;
501|            pos++;
502|            while (pos < n && query[pos] != '\'') {
503|                if (query[pos] == '\\' && pos + 1 < n) pos++;
504|                pos++;
505|            }
506|            if (pos < n) pos++; // closing quote
507|            tokens.push_back({query.substr(start, pos - start), "StringLiteral", false, true});
508|            continue;
509|        }
510|
511|        // Quoted identifier
512|        if (query[pos] == '`' || query[pos] == '"') {
513|            char quote = query[pos];
514|            size_t start = pos;
515|            pos++;
516|            while (pos < n && query[pos] != quote) {
517|                if (query[pos] == '\\' && pos + 1 < n) pos++;
518|                pos++;
519|            }
520|            if (pos < n) pos++;
521|            tokens.push_back({query.substr(start, pos - start), "QuotedIdentifier", false, false});
522|            tokens.back().is_ident = true;
523|            continue;
524|        }
525|
526|        // Numbers
527|        if (isdigit(query[pos]) || (query[pos] == '.' && pos + 1 < n && isdigit(query[pos+1]))) {
528|            size_t start = pos;
529|            while (pos < n && (isalnum(query[pos]) || query[pos] == '.')) pos++;
530|            tokens.push_back({query.substr(start, pos - start), "Number", false, false});
531|            continue;
532|        }
533|
534|        // BareWord (identifier or keyword)
535|        if (isalpha(query[pos]) || query[pos] == '_') {
536|            size_t start = pos;
537|            while (pos < n && (isalnum(query[pos]) || query[pos] == '_')) pos++;
538|            std::string word = query.substr(start, pos - start);
539|            if (is_keyword(word)) {
540|                tokens.push_back({word, "Keyword", false, false});
541|            } else {
542|                tokens.push_back({word, "BareWord", true, false});
543|            }
544|            continue;
545|        }
546|
547|        // Multi-char operators
548|        if (pos + 1 < n) {
549|            std::string two = query.substr(pos, 2);
550|            if (two == "::" || two == "->" || two == "<=" || two == ">=" ||
551|                two == "!=" || two == "||" || two == "@@" || two == "<>") {
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/export/CMakeLists.txt" start_line="1" end_line="24" total_lines="24">
  1|cmake_minimum_required(VERSION 3.14)
  2|project(chai_clickhouse_autocomplete CXX)
  3|
  4|set(CMAKE_CXX_STANDARD 17)
  5|set(CMAKE_CXX_STANDARD_REQUIRED ON)
  6|
  7|# Native library (for ClickHouse embedding)
  8|add_library(chai_native STATIC
  9|    ch_inference.h  # header-only library
 10|)
 11|target_include_directories(chai_native PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
 12|
 13|# Test binary
 14|add_executable(chai_test test_main.cpp)
 15|target_link_libraries(chai_test PRIVATE chai_native)
 16|
 17|# WebAssembly target (requires emscripten)
 18|if(EMSCRIPTEN)
 19|    add_executable(chai_wasm test_main.cpp)
 20|    target_link_libraries(chai_wasm PRIVATE chai_native)
 21|    set_target_properties(chai_wasm PROPERTIES
 22|        LINK_FLAGS "-s EXPORTED_FUNCTIONS=['_chai_load_model','_chai_autocomplete','_chai_free_model'] -s EXPORTED_RUNTIME_METHODS=['ccall','cwrap']"
 23|    )
 24|endif()
</file-view>

/Users/root1/Downloads/projects/clickhouse-github/tokenizer/ch_tokenizer.py

/Users/root1/Downloads/projects/clickhouse-github/model/config.py

Found 4 match(es) for pattern 'import' in /Users/root1/Downloads/projects/clickhouse-github/tokenizer:
-- 4 matches in /Users/root1/Downloads/projects/clickhouse-github/tokenizer/ch_tokenizer.py
20|import re
21|import json
22|from dataclasses import dataclass, field
23|from typing import Optional

Found 25 match(es) for pattern 'import' in /Users/root1/Downloads/projects/clickhouse-github/model:
-- 13 matches in /Users/root1/Downloads/projects/clickhouse-github/model/training.py
  9|import json
 10|import math
 11|import os
 12|import random
 13|import torch
 14|from torch.utils.data import Dataset, DataLoader
 15|from pathlib import Path
 16|from typing import Optional
 18|import sys
 22|from ch_tokenizer import (
 27|from config import ModelConfig, ClickHouseGPT, create_model
382|    import tempfile
414|    from torch.utils.data import DataLoader
-- 12 matches in /Users/root1/Downloads/projects/clickhouse-github/model/config.py
 20|import math
 21|import torch
 22|import torch.nn as nn
 23|import torch.nn.functional as F
 24|from dataclasses import dataclass, asdict
 25|from typing import Optional
 27|import sys
 28|sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent / "tokenizer"))
 29|from ch_tokenizer import VOCAB_SIZE, PAD_ID, BOS_ID, EOS_ID
313|    import sys
314|    sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent / "tokenizer"))
315|    from ch_tokenizer import tokenize_to_ids

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_01_basic_select.jsonl" start_line="20" end_line="20" total_lines="20">
 20|{"id":"traj_020","category":"basic_select","subcategory":"simple_select_count","description":"Simple select with count() function and WHERE","query":"SELECT count() AS total_events FROM events WHERE event_type = 'click'","tables":["events"],"columns":{"events":["id","event_type","user_id","timestamp","data"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"count() AS total_events FROM events WHERE event_type = 'click'","next_token":"count()"},{"prefix":"SELECT count() AS total_events ","completion":"FROM events WHERE event_type = 'click'","next_token":"FROM"},{"prefix":"SELECT count() AS total_events FROM ","completion":"events WHERE event_type = 'click'","next_token":"events"},{"prefix":"SELECT count() AS total_events FROM events ","completion":"WHERE event_type = 'click'","next_token":"WHERE"}]}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_02_aggregation.jsonl" start_line="20" end_line="20" total_lines="20">
 20|{"id":"traj_040","category":"aggregation","subcategory":"multi_feature_aggregation","description":"Complex query combining GROUP BY WITH TOTALS, aggregate combinators, window functions, and SETTINGS","query":"SELECT toStartOfWeek(event_date) AS week, channel, count() AS total_events, countIf(is_converted = 1) AS conversions, sumIf(revenue, is_converted = 1) AS total_revenue, avgIf(revenue, is_converted = 1) AS avg_revenue, uniq(user_id) AS unique_users, groupUniqArray(campaign_id) AS campaigns, sum(total_revenue) OVER (PARTITION BY channel ORDER BY week ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cumulative_revenue FROM marketing_events WHERE event_date >= '2024-01-01' GROUP BY week, channel WITH TOTALS HAVING total_events > 10 ORDER BY week, channel SETTINGS max_threads = 8, max_memory_usage = 1000000000","tables":["marketing_events"],"columns":{"marketing_events":["id","event_date","channel","is_converted","revenue","user_id","campaign_id","country"]},"dictionaries":[],"settings":{"max_threads":"8","max_memory_usage":"1000000000"},"completion_points":[{"prefix":"SELECT ","completion":"toStartOfWeek(event_date) AS week, channel, count() AS total_events, countIf(is_converted = 1) AS conversions, sumIf(revenue, is_converted = 1) AS total_revenue, avgIf(revenue, is_converted = 1) AS avg_revenue, uniq(user_id) AS unique_users, groupUniqArray(campaign_id) AS campaigns, sum(total_revenue) OVER (PARTITION BY channel ORDER BY week ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cumulative_revenue ","next_token":"toStartOfWeek"},{"prefix":"SELECT toStartOfWeek(event_date) AS week, channel, count() AS total_events, countIf(is_converted = 1) AS conversions, sumIf(revenue, is_converted = 1) AS total_revenue, avgIf(revenue, is_converted = 1) AS avg_revenue, uniq(user_id) AS unique_users, groupUniqArray(campaign_id) AS campaigns, sum(total_revenue) OVER (PARTITION BY channel ORDER BY week ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cumulative_revenue FROM ... [truncated 3598 chars]
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_03_joins.jsonl" start_line="20" end_line="20" total_lines="20">
 20|{"id":"traj_060","category":"joins","subcategory":"asof_join_multiple","description":"ASOF join with additional INNER JOIN for enriched time-series analysis","query":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.device_type, u.username FROM events AS e ASOF LEFT JOIN sessions AS s ON e.user_id = s.user_id AND e.event_time > s.started_at INNER JOIN users AS u ON e.user_id = u.user_id WHERE e.event_date BETWEEN '2024-06-01' AND '2024-06-07' ORDER BY e.event_time DESC LIMIT 1000","tables":["events","sessions","users"],"columns":{"events":["event_id","user_id","event_time","event_date","event_type","properties"],"sessions":["session_id","user_id","started_at","ended_at","device_type"],"users":["user_id","username","email","is_active","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"e.event_id, e.event_time, e.event_type, s.session_id, s.device_type, u.username ","next_token":"e"},{"prefix":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.device_type, u.username FROM ","completion":"events AS e ","next_token":"events"},{"prefix":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.device_type, u.username FROM events AS e ASOF LEFT JOIN ","completion":"sessions AS s ON e.user_id = s.user_id AND e.event_time > s.started_at ","next_token":"sessions"},{"prefix":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.device_type, u.username FROM events AS e ASOF LEFT JOIN sessions AS s ON e.user_id = s.user_id AND e.event_time > s.started_at INNER JOIN ","completion":"users AS u ON e.user_id = u.user_id ","next_token":"users"},{"prefix":"SELECT e.event_id, e.event_time, e.event_type, s.session_id, s.device_type, u.username FROM events AS e ASOF LEFT JOIN sessions AS s ON e.user_id = s.user_id AND e.event_time > s.started_at INNER JOIN users AS u ON e.user_id = u.user_id WHERE ","completion":"e.event_date BETWEEN '2024-06-01' AND '2024-06-07' ","next_token":"e"},{"prefix":"... [truncated 358 chars]
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_04_ddl_dml.jsonl" start_line="20" end_line="20" total_lines="20">
 20|{"id":"traj_080","category":"ddl_dml","subcategory":"explain_query","description":"EXPLAIN SYNTAX with complex inner SELECT including WHERE, GROUP BY, ORDER BY, and LIMIT","query":"EXPLAIN SYNTAX SELECT event_type, count() AS cnt FROM analytics.events WHERE event_time >= '2024-01-01 00:00:00' GROUP BY event_type ORDER BY cnt DESC LIMIT 10","tables":["analytics.events"],"columns":{"analytics.events":["event_id","user_id","event_type","event_data","event_time","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"EXPLAIN ","completion":"SYNTAX ","next_token":"SYNTAX"},{"prefix":"EXPLAIN SYNTAX ","completion":"SELECT event_type, count() AS cnt ","next_token":"SELECT"},{"prefix":"EXPLAIN SYNTAX SELECT event_type, count() AS cnt FROM ","completion":"analytics.events ","next_token":"analytics"},{"prefix":"EXPLAIN SYNTAX SELECT event_type, count() AS cnt FROM analytics.events WHERE ","completion":"event_time >= '2024-01-01 00:00:00' ","next_token":"event_time"},{"prefix":"EXPLAIN SYNTAX SELECT ... FROM analytics.events WHERE event_time >= '2024-01-01 00:00:00' GROUP BY ","completion":"event_type ","next_token":"event_type"},{"prefix":"EXPLAIN SYNTAX SELECT ... FROM analytics.events WHERE event_time >= '2024-01-01 00:00:00' GROUP BY event_type ORDER BY ","completion":"cnt DESC ","next_token":"cnt"},{"prefix":"EXPLAIN SYNTAX SELECT ... GROUP BY event_type ORDER BY cnt DESC LIMIT ","completion":"10","next_token":"10"}]}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl" start_line="20" end_line="20" total_lines="20">
 20|{"id":"traj_100","category":"advanced","subcategory":"settings_interval_datetime","description":"Complex query with SETTINGS, interval arithmetic, parseDateTime, and QUALIFY for top-N per group","query":"SELECT parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') AS event_dt, user_id, event_type, event_value FROM raw_events WHERE parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') >= now() - INTERVAL 1 HOUR QUALIFY row_number() OVER (PARTITION BY user_id ORDER BY parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') DESC) <= 5 SETTINGS max_threads = 8, max_execution_time = 30","tables":["raw_events"],"columns":{"raw_events":["event_id","user_id","event_type","event_value","event_time_str"]},"dictionaries":[],"settings":{"max_threads":"8","max_execution_time":"30"},"completion_points":[{"prefix":"SELECT ","completion":"parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') AS event_dt, user_id, event_type, event_value ","next_token":"parseDateTime"},{"prefix":"SELECT parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') AS event_dt, user_id, event_type, event_value FROM ","completion":"raw_events ","next_token":"raw_events"},{"prefix":"SELECT parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') AS event_dt, user_id, event_type, event_value FROM raw_events WHERE ","completion":"parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') >= now() - INTERVAL 1 HOUR ","next_token":"parseDateTime"},{"prefix":"SELECT parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') AS event_dt, user_id, event_type, event_value FROM raw_events WHERE parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') >= now() - INTERVAL 1 HOUR QUALIFY ","completion":"row_number() OVER (PARTITION BY user_id ORDER BY parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') DESC) <= 5 ","next_token":"row_number"},{"prefix":"SELECT parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') AS event_dt, user_id, event_type, event_value FROM raw_events WHERE parseDateTime(event_time_str, '%Y-%m-%d %H:%i:%s') >= now() - INTERVAL 1 HOUR QUALIFY row_nu... [truncated 201 chars]
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/tokenizer/ch_tokenizer.py" start_line="1" end_line="514" total_lines="514">
  1|"""
  2|ClickHouse Lexer-based tokenizer with lookbehind copies.
  3|
  4|This tokenizer mirrors the ClickHouse Lexer (src/Parsers/Lexer.h) token types
  5|but replaces identifier and string-literal values with "lookbehind copy" tokens.
  6|
  7|Instead of encoding identifier/string values literally (which would require an
  8|unbounded vocabulary), each identifier or string is represented as either:
  9|  - NEW_IDENT / NEW_STRING  — first occurrence in the sequence
 10|  - COPY_BACK_N / COPY_STR_BACK_N — a repeat of a value seen N tokens ago
 11|
 12|This keeps the vocabulary small (~350 tokens) while allowing the model to
 13|reproduce arbitrary identifiers and strings via a pointer/copy mechanism at
 14|inference time.
 15|
 16|The tokenizer also stores the raw text for each identifier/string so that the
 17|inference engine can resolve copy references.
 18|"""
 19|
 20|import re
 21|import json
 22|from dataclasses import dataclass, field
 23|from typing import Optional
 24|
 25|
 26|# ─── ClickHouse keywords (from utils/antlr/ClickHouseLexer.g4) ───────────────
 27|
 28|KEYWORDS = [
 29|    "ACCESS", "ADD", "AFTER", "ALIAS", "ALL", "ALTER", "AND", "ANTI", "ANY",
 30|    "ARRAY", "AS", "ASC", "ASCENDING", "ASOF", "AST", "ASYNC", "ATTACH",
 31|    "BETWEEN", "BOTH", "BY", "CACHES", "CASE", "CAST", "CHECK", "CLEAR",
 32|    "CLUSTER", "CLUSTERS", "CODEC", "COLLATE", "COLUMN", "COLUMNS", "COMMENT",
 33|    "CONSTRAINT", "CREATE", "CROSS", "CUBE", "CURRENT", "CURRENT_USER",
 34|    "CHANGED", "DATABASE", "DATABASES", "DATE", "DAY", "DEDUPLICATE",
 35|    "DEFAULT", "DELAY", "DELETE", "DESC", "DESCENDING", "DESCRIBE", "DETACH",
 36|    "DICTIONARIES", "DICTIONARY", "DISK", "DISTINCT", "DISTRIBUTED", "DROP",
 37|    "ELSE", "ENABLED", "END", "ENGINE", "ENGINES", "ESTIMATE", "EVENTS",
 38|    "EXCEPT", "EXISTS", "EXPLAIN", "EXPRESSION", "EXTENDED", "EXTRACT",
 39|    "FETCHES", "FIELDS", "FILESYSTEM", "FILL", "FINAL", "FIRST", "FLUSH",
 40|    "FOLLOWING", "FOR", "FORMAT", "FREEZE", "FROM", "FULL", "FUNCTION",
 41|    "FUNCTIONS", "GLOBAL", "GRANULARITY", "GRANTS", "GROUP", "GROUPING",
 42|    "HAVING", "HIERARCHICAL", "HOUR", "ID", "IF", "ILIKE", "IMPLICIT", "IN",
 43|    "INDEX", "INDEXES", "INDICES", "INF", "INFINITY", "INJECTIVE", "INNER",
 44|    "INSERT", "INTERPOLATE", "INTERVAL", "INTO", "IS", "IS_OBJECT_ID", "JOIN",
 45|    "KEY", "KEYS", "KILL", "LAST", "LAYOUT", "LEADING", "LEFT", "LIFETIME",
 46|    "LIKE", "LIMIT", "LIVE", "LOCAL", "LOGS", "MATERIALIZE", "MATERIALIZED",
 47|    "MAX", "MERGES", "MICROSECOND", "MILLISECOND", "MIN", "MINUTE", "MODIFY",
 48|    "MONTH", "MOVE", "MUTATION", "NAN", "NANOSECOND", "NO", "NOT", "NULL",
 49|    "NULLS", "OFFSET", "ON", "OPTIMIZE", "OR", "ORDER", "OUTER", "OUTFILE",
 50|    "OVER", "OVERRIDE", "PARTITION", "PIPELINE", "PLAN", "POLICY", "POLICIES",
 51|    "POPULATE", "PRECEDING", "PREWHERE", "PRIMARY", "PRIVILEGES",
 52|    "PROCESSLIST", "PROFILE", "PROFILES", "PROJECTION", "QUARTER", "QUOTA",
 53|    "QUOTAS", "RANGE", "RECURSIVE", "RELOAD", "REMOVE", "RENAME", "REPLACE",
 54|    "REPLICA", "REPLICATED", "RIGHT", "ROLE", "ROLES", "ROLLUP", "ROW",
 55|    "ROWS", "SAMPLE", "SECOND", "SELECT", "SEMI", "SENDS", "SET", "SETS",
 56|    "SETTING", "SETTINGS", "SHOW", "SOURCE", "START", "STOP", "SUBSTRING",
 57|    "SYNC", "SYNTAX", "SYSTEM", "STEP", "TABLE", "TABLES", "TEMPORARY",
 58|    "TEST", "THEN", "TIES", "TIMEOUT", "TIMESTAMP", "TO", "TOP", "TOTALS",
 59|    "TRAILING", "TREE", "TRIM", "TRUNCATE", "TTL", "TYPE", "UNBOUNDED",
 60|    "UNION", "UPDATE", "USE", "USER", "USERS", "USING", "UUID", "VALUES",
 61|    "VIEW", "VOLUME", "WATCH", "WEEK", "WHEN", "WHERE", "WINDOW", "WITH",
 62|    "YEAR", "YYYY", "FALSE", "TRUE",
 63|]
 64|
 65|# Build a case-insensitive lookup set (ClickHouse keywords are case-insensitive)
 66|KEYWORD_SET = {kw.upper() for kw in KEYWORDS}
 67|
 68|
 69|# ─── Token type enum (mirrors ClickHouse TokenType) ──────────────────────────
 70|
 71|# Significant token types from the ClickHouse lexer that we map to vocab entries
 72|SIGNIFICANT_TOKEN_TYPES = [
 73|    "BareWord", "Number", "StringLiteral", "QuotedIdentifier",
 74|    "OpeningRoundBracket", "ClosingRoundBracket",
 75|    "OpeningSquareBracket", "ClosingSquareBracket",
 76|    "OpeningCurlyBrace", "ClosingCurlyBrace",
 77|    "Comma", "Semicolon", "Dot", "Asterisk",
 78|    "Plus", "Minus", "Slash", "Percent",
 79|    "Arrow", "QuestionMark", "Colon", "Caret", "DoubleColon",
 80|    "Equals", "NotEquals", "Less", "Greater",
 81|    "LessOrEquals", "GreaterOrEquals", "Spaceship",
 82|    "Concatenation", "At", "DoubleAt",
 83|]
 84|
 85|# ─── Vocabulary construction ─────────────────────────────────────────────────
 86|
 87|MAX_COPY_BACK = 32  # maximum lookbehind distance for copy tokens
 88|
 89|SPECIAL_TOKENS = ["<PAD>", "<BOS>", "<EOS>", "<UNK>"]
 90|
 91|# Operator/punctuation tokens with their string representations
 92|OPERATOR_TOKENS = {
 93|    "LPAREN": "(", "RPAREN": ")",
 94|    "LBRACKET": "[", "RBRACKET": "]",
 95|    "LBRACE": "{", "RBRACE": "}",
 96|    "COMMA": ",", "SEMICOLON": ";",
 97|    "DOT": ".", "ASTERISK": "*",
 98|    "PLUS": "+", "MINUS": "-",
 99|    "SLASH": "/", "PERCENT": "%",
100|    "ARROW": "->",
101|    "QUESTION": "?", "COLON": ":",
102|    "CARET": "^", "DOUBLE_COLON": "::",
103|    "EQUALS": "=", "NOT_EQUALS": "!=",
104|    "LESS": "<", "GREATER": ">",
105|    "LESS_OR_EQUALS": "<=", "GREATER_OR_EQUALS": ">=",
106|    "SPACESHIP": "<=>",
107|    "CONCAT": "||",
108|    "AT": "@", "DOUBLE_AT": "@@",
109|}
110|
111|NUMBER_TOKENS = ["NUM_INT", "NUM_FLOAT", "NUM_HEX"]
112|
113|# Identifier copy tokens
114|IDENT_COPY_TOKENS = ["NEW_IDENT"] + [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK + 1)]
115|
116|# String copy tokens
117|STRING_COPY_TOKENS = ["NEW_STRING"] + [f"COPY_STR_BACK_{i}" for i in range(1, MAX_COPY_BACK + 1)]
118|
119|
120|def build_vocab():
121|    """Build the vocabulary mapping token string -> integer id."""
122|    vocab = {}
123|    idx = 0
124|    for tok in SPECIAL_TOKENS:
125|        vocab[tok] = idx
126|        idx += 1
127|    for kw in KEYWORDS:
128|        vocab[f"KW_{kw}"] = idx
129|        idx += 1
130|    for tok in OPERATOR_TOKENS:
131|        vocab[tok] = idx
132|        idx += 1
133|    for tok in NUMBER_TOKENS:
134|        vocab[tok] = idx
135|        idx += 1
136|    for tok in IDENT_COPY_TOKENS:
137|        vocab[tok] = idx
138|        idx += 1
139|    for tok in STRING_COPY_TOKENS:
140|        vocab[tok] = idx
141|        idx += 1
142|    return vocab
143|
144|
145|VOCAB = build_vocab()
146|VOCAB_SIZE = len(VOCAB)
147|ID_TO_TOKEN = {v: k for k, v in VOCAB.items()}
148|
149|# Special token IDs
150|PAD_ID = VOCAB["<PAD>"]
151|BOS_ID = VOCAB["<BOS>"]
152|EOS_ID = VOCAB["<EOS>"]
153|UNK_ID = VOCAB["<UNK>"]
154|
155|
156|# ─── Lexer (Python re-based, mirrors ClickHouse Lexer.cpp) ───────────────────
157|
158|# Token patterns in priority order
159|_TOKEN_PATTERNS = [
160|    ("Whitespace",      r'[ \t\r\n]+'),
161|    ("Comment",         r'--[^\n\r]*|/\*.*?\*/'),
162|    ("HereDoc",         r"\$\$\w*\$\$"),
163|    # String literal: 'text' with '' escape and \' escape
164|    ("StringLiteral",   r"'(?:[^'\\]|\\.|'')*'"),
165|    # Quoted identifier: `text` or "text"
166|    ("QuotedIdentifier",r'`(?:[^`\\]|\\.|``)*`|"(?:[^"\\]|\\.|"")*"'),
167|    # Hex string literal: X'...'
168|    ("HexStringLiteral",r"X'[0-9a-fA-F]+'"),
169|    # Binary string literal: B'...'
170|    ("BinaryStringLit", r"B'[01]+'"),
171|    # Numbers: hex (0x...), binary (0b...), octal (0o...), float, int
172|    ("Number",          r'0[xX][0-9a-fA-F]+(?:[pP][+-]?[0-9]+)?'
173|                        r'|0[bB][01]+'
174|                        r'|0[oO][0-7]+'
175|                        r'|\d+\.\d*(?:[eE][+-]?\d+)?'
176|                        r'|\.\d+(?:[eE][+-]?\d+)?'
177|                        r'|\d+(?:[eE][+-]?\d+)?'),
178|    # BareWord: identifier or keyword
179|    ("BareWord",        r'[a-zA-Z_][a-zA-Z_0-9]*'),
180|    # Multi-char operators (must come before single-char)
181|    ("DoubleColon",     r'::'),
182|    ("Arrow",           r'->'),
183|    ("Spaceship",       r'<=>'),
184|    ("Concatenation",   r'\|\|'),
185|    ("DoubleAt",        r'@@'),
186|    ("NotEquals",       r'!=|<>'),
187|    ("LessOrEquals",    r'<='),
188|    ("GreaterOrEquals", r'>='),
189|    ("Equals",          r'==|='),  # ClickHouse accepts both = and ==
190|    # Single-char operators
191|    ("OpeningRoundBracket", r'\('),
192|    ("ClosingRoundBracket", r'\)'),
193|    ("OpeningSquareBracket", r'\['),
194|    ("ClosingSquareBracket", r'\]'),
195|    ("OpeningCurlyBrace", r'\{'),
196|    ("ClosingCurlyBrace", r'\}'),
197|    ("Comma",           r','),
198|    ("Semicolon",       r';'),
199|    ("Dot",             r'\.'),
200|    ("Asterisk",        r'\*'),
201|    ("Plus",            r'\+'),
202|    ("Minus",           r'-'),
203|    ("Slash",           r'/'),
204|    ("Percent",         r'%'),
205|    ("QuestionMark",    r'\?'),
206|    ("Colon",           r':'),
207|    ("Caret",           r'\^'),
208|    ("At",              r'@'),
209|]
210|
211|_COMPILED_PATTERNS = [(name, re.compile(pat, re.DOTALL)) for name, pat in _TOKEN_PATTERNS]
212|
213|
214|@dataclass
215|class LexToken:
216|    """A raw lexed token (before lookbehind encoding)."""
217|    type: str        # ClickHouse TokenType name
218|    text: str        # raw text
219|    start: int       # character offset in source
220|    end: int         # character offset (exclusive)
221|
222|
223|def lex(query: str) -> list[LexToken]:
224|    """Lex a ClickHouse query string into raw tokens (mirrors ClickHouse Lexer).
225|
226|    Whitespace and comments are skipped (matching isSignificant() in the C++ lexer).
227|    """
228|    tokens: list[LexToken] = []
229|    pos = 0
230|    n = len(query)
231|    while pos < n:
232|        matched = False
233|        for ttype, pattern in _COMPILED_PATTERNS:
234|            m = pattern.match(query, pos)
235|            if m:
236|                text = m.group(0)
237|                if ttype not in ("Whitespace", "Comment"):
238|                    tokens.append(LexToken(ttype, text, pos, m.end()))
239|                pos = m.end()
240|                matched = True
241|                break
242|        if not matched:
243|            # Unrecognized character — skip it (could also raise)
244|            pos += 1
245|    return tokens
246|
247|
248|# ─── Lookbehind copy encoding ────────────────────────────────────────────────
249|
250|@dataclass
251|class EncodedToken:
252|    """A token after lookbehind copy encoding."""
253|    token_id: int           # vocabulary index
254|    token_str: str          # vocabulary token string
255|    raw_text: str           # original text (for identifiers/strings; empty for keywords/ops)
256|    lex_type: str           # original lexer token type
257|    is_ident: bool          # whether this is an identifier (for copy resolution)
258|    is_string: bool         # whether this is a string literal (for copy resolution)
259|
260|
261|def _classify_number(text: str) -> str:
262|    """Classify a number token as NUM_INT, NUM_FLOAT, or NUM_HEX."""
263|    lower = text.lower()
264|    if lower.startswith("0x"):
265|        return "NUM_HEX"
266|    if "." in text or "e" in lower or "p" in lower:
267|        return "NUM_FLOAT"
268|    return "NUM_INT"
269|
270|
271|def encode_tokens(lex_tokens: list[LexToken]) -> list[EncodedToken]:
272|    """Encode raw lexed tokens into vocabulary tokens with lookbehind copies.
273|
274|    For identifiers (BareWord that's not a keyword, QuotedIdentifier):
275|      - First occurrence → NEW_IDENT
276|      - Repeat within MAX_COPY_BACK distance → COPY_BACK_N
277|      - Repeat beyond MAX_COPY_BACK → NEW_IDENT (treated as new)
278|
279|    For string literals:
280|      - First occurrence → NEW_STRING
281|      - Repeat within MAX_COPY_BACK distance → COPY_STR_BACK_N
282|      - Repeat beyond MAX_COPY_BACK → NEW_STRING
283|
284|    The raw_text is preserved so the inference engine can resolve copies.
285|    """
286|    result: list[EncodedToken] = []
287|    # Track recent identifier texts and their positions (token index in result)
288|    ident_history: list[tuple[str, int]] = []  # (text_lower, token_index)
289|    string_history: list[tuple[str, int]] = []  # (text, token_index)
290|
291|    for lt in lex_tokens:
292|        et = EncodedToken(
293|            token_id=UNK_ID, token_str="<UNK>", raw_text="",
294|            lex_type=lt.type, is_ident=False, is_string=False,
295|        )
296|
297|        if lt.type == "BareWord":
298|            upper = lt.text.upper()
299|            if upper in KEYWORD_SET:
300|                kw_tok = f"KW_{upper}"
301|                et.token_id = VOCAB.get(kw_tok, UNK_ID)
302|                et.token_str = kw_tok
303|            else:
304|                # Identifier — lookbehind copy encoding
305|                lower = lt.text.lower()
306|                copy_dist = _find_copy_distance(ident_history, lower, len(result))
307|                if copy_dist is not None and copy_dist <= MAX_COPY_BACK:
308|                    tok_str = f"COPY_BACK_{copy_dist}"
309|                else:
310|                    tok_str = "NEW_IDENT"
311|                et.token_id = VOCAB[tok_str]
312|                et.token_str = tok_str
313|                et.raw_text = lt.text
314|                et.is_ident = True
315|                ident_history.append((lower, len(result)))
316|        elif lt.type == "QuotedIdentifier":
317|            # Quoted identifier — strip quotes for comparison
318|            inner = lt.text
319|            if inner.startswith("`") and inner.endswith("`"):
320|                inner = inner[1:-1].replace("``", "`")
321|            elif inner.startswith('"') and inner.endswith('"'):
322|                inner = inner[1:-1].replace('""', '"')
323|            lower = inner.lower()
324|            copy_dist = _find_copy_distance(ident_history, lower, len(result))
325|            if copy_dist is not None and copy_dist <= MAX_COPY_BACK:
326|                tok_str = f"COPY_BACK_{copy_dist}"
327|            else:
328|                tok_str = "NEW_IDENT"
329|            et.token_id = VOCAB[tok_str]
330|            et.token_str = tok_str
331|            et.raw_text = lt.text
332|            et.is_ident = True
333|            ident_history.append((lower, len(result)))
334|        elif lt.type == "StringLiteral":
335|            # String literal — strip quotes for comparison
336|            inner = lt.text[1:-1].replace("''", "'")
337|            copy_dist = _find_copy_distance(string_history, inner, len(result))
338|            if copy_dist is not None and copy_dist <= MAX_COPY_BACK:
339|                tok_str = f"COPY_STR_BACK_{copy_dist}"
340|            else:
341|                tok_str = "NEW_STRING"
342|            et.token_id = VOCAB[tok_str]
343|            et.token_str = tok_str
344|            et.raw_text = lt.text
345|            et.is_string = True
346|            string_history.append((inner, len(result)))
347|        elif lt.type == "Number":
348|            num_type = _classify_number(lt.text)
349|            et.token_id = VOCAB[num_type]
350|            et.token_str = num_type
351|        elif lt.type in ("HexStringLiteral", "BinaryStringLit"):
352|            et.token_id = VOCAB["NEW_STRING"]
353|            et.token_str = "NEW_STRING"
354|            et.raw_text = lt.text
355|            et.is_string = True
356|        elif lt.type == "HereDoc":
357|            et.token_id = VOCAB["NEW_STRING"]
358|            et.token_str = "NEW_STRING"
359|            et.raw_text = lt.text
360|            et.is_string = True
361|        else:
362|            # Operator/punctuation
363|            op_tok = _LEX_TYPE_TO_OP.get(lt.type)
364|            if op_tok is not None:
365|                et.token_id = VOCAB[op_tok]
366|                et.token_str = op_tok
367|            else:
368|                et.token_id = UNK_ID
369|                et.token_str = "<UNK>"
370|
371|        result.append(et)
372|
373|    return result
374|
375|
376|# Map lexer token types to operator vocabulary tokens
377|_LEX_TYPE_TO_OP = {
378|    "OpeningRoundBracket": "LPAREN",
379|    "ClosingRoundBracket": "RPAREN",
380|    "OpeningSquareBracket": "LBRACKET",
381|    "ClosingSquareBracket": "RBRACKET",
382|    "OpeningCurlyBrace": "LBRACE",
383|    "ClosingCurlyBrace": "RBRACE",
384|    "Comma": "COMMA",
385|    "Semicolon": "SEMICOLON",
386|    "Dot": "DOT",
387|    "Asterisk": "ASTERISK",
388|    "Plus": "PLUS",
389|    "Minus": "MINUS",
390|    "Slash": "SLASH",
391|    "Percent": "PERCENT",
392|    "Arrow": "ARROW",
393|    "QuestionMark": "QUESTION",
394|    "Colon": "COLON",
395|    "Caret": "CARET",
396|    "DoubleColon": "DOUBLE_COLON",
397|    "Equals": "EQUALS",
398|    "NotEquals": "NOT_EQUALS",
399|    "Less": "LESS",
400|    "Greater": "GREATER",
401|    "LessOrEquals": "LESS_OR_EQUALS",
402|    "GreaterOrEquals": "GREATER_OR_EQUALS",
403|    "Spaceship": "SPACESHIP",
404|    "Concatenation": "CONCAT",
405|    "At": "AT",
406|    "DoubleAt": "DOUBLE_AT",
407|}
408|
409|
410|def _find_copy_distance(history: list[tuple[str, int]], text: str, current_idx: int) -> Optional[int]:
411|    """Find the most recent occurrence of text in history and return the distance back."""
412|    # Search from most recent backwards
413|    for i in range(len(history) - 1, -1, -1):
414|        hist_text, hist_idx = history[i]
415|        if hist_text == text:
416|            return current_idx - hist_idx
417|    return None
418|
419|
420|# ─── Public API ──────────────────────────────────────────────────────────────
421|
422|def tokenize(query: str) -> list[EncodedToken]:
423|    """Tokenize a ClickHouse query into encoded tokens with lookbehind copies."""
424|    lex_tokens = lex(query)
425|    return encode_tokens(lex_tokens)
426|
427|
428|def tokenize_to_ids(query: str, add_bos: bool = True, add_eos: bool = True) -> list[int]:
429|    """Tokenize a query and return a list of vocabulary token IDs."""
430|    encoded = tokenize(query)
431|    ids = [et.token_id for et in encoded]
432|    if add_bos:
433|        ids = [BOS_ID] + ids
434|    if add_eos:
435|        ids = ids + [EOS_ID]
436|    return ids
437|
438|
439|def detokenize(encoded_tokens: list[EncodedToken]) -> str:
440|    """Reconstruct the original query text from encoded tokens.
441|
442|    This uses the raw_text stored in each EncodedToken for identifiers and strings.
443|    For copy tokens, the raw_text is the actual text (stored at encode time).
444|    """
445|    parts: list[str] = []
446|    for et in encoded_tokens:
447|        if et.is_ident or et.is_string:
448|            parts.append(et.raw_text)
449|        elif et.token_str.startswith("KW_"):
450|            parts.append(et.token_str[3:])
451|        elif et.token_str in NUMBER_TOKENS:
452|            parts.append(et.raw_text if et.raw_text else "0")
453|        elif et.token_str in OPERATOR_TOKENS:
454|            parts.append(OPERATOR_TOKENS[et.token_str])
455|        elif et.token_str in SPECIAL_TOKENS:
456|            continue
457|        else:
458|            parts.append(et.token_str)
459|    # Simple spacing heuristic
460|    return _join_with_spacing(parts)
461|
462|
463|def _join_with_spacing(parts: list[str]) -> str:
464|    """Join token parts with reasonable spacing."""
465|    result = ""
466|    for i, part in enumerate(parts):
467|        if i == 0:
468|            result = part
469|            continue
470|        prev = parts[i - 1]
471|        # No space after opening brackets or before closing brackets/comma/semicolon/dot
472|        no_space_before = part in (")", "]", "}", ",", ";", ".")
473|        no_space_after = prev in ("(", "[", "{", ".")
474|        if no_space_before or no_space_after:
475|            result += part
476|        else:
477|            result += " " + part
478|    return result
479|
480|
481|def save_vocab(path: str):
482|    """Save the vocabulary to a JSON file."""
483|    with open(path, "w") as f:
484|        json.dump({
485|            "vocab": VOCAB,
486|            "vocab_size": VOCAB_SIZE,
487|            "special_tokens": {
488|                "pad": PAD_ID, "bos": BOS_ID, "eos": EOS_ID, "unk": UNK_ID,
489|            },
490|            "max_copy_back": MAX_COPY_BACK,
491|            "keywords": KEYWORDS,
492|            "operators": OPERATOR_TOKENS,
493|            "number_tokens": NUMBER_TOKENS,
494|            "ident_copy_tokens": IDENT_COPY_TOKENS,
495|            "string_copy_tokens": STRING_COPY_TOKENS,
496|        }, f, indent=2)
497|
498|
499|if __name__ == "__main__":
500|    # Quick self-test
501|    test_queries = [
502|        "SELECT count(), avg(price) FROM orders WHERE region = 'US' GROUP BY city ORDER BY count() DESC LIMIT 10",
503|        "SELECT * FROM events WHERE event_type = 'click' AND user_id IN (SELECT user_id FROM users WHERE country = 'US') ORDER BY timestamp DESC LIMIT 100",
504|        "CREATE TABLE IF NOT EXISTS analytics.events (id UUID, event_time DateTime) ENGINE = MergeTree() ORDER BY (event_time)",
505|    ]
506|    for q in test_queries:
507|        print(f"\nQuery: {q}")
508|        encoded = tokenize(q)
509|        for et in encoded:
510|            extra = f" [{et.raw_text}]" if (et.is_ident or et.is_string) else ""
511|            print(f"  {et.token_str}{extra}")
512|        ids = tokenize_to_ids(q)
513|        print(f"  IDs: {ids}")
514|    print(f"\nVocab size: {VOCAB_SIZE}")
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/model/config.py" start_line="1" end_line="329" total_lines="329">
  1|"""
  2|GPT-based Small Language Model for ClickHouse query autocomplete.
  3|
  4|Design goals:
  5|  - ~10 MB model size (float32) → ~2.7M parameters
  6|  - CPU-optimized: small dimensions, weight tying, pre-LayerNorm
  7|  - Embeddable in C++ (ClickHouse binary) and exportable to WebAssembly
  8|  - Constrained sampling at inference time (syntactic correctness + table dicts)
  9|
 10|Architecture:
 11|  - d_model = 192, n_layers = 6, n_heads = 6, d_ff = 768
 12|  - max_seq_len = 256
 13|  - Weight tying (input embedding == output projection)
 14|  - Pre-LayerNorm transformer blocks
 15|  - Learned positional embeddings
 16|  - GELU activation in FFN
 17|  - ~2.77M params ≈ 10.6 MB (float32), ~5.3 MB (float16), ~2.7 MB (int8)
 18|"""
 19|
 20|import math
 21|import torch
 22|import torch.nn as nn
 23|import torch.nn.functional as F
 24|from dataclasses import dataclass, asdict
 25|from typing import Optional
 26|
 27|import sys
 28|sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent / "tokenizer"))
 29|from ch_tokenizer import VOCAB_SIZE, PAD_ID, BOS_ID, EOS_ID
 30|
 31|
 32|@dataclass
 33|class ModelConfig:
 34|    """Configuration for the ClickHouse autocomplete GPT model."""
 35|    vocab_size: int = VOCAB_SIZE
 36|    d_model: int = 192
 37|    n_layers: int = 6
 38|    n_heads: int = 6
 39|    d_ff: int = 768
 40|    max_seq_len: int = 256
 41|    n_heads_kv: int = 6       # no GQA (d_model is small)
 42|    dropout: float = 0.1
 43|    weight_tying: bool = True
 44|    pad_token_id: int = PAD_ID
 45|    bos_token_id: int = BOS_ID
 46|    eos_token_id: int = EOS_ID
 47|
 48|    def param_count(self) -> int:
 49|        """Estimate total parameter count."""
 50|        embed = self.vocab_size * self.d_model
 51|        pos = self.max_seq_len * self.d_model
 52|        per_layer = 12 * self.d_model * self.d_model + 4 * self.d_model  # approx
 53|        layers = self.n_layers * per_layer
 54|        output = 0 if self.weight_tying else self.vocab_size * self.d_model
 55|        return embed + pos + layers + output
 56|
 57|    def model_size_mb(self, bytes_per_param: int = 4) -> float:
 58|        return self.param_count() * bytes_per_param / (1024 * 1024)
 59|
 60|    def to_dict(self) -> dict:
 61|        return asdict(self)
 62|
 63|    @classmethod
 64|    def from_dict(cls, d: dict) -> "ModelConfig":
 65|        return cls(**d)
 66|
 67|
 68|class CausalSelfAttention(nn.Module):
 69|    """Multi-head causal self-attention with pre-LayerNorm."""
 70|
 71|    def __init__(self, config: ModelConfig):
 72|        super().__init__()
 73|        assert config.d_model % config.n_heads == 0
 74|        self.n_heads = config.n_heads
 75|        self.d_head = config.d_model // config.n_heads
 76|        self.d_model = config.d_model
 77|        self.scale = 1.0 / math.sqrt(self.d_head)
 78|
 79|        # Fused QKV projection
 80|        self.qkv = nn.Linear(config.d_model, 3 * config.d_model, bias=False)
 81|        self.proj = nn.Linear(config.d_model, config.d_model, bias=False)
 82|        self.dropout = config.dropout
 83|
 84|    def forward(self, x: torch.Tensor) -> torch.Tensor:
 85|        B, T, C = x.shape
 86|        qkv = self.qkv(x)  # (B, T, 3*C)
 87|        qkv = qkv.view(B, T, 3, self.n_heads, self.d_head)
 88|        qkv = qkv.permute(2, 0, 3, 1, 4)  # (3, B, n_heads, T, d_head)
 89|        q, k, v = qkv[0], qkv[1], qkv[2]
 90|
 91|        # Scaled dot-product attention with causal mask
 92|        att = torch.matmul(q, k.transpose(-2, -1)) * self.scale  # (B, n_heads, T, T)
 93|        # Causal mask
 94|        mask = torch.triu(torch.ones(T, T, device=x.device, dtype=torch.bool), diagonal=1)
 95|        att = att.masked_fill(mask, float("-inf"))
 96|        att = F.softmax(att, dim=-1)
 97|        if self.dropout > 0 and self.training:
 98|            att = F.dropout(att, p=self.dropout)
 99|
100|        out = torch.matmul(att, v)  # (B, n_heads, T, d_head)
101|        out = out.transpose(1, 2).contiguous().view(B, T, C)
102|        out = self.proj(out)
103|        if self.dropout > 0 and self.training:
104|            out = F.dropout(out, p=self.dropout)
105|        return out
106|
107|
108|class FeedForward(nn.Module):
109|    """Position-wise feed-forward network with GELU."""
110|
111|    def __init__(self, config: ModelConfig):
112|        super().__init__()
113|        self.fc1 = nn.Linear(config.d_model, config.d_ff, bias=False)
114|        self.fc2 = nn.Linear(config.d_ff, config.d_model, bias=False)
115|        self.dropout = config.dropout
116|
117|    def forward(self, x: torch.Tensor) -> torch.Tensor:
118|        x = self.fc1(x)
119|        x = F.gelu(x)
120|        if self.dropout > 0 and self.training:
121|            x = F.dropout(x, p=self.dropout)
122|        x = self.fc2(x)
123|        if self.dropout > 0 and self.training:
124|            x = F.dropout(x, p=self.dropout)
125|        return x
126|
127|
128|class TransformerBlock(nn.Module):
129|    """Pre-LayerNorm transformer block."""
130|
131|    def __init__(self, config: ModelConfig):
132|        super().__init__()
133|        self.ln1 = nn.LayerNorm(config.d_model, bias=False)
134|        self.attn = CausalSelfAttention(config)
135|        self.ln2 = nn.LayerNorm(config.d_model, bias=False)
136|        self.ffn = FeedForward(config)
137|
138|    def forward(self, x: torch.Tensor) -> torch.Tensor:
139|        x = x + self.attn(self.ln1(x))
140|        x = x + self.ffn(self.ln2(x))
141|        return x
142|
143|
144|class ClickHouseGPT(nn.Module):
145|    """GPT model for ClickHouse query autocomplete.
146|
147|    Architecture:
148|      - Token embedding + learned positional embedding
149|      - N pre-LayerNorm transformer blocks
150|      - Final LayerNorm
151|      - Output projection (tied with token embedding if weight_tying=True)
152|    """
153|
154|    def __init__(self, config: ModelConfig):
155|        super().__init__()
156|        self.config = config
157|        self.tok_emb = nn.Embedding(config.vocab_size, config.d_model, padding_idx=config.pad_token_id)
158|        self.pos_emb = nn.Embedding(config.max_seq_len, config.d_model)
159|        self.blocks = nn.ModuleList([TransformerBlock(config) for _ in range(config.n_layers)])
160|        self.ln_f = nn.LayerNorm(config.d_model, bias=False)
161|        self.dropout = config.dropout
162|
163|        if config.weight_tying:
164|            # Tie output projection weights to token embedding
165|            self.lm_head = None  # use tok_emb.weight in forward
166|        else:
167|            self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)
168|
169|        # Initialize weights
170|        self.apply(self._init_weights)
171|
172|    def _init_weights(self, module: nn.Module):
173|        if isinstance(module, nn.Linear):
174|            nn.init.normal_(module.weight, mean=0.0, std=0.02)
175|            if module.bias is not None:
176|                nn.init.zeros_(module.bias)
177|        elif isinstance(module, nn.Embedding):
178|            nn.init.normal_(module.weight, mean=0.0, std=0.02)
179|            if module.padding_idx is not None:
180|                with torch.no_grad():
181|                    module.weight[module.padding_idx].fill_(0)
182|
183|    def forward(self, idx: torch.Tensor, targets: Optional[torch.Tensor] = None) -> dict:
184|        """
185|        Args:
186|            idx: (B, T) token IDs
187|            targets: (B, T) target token IDs for loss computation
188|
189|        Returns:
190|            dict with 'logits' (B, T, V) and optionally 'loss'
191|        """
192|        B, T = idx.shape
193|        assert T <= self.config.max_seq_len, f"Sequence length {T} exceeds max {self.config.max_seq_len}"
194|
195|        # Embeddings
196|        tok = self.tok_emb(idx)  # (B, T, d_model)
197|        pos = self.pos_emb(torch.arange(T, device=idx.device).unsqueeze(0))  # (1, T, d_model)
198|        x = tok + pos
199|        if self.dropout > 0 and self.training:
200|            x = F.dropout(x, p=self.dropout)
201|
202|        # Transformer blocks
203|        for block in self.blocks:
204|            x = block(x)
205|
206|        x = self.ln_f(x)  # (B, T, d_model)
207|
208|        # Output projection
209|        if self.config.weight_tying:
210|            logits = x @ self.tok_emb.weight.T  # (B, T, vocab_size)
211|        else:
212|            logits = self.lm_head(x)
213|
214|        result = {"logits": logits}
215|
216|        if targets is not None:
217|            # Cross-entropy loss, ignoring padding
218|            loss = F.cross_entropy(
219|                logits.view(-1, logits.size(-1)),
220|                targets.view(-1),
221|                ignore_index=self.config.pad_token_id,
222|            )
223|            result["loss"] = loss
224|
225|        return result
226|
227|    @torch.no_grad()
228|    def generate(
229|        self,
230|        idx: torch.Tensor,
231|        max_new_tokens: int,
232|        temperature: float = 1.0,
233|        top_k: int = 0,
234|        top_p: float = 0.0,
235|        eos_token_id: Optional[int] = None,
236|    ) -> torch.Tensor:
237|        """Autoregressive generation with optional top-k / top-p sampling.
238|
239|        Args:
240|            idx: (1, T) starting token IDs (the prefix)
241|            max_new_tokens: maximum number of tokens to generate
242|            temperature: sampling temperature (1.0 = no change, <1 = sharper)
243|            top_k: if > 0, only sample from top-k logits
244|            top_p: if > 0, use nucleus sampling
245|            eos_token_id: if set, stop generation when this token is produced
246|
247|        Returns:
248|            (1, T + max_new_tokens) token IDs
249|        """
250|        self.eval()
251|        eos = eos_token_id if eos_token_id is not None else self.config.eos_token_id
252|
253|        for _ in range(max_new_tokens):
254|            # Crop context if too long
255|            idx_cond = idx if idx.size(1) <= self.config.max_seq_len else idx[:, -self.config.max_seq_len:]
256|
257|            logits = self(idx_cond)["logits"]  # (1, T, V)
258|            logits = logits[:, -1, :] / max(temperature, 1e-8)  # (1, V)
259|
260|            # Apply top-k filtering
261|            if top_k > 0:
262|                v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
263|                logits[logits < v[:, [-1]]] = float("-inf")
264|
265|            # Apply top-p (nucleus) filtering
266|            if top_p > 0.0:
267|                sorted_logits, sorted_indices = torch.sort(logits, descending=True)
268|                cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
269|                sorted_indices_to_remove = cumulative_probs > top_p
270|                sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
271|                sorted_indices_to_remove[..., 0] = False
272|                indices_to_remove = sorted_indices_to_remove.scatter(
273|                    1, sorted_indices, sorted_indices_to_remove
274|                )
275|                logits = logits.masked_fill(indices_to_remove, float("-inf"))
276|
277|            probs = F.softmax(logits, dim=-1)
278|            next_token = torch.multinomial(probs, num_samples=1)  # (1, 1)
279|            idx = torch.cat([idx, next_token], dim=1)
280|
281|            if eos is not None and next_token.item() == eos:
282|                break
283|
284|        return idx
285|
286|    def num_parameters(self) -> int:
287|        return sum(p.numel() for p in self.parameters())
288|
289|    def model_size_mb(self, bytes_per_param: int = 4) -> float:
290|        return self.num_parameters() * bytes_per_param / (1024 * 1024)
291|
292|
293|def create_model(config: Optional[ModelConfig] = None) -> ClickHouseGPT:
294|    """Create a ClickHouseGPT model with the given or default config."""
295|    if config is None:
296|        config = ModelConfig()
297|    model = ClickHouseGPT(config)
298|    return model
299|
300|
301|if __name__ == "__main__":
302|    config = ModelConfig()
303|    model = create_model(config)
304|    print(f"Model config: {config.to_dict()}")
305|    print(f"Parameters: {model.num_parameters():,}")
306|    print(f"Model size (float32): {model.model_size_mb(4):.2f} MB")
307|    print(f"Model size (float16): {model.model_size_mb(2):.2f} MB")
308|    print(f"Model size (int8):    {model.model_size_mb(1):.2f} MB")
309|    print(f"Estimated params:     {config.param_count():,}")
310|    print(f"Estimated size (f32): {config.model_size_mb(4):.2f} MB")
311|
312|    # Quick forward pass test
313|    import sys
314|    sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent / "tokenizer"))
315|    from ch_tokenizer import tokenize_to_ids
316|
317|    query = "SELECT count(), avg(price) FROM orders WHERE region = 'US' GROUP BY city ORDER BY count() DESC LIMIT 10"
318|    ids = tokenize_to_ids(query)
319|    print(f"\nTokenized query ({len(ids)} tokens): {ids[:10]}...")
320|
321|    x = torch.tensor([ids])
322|    out = model(x, targets=x)
323|    print(f"Logits shape: {out['logits'].shape}")
324|    print(f"Loss: {out['loss'].item():.4f}")
325|
326|    # Generation test
327|    prefix = torch.tensor([[BOS_ID]])
328|    generated = model.generate(prefix, max_new_tokens=20, temperature=0.8, top_k=10)
329|    print(f"Generated {generated.size(1)} tokens")
</file-view>

# Web Search Results for "ClickHouse lexer token types identifier string literal"

## 1. src/Parsers/Lexer.h at 77a7d29d · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/77a7d29d/src/Parsers/Lexer.h

#define APPLY_FOR_TOKENS(M) \
    M(Whitespace) \
    M(Comment) \
    \
    M(BareWord)               /** Either keyword (SELECT) or identifier (column) */ \
    \
    M(Number)                 /** Always non-negative. No leading plus. 123 or something like 123.456e12, 0x123p12 */ \
    M(StringLiteral)          /** 'hello word', 'hello''word', 'hello\'word\\' */ \
    \
    M(QuotedIdentifier)       /** "x", `x` */ \
    \
    M(OpeningRoundBracket) \
    M(ClosingRoundBracket) \
    \
    M(OpeningSquareBracket) \
    M(ClosingSquareBracket) \
    \
    M(OpeningCurlyBrace) \
    M(ClosingCurlyBrace) \
    \
    M(Comma) \
    M(Semicolon) \
    M(VerticalDelimiter)      /** Vertical delimiter \G */ \
    M(Dot)                    /** Compound identifiers, like a.b or tuple access operator a.1, (x, y).2. */ \
                              /** Need to be distinguished from floating point number with omitted integer part: .1 */ \
    \
    M(Asterisk)               /** Could be used as multiplication operator or on it's own: "SELECT *" */ \
    \
    M(HereDoc) \
    \
    M(DollarSign) \
    M(Plus) \
    M(Minus) \
    M(Slash) \
    M(Percent) \
    M(Arrow)                  /** ->. Should be distinguished from minus operator. */ \
    M(QuestionMark) \
    M(Colon) \
    M(Caret) \
    M(DoubleColon) \
    M(Equals) \
    M(NotEquals) \
    M(Less) \
    M(Greater) \
    M(LessOrEquals) \
    M(GreaterOrEquals) \
    M(Spaceship)              /** <=>. Used in MySQL for NULL-safe equality comparison. */ \
    M(PipeMark) \
    M(Concatenation)          /** String concatenation operator: || */ \
    \
    M(At)                     /** @. Used for specifying user names and also for MySQL-style variables. */ \
    M(DoubleAt)               /** @@. Used for MySQL-style global variables. */ \
    \
    /** Order is important. EndOfStream goes after all usual tokens, and special error tokens goes after EndOfStream. */ \
    \
    M(EndOfStream) \
    \
    /** Somethi...

## 2. utils/antlr/ClickHouseLexer.g4
URL: https://github.com/ClickHouse/ClickHouse/blob/master/utils/antlr/ClickHouseLexer.g4

HEXADECIMAL_STRING_LITERAL: X QUOTE_SINGLE HEX_DIGIT+ QUOTE_SINGLE;
...
BINARY_STRING_LITERAL: B QUOTE_SINGLE BINARY_DIGIT+ QUOTE_SINGLE;
...
IDENTIFIER
 : (LETTER | UNDERSCORE) (LETTER | UNDERSCORE | DEC_DIGIT)*
 | BACKQUOTE ( ~([\\`]) | (BACKSLASH .) | (BACKQUOTE BACKQUOTE) )* BACKQUOTE
 | QUOTE_DOUBLE ( ~([\\"]) | (BACKSLASH .) | (QUOTE_DOUBLE QUOTE_DOUBLE) )* QUOTE_DOUBLE
 ;
...
FLOATING_LITERAL
 : HEXADECIMAL_NUMERIC_LITERAL DOT HEX_DIGIT* (P | E) (PLUS | DASH)? DEC_DIGIT+
 | HEXADECIMAL_NUMERIC_LITERAL (P | E) (PLUS | DASH)? DEC_DIGIT+
 | DECIMAL_LITERAL DOT DEC_DIGIT* E (PLUS | DASH)? DEC_DIGIT+
 | DOT DECIMAL_LITERAL E (PLUS | DASH)? DEC_DIGIT+
 | DECIMAL_LITERAL E (PLUS | DASH)? DEC_DIGIT+
 ;
...
OCTAL_LITERAL: '0' OCT_DIGIT+;
...
DECIMAL_LITERAL: DEC_DIGIT+;
...
HEXADECIMAL_NUMERIC_LITERAL: '0' X HEX_DIGIT+;
...
BINARY_NUMERIC_LITERAL: '0' B BINARY_DIGIT+;
...
// It's important that quote-symbol is a single character.
...
STRING_LITERAL: QUOTE_SINGLE ( ~([\\']) | (BACKSLASH .) | (QUOTE_SINGLE QUOTE_SINGLE) )* QUOTE_SINGLE;

## 3. src/Parsers/Lexer.cpp at 77a7d29d · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/77a7d29d/src/Parsers/Lexer.cpp

Token quotedHexOrBinString(const char *& pos, const char * const token_begin, const char * const end)
{
    constexpr char
...
= '\'';

    chassert(pos[1] == quote);

    bool hex = (*pos == 'x' || *pos == 'X');

    pos += 2;

    if (hex)
    {
        while (pos < end && isHexDigit(*pos))
            ++pos;
    }
    else
    {
        pos = find_first_not_symbols<'0', '1'>(pos, end);
    }

    if (pos >= end || *pos != quote)
    {
        pos = end;
        return Token(TokenType::ErrorSingleQuoteIsNotClosed, token_begin, end);
    }

    ++pos;
    return Token(TokenType::StringLiteral, token_begin, pos);
}

}
...
exponentiation (base
...
10 or base
...
1 < end
...
(*pos ==
...
p' || *pos ==
...
P') : (*pos == 'e' || *pos
...
start_of_block = true;
                    ++pos
...
/// sign of exponent.
...
is always decimal.
...
1 < end && (*pos
...
'-' || *pos
...
String<'\''>(pos,
...
, TokenType::StringLiteral, TokenType::ErrorSingle
...
case '"':
...
quotedString<
...
'>(pos,
...
, TokenType::QuotedIdentifier
...
case '`':
...
'>(pos,
...
, TokenType::QuotedIdentifier, TokenType::
...
BackQuoteIsNotClosed);

        case '(':
...
Token(TokenType::OpeningRoundBracket, token
...
begin, ++pos);
        case ')':
            return Token(TokenType::ClosingRoundBracket, token_begin, ++pos);
        case '[':
            return Token(TokenType::OpeningSquareBracket, token_begin, ++pos);
        case ']':
            return Token(TokenType::ClosingSquareBracket, token_begin, ++pos);
        case '{':
            return Token(TokenType::OpeningCurlyBrace, token_begin, ++pos);
        case '}':
            return Token(TokenType::ClosingCurlyBrace, token_begin, ++pos);
        case ',':
            return Token(TokenType::Comma, token_begin, ++pos);
        case ';':
            return Token(TokenType::Semicolon, token_begin, ++pos);

        case '.':   /// qualifier, tuple access operator or start of floating point number
        {
            /// Just after ident...

## 4. src/Parsers/IParser.h at 880f38dc · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/880f38dc/src/Parsers/IParser.h

/// Highlight types for syntax highlighting of parsed queries.
/// APPLY_FOR_HIGHLIGHTS lists the publicly visible types (used in the output of `highlightQuery`).
/// string_like and string_regexp are internal types used by the parser before expansion.
#define APPLY_FOR_HIGHLIGHTS(M) \
    M(none) \
    M(keyword) \
    M(identifier) \
    M(function) \
    M(alias) \
    M(substitution) \
    M(number) \
    M(string) \
    M(string_escape) \
    M(string_metacharacter)

enum class Highlight : uint8_t
{
#define M(NAME) NAME,
    APPLY_FOR_HIGHLIGHTS(M)
#undef M
    /// These two are used internally by the parser to mark LIKE/REGEXP string ranges.
    /// They are expanded into string/string_escape/string_metacharacter by `expandHighlights`.
    string_like,
    string_regexp,
};
...
/// Expand string_like and string_regexp ranges into character-level sub-ranges
/// with string, string_escape, and string_metacharacter highlight types.
/// Other ranges are passed through unchanged.
std::vector<HighlightedRange> expandHighlights(const std::set<HighlightedRange> & highlights);
...
/** Collects variants, how parser could proceed further at rightmost position.
  * Also collects a mapping of parsed ranges for highlighting,
  * which is accumulated through the parsing.
  */
struct Expected
{
    absl::InlinedVector<const char *, 7> variants;
    const char * max_parsed_pos = nullptr;

    bool enable_highlighting = false;
    std::set<HighlightedRange> highlights;

    /// Optional map for capturing literal token positions during parsing.
    /// Used by ValuesBlockInputFormat for ConstantExpressionTemplate construction
    /// and for LIKE/REGEXP syntax highlighting.
    /// The caller must allocate and manage the map's lifetime.
    LiteralTokenMap * literal_token_map = nullptr;

    /// 'description' should be statically allocated string.
    ALWAYS_INLINE void add(const char * current_pos, const char * description)
    {
        if (!max_parsed_pos || current_pos > max...

## 5. Add tokenizeQuery and highlightQuery functions · Pull Request #101054 · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/pull/101054

`tokenizeQuery(query)` takes a query string and returns an array of named tuples `(begin UInt64, end UInt64, type Enum8(...))` with lexer tokens. `highlightQuery(query)` does the same but returns syntax highlighting ranges from the parser, useful for UI editors and query log analysis.
...
Add `tokenizeQuery` and `highlightQuery` functions for SQL query tokenization and syntax highlighting. `tokenizeQuery` returns lexer tokens with byte offsets and token types; `highlightQuery` returns parser-based syntax highlighting ranges with highlight categories (keyword, identifier, function, number, string, etc.).
...
- `tokenizeQuery(query)` — Tokenizes a ClickHouse SQL query and returns `Array(Tuple(begin UInt64, end UInt64, type Enum8(...)))` with all lexer tokens including whitespace and comments.
- `highlightQuery(query)` — Parses a ClickHouse SQL query and returns `Array(Tuple(begin UInt64, end UInt64, type Enum8(...)))` with syntax highlighting ranges. Highlight types: `keyword`, `identifier`, `function`, `alias`, `substitution`, `number`, `string`, `string_like`, `string_regexp`.

# Web Search Results for "GPT transformer attention mask causal implementation"

## 1. src/transformers/modeling_attn_mask_utils.py
URL: https://github.com/huggingface/transformers/blob/25245ec26dc29bcf6102e1b4ddd0dfd02e720cf5/src/transformers/modeling_attn_mask_utils.py

@dataclass
class AttentionMaskConverter:
 """
 A utility attention mask class that allows one to:
 - Create a causal 4d mask
 - Create a causal 4d mask with slided window
 - Convert a 2d attention mask (batch_size, query_length) to a 4d attention mask (batch_size, 1, query_length,
 key_value_length) that can be multiplied with attention scores

 Examples:

    ```python
    >>> import torch
    >>> from transformers.modeling_attn_mask_utils import AttentionMaskConverter
...
causal_4d(
 self,
 batch_size: int,
 query_length: int,
 key_value_length: int,
 dtype: torch.dtype,
 device: Union[torch.device, "str"] = "cpu",
 ) -> Optional[torch.Tensor]:
 """
 Creates a causal 4D mask of (bsz, head_dim=1, query_length, key_value_length) shape and adds large negative
 bias to upper right hand triangular matrix (causal mask).
 """
 if not self.is_causal:
 raise ValueError(f"Please use `to_causal_4d` only if {self.__class__} has `is_causal` set to True.")

 # If shape is not cached, create a new causal mask and cache it
 input_shape = (batch_size, query_length)
 past_key_values_length = key_value_length - query_length
...
causal_4d_mask

 def to_4d(
 self,
 attention_mask_2d: torch.Tensor,
 query_length: int,
 dtype: torch.dtype,
 key_value_length: Optional[int] = None,
 ) -> torch.Tensor:
 """
 Converts 2D attention mask to 4D attention mask by expanding mask to (bsz, head_dim=1, query_length,
 key_value_length) shape and by adding a large negative bias to not-attended positions. If attention_mask is
 causal, a causal mask will be added.
 """
 input_shape = (attention_mask_2d.shape[0], query_length)
...
# create causal mask
...
past_key_values_length = key_value_length - query_length
...
def _prepare_4d_causal_attention_mask(
 attention_mask: Optional[torch.Tensor],
 input_shape: Union[torch.Size, Tuple, List],
 inputs_embeds: torch.Tensor,
 past_key_values_length: int,
 sliding_window: Optional[int] = None,
):
 """
 Creates a causal 4D mask of shape `(batch_size, 1, query_le...

## 2. Causal Self-Attention - TorchCode
URL: https://albanna-tutorials.com/torchcode/09_causal_attention.html

Causal Self-Attention - TorchCode

# Causal Self-Attention

Autoregressive masking with -inf, GPT-style

Hard Attention

## Problem Description

Implement causal (masked) self-attention — the attention used in GPT-style decoders.

Same as softmax attention, but each position can only attend to itself and earlier positions (no peeking at future tokens).

$$\text{scores}_{ij} = \begin{cases} \frac{Q_i \cdot K_j}{\sqrt{d_k}} & \text{if } j \le i \\ -\infty & \text{if } j > i \end{cases}$$

### Signature

Copydef causal_attention(Q, K, V): # Q, K, V: (batch, seq, d) → output: (batch, seq, d_v)

### Rules

• Do NOT use`F.scaled_dot_product_attention`

• Position i can only attend to positions \le i

• You may use`torch.softmax`,`torch.bmm`,`torch.triu`

## Template

Implement the function below. Use only basic PyTorch operations.

Copy# ✏️ YOUR IMPLEMENTATION HERE def causal_attention(Q, K, V): pass # Replace this

## Test Your Implementation

Use this code to debug before submitting.

Copy# 🧪 Debug torch.manual_seed(0) Q = torch.randn(1, 4, 8) K = torch.randn(1, 4, 8) V = torch.randn(1, 4, 8) out = causal_attention(Q, K, V) print("Output shape:", out.shape) # (1, 4, 8) print("Pos 0 == V[0]?", torch.allclose(out[:, 0], V[:, 0], atol=1e-5)) # should be True

## Reference Solution

Try solving it yourself first! Click below to reveal the solution.

Show Solution

Copy# ✅ SOLUTION def causal_attention(Q, K, V): d_k = K.size(-1) scores = torch.bmm(Q, K.transpose(1, 2)) / math.sqrt(d_k) S = scores.size(-1) mask = torch.triu(torch.ones(S, S, device=scores.device, dtype=torch.bool), diagonal=1) scores = scores.masked_fill(mask.unsqueeze(0), float('-inf')) weights = torch.softmax(scores, dim=-1) return torch.bmm(weights, V)

## Tips

### Run Locally

For interactive practice with auto-grading, run TorchCode locally:`pip install torch-judge` then use`check("causal_attention")`

### Key Concepts

Autoregressive masking with -inf, GPT-style

## 3. shaheennabi/Decoder-Only-Transformer-Architecture-Implementation-in-Pytorch
URL: https://github.com/shaheennabi/Decoder-Only-Transformer-Architecture-Implementation-in-Pytorch

A complete implementation of a Decoder-Only Transformer (GPT-style) built using PyTorch, without relying on high-level abstractions. This implementation includes all core components: token embeddings, positional embeddings, multi-head self-attention, feedforward networks, causal masking, and output logits generation.
...
A complete implementation of a **Decoder-Only Transformer** (GPT-style) built from the ground up using PyTorch, without relying on high-level abstractions. This implementation includes all core components: token embeddings, positional embeddings, multi-head self-attention, feedforward networks, causal masking, and output logits generation.
...
✅ Pure PyTorch implementation from scratch 
✅ Multi-head self-attention with causal masking 
✅ Layer normalization and residual connections 
✅ GELU activation function 
✅ Learnable token and positional embeddings 
✅ Configurable model dimensions and layers 

---
...
### 4. **Causal Masking**
...
Prevents the model from attending to future tokens during training/inference (autoregressive constraint).
...
blocks:

  ┌─────────────────────────────────────────┐
  │                                         │
  │  [STEP 4a] MULTI-HEAD SELF-ATTENTION  │
  │  ─────────────────────────────────     │
  │  • Compute Q, K, V from input          │
  │  • Split into n_heads (e.
...
., 8)        │
  │  • Each head: Attention(Q,K,V)         │
  │  • Apply CAUSAL
...
)  │
  │  • Concatenate heads                   │
  │  • Output projection                   │
  │  Residual
...
(x)│
  │
...
LAYER NORM
...
] FEEDFORWARD NETWORK
...
──────────────────────────────        │
  │
...
│
  │  • GELU
...
│
  │
...
Linear(2048 → 512)                  │
  │  Residual connection: x + FFN(x)      │
  │                                        │
  │  Output: (1, 4, 512)                   │
  └─────────────────────────────────────────┘
       ↓
...
repeat for N layers
...
### Causal Masking
...
**Purpose**: Enforce autoregressive constraint (to...

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/tokenizer/ch_tokenizer.py" start_line="1" end_line="100" total_lines="514">
  1|"""
  2|ClickHouse Lexer-based tokenizer with lookbehind copies.
  3|
  4|This tokenizer mirrors the ClickHouse Lexer (src/Parsers/Lexer.h) token types
  5|but replaces identifier and string-literal values with "lookbehind copy" tokens.
  6|
  7|Instead of encoding identifier/string values literally (which would require an
  8|unbounded vocabulary), each identifier or string is represented as either:
  9|  - NEW_IDENT / NEW_STRING  — first occurrence in the sequence
 10|  - COPY_BACK_N / COPY_STR_BACK_N — a repeat of a value seen N tokens ago
 11|
 12|This keeps the vocabulary small (~350 tokens) while allowing the model to
 13|reproduce arbitrary identifiers and strings via a pointer/copy mechanism at
 14|inference time.
 15|
 16|The tokenizer also stores the raw text for each identifier/string so that the
 17|inference engine can resolve copy references.
 18|"""
 19|
 20|import re
 21|import json
 22|from dataclasses import dataclass, field
 23|from typing import Optional
 24|
 25|
 26|# ─── ClickHouse keywords (from utils/antlr/ClickHouseLexer.g4) ───────────────
 27|
 28|KEYWORDS = [
 29|    "ACCESS", "ADD", "AFTER", "ALIAS", "ALL", "ALTER", "AND", "ANTI", "ANY",
 30|    "ARRAY", "AS", "ASC", "ASCENDING", "ASOF", "AST", "ASYNC", "ATTACH",
 31|    "BETWEEN", "BOTH", "BY", "CACHES", "CASE", "CAST", "CHECK", "CLEAR",
 32|    "CLUSTER", "CLUSTERS", "CODEC", "COLLATE", "COLUMN", "COLUMNS", "COMMENT",
 33|    "CONSTRAINT", "CREATE", "CROSS", "CUBE", "CURRENT", "CURRENT_USER",
 34|    "CHANGED", "DATABASE", "DATABASES", "DATE", "DAY", "DEDUPLICATE",
 35|    "DEFAULT", "DELAY", "DELETE", "DESC", "DESCENDING", "DESCRIBE", "DETACH",
 36|    "DICTIONARIES", "DICTIONARY", "DISK", "DISTINCT", "DISTRIBUTED", "DROP",
 37|    "ELSE", "ENABLED", "END", "ENGINE", "ENGINES", "ESTIMATE", "EVENTS",
 38|    "EXCEPT", "EXISTS", "EXPLAIN", "EXPRESSION", "EXTENDED", "EXTRACT",
 39|    "FETCHES", "FIELDS", "FILESYSTEM", "FILL", "FINAL", "FIRST", "FLUSH",
 40|    "FOLLOWING", "FOR", "FORMAT", "FREEZE", "FROM", "FULL", "FUNCTION",
 41|    "FUNCTIONS", "GLOBAL", "GRANULARITY", "GRANTS", "GROUP", "GROUPING",
 42|    "HAVING", "HIERARCHICAL", "HOUR", "ID", "IF", "ILIKE", "IMPLICIT", "IN",
 43|    "INDEX", "INDEXES", "INDICES", "INF", "INFINITY", "INJECTIVE", "INNER",
 44|    "INSERT", "INTERPOLATE", "INTERVAL", "INTO", "IS", "IS_OBJECT_ID", "JOIN",
 45|    "KEY", "KEYS", "KILL", "LAST", "LAYOUT", "LEADING", "LEFT", "LIFETIME",
 46|    "LIKE", "LIMIT", "LIVE", "LOCAL", "LOGS", "MATERIALIZE", "MATERIALIZED",
 47|    "MAX", "MERGES", "MICROSECOND", "MILLISECOND", "MIN", "MINUTE", "MODIFY",
 48|    "MONTH", "MOVE", "MUTATION", "NAN", "NANOSECOND", "NO", "NOT", "NULL",
 49|    "NULLS", "OFFSET", "ON", "OPTIMIZE", "OR", "ORDER", "OUTER", "OUTFILE",
 50|    "OVER", "OVERRIDE", "PARTITION", "PIPELINE", "PLAN", "POLICY", "POLICIES",
 51|    "POPULATE", "PRECEDING", "PREWHERE", "PRIMARY", "PRIVILEGES",
 52|    "PROCESSLIST", "PROFILE", "PROFILES", "PROJECTION", "QUARTER", "QUOTA",
 53|    "QUOTAS", "RANGE", "RECURSIVE", "RELOAD", "REMOVE", "RENAME", "REPLACE",
 54|    "REPLICA", "REPLICATED", "RIGHT", "ROLE", "ROLES", "ROLLUP", "ROW",
 55|    "ROWS", "SAMPLE", "SECOND", "SELECT", "SEMI", "SENDS", "SET", "SETS",
 56|    "SETTING", "SETTINGS", "SHOW", "SOURCE", "START", "STOP", "SUBSTRING",
 57|    "SYNC", "SYNTAX", "SYSTEM", "STEP", "TABLE", "TABLES", "TEMPORARY",
 58|    "TEST", "THEN", "TIES", "TIMEOUT", "TIMESTAMP", "TO", "TOP", "TOTALS",
 59|    "TRAILING", "TREE", "TRIM", "TRUNCATE", "TTL", "TYPE", "UNBOUNDED",
 60|    "UNION", "UPDATE", "USE", "USER", "USERS", "USING", "UUID", "VALUES",
 61|    "VIEW", "VOLUME", "WATCH", "WEEK", "WHEN", "WHERE", "WINDOW", "WITH",
 62|    "YEAR", "YYYY", "FALSE", "TRUE",
 63|]
 64|
 65|# Build a case-insensitive lookup set (ClickHouse keywords are case-insensitive)
 66|KEYWORD_SET = {kw.upper() for kw in KEYWORDS}
 67|
 68|
 69|# ─── Token type enum (mirrors ClickHouse TokenType) ──────────────────────────
 70|
 71|# Significant token types from the ClickHouse lexer that we map to vocab entries
 72|SIGNIFICANT_TOKEN_TYPES = [
 73|    "BareWord", "Number", "StringLiteral", "QuotedIdentifier",
 74|    "OpeningRoundBracket", "ClosingRoundBracket",
 75|    "OpeningSquareBracket", "ClosingSquareBracket",
 76|    "OpeningCurlyBrace", "ClosingCurlyBrace",
 77|    "Comma", "Semicolon", "Dot", "Asterisk",
 78|    "Plus", "Minus", "Slash", "Percent",
 79|    "Arrow", "QuestionMark", "Colon", "Caret", "DoubleColon",
 80|    "Equals", "NotEquals", "Less", "Greater",
 81|    "LessOrEquals", "GreaterOrEquals", "Spaceship",
 82|    "Concatenation", "At", "DoubleAt",
 83|]
 84|
 85|# ─── Vocabulary construction ─────────────────────────────────────────────────
 86|
 87|MAX_COPY_BACK = 32  # maximum lookbehind distance for copy tokens
 88|
 89|SPECIAL_TOKENS = ["<PAD>", "<BOS>", "<EOS>", "<UNK>"]
 90|
 91|# Operator/punctuation tokens with their string representations
 92|OPERATOR_TOKENS = {
 93|    "LPAREN": "(", "RPAREN": ")",
 94|    "LBRACKET": "[", "RBRACKET": "]",
 95|    "LBRACE": "{", "RBRACE": "}",
 96|    "COMMA": ",", "SEMICOLON": ";",
 97|    "DOT": ".", "ASTERISK": "*",
 98|    "PLUS": "+", "MINUS": "-",
 99|    "SLASH": "/", "PERCENT": "%",
100|    "ARROW": "->",
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/model/config.py" start_line="1" end_line="100" total_lines="329">
  1|"""
  2|GPT-based Small Language Model for ClickHouse query autocomplete.
  3|
  4|Design goals:
  5|  - ~10 MB model size (float32) → ~2.7M parameters
  6|  - CPU-optimized: small dimensions, weight tying, pre-LayerNorm
  7|  - Embeddable in C++ (ClickHouse binary) and exportable to WebAssembly
  8|  - Constrained sampling at inference time (syntactic correctness + table dicts)
  9|
 10|Architecture:
 11|  - d_model = 192, n_layers = 6, n_heads = 6, d_ff = 768
 12|  - max_seq_len = 256
 13|  - Weight tying (input embedding == output projection)
 14|  - Pre-LayerNorm transformer blocks
 15|  - Learned positional embeddings
 16|  - GELU activation in FFN
 17|  - ~2.77M params ≈ 10.6 MB (float32), ~5.3 MB (float16), ~2.7 MB (int8)
 18|"""
 19|
 20|import math
 21|import torch
 22|import torch.nn as nn
 23|import torch.nn.functional as F
 24|from dataclasses import dataclass, asdict
 25|from typing import Optional
 26|
 27|import sys
 28|sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent.parent / "tokenizer"))
 29|from ch_tokenizer import VOCAB_SIZE, PAD_ID, BOS_ID, EOS_ID
 30|
 31|
 32|@dataclass
 33|class ModelConfig:
 34|    """Configuration for the ClickHouse autocomplete GPT model."""
 35|    vocab_size: int = VOCAB_SIZE
 36|    d_model: int = 192
 37|    n_layers: int = 6
 38|    n_heads: int = 6
 39|    d_ff: int = 768
 40|    max_seq_len: int = 256
 41|    n_heads_kv: int = 6       # no GQA (d_model is small)
 42|    dropout: float = 0.1
 43|    weight_tying: bool = True
 44|    pad_token_id: int = PAD_ID
 45|    bos_token_id: int = BOS_ID
 46|    eos_token_id: int = EOS_ID
 47|
 48|    def param_count(self) -> int:
 49|        """Estimate total parameter count."""
 50|        embed = self.vocab_size * self.d_model
 51|        pos = self.max_seq_len * self.d_model
 52|        per_layer = 12 * self.d_model * self.d_model + 4 * self.d_model  # approx
 53|        layers = self.n_layers * per_layer
 54|        output = 0 if self.weight_tying else self.vocab_size * self.d_model
 55|        return embed + pos + layers + output
 56|
 57|    def model_size_mb(self, bytes_per_param: int = 4) -> float:
 58|        return self.param_count() * bytes_per_param / (1024 * 1024)
 59|
 60|    def to_dict(self) -> dict:
 61|        return asdict(self)
 62|
 63|    @classmethod
 64|    def from_dict(cls, d: dict) -> "ModelConfig":
 65|        return cls(**d)
 66|
 67|
 68|class CausalSelfAttention(nn.Module):
 69|    """Multi-head causal self-attention with pre-LayerNorm."""
 70|
 71|    def __init__(self, config: ModelConfig):
 72|        super().__init__()
 73|        assert config.d_model % config.n_heads == 0
 74|        self.n_heads = config.n_heads
 75|        self.d_head = config.d_model // config.n_heads
 76|        self.d_model = config.d_model
 77|        self.scale = 1.0 / math.sqrt(self.d_head)
 78|
 79|        # Fused QKV projection
 80|        self.qkv = nn.Linear(config.d_model, 3 * config.d_model, bias=False)
 81|        self.proj = nn.Linear(config.d_model, config.d_model, bias=False)
 82|        self.dropout = config.dropout
 83|
 84|    def forward(self, x: torch.Tensor) -> torch.Tensor:
 85|        B, T, C = x.shape
 86|        qkv = self.qkv(x)  # (B, T, 3*C)
 87|        qkv = qkv.view(B, T, 3, self.n_heads, self.d_head)
 88|        qkv = qkv.permute(2, 0, 3, 1, 4)  # (3, B, n_heads, T, d_head)
 89|        q, k, v = qkv[0], qkv[1], qkv[2]
 90|
 91|        # Scaled dot-product attention with causal mask
 92|        att = torch.matmul(q, k.transpose(-2, -1)) * self.scale  # (B, n_heads, T, T)
 93|        # Causal mask
 94|        mask = torch.triu(torch.ones(T, T, device=x.device, dtype=torch.bool), diagonal=1)
 95|        att = att.masked_fill(mask, float("-inf"))
 96|        att = F.softmax(att, dim=-1)
 97|        if self.dropout > 0 and self.training:
 98|            att = F.dropout(att, p=self.dropout)
 99|
100|        out = torch.matmul(att, v)  # (B, n_heads, T, d_head)
</file-view>

/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_01_basic_select.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_02_aggregation.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_03_joins.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_04_ddl_dml.jsonl
/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_05_advanced.jsonl

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/inference/autocomplete.py" start_line="1" end_line="100" total_lines="497">
  1|"""
  2|Inference engine for ClickHouse query autocomplete.
  3|
  4|Features:
  5|  - Constrained sampling based on syntactic correctness
  6|  - Table/column dictionary awareness for identifier prediction
  7|  - Copy-back resolution for identifiers and strings
  8|  - Top-k / top-p sampling with temperature
  9|  - Beam search option
 10|
 11|The constrained sampler uses a transition table that defines which token types
 12|are valid after the current token, based on ClickHouse SQL grammar rules.
 13|"""
 14|
 15|import sys
 16|import torch
 17|import torch.nn.functional as F
 18|from pathlib import Path
 19|from typing import Optional
 20|from dataclasses import dataclass, field
 21|
 22|sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tokenizer"))
 23|sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "model"))
 24|
 25|from ch_tokenizer import (
 26|    tokenize, lex, encode_tokens, EncodedToken,
 27|    VOCAB, VOCAB_SIZE, ID_TO_TOKEN,
 28|    BOS_ID, EOS_ID, PAD_ID, UNK_ID,
 29|    KEYWORDS, KEYWORD_SET, OPERATOR_TOKENS,
 30|    IDENT_COPY_TOKENS, STRING_COPY_TOKENS, NUMBER_TOKENS,
 31|    MAX_COPY_BACK,
 32|)
 33|from config import ModelConfig, ClickHouseGPT, create_model
 34|
 35|
 36|# ─── Grammar-based valid next token sets ─────────────────────────────────────
 37|
 38|# Token categories for constraint checking
 39|KW_CLAUSES = {"KW_SELECT", "KW_FROM", "KW_WHERE", "KW_GROUP", "KW_BY",
 40|              "KW_HAVING", "KW_ORDER", "KW_LIMIT", "KW_OFFSET", "KW_UNION",
 41|              "KW_JOIN", "KW_LEFT", "KW_RIGHT", "KW_INNER", "KW_FULL",
 42|              "KW_CROSS", "KW_ON", "KW_USING", "KW_AS", "KW_AND", "KW_OR",
 43|              "KW_NOT", "KW_IN", "KW_BETWEEN", "KW_LIKE", "KW_IS",
 44|              "KW_NULL", "KW_DISTINCT", "KW_ALL", "KW_ASC", "KW_DESC",
 45|              "KW_INSERT", "KW_INTO", "KW_VALUES", "KW_CREATE", "KW_TABLE",
 46|              "KW_IF", "KW_EXISTS", "KW_ENGINE", "KW_PARTITION", "KW_TTL",
 47|              "KW_SETTINGS", "KW_ALTER", "KW_ADD", "KW_DROP", "KW_MODIFY",
 48|              "KW_COLUMN", "KW_INDEX", "KW_WITH", "KW_CASE", "KW_WHEN",
 49|              "KW_THEN", "KW_ELSE", "KW_END", "KW_OVER", "KW_PARTITION",
 50|              "KW_SAMPLE", "KW_FINAL", "KW_PREWHERE", "KW_ARRAY", "KW_SAMPLE"}
 51|
 52|KW_START = {"KW_SELECT", "KW_CREATE", "KW_INSERT", "KW_ALTER", "KW_DROP",
 53|            "KW_WITH", "KW_SHOW", "KW_DESCRIBE", "KW_EXPLAIN", "KW_SYSTEM",
 54|            "KW_OPTIMIZE", "KW_TRUNCATE", "KW_USE", "KW_SET", "KW_KILL",
 55|            "KW_ATTACH", "KW_DETACH", "KW_RENAME", "KW_GRANT", "KW_REVOKE"}
 56|
 57|# Valid tokens after BOS or start of query
 58|VALID_AFTER_START = set()
 59|for tok_str in KW_START:
 60|    if tok_str in VOCAB:
 61|        VALID_AFTER_START.add(VOCAB[tok_str])
 62|
 63|# Valid tokens after SELECT (column list start)
 64|VALID_AFTER_SELECT = set()
 65|for tok_str in ["NEW_IDENT", "ASTERISK", "KW_DISTINCT", "KW_ALL"] + \
 66|                 [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
 67|    if tok_str in VOCAB:
 68|        VALID_AFTER_SELECT.add(VOCAB[tok_str])
 69|
 70|# Valid tokens after FROM (table name)
 71|VALID_AFTER_FROM = set()
 72|for tok_str in ["NEW_IDENT"] + [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
 73|    if tok_str in VOCAB:
 74|        VALID_AFTER_FROM.add(VOCAB[tok_str])
 75|
 76|# Valid tokens after a clause keyword (WHERE, HAVING, ON, etc.)
 77|VALID_AFTER_CONDITION = set()
 78|for tok_str in ["NEW_IDENT", "KW_NOT", "KW_CASE"] + \
 79|                 [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)] + \
 80|                 ["NUM_INT", "NUM_FLOAT", "NUM_HEX"] + \
 81|                 ["NEW_STRING"] + [f"COPY_STR_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
 82|    if tok_str in VOCAB:
 83|        VALID_AFTER_CONDITION.add(VOCAB[tok_str])
 84|
 85|# Valid tokens after an identifier (operators, AS, comma, etc.)
 86|VALID_AFTER_IDENT = set()
 87|for tok_str in ["DOT", "COMMA", "LPAREN", "EQUALS", "NOT_EQUALS", "LESS", "GREATER",
 88|                "LESS_OR_EQUALS", "GREATER_OR_EQUALS", "KW_AS", "KW_AND", "KW_OR",
 89|                "KW_IN", "KW_NOT", "KW_BETWEEN", "KW_LIKE", "KW_IS", "KW_FROM",
 90|                "KW_WHERE", "KW_GROUP", "KW_ORDER", "KW_HAVING", "KW_LIMIT",
 91|                "KW_JOIN", "KW_LEFT", "KW_RIGHT", "KW_INNER", "KW_FULL",
 92|                "KW_UNION", "KW_ON", "KW_USING", "KW_ASC", "KW_DESC",
 93|                "PLUS", "MINUS", "ASTERISK", "SLASH", "PERCENT",
 94|                "SEMICOLON", "EOS_ID"] + \
 95|                [f"COPY_BACK_{i}" for i in range(1, MAX_COPY_BACK+1)]:
 96|    if tok_str in VOCAB:
 97|        VALID_AFTER_IDENT.add(VOCAB[tok_str])
 98|
 99|# Valid tokens after a string literal
100|VALID_AFTER_STRING = VALID_AFTER_IDENT
</file-view>

# Web Search Results for "ClickHouse SQL syntax SAMPLE FINAL clause order"

## 1. SELECT Query | ClickHouse Docs
URL: https://clickhouse.com/docs/sql-reference/statements/select

```sql
[WITH expr_list(subquery)]
SELECT [DISTINCT [ON (column1, column2, ...)]] expr_list
[FROM [db.]table | (subquery) | table_function] [FINAL]
[SAMPLE sample_coeff]
[ARRAY JOIN ...]
[GLOBAL] [ANY|ALL|ASOF] [INNER|LEFT|RIGHT|FULL|CROSS] [OUTER|SEMI|ANTI] JOIN (subquery)|table [(alias1 [, alias2 ...])] (ON <expr_list>)|(USING <column_list>)
[PREWHERE expr]
[WHERE expr]
[GROUP BY expr_list] [WITH ROLLUP|WITH CUBE] [WITH TOTALS]
[HAVING expr]
[WINDOW window_expr_list]
[QUALIFY expr]
[ORDER BY expr_list] [WITH FILL] [FROM expr] [TO expr] [STEP expr] [INTERPOLATE [(expr_list)]]
...
[LIMIT [offset_value, ]n BY columns]
[LIMIT [n, ]m] [WITH TIES]
[SETTINGS ...]
[UNION  ...]
[INTO OUTFILE filename [TRUNCATE] [COMPRESSION type [LEVEL level]] ]
...
[FORMAT format]
...
All clauses are optional, except for the
...
list of expressions immediately after`SELECT` which is
...
in more detail below.

## 2. FROM Clause | ClickHouse Docs
URL: https://clickhouse.com/docs/sql-reference/statements/select/from

## FINAL Modifier​
...
When `FINAL` is specified, ClickHouse fully merges the data before returning the result. This also performs all data transformations that happen during merges for the given table engine.
...
### Example Usage​
...
Using the `FINAL` keyword
...
```sql
SELECT x, y FROM mytable FINAL WHERE x > 1;

```
...
Using `FINAL` as a query-level setting
...
```sql
SELECT x, y FROM mytable WHERE x > 1 SETTINGS final = 1;

```
...
Using `FINAL` as a session-level setting
...
```sql
SET final = 1;
SELECT x, y FROM mytable WHERE x > 1;

```
...
and FINAL​
...
When a table has an alias, `FINAL` comes after the alias. This is most visible in JOIN queries, where tables are usually aliased:
...
```sql
SELECT t1.id, t2.name
FROM table1 AS t1 FINAL
INNER JOIN table2 AS t2 FINAL ON t1.id = t2.id;

```
...
`FINAL` is a modifier on the table reference, so it must follow the full `table [AS alias]` expression. Placing it before the alias (`FROM table1 FINAL AS t1`) is a syntax error.

## 3. SAMPLE Clause | ClickHouse Docs
URL: https://clickhouse.com/docs/sql-reference/statements/select/sample

For the `SAMPLE` clause the following syntax is supported:
...
`SAMPLE k`
...
Here `k` is the number from 0 to 1. The query is executed on `k` fraction of data. For example, `SAMPLE 0.1` runs the query on 10% of data. Read more
...
`SAMPLE n`
...
Here `n` is a sufficiently large integer. The query is executed on a sample of at least `n` rows (but not significantly more than this). For example, `SAMPLE 10000000` runs the query on a minimum of 10,000,000 rows. Read more
...
`SAMPLE k OFFSET m`
...
Here `k` and `m` are the numbers from 0 to 1. The query is executed on a sample of `k` fraction of the data. The data used for the sample is offset by `m` fraction. Read more
...
## SAMPLE K​
...
Here `k` is the number from 0 to 1 (both fractional and decimal notations are supported). For example, `SAMPLE 1/2` or `SAMPLE 0.5`.
...
In a `SAMPLE k` clause, the sample is taken from the `k` fraction of data. The example is shown below:
...
```sql
SELECT
    Title,
    count() * 10 AS PageViews
FROM hits_distributed
SAMPLE 0.1
WHERE
    CounterID = 34
GROUP BY Title
ORDER BY PageViews DESC LIMIT 1000

```
...
In this example, the query is executed on a sample from 0.1 (10%) of data. Values of aggregate functions are not corrected automatically, so to get an approximate result, the value `count()` is manually multiplied by 10.
...
## SAMPLE N​
...
Here `n` is a sufficiently large integer. For example, `SAMPLE 10000000`.
...
In this case, the query is executed on a sample of at least `n` rows (but not significantly more than this). For example, `SAMPLE 10000000` runs the query on a minimum of 10,000,000 rows.
...
When using the `SAMPLE n` clause, you do not know which relative percent of data was processed. So you do not know the coefficient the aggregate functions should be multiplied by. Use the `_sample_factor` virtual column to get the approximate result.
...
## SAMPLE K OFFSET M​
...
Here `k` and `m` are numbers from 0 to 1. Examples are shown below.
...
```sql
SAMPLE 1/10
......

## 4. docs/en/sql-reference/statements/select/index.md at master · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/master/docs/en/sql-reference/statements/select/index.md

```sql
[WITH expr_list(subquery)]
SELECT [DISTINCT [ON (column1, column2, ...)]] expr_list
[FROM [db.]table | (subquery) | table_function] [FINAL]
[SAMPLE sample_coeff]
[ARRAY JOIN ...]
[GLOBAL] [ANY|ALL|ASOF] [INNER|LEFT|RIGHT|FULL|CROSS] [OUTER|SEMI|ANTI] JOIN (subquery)|table [(alias1 [, alias2 ...])] (ON <expr_list>)|(USING <column_list>)
...
[PREWHERE expr]
[WHERE expr]
[GROUP BY expr_list] [WITH ROLLUP|WITH CUBE] [WITH TOTALS]
[HAVING expr]
[WINDOW window_expr_list]
[QUALIFY expr]
[ORDER BY expr_list] [WITH FILL] [FROM expr] [TO expr] [STEP expr] [INTERPOLATE [(expr_list)]]
...
[LIMIT [offset_value
...
]n BY columns]
...
All clauses are optional, except for the required list of expressions immediately after `SELECT` which is
...
in more detail [below](#select-clause).
...
Specifics of each optional clause are covered in separate sections, which are listed in the same order as they are executed:
...
- [WITH clause](../../../sql-reference/statements/select/with.md)
- [SELECT clause](#select-clause)
- [DISTINCT clause](../../../sql-reference/statements/select/distinct.md)
- [FROM clause](../../../sql-reference/statements/select/from.md)
- [SAMPLE clause](../../../sql-reference/statements/select/sample.md)
- [JOIN clause](../../../sql-reference/statements/select/join.md)
- [PREWHERE clause](../../../sql-reference/statements/select/prewhere.md)
- [WHERE clause](../../../sql-reference/statements/select/where.md)
- [WINDOW clause](../../../sql-reference/window-functions/index.md)
- [GROUP BY clause](/sql-reference/statements/select/group-by)
- [LIMIT BY clause](../../../sql-reference/statements/select/limit-by.md)
- [HAVING clause](../../../sql-reference/statements/select/having.md)
- [QUALIFY clause](../../../sql-reference/statements/select/qualify.md)
- [LIMIT clause](../../../sql-reference/statements/select/limit.md)
- [OFFSET clause](../../../sql-reference/statements/select/offset.md)
- [UNION clause](../../../sql-reference/statements/select/union.md)
- [INTERSEC...

## 5. FINAL SAMPLE · Pull Request #7907 · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/pull/7907

# Pull Request: ClickHouse/ClickHouse #7907

- Repository: ClickHouse/ClickHouse | ClickHouse® is a real-time analytics database management system | 47K stars | C++

## FINAL SAMPLE

- Author: [@KochetovNicolai](https://github.com/KochetovNicolai)
- Association: MEMBER
- State: merged
- Labels: pr-improvement
- Source branch: sample_final
- Target branch: master
- Mergeable: unknown
- Commits: 6
- Additions: 57
- Deletions: 13
- Changed files: 3
- Created: 2019-11-25T13:16:32Z
- Updated: 2019-11-26T06:21:39Z
- Closed: 2019-11-26T06:21:39Z
- Merged: 2019-11-26T06:21:39Z
- Merged by: [@alexey-milovidov](https://github.com/alexey-milovidov)

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

- Improvement

Changelog entry (up to few sentences, required except for Non-significant/Documentation categories):
Fix error `Column ... already exists` while using `FINAL` and `SAMPLE` together, e.g. `select count() from table final sample 1/2`. Fixes #5186
Detailed description (optional):
It happened because sample expression is a part of PK, and it was calculated twice if we use final.

---

### Timeline

**Nikolai Kochetov** pushed commit `f770934`: Support SAMPLE and FINAL together in MergeTree. · Nov 25, 2019 at 12:05pm

**Nikolai Kochetov** pushed commit `9b42e93`: Support SAMPLE and FINAL together in MergeTree. · Nov 25, 2019 at 12:49pm

**Nikolai Kochetov** pushed commit `d3118f0`: Support SAMPLE and FINAL together in MergeTree. · Nov 25, 2019 at 12:57pm

**Nikolai Kochetov** pushed commit `6587a87`: Added test with sample and final. · Nov 25, 2019 at 1:09pm

**alexey-milovidov** reviewed: commented · Nov 25, 2019 at 1:23pm

**Nikolai Kochetov** pushed commit `c97212d`: Remove unused variable. · Nov 25, 2019 at 1:35pm

**alexey-milovidov** reviewed: approved · Nov 25, 2019 at 1:41pm

**KochetovNicolai** added label `pr-improvement` · Nov 25, 2019 at 2:40pm

**Nikolai Kochetov** pushed commit `f90a8ce...

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/trajectories/data/trajectories_01_basic_select.jsonl" start_line="1" end_line="12" total_lines="20">
  1|{"id":"traj_001","category":"basic_select","subcategory":"simple_select","description":"Simple select all with limit","query":"SELECT * FROM events LIMIT 10","tables":["events"],"columns":{"events":["id","event_type","user_id","timestamp","data"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"* FROM events LIMIT 10","next_token":"*"},{"prefix":"SELECT * ","completion":"FROM events LIMIT 10","next_token":"FROM"},{"prefix":"SELECT * FROM ","completion":"events LIMIT 10","next_token":"events"},{"prefix":"SELECT * FROM events ","completion":"LIMIT 10","next_token":"LIMIT"}]}
  2|{"id":"traj_002","category":"basic_select","subcategory":"where_and_orderby","description":"Select with WHERE multiple conditions and ORDER BY","query":"SELECT user_id, event_type, timestamp FROM events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","tables":["events"],"columns":{"events":["id","event_type","user_id","timestamp","data"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"user_id, event_type, timestamp FROM events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"user_id"},{"prefix":"SELECT user_id, event_type, timestamp ","completion":"FROM events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"FROM"},{"prefix":"SELECT user_id, event_type, timestamp FROM ","completion":"events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"events"},{"prefix":"SELECT user_id, event_type, timestamp FROM events ","completion":"WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"WHERE"},{"prefix":"SELECT user_id, event_type, timestamp FROM events WHERE ","completion":"event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC LIMIT 50","next_token":"event_type"},{"prefix":"SELECT user_id, event_type, timestamp FROM events WHERE event_type = 'click' AND user_id = 1001 ","completion":"ORDER BY timestamp DESC LIMIT 50","next_token":"ORDER"},{"prefix":"SELECT user_id, event_type, timestamp FROM events WHERE event_type = 'click' AND user_id = 1001 ORDER BY timestamp DESC ","completion":"LIMIT 50","next_token":"LIMIT"}]}
  3|{"id":"traj_003","category":"basic_select","subcategory":"distinct","description":"Select distinct values with limit","query":"SELECT DISTINCT country, region FROM users ORDER BY country ASC, region ASC LIMIT 100","tables":["users"],"columns":{"users":["id","username","email","country","region","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"DISTINCT country, region FROM users ORDER BY country ASC, region ASC LIMIT 100","next_token":"DISTINCT"},{"prefix":"SELECT DISTINCT ","completion":"country, region FROM users ORDER BY country ASC, region ASC LIMIT 100","next_token":"country"},{"prefix":"SELECT DISTINCT country, region ","completion":"FROM users ORDER BY country ASC, region ASC LIMIT 100","next_token":"FROM"},{"prefix":"SELECT DISTINCT country, region FROM ","completion":"users ORDER BY country ASC, region ASC LIMIT 100","next_token":"users"},{"prefix":"SELECT DISTINCT country, region FROM users ","completion":"ORDER BY country ASC, region ASC LIMIT 100","next_token":"ORDER"},{"prefix":"SELECT DISTINCT country, region FROM users ORDER BY country ASC, region ASC ","completion":"LIMIT 100","next_token":"LIMIT"}]}
  4|{"id":"traj_004","category":"basic_select","subcategory":"limit_offset","description":"Select with LIMIT and OFFSET for pagination","query":"SELECT id, title, price FROM products WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","tables":["products"],"columns":{"products":["id","title","price","category","stock","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, title, price FROM products WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"id"},{"prefix":"SELECT id, title, price ","completion":"FROM products WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"FROM"},{"prefix":"SELECT id, title, price FROM ","completion":"products WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"products"},{"prefix":"SELECT id, title, price FROM products ","completion":"WHERE price > 50 ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"WHERE"},{"prefix":"SELECT id, title, price FROM products WHERE price > 50 ","completion":"ORDER BY price DESC LIMIT 20 OFFSET 40","next_token":"ORDER"},{"prefix":"SELECT id, title, price FROM products WHERE price > 50 ORDER BY price DESC ","completion":"LIMIT 20 OFFSET 40","next_token":"LIMIT"},{"prefix":"SELECT id, title, price FROM products WHERE price > 50 ORDER BY price DESC LIMIT 20 ","completion":"OFFSET 40","next_token":"OFFSET"}]}
  5|{"id":"traj_005","category":"basic_select","subcategory":"prewhere","description":"Select with PREWHERE optimization for MergeTree table","query":"SELECT user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","tables":["logs"],"columns":{"logs":["user_id","event_name","event_time","event_date","ip_address","user_agent"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"user_id"},{"prefix":"SELECT user_id, event_name, event_time ","completion":"FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"FROM"},{"prefix":"SELECT user_id, event_name, event_time FROM ","completion":"logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"logs"},{"prefix":"SELECT user_id, event_name, event_time FROM logs ","completion":"PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"PREWHERE"},{"prefix":"SELECT user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' ","completion":"WHERE event_name = 'login' ORDER BY event_time DESC LIMIT 100","next_token":"WHERE"},{"prefix":"SELECT user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ","completion":"ORDER BY event_time DESC LIMIT 100","next_token":"ORDER"},{"prefix":"SELECT user_id, event_name, event_time FROM logs PREWHERE event_date = '2024-01-15' WHERE event_name = 'login' ORDER BY event_time DESC ","completion":"LIMIT 100","next_token":"LIMIT"}]}
  6|{"id":"traj_006","category":"basic_select","subcategory":"final","description":"Select with FINAL keyword for CollapsingMergeTree deduplication","query":"SELECT id, user_id, amount, status FROM transactions FINAL WHERE status = 'completed' ORDER BY id LIMIT 200","tables":["transactions"],"columns":{"transactions":["id","user_id","amount","status","currency","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, user_id, amount, status FROM transactions FINAL WHERE status = 'completed' ORDER BY id LIMIT 200","next_token":"id"},{"prefix":"SELECT id, user_id, amount, status ","completion":"FROM transactions FINAL WHERE status = 'completed' ORDER BY id LIMIT 200","next_token":"FROM"},{"prefix":"SELECT id, user_id, amount, status FROM ","completion":"transactions FINAL WHERE status = 'completed' ORDER BY id LIMIT 200","next_token":"transactions"},{"prefix":"SELECT id, user_id, amount, status FROM transactions ","completion":"FINAL WHERE status = 'completed' ORDER BY id LIMIT 200","next_token":"FINAL"},{"prefix":"SELECT id, user_id, amount, status FROM transactions FINAL ","completion":"WHERE status = 'completed' ORDER BY id LIMIT 200","next_token":"WHERE"},{"prefix":"SELECT id, user_id, amount, status FROM transactions FINAL WHERE status = 'completed' ","completion":"ORDER BY id LIMIT 200","next_token":"ORDER"},{"prefix":"SELECT id, user_id, amount, status FROM transactions FINAL WHERE status = 'completed' ORDER BY id ","completion":"LIMIT 200","next_token":"LIMIT"}]}
  7|{"id":"traj_007","category":"basic_select","subcategory":"sample","description":"Select with SAMPLE clause for approximate query","query":"SELECT session_id, user_id, duration FROM sessions SAMPLE 0.1 WHERE duration > 30 ORDER BY duration DESC LIMIT 50","tables":["sessions"],"columns":{"sessions":["session_id","user_id","duration","start_time","end_time","page_count"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"session_id, user_id, duration FROM sessions SAMPLE 0.1 WHERE duration > 30 ORDER BY duration DESC LIMIT 50","next_token":"session_id"},{"prefix":"SELECT session_id, user_id, duration ","completion":"FROM sessions SAMPLE 0.1 WHERE duration > 30 ORDER BY duration DESC LIMIT 50","next_token":"FROM"},{"prefix":"SELECT session_id, user_id, duration FROM ","completion":"sessions SAMPLE 0.1 WHERE duration > 30 ORDER BY duration DESC LIMIT 50","next_token":"sessions"},{"prefix":"SELECT session_id, user_id, duration FROM sessions ","completion":"SAMPLE 0.1 WHERE duration > 30 ORDER BY duration DESC LIMIT 50","next_token":"SAMPLE"},{"prefix":"SELECT session_id, user_id, duration FROM sessions SAMPLE 0.1 ","completion":"WHERE duration > 30 ORDER BY duration DESC LIMIT 50","next_token":"WHERE"},{"prefix":"SELECT session_id, user_id, duration FROM sessions SAMPLE 0.1 WHERE duration > 30 ","completion":"ORDER BY duration DESC LIMIT 50","next_token":"ORDER"},{"prefix":"SELECT session_id, user_id, duration FROM sessions SAMPLE 0.1 WHERE duration > 30 ORDER BY duration DESC ","completion":"LIMIT 50","next_token":"LIMIT"}]}
  8|{"id":"traj_008","category":"basic_select","subcategory":"union_all","description":"UNION ALL combining results from two simple selects","query":"SELECT id, event_type, timestamp FROM events WHERE event_type = 'click' UNION ALL SELECT id, event_type, timestamp FROM events WHERE event_type = 'view' ORDER BY timestamp DESC LIMIT 100","tables":["events"],"columns":{"events":["id","event_type","user_id","timestamp","data"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, event_type, timestamp FROM events WHERE event_type = 'click' UNION ALL SELECT id, event_type, timestamp FROM events WHERE event_type = 'view' ORDER BY timestamp DESC LIMIT 100","next_token":"id"},{"prefix":"SELECT id, event_type, timestamp ","completion":"FROM events WHERE event_type = 'click' UNION ALL SELECT id, event_type, timestamp FROM events WHERE event_type = 'view' ORDER BY timestamp DESC LIMIT 100","next_token":"FROM"},{"prefix":"SELECT id, event_type, timestamp FROM ","completion":"events WHERE event_type = 'click' UNION ALL SELECT id, event_type, timestamp FROM events WHERE event_type = 'view' ORDER BY timestamp DESC LIMIT 100","next_token":"events"},{"prefix":"SELECT id, event_type, timestamp FROM events ","completion":"WHERE event_type = 'click' UNION ALL SELECT id, event_type, timestamp FROM events WHERE event_type = 'view' ORDER BY timestamp DESC LIMIT 100","next_token":"WHERE"},{"prefix":"SELECT id, event_type, timestamp FROM events WHERE event_type = 'click' ","completion":"UNION ALL SELECT id, event_type, timestamp FROM events WHERE event_type = 'view' ORDER BY timestamp DESC LIMIT 100","next_token":"UNION"},{"prefix":"SELECT id, event_type, timestamp FROM events WHERE event_type = 'click' UNION ALL SELECT id, event_type, timestamp FROM events WHERE event_type = 'view' ","completion":"ORDER BY timestamp DESC LIMIT 100","next_token":"ORDER"},{"prefix":"SELECT id, event_type, timestamp FROM events WHERE event_type = 'click' UNION ALL SELECT id, ... [truncated 134 chars]
  9|{"id":"traj_009","category":"basic_select","subcategory":"in_subquery","description":"Select with IN subquery in WHERE clause","query":"SELECT id, user_id, amount FROM orders WHERE user_id IN (SELECT id FROM users WHERE country = 'US') ORDER BY amount DESC LIMIT 50","tables":["orders","users"],"columns":{"orders":["id","user_id","amount","status","created_at"],"users":["id","username","email","country","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, user_id, amount FROM orders WHERE user_id IN (SELECT id FROM users WHERE country = 'US') ORDER BY amount DESC LIMIT 50","next_token":"id"},{"prefix":"SELECT id, user_id, amount ","completion":"FROM orders WHERE user_id IN (SELECT id FROM users WHERE country = 'US') ORDER BY amount DESC LIMIT 50","next_token":"FROM"},{"prefix":"SELECT id, user_id, amount FROM ","completion":"orders WHERE user_id IN (SELECT id FROM users WHERE country = 'US') ORDER BY amount DESC LIMIT 50","next_token":"orders"},{"prefix":"SELECT id, user_id, amount FROM orders ","completion":"WHERE user_id IN (SELECT id FROM users WHERE country = 'US') ORDER BY amount DESC LIMIT 50","next_token":"WHERE"},{"prefix":"SELECT id, user_id, amount FROM orders WHERE ","completion":"user_id IN (SELECT id FROM users WHERE country = 'US') ORDER BY amount DESC LIMIT 50","next_token":"user_id"},{"prefix":"SELECT id, user_id, amount FROM orders WHERE user_id IN (SELECT id FROM users WHERE country = 'US') ","completion":"ORDER BY amount DESC LIMIT 50","next_token":"ORDER"},{"prefix":"SELECT id, user_id, amount FROM orders WHERE user_id IN (SELECT id FROM users WHERE country = 'US') ORDER BY amount DESC ","completion":"LIMIT 50","next_token":"LIMIT"}]}
 10|{"id":"traj_010","category":"basic_select","subcategory":"between_like","description":"Select with BETWEEN and LIKE conditions","query":"SELECT id, title, price, created_at FROM products WHERE price BETWEEN 10 AND 100 AND title LIKE '%phone%' ORDER BY price ASC LIMIT 30","tables":["products"],"columns":{"products":["id","title","price","category","stock","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, title, price, created_at FROM products WHERE price BETWEEN 10 AND 100 AND title LIKE '%phone%' ORDER BY price ASC LIMIT 30","next_token":"id"},{"prefix":"SELECT id, title, price, created_at ","completion":"FROM products WHERE price BETWEEN 10 AND 100 AND title LIKE '%phone%' ORDER BY price ASC LIMIT 30","next_token":"FROM"},{"prefix":"SELECT id, title, price, created_at FROM ","completion":"products WHERE price BETWEEN 10 AND 100 AND title LIKE '%phone%' ORDER BY price ASC LIMIT 30","next_token":"products"},{"prefix":"SELECT id, title, price, created_at FROM products ","completion":"WHERE price BETWEEN 10 AND 100 AND title LIKE '%phone%' ORDER BY price ASC LIMIT 30","next_token":"WHERE"},{"prefix":"SELECT id, title, price, created_at FROM products WHERE ","completion":"price BETWEEN 10 AND 100 AND title LIKE '%phone%' ORDER BY price ASC LIMIT 30","next_token":"price"},{"prefix":"SELECT id, title, price, created_at FROM products WHERE price BETWEEN 10 AND 100 AND title LIKE '%phone%' ","completion":"ORDER BY price ASC LIMIT 30","next_token":"ORDER"},{"prefix":"SELECT id, title, price, created_at FROM products WHERE price BETWEEN 10 AND 100 AND title LIKE '%phone%' ORDER BY price ASC ","completion":"LIMIT 30","next_token":"LIMIT"}]}
 11|{"id":"traj_011","category":"basic_select","subcategory":"is_null_not_in","description":"Select with IS NULL and NOT IN conditions","query":"SELECT id, email, phone, created_at FROM users WHERE phone IS NULL AND email NOT IN ('test@test.com', 'admin@test.com') ORDER BY created_at DESC LIMIT 25","tables":["users"],"columns":{"users":["id","username","email","phone","country","created_at"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, email, phone, created_at FROM users WHERE phone IS NULL AND email NOT IN ('test@test.com', 'admin@test.com') ORDER BY created_at DESC LIMIT 25","next_token":"id"},{"prefix":"SELECT id, email, phone, created_at ","completion":"FROM users WHERE phone IS NULL AND email NOT IN ('test@test.com', 'admin@test.com') ORDER BY created_at DESC LIMIT 25","next_token":"FROM"},{"prefix":"SELECT id, email, phone, created_at FROM ","completion":"users WHERE phone IS NULL AND email NOT IN ('test@test.com', 'admin@test.com') ORDER BY created_at DESC LIMIT 25","next_token":"users"},{"prefix":"SELECT id, email, phone, created_at FROM users ","completion":"WHERE phone IS NULL AND email NOT IN ('test@test.com', 'admin@test.com') ORDER BY created_at DESC LIMIT 25","next_token":"WHERE"},{"prefix":"SELECT id, email, phone, created_at FROM users WHERE ","completion":"phone IS NULL AND email NOT IN ('test@test.com', 'admin@test.com') ORDER BY created_at DESC LIMIT 25","next_token":"phone"},{"prefix":"SELECT id, email, phone, created_at FROM users WHERE phone IS NULL AND email NOT IN ('test@test.com', 'admin@test.com') ","completion":"ORDER BY created_at DESC LIMIT 25","next_token":"ORDER"},{"prefix":"SELECT id, email, phone, created_at FROM users WHERE phone IS NULL AND email NOT IN ('test@test.com', 'admin@test.com') ORDER BY created_at DESC ","completion":"LIMIT 25","next_token":"LIMIT"}]}
 12|{"id":"traj_012","category":"basic_select","subcategory":"or_conditions","description":"Select with OR conditions and parentheses","query":"SELECT id, event_type, user_id, timestamp FROM events WHERE (event_type = 'click' OR event_type = 'scroll') AND user_id > 500 ORDER BY timestamp DESC LIMIT 100","tables":["events"],"columns":{"events":["id","event_type","user_id","timestamp","data"]},"dictionaries":[],"settings":{},"completion_points":[{"prefix":"SELECT ","completion":"id, event_type, user_id, timestamp FROM events WHERE (event_type = 'click' OR event_type = 'scroll') AND user_id > 500 ORDER BY timestamp DESC LIMIT 100","next_token":"id"},{"prefix":"SELECT id, event_type, user_id, timestamp ","completion":"FROM events WHERE (event_type = 'click' OR event_type = 'scroll') AND user_id > 500 ORDER BY timestamp DESC LIMIT 100","next_token":"FROM"},{"prefix":"SELECT id, event_type, user_id, timestamp FROM ","completion":"events WHERE (event_type = 'click' OR event_type = 'scroll') AND user_id > 500 ORDER BY timestamp DESC LIMIT 100","next_token":"events"},{"prefix":"SELECT id, event_type, user_id, timestamp FROM events ","completion":"WHERE (event_type = 'click' OR event_type = 'scroll') AND user_id > 500 ORDER BY timestamp DESC LIMIT 100","next_token":"WHERE"},{"prefix":"SELECT id, event_type, user_id, timestamp FROM events WHERE ","completion":"(event_type = 'click' OR event_type = 'scroll') AND user_id > 500 ORDER BY timestamp DESC LIMIT 100","next_token":"("},{"prefix":"SELECT id, event_type, user_id, timestamp FROM events WHERE (event_type = 'click' OR event_type = 'scroll') AND user_id > 500 ","completion":"ORDER BY timestamp DESC LIMIT 100","next_token":"ORDER"},{"prefix":"SELECT id, event_type, user_id, timestamp FROM events WHERE (event_type = 'click' OR event_type = 'scroll') AND user_id > 500 ORDER BY timestamp DESC ","completion":"LIMIT 100","next_token":"LIMIT"}]}
</file-view>

# Web Search Results for "ClickHouse ASOF JOIN syntax event_time > started_at"

## 1. docs/en/sql-reference/statements/select/join.md at master · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/blob/master/docs/en/sql-reference/statements/select/join.md

| Description |
|---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| `
...
|
| `LEFT ANY JOIN`, `RIGHT ANY
...
. |
| `ASOF JOIN`, `LEFT
...
Joining sequences with a
...
. `ASOF JOIN
...
below. |
| `
...
## ASOF JOIN usage {#asof-join-usage}
...
`ASOF JOIN` is useful when you need to join records that have no exact match.
...
This JOIN algorithm requires a special column in tables. This column:
...
- Must contain an ordered sequence.
- Can be one of the following types: [Int, UInt](../../../sql-reference/data-types/int-uint.md), [Float](../../../sql-
...
/data-
...
float.md), [Date
...
../sql-
...
/decimal.md).
- For the `hash` join algorithm it can't be the only column in the `JOIN` clause.
...
Syntax `ASOF JOIN ... ON`:
...
```sql
SELECT expressions_list
FROM table_1
ASOF LEFT JOIN table_2
ON equi_cond AND closest_match_cond
...
You can use any number of equality conditions and exactly one closest match condition. For example, `SELECT count() FROM table_1 ASOF LEFT JOIN table_2 ON table_1.a == table_2.b AND table_2.t <= table_1.t`.
...
Conditions supported for the closest match: `>`, `>=`, `<`, `<=`.
...
Syntax `ASOF JOIN ... USING`:
...
```sql
SELECT expressions_list
FROM table_1
ASOF JOIN table_2
USING (equi_column1, ... equi_columnN, asof_column)
...
`ASOF JOIN` uses `equi_columnX` for joining on equality and `asof_column` for joining on the closest match with the `table_1.asof_column >= table_2.asof_column` condition. The `asof_column` column is always the last one in the `USING` clause.
...
consider the following tables:
...
```text
         table_1                           table_2
      event   | ev
...
time | user_id       event   |
...
_time | user_id
    ----------|---------|----------   ----------|---------|----------
                  ...                               ...
    event_1_1 |  12:00  |  42         event_2_1...

## 2. docs/guides/starter%5Fguides/working%5Fwith%5Fjoins.md at main · ClickHouse/clickhouse-docs
URL: https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/guides/starter%5Fguides/working%5Fwith%5Fjoins.md

## ASOF JOIN {#asof-join}
...
The `ASOF JOIN`, provides non-exact matching capabilities.
If a row from the left table doesn’t have an exact match in the right table, then the closest matching row from the right table is used as a match instead.

This is particularly useful for time-series analytics and can drastically reduce query complexity.

 

The following example performs time-series analytics of stock market data.
A `quotes` table contains stock symbol quotes based on specific times of the day.
The price is updated every 10 seconds in the example data.
A `trades` table lists symbol trades - a specific volume of a symbol got bought at a specific time:

 

In order to calculate the concrete cost of each trade, we need to match the trades with their closest quote time.

This is easy and compact with the `ASOF JOIN`, where you use the `ON` clause for specifying an exact match condition and the `AND` clause for specifying the closest match condition - for a specific symbol (exact match) you're looking for the row with the ‘closest’ time from the `quotes` table at exactly or before the time (non-exact match) of a trade of that symbol:
...
```sql
SELECT
    t.symbol,
    t.volume,
    t.time AS trade_time,
    q.time AS closest_quote_time,
    q.price AS quote_price,
    t.volume * q.price AS final_price
FROM trades t
ASOF LEFT JOIN quotes q ON t.symbol = q.symbol AND t.time >= q.time
FORMAT Vertical;
...
```response
Row 1:
──────
symbol:             ABC
volume:             200
trade_time:         2023-02-22 14:09:05
closest_quote_time: 2023-02-22 14:09:00
quote_price:        32.11
final_price:        6422
...
Row 2:
──────
symbol:             ABC
volume:             300
trade_time:         2023-02-22 14:09:28
closest_quote_time: 2023-02-22 14:09:20
quote_price:        32.15
final_price:        9645
...
The `ON` clause of the `ASOF JOIN` is required and specifies an exact match condition next to the non-exact match condition of the `AND` clause.
:::...

## 3. Support ASOF JOIN ON syntax · Pull Request #6211 · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/pull/6211

# Pull Request: ClickHouse/ClickHouse #6211

- Repository: ClickHouse/ClickHouse | ClickHouse® is a real-time analytics database management system | 47K stars | C++

## Support ASOF JOIN ON syntax

- Author: [@4ertus2](https://github.com/4ertus2)
- Association: CONTRIBUTOR
- State: merged
- Labels: pr-improvement
- Source branch: asof
- Target branch: master
- Mergeable: unknown
- Commits: 1
- Additions: 267
- Deletions: 151
- Changed files: 6
- Created: 2019-07-29T15:02:35Z
- Updated: 2019-09-19T13:21:17Z
- Closed: 2019-07-31T03:00:23Z
- Merged: 2019-07-31T03:00:23Z
- Merged by: [@alexey-milovidov](https://github.com/alexey-milovidov)

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Category (leave one):

- Improvement

Short description (up to few sentences):
Support ASOF JOIN with ON section.

Detailed description (optional):
Ex. ```select * from t1 asof join t2 on t1.event = t2.event and t1.dt >= t2.dt```

---

### Timeline

**chertus** pushed commit `7b4ddd2`: support ASOF JOIN ON syntax · Jul 29, 2019 at 2:58pm

**alexey-milovidov** merged this pull request; closed this · Jul 31, 2019 at 3am

**@filimonov** commented · Aug 5, 2019 at 1:03pm

> Related: #6337

**BayoNet** mentioned this in PR [#6396: DOCAPI-7984: ASOF JOIN ON docs](https://github.com/ClickHouse/ClickHouse/pull/6396) · Aug 8, 2019 at 8:38am

**KochetovNicolai** added label `pr-improvement` · Sep 19, 2019 at 1:21pm

**BayoNet** mentioned this in PR [#7035: DOCAPI-7984: EN review, RU translation. ASOF JOIN ON docs](https://github.com/ClickHouse/ClickHouse/pull/7035) · Sep 21, 2019 at 1:12pm

## 4. ClickHouse ASOF JOIN
URL: https://pulse.support/kb/clickhouse-asof-join

ClickHouse join
...
row from the left table to the
...
table based on an inequality on an ordered column, rather than requiring
...
. It is the
...
trade at 10:05:03,
...
recent quote at or before that
...
for the same symbol.
...
ASOF JOIN requires at least one equality condition and exactly one inequality (closest-match) condition. Purely inequality-based joins with no equality key are not supported.
...
```
SELECT
    t.symbol,
    t.volume,
    t.time    AS trade_time,
    q.time    AS quote_time,
    q.price   AS quote_price,
    t.volume * q.price AS value
FROM trades AS t
...
ASOF LEFT JOIN quotes AS
...
t.symbol
...
t.time >= q.time;
...
`t.symbol = q.symbol` is the equality condition. You can supply multiple equality conditions.`t.time >= q.time` is the closest-match condition: ClickHouse selects the row in`quotes` with the greatest`q.time` that is still less than or equal to`t.time`, per matching symbol. Exactly one closest-match condition is permitted.
...
Supported inequality operators:`>`,`>=`,`<`,`<=`. The operator controls direction:
...
- `>=` or`>`: find the nearest prior row (most common — "last known value before this timestamp")
- `<=` or`<`: find the nearest future row (forward lookup — "next scheduled event after this timestamp")
...
With`USING`, equality columns come first and the ASOF column must be last. The comparison operator defaults to`>=`. This form is shorter but cannot express`<` or`>` direction.
...
```
SELECT
    mt.user_id,
    mt.event_time,
    dv.value  AS domains_visited_per_hour,
    uips.value AS unique_ips_per_hour,
    mt.label
FROM events AS mt FINAL
ASOF JOIN domains_feature AS dv
    ON mt.user_id = dv.user_id AND mt.event_time >= dv.event_time
ASOF JOIN ip_feature AS uips
    ON mt.domain = uips.domain AND mt.event_time >= uips.event_time;
...
```
SELECT t.user_id, t.event_time, p.price
FROM events AS t
ASOF JOIN prices AS p
    ON t.user_id = p.user_id AND t.event_time >= p.price_time
SETTINGS
...
_algorithm = 'full_so...

## 5. ASOF JOIN inequalities · Pull Request #7282 · ClickHouse/ClickHouse
URL: https://github.com/ClickHouse/ClickHouse/pull/7282

# Pull Request: ClickHouse/ClickHouse #7282

- Repository: ClickHouse/ClickHouse | ClickHouse® is a real-time analytics database management system | 47K stars | C++

## ASOF JOIN inequalities

- Author: [@4ertus2](https://github.com/4ertus2)
- Association: CONTRIBUTOR
- State: merged
- Labels: pr-feature
- Source branch: asof
- Target branch: master
- Mergeable: unknown
- Commits: 2
- Additions: 140
- Deletions: 54
- Changed files: 10
- Created: 2019-10-11T18:01:32Z
- Updated: 2025-07-11T22:03:07Z
- Closed: 2019-10-16T10:46:53Z
- Merged: 2019-10-16T10:46:53Z
- Merged by: [@4ertus2](https://github.com/4ertus2)

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

For changelog. Remove if this is non-significant change.

Category (leave one):

- Improvement

Short description (up to few sentences):
Support missing inequalities for ASOF JOIN. It's possible to join less-or-equal variant and strict greater and less variants for ASOF column in ON syntax.

Detailed description (optional):
Add support ```left.x > right.x```, ```left.x < right.x```, ```left.x <= right.x``` variants for ASOF JOIN.

---

### Timeline

**chertus** pushed commit `91c7ae8`: asof join inequalities · Oct 11, 2019 at 5:56pm

**4ertus2** mentioned this in issue [#6337: Support for `<`, `<=`, `>`, `>=` operator with ASOF join.](https://github.com/ClickHouse/ClickHouse/issues/6337) · Oct 11, 2019 at 6:01pm

**blinkov** added label `pr-feature` · Oct 11, 2019 a… (8 chars truncated)
… (12 lines truncated)
<truncation_notice>
Full output written to: /var/folders/d9/mzhyr7lx6xbf236j5ll9v_300000gn/T/devin-overflows-501/049efd3d/content.txt
</truncation_notice>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/export/build_wasm.sh" start_line="1" end_line="33" total_lines="33">
  1|#!/bin/bash
  2|# Build CHAI (ClickHouse Autocomplete Inference) as WebAssembly
  3|# Requires emscripten (emcc) to be installed.
  4|#
  5|# Install emscripten:
  6|#   git clone https://github.com/emscripten-core/emsdk.git
  7|#   cd emsdk && ./emsdk install latest && ./emsdk activate latest
  8|#   source ./emsdk_env.sh
  9|
 10|set -e
 11|
 12|echo "Building CHAI WebAssembly..."
 13|
 14|# Compile to WASM
 15|emcc -std=c++17 -O2 \
 16|    -s WASM=1 \
 17|    -s EXPORTED_FUNCTIONS="['_chai_load_model','_chai_autocomplete','_chai_free_model']" \
 18|    -s EXPORTED_RUNTIME_METHODS="['ccall','cwrap']" \
 19|    -s ALLOW_MEMORY_GROWTH=1 \
 20|    -s MODULARIZE=1 \
 21|    -s EXPORT_ES6=1 \
 22|    -o chai_wasm.js \
 23|    test_main.cpp
 24|
 25|echo "WASM build complete: chai_wasm.js + chai_wasm.wasm"
 26|echo ""
 27|echo "Usage in JavaScript:"
 28|echo "  const Module = require('./chai_wasm.js');"
 29|echo "  const mod = await Module();"
 30|echo "  const loadModel = mod.cwrap('chai_load_model', 'number', ['string']);"
 31|echo "  const autocomplete = mod.cwrap('chai_autocomplete', 'string', ['string', 'number']);"
 32|echo "  loadModel('ch_autocomplete.bin');"
 33|echo "  const result = autocomplete('SELECT ', 20);"
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/export/test_main.cpp" start_line="1" end_line="45" total_lines="45">
  1|// Test program for CHAI (ClickHouse Autocomplete Inference)
  2|#include "ch_inference.h"
  3|#include <iostream>
  4|#include <chrono>
  5|
  6|int main(int argc, char* argv[]) {
  7|    std::string model_path = "ch_autocomplete.bin";
  8|    if (argc > 1) model_path = argv[1];
  9|
 10|    std::cerr << "Loading model from: " << model_path << std::endl;
 11|    auto model = CHAI::load_model(model_path);
 12|    if (!model) {
 13|        std::cerr << "Failed to load model!" << std::endl;
 14|        return 1;
 15|    }
 16|
 17|    std::cerr << "Model loaded: "
 18|              << "d_model=" << model->config.d_model
 19|              << ", n_layers=" << model->config.n_layers
 20|              << ", vocab_size=" << model->config.vocab_size
 21|              << std::endl;
 22|
 23|    std::vector<std::string> test_prefixes = {
 24|        "SELECT ",
 25|        "SELECT count() FROM ",
 26|        "SELECT * FROM orders WHERE ",
 27|        "SELECT city, count() FROM orders GROUP BY ",
 28|        "CREATE TABLE ",
 29|    };
 30|
 31|    for (const auto& prefix : test_prefixes) {
 32|        auto start = std::chrono::high_resolution_clock::now();
 33|        auto result = CHAI::autocomplete(*model, prefix, 20, 0.5f, 15);
 34|        auto end = std::chrono::high_resolution_clock::now();
 35|        auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
 36|
 37|        std::cout << "Prefix:     " << prefix << std::endl;
 38|        std::cout << "Completion: " << result.completion << std::endl;
 39|        std::cout << "Score:      " << result.score << std::endl;
 40|        std::cout << "Time:       " << ms.count() << "ms" << std::endl;
 41|        std::cout << std::endl;
 42|    }
 43|
 44|    return 0;
 45|}
</file-view>

<file-view path="/Users/root1/Downloads/projects/clickhouse-github/train.py" start_line="60" end_line="79" total_lines="182">
 60|
 61|    # Determine device
 62|    device = args.device
 63|    if device == "mps" and not hasattr(torch.backends, "mps"):
 64|        device = "cpu"
 65|    if device == "cuda" and not torch.cuda.is_available():
 66|        device = "cpu"
 67|    print(f"Device: {device}")
 68|
 69|    # Create datasets
 70|    traj_dir = str(PROJECT_ROOT / args.trajectory_dir)
 71|    print(f"\nLoading trajectories from: {traj_dir}")
 72|    train_dataset, val_dataset = create_data_loaders(
 73|        traj_dir,
 74|        batch_size=args.batch_size,
 75|        max_seq_len=args.max_seq_len,
 76|    )
 77|    print(f"Train samples: {len(train_dataset)}")
 78|    print(f"Val samples: {len(val_dataset)}")
</file-view>

/Users/root1/Downloads/projects/clickhouse-github/export/export_cpp.py
/Users/root1/Downloads/projects/clickhouse-github/inference/autocomplete.py
/Users/root1/Downloads/projects/clickhouse-github/model/config.py
/Users/root1/Downloads/projects/clickhouse-github/model/training.py
/Users/root1/Downloads/projects/clickhouse-github/test_autocomplete.py
/Users/root1/Downloads/projects/clickhouse-github/tokenizer/ch_tokenizer.py
/Users/root1/Downloads/projects/clickhouse-github/train.py

Subagent cb8c059d completed successfully: