JanusIR-500M: Teaching Neural Networks How Compilers Actually Think

NEW Sep 30, 2026 · 16 min read

machine-learningcompilersllvmgraph-neural-networksdeep-learningopen-sourcehugging-facepytorch

JanusIR-500M: Teaching Neural Networks How Compilers Actually Think

Why standard code LLMs struggle with program logic, and how dual-stream relational graph attention unlocks structured compiler reasoning.

I have published JanusIR-500M-Research on Hugging Face: a 545.7M-parameter compiler-aware autoregressive hybrid architecture integrating a causal Transformer language pathway with a relational graph attention network (RGAT) and cross-modal text-graph fusion.

When you ask a modern Large Language Model to write a Python function, it often feels like magic. But ask that same model to do something a real compiler does every second: check whether a block of code is dead, trace where a variable actually gets consumed, or resolve an ambiguous register across branching paths: and the illusion quickly breaks down.

The reason is simple. Large Language Models read code like a book: word by word, from left to right. But to a compiler, code is never a line of text. Code is a highway system: full of intersections, loop detours, and data pipelines. When a model only looks at words in a row, it easily gets lost at the exits.

JanusIR solves this with a dual-stream hybrid architecture. It combines a 20-layer causal Transformer backbone (which reads the text) with a 4-layer Relational Graph Attention Network (which navigates the roadmap). By fusing these two streams with gated cross-modal text-graph attention, JanusIR learns to reason about code the same way a real compiler does.

In this blog, I will explain the core intuition behind compiler representations, walk through the hybrid architecture, detail the 5-stage training curriculum, examine the empirical evaluation metrics fetched directly from Hugging Face, and show how compiler reasoning results are shared in sub-second latency (milliseconds).

1. The Core Intuition: Why Code Is a Map, Not a Sentence

To understand why standard AI models struggle with compilers, we need to look at what compilers actually work with: Intermediate Representation (IR).

When you compile a C, C++, or Rust program using LLVM or Clang, the compiler translates your human-readable source code into LLVM IR. LLVM IR is a clean, assembly-like language with instructions like additions, comparisons, and branches.

Standard code models (such as CodeLlama, StarCoder, or DeepSeek-Coder) process LLVM IR using standard self-attention:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

This formula calculates how much attention every word should pay to every other word based on their positions in the text. This works wonderfully for natural language because sentences flow in sequence. But programs do not flow in simple sequences.

Compilers organize code into three foundational structures:

1. The Control-Flow Graph (CFG)

Think of the CFG as the road map of execution. Code is divided into basic blocks: chunks of instructions that run sequentially without jumps. At the end of each block, a branch instruction decides where to go next based on a condition. An if/else creates a fork in the road; a while loop creates a roundabout. In linear text, two blocks might sit next to each other on the page, yet in execution they belong to completely different alternate universes.

2. The Data-Flow Graph (DFG)

Think of the DFG as a supply chain. When instruction A creates a value in register %x, instructions B, C, and D might consume %x much later. A compiler needs to know exact def-use chains (where a value is defined, and where it is used) to know if a calculation can be simplified or thrown away.

3. Static Single Assignment (SSA) and Phi Nodes

In LLVM IR, every register is written to exactly once. If a variable gets modified inside an if/else block, how do you know which value to use once the branches merge back together?

Compilers solve this with a special junction called a $\phi$ (phi) node:

%res = phi i32 [ %val_true, %then_block ], [ %val_false, %else_block ]

The phi node tells the CPU: “If execution arrived from %then_block, use %val_true. If execution arrived from %else_block, use %val_false.”

The Structural Gap

Linear 1D Sequence (Standard LLM): [define] -> [i32] -> [@foo] -> [%x] -> [br] -> [%bb1] -> [%bb2]. The model sees only flat text tokens. Discovering whether %bb2 can jump to %bb1 requires computing attention across distant token slots, often resulting in hallucinations.
Compiler Topology (JanusIR): %entry conditionally branches to %bb_neg or %bb_pos, values propagate directly through def-use links, and converging paths meet at %bb_end where the phi node selects the active value (%out = phi [ %abs, %bb_neg ], [ %double, %bb_pos ]).

JanusIR provides this structural awareness natively by running a graph neural network alongside the transformer decoder.

2. The JanusIR Architecture: Two Brains, One Model

At its architectural core, JanusIR-500M is a 545.7M-parameter compiler-aware autoregressive hybrid architecture integrating a causal Transformer language pathway with a relational graph attention network (RGAT) and cross-modal text-graph fusion.

In Roman mythology, Janus is the two-faced god who looks in two directions at once: one face looks to the past, the other to the future.

JanusIR-500M follows this philosophy. It possesses two complementary pathways:

Path A (The Sequence Reader): A 20-layer causal Transformer decoder that reads the tokenized LLVM IR code text.
Path B (The Graph Navigator): A 4-layer Relational Graph Attention Network (RGAT) that moves through instructions as connected graph nodes.
The Bridge (Gated Cross-Modal Fusion): Three cross-attention layers located in the middle of the transformer stack, allowing the sequence reader to check the roadmap before making predictions.

Architecture Specifications at a Glance

Parameter	Value	Role in the Model
Active Parameters	545.7M (545,692,999)	Full model parameter capacity
Model Dimension ( $d_{model}$ )	1280	Sequence hidden representation width
Transformer Layers ( $n_{layers}$ )	20	Depth of causal sequence decoder
Attention Heads ( $n_{heads}$ )	16	Multi-head attention count
Feed-Forward Dimension ( $d_{ff}$ )	5120	SwiGLU projection width
Context Window	1024 tokens	Input IR context length
Vocabulary Size	4096 tokens	Compiler-tailored BPE vocabulary
Graph Hidden Dimension ( $d_{graph}$ )	384	RGAT node feature width
Graph Layers	4	Relational graph attention depth
Graph Heads	8	Multi-head graph attention
Relational Edge Types	4	CFG Sequential, CFG Cond, DFG, SSA-Phi
Fusion Layers	3	Gated cross-modal injection blocks
Positional Embeddings	RoPE ( $\theta = 10000.0$ )	Rotary positional encoding
Normalization	RMSNorm ( $\epsilon = 10^{-5}$ )	Pre-normalization

3. Mathematical Formulation: How the Pathways Communicate

For practitioners and machine learning engineers, here is the exact formulation governing how the graph encoder and language decoder interact.

3.1 Relational Graph Attention (RGAT)

The compiler program graph $G = (V, E, \mathcal{R})$ represents instructions as nodes $v_i \in V$ and compiler dependencies as directed typed edges $(v_i, v_j, r) \in E$ .

We define four discrete relation types $r \in \mathcal{R} = \{0, 1, 2, 3\}$ :

$r=0$ : CFG Direct / Sequential Fallthrough
$r=1$ : CFG Conditional Branch Target
$r=2$ : Data-Flow Def-Use Dependency
$r=3$ : SSA $\phi$ -node Provenance

At each graph layer $l$ , node $i$ gathers messages from its neighbors $\mathcal{N}_i^r$ across each relationship type:

$h_i^{(l+1)} = \text{RMSNorm}\left(\text{SiLU}\left(W_{\text{self}} h_i^{(l)} + \sum_{r \in \mathcal{R}} \frac{1}{|\mathcal{N}_i^r|} \sum_{j \in \mathcal{N}_i^r} W_r h_j^{(l)}\right)\right)$

Notice how each relation type $r$ uses its own dedicated weight matrix $W_r$ . A control jump is interpreted differently from a data dependency or a phi node. The degree normalization $\frac{1}{|\mathcal{N}_i^r|}$ ensures that instructions with dozens of incoming def-use edges do not overwhelm the gradient signal.

Halfway through the 20-layer sequence transformer (after layer 10), we pause sequence generation to inject the graph states.

Let $H_{\text{lang}} \in \mathbb{R}^{B \times T \times d_{model}}$ be the sequence hidden states, and $H_{\text{graph}} \in \mathbb{R}^{N \times d_{graph}}$ be the output of the RGAT.

We project the graph states into the language dimension using a projection matrix $W_{proj}$ , then compute multi-head cross-attention:

$\text{Fused} = \text{CrossAttention}\left(Q = H_{\text{lang}}, K = H_{\text{graph}} W_{proj}, V = H_{\text{graph}} W_{proj}\right)$

If we simply added this fused representation directly to the transformer states, early pretraining gradients could destabilize the language backbone. To prevent this, we introduce a learnable scalar gate $\alpha$ , initialized at zero:

$H_{\text{out}} = \text{RMSNorm}\left(H_{\text{lang}} + \sigma(\alpha) \cdot \text{Fused}\right)$

Because $\alpha = 0.0$ initially, $\sigma(\alpha) = 0.5$ . The model starts with equal access to both modalities and automatically learns how much graph context each token requires.

4. The 5-Stage Training Curriculum

You cannot teach a model complex compiler optimization without first teaching it syntax. We structured training into five disciplined stages:

Curriculum Stage	Primary Modality	Objectives and Training Methodology
Stage A: Lexical Language Pretraining	Causal Token Sequence	Pretrained on LLVM IR modules from AnghaBench and open-source systems libraries to master instruction grammar, opcode vocabularies, register naming conventions, and syntax.
Stage B: Structural Graph Pretraining	Multi-Relational Graph	Relational Graph Attention Network (RGAT) pretrained via link prediction and relation classification across CFG, DFG, and SSA edges.
Stage C: Joint Dual-Stream Pretraining	Fused Cross-Modal Streams	Joint end-to-end training fusing sequence representations with graph embeddings using gated cross-attention and structural alignment regularization: $\mathcal{L}_{\text{joint}} = \mathcal{L}_{\text{LM}} + 0.5 \mathcal{L}_{\text{struct}} + 0.2 \mathcal{L}_{\text{align}}$ .
Stage D: Compiler Reasoning SFT	Supervised Instruction Tasks	Fine-tuning on CFG reachability, def-use value tracing, and SSA phi invariant resolution with completion-only loss masking.
Stage E: Domain Alignment Investigation	Preference Pairs	Evaluation of compiler-verified reasoning pairs versus hallucinated transformations, analyzing alignment tax on exact symbolic reasoning.

Stage A: Lexical Language Pretraining

We trained the sequence transformer on millions of LLVM IR functions extracted from AnghaBench and open-source system repositories. The model learned what valid LLVM IR looks like: types, opcodes, alignments, and instruction formats.

Stage B: Structural Graph Pretraining

Before connecting the graph encoder to the language model, we trained the RGAT independently. The task was link prediction and relation classification: given two instruction nodes, could the RGAT predict whether an edge connected them, and whether that edge was a CFG branch, a data dependency, or a phi node?

With both halves pretrained, we activated the gated cross-attention bridge and trained the entire system end-to-end. The objective combined language modeling with structural preservation:

$\mathcal{L}_{\text{joint}} = \lambda_{\text{LM}} \mathcal{L}_{\text{LM}} + \lambda_{\text{struct}} \mathcal{L}_{\text{struct}} + \lambda_{\text{align}} \mathcal{L}_{\text{align}}$

We set $\lambda_{\text{LM}} = 1.0$ , $\lambda_{\text{struct}} = 0.5$ , and $\lambda_{\text{align}} = 0.2$ .

Stage D: Compiler Reasoning SFT (The Champion Variant)

In Supervised Fine-Tuning, we taught the model to answer structured compiler queries:

Which basic blocks exist in this function, and which blocks can reach block X?
Which instructions consume the value in register %1?
What are the incoming operands and predecessor labels for phi node %res?

Critical Technique: Completion-Only Loss Masking

When fine-tuning on code reasoning tasks, standard cross-entropy loss trains the model to predict both the prompt and the answer. But the prompt is just the input LLVM IR assembly, which the model already knows. Wasting gradient capacity on re-predicting known code degrades reasoning performance.

We implemented completion-only loss masking: setting the target index for all prompt positions to ignore_index = -100 so that backpropagation updates weights exclusively based on the reasoning output:

# Completion-only loss masking across the batch
for b_idx, p_len in enumerate(item["prompt_lens"]):
    mask_boundary = max(0, p_len - 1)
    if mask_boundary < shift_targets.shape[1]:
        shift_targets[b_idx, :mask_boundary] = -100

loss = F.cross_entropy(
    shift_logits.view(-1, vocab_size), 
    shift_targets.view(-1), 
    ignore_index=-100
)

5. Empirical Benchmark Results and Evaluation Metrics

We evaluated JanusIR-500M on 50 held-out, real-world C compiler benchmark programs compiled to LLVM IR. The test suite evaluated exact compiler-grounded predictions using deterministic greedy generation ( $\text{temperature} = 0.0$ ).

Here are the verified metrics fetched directly from Hugging Face:

Figure 1: Multi-metric evaluation of the Champion Variant (Full JanusIR + SFT) across Control-Flow Graph (CFG), Data-Flow Graph (DFG), SSA Phi Invariants, and Overall Combined metrics. Source: Hugging Face.

Detailed Quantitative Benchmark Breakdown

Benchmark Evaluation Task	Accuracy (Jaccard)	Precision	Recall	F1 Score
Control-Flow Graph (CFG)	33.74%	60.77%	39.43%	43.37%
Data-Flow Graph (DFG)	22.47%	30.00%	27.13%	25.28%
SSA Phi Invariants	43.06%	46.80%	45.28%	44.58%
Overall Combined	33.09%	45.86%	37.28%	37.74%

Understanding the Results

100.0% Syntactic Validity: Across all 50 test programs, the model never generated broken tokens or malformed IR.
60.77% Precision on CFG: When JanusIR asserts that a basic block is reachable or transitions to another block, it is correct over 6 times out of 10. The graph pathway dramatically curtails control-flow hallucinations.
Strongest Performance on SSA Phi Invariants (44.58% F1 / 43.06% Accuracy): In standard language models, phi resolution is notoriously difficult because you must correlate data registers with branching labels simultaneously. Because the RGAT explicitly tracks phi edges, JanusIR achieves its highest score on this task.
Sub-Second Latency and Millisecond Speed: Compiler optimization passes and developer tooling cannot wait seconds for answers. Because JanusIR-500M is compact (545.7M parameters) and uses native vectorized PyTorch graph message passing, compiler reasoning queries and graph extraction run in sub-second time (milliseconds), enabling real-time compiler verification.

6. Scientific Architectural Ablations and Failure Analysis

To understand what each architectural component actually contributed, we ran a rigorous ablation study comparing six configurations:

Variant A: Token-Only Baseline (Pure Causal Transformer, no graph pathway).
Variant B: Token + CFG (Transformer + RGAT with CFG edges only).
Variant C: Token + CFG + DFG (Adding Def-Use relational edges).
Variant D: Full JanusIR (+SSA edges, pre-SFT).
Variant E: Full JanusIR + SFT (Champion Model).
Variant F: Full JanusIR + SFT + [Excluded] DPO (Both variants are similar as DPO was excluded here).

Figure 2: (Left) Accuracy score comparison across architectural variants A through F for CFG, DFG, and SSA reasoning tasks. (Right) Dynamic failure mode distribution across held-out programs. Source: Hugging Face.

What the Ablation Matrix Teaches Us

The Token-Only Failure (Variant A): Without the graph pathway, the pure transformer achieved 0.00% CFG accuracy and only 7.00% SSA accuracy. A flat sequence model simply cannot compute multi-hop branch reachability from raw text tokens alone.
The Power of Explicit Edges (Variants B, C, and D): Adding CFG edges immediately provided a foundation for reachability. Adding DFG and SSA edges enabled cross-modal alignment.
The SFT Catalyst (Variant E): Fine-tuning on structured reasoning queries with completion-only loss masking unlocked the model’s full potential, jumping CFG precision to 60.77% and SSA F1 to 44.58%.
Why Variant E and Variant F Show Similar Scores: In the evaluation chart, Variant E and Variant F show virtually identical performance because DPO was excluded here. Without active preference optimization, Variant F functions identically to the supervised fine-tuned checkpoint, making Variant E (Full JanusIR + SFT) the verified champion model for exact compiler static analysis.

Error Distribution and Failure Categorization

Looking at the right panel of Figure 2:

Control-Flow Misses (<100%): 22 cases, typically involving complex multi-way switch statements and heavily nested exception handling blocks.
Data-Flow Tracing Errors (<100%): 20 cases, primarily caused by pointer dereferencing and array indexing (getelementptr), where memory values cannot be traced from register names alone.
SSA Phi Inconsistencies (<100%): 15 cases, almost exclusively occurring in loops with 4 or more predecessor paths merging at a single junction.
Severe CFG Errors (<25%): 14 cases.
Fully Accurate Joint Inferences: 2 held-out programs achieved 100% exact match across all three dimensions simultaneously.

7. Running JanusIR on Hugging Face

The complete open-weights release is accessible on the Hugging Face Hub:

Hugging Face Model Hub: the-jashthakkar/JanusIR-500M-Research
Model Checkpoint: janusir_model.pt (2.2 GB / 545.7M parameters)
Architecture Config: config.json
Compiler Vocab: tokenizer_vocab.json (4,096 tokens)
Model Card and Documentation: README.md

7.1 Sub-Second Compiler Reasoning: Results Shared in Milliseconds (ms)

In real-world compiler engineering, speed is critical. If a static analysis check, alias verifier, or optimization oracle takes several seconds waiting for a large 70B parameter model, it cannot be practically integrated into a continuous integration pipeline, a live compiler pass, or an interactive IDE linter.

JanusIR-500M solves this latency challenge by delivering structured compiler reasoning with sub-second performance:

Sub-Second Results (Milliseconds): Because JanusIR has 545.7M parameters and uses native vectorized PyTorch graph convolutions (index_add_), program graph extraction and causal inference queries execute in milliseconds (ms). Results and static analysis findings are shared in sub-second response times.
Fast Static Analysis Passes: Identifying reachable control-flow blocks and extracting SSA phi invariants occurs in sub-second time per function, allowing compiler tools to execute hundreds of verification queries without bottlenecking compilation.
Lean Resource Footprint: The model fits comfortably on standard consumer GPUs or CPU instances (using approximately 2.2 GB VRAM in FP32, or under 1.2 GB in BF16/FP16), eliminating the need for expensive multi-GPU clusters.

7.2 Quick Start Code

# Clone the repository and weights
git lfs install
git clone https://huggingface.co/the-jashthakkar/JanusIR-500M-Research

import json
import torch

# Load configuration and vocabulary
with open("JanusIR-500M-Research/config.json", "r") as f:
    config_dict = json.load(f)

with open("JanusIR-500M-Research/tokenizer_vocab.json", "r") as f:
    vocab = json.load(f)

# Load PyTorch model weights
device = "cuda" if torch.cuda.is_available() else "cpu"
state_dict = torch.load("JanusIR-500M-Research/janusir_model.pt", map_location=device)
print(f"Loaded JanusIR-500M successfully with {len(state_dict)} tensor keys!")

8. Summary and What’s Next

JanusIR-500M demonstrates that structural inductive biases can dramatically elevate reasoning on compiler artifacts.

Key Takeaways:

Flat text fails on compilers: Causal token attention cannot reliably infer non-local control and data flow from 1D sequences alone.
Relational graph fusion works: Equipping a 20-layer causal transformer with a 4-layer multi-relational graph attention network (RGAT) boosts CFG precision to 60.77% and SSA F1 to 44.58%.
Completion-only loss masking: Masking the prompt IR during SFT focuses 100% of gradient updates on structured reasoning outputs, preventing the model from wasting capacity memorizing assembly text.
Sub-second reasoning speed: Results and static analysis queries are shared in sub-second latency (milliseconds), enabling interactive static analysis passes and real-time compiler verification.
Open-weights release: The 545.7M parameter checkpoint, vocab, evaluation metrics, and comprehensive documentation are fully accessible on Hugging Face for the systems and ML community.

Future Roadmap

Scaling to 1B and 3B Parameters: Expanding the causal decoder depth while increasing graph attention capacity.
Memory Dependency Graphs (MDG): Incorporating pointer alias analysis and points-to graph edges to resolve indirect memory dereferences.
Direct LLVM opt Integration: Packaging JanusIR as an interactive guidance oracle for LLVM pass pipelines, assisting heuristics in loop unrolling, vectorization, and dead-code elimination.

Sources & References

Hugging Face Repository: the-jashthakkar/JanusIR-500M-Research
LLVM Project Documentation: LLVM Language Reference Manual
Schlichtkrull et al.: Modeling Relational Data with Graph Convolutional Networks (RGCN)
Veličković et al.: Graph Attention Networks (GAT)
Cummins et al.: ProGraML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations (ICML)
Su et al.: RoFormer: Enhanced Transformer with Rotary Position Embedding

JanusIR-500M: Teaching Neural Networks How Compilers Actually Think

1. The Core Intuition: Why Code Is a Map, Not a Sentence

1. The Control-Flow Graph (CFG)

2. The Data-Flow Graph (DFG)

3. Static Single Assignment (SSA) and Phi Nodes

The Structural Gap

2. The JanusIR Architecture: Two Brains, One Model

Architecture Specifications at a Glance

3. Mathematical Formulation: How the Pathways Communicate

3.1 Relational Graph Attention (RGAT)

3.2 Gated Cross-Modal Fusion

4. The 5-Stage Training Curriculum

Stage A: Lexical Language Pretraining

Stage B: Structural Graph Pretraining

Stage C: Joint Cross-Modal Pretraining

Stage D: Compiler Reasoning SFT (The Champion Variant)

Critical Technique: Completion-Only Loss Masking

5. Empirical Benchmark Results and Evaluation Metrics

Detailed Quantitative Benchmark Breakdown

Understanding the Results

6. Scientific Architectural Ablations and Failure Analysis

What the Ablation Matrix Teaches Us

Error Distribution and Failure Categorization

7. Running JanusIR on Hugging Face

7.1 Sub-Second Compiler Reasoning: Results Shared in Milliseconds (ms)

7.2 Quick Start Code

8. Summary and What’s Next

Future Roadmap

Sources & References