Tag: ai

For a software, solution, or enterprise architect, documentation tools generally fall into four main categories: Diagrams-as-Code (version-controlled), Visual Canvas Tools (collaborative/presentation-ready), Specialized Architecture Modeling Tools (formal models like C4/ArchiMate), and Documentation & ADR Platforms.

1. Diagrams-as-Code (Version-Controlled & Dev-Friendly)

These tools let you define architecture diagrams in plain text or code, store them alongside source code in Git, and render them automatically.

2. Visual Canvas & Collaborative Diagramming Tools

Best for whiteboarding, interactive stakeholder walkthroughs, and high-fidelity presentations.

• Draw.io (diagrams.net):
  • Strengths: Free, open-source, works offline, saves directly to Git/Google Drive/OneDrive, includes official AWS, Azure, GCP, and C4 shape libraries.
• Lucidchart:
  • Strengths: Enterprise standard for technical architecture. Supports live data linking, automated architecture generation from AWS/GCP/Azure configs, and team collaboration.
• Miro / FigJam / Mural:
  • Strengths: Infinite digital whiteboards. Ideal for Event Storming, Domain-Driven Design (DDD) workshops, and early-stage exploratory system architecture.
• IcePanel:
  • Strengths: An interactive, zoomable map specifically built around the C4 model. Bridges the gap between loose whiteboards and overly rigid enterprise modeling tools.

3. Formal Enterprise Architecture & Modeling Tools

For large enterprise teams operating under formal frameworks like TOGAF, ArchiMate, or strict UML/SysML standards.

• Archi: Open-source, dedicated ArchiMate modeling tool for enterprise architects mapping business capabilities, application layers, and technology infrastructure.
• Sparx Systems Enterprise Architect: Heavyweight, industrial-grade tool for deep UML, SysML, database schema, and process modeling.
• Bizzdesign / LeanIX: SaaS platforms for Application Portfolio Management (APM), capability mapping, and enterprise IT governance.

4. Documentation Frameworks & Decision Records (ADRs)

Diagrams alone are rarely enough—architectural rationale needs structured documentation.

A. Architecture Decision Records (ADRs)

Document the why behind decisions (context, options considered, trade-offs, final decision).

• adr-tools / Log4brains: CLI tools to maintain Markdown ADRs directly in your Git repository.
• MADR (Markdown Any Architecture Decision Records): Standardized, lightweight Markdown templates for engineering teams.

B. Docs-as-Code & Knowledge Bases

• arc42 Template: The industry-standard 12-section architecture documentation template (can be used in Markdown, AsciiDoc, or Confluence).
• Docusaurus / MkDocs (with mkdocs-material): Static site generators that convert Markdown + Mermaid/PlantUML files into searchable architecture portals.
• Backstage (by Spotify): Open-source developer portal framework that centralizes architecture documentation, API specs (OpenAPI/AsyncAPI), and service catalogs.

Recommended Architecture Toolchain by Team Style

When designing and implementing REST APIs in Java, architecture is typically evaluated across three dimensions:

1. Application / Codebase Structure (How internal code is organized)
2. System & Deployment Topology (How the system runs and scales across infrastructure)
3. Execution & Concurrency Model (How Java processes HTTP requests and I/O)

1. Codebase & Application-Level Architectures

These patterns dictate how your Java code is structured inside a repository/service.

A. Layered (Controller–Service–Repository / N-Tier)

• How it works: Code is arranged horizontally by technical concern:
  • Controller Layer (@RestController): Handles HTTP serialization/deserialization, validation, routing, and HTTP status codes.
  • Service Layer (@Service): Contains business logic, orchestration, and transactional boundaries (@Transactional).
  • Repository/Data Layer (@Repository): Communicates with the database (Spring Data JPA, Hibernate, jOOQ).
• Common Java Stack: Standard Spring Boot, Jakarta EE (JAX-RS + EJB/CDI + JPA), Micronaut.
• Best for: Small to medium applications, CRUD-heavy services, fast development cycles.

B. Hexagonal Architecture (Ports and Adapters) / Clean / Onion Architecture

• How it works: The business domain is placed at the center, completely independent of frameworks, databases, and HTTP transports:
  • Inbound Ports & Adapters: Inbound ports define what the app can do; the REST controller is an inbound adapter that translates HTTP requests to call the port.
  • Outbound Ports & Adapters: Outbound ports define interfaces for data persistence/external APIs; repositories are outbound adapters.
• Common Java Stack: Core Java domain (POJOs, no framework annotations), with Spring Boot/Quarkus wiring adapters via Dependency Injection.
• Best for: Complex domain logic, long-lived enterprise applications, and scenarios where you want high testability without spinning up Spring contexts or databases.

C. Modular Monolith Architecture

• How it works: A single deployable unit structured strictly into domain-bounded modules with explicit API boundaries between modules. It prevents the “spaghetti monolith” while avoiding the operational overhead of microservices.
• Common Java Stack:Spring Modulith, ArchUnit (for boundary enforcement), Maven/Gradle multi-module builds.
• Best for: Teams wanting clean domain boundaries and independence without managing distributed infrastructure.

D. CQRS (Command Query Responsibility Segregation) & Event Sourcing

• How it works: Splits REST write endpoints (Commands: POST, PUT, DELETE) from read endpoints (Queries: GET). The read model and write model can use different data stores optimized for queries (e.g., Elasticsearch/Redis) and transactional writes (PostgreSQL).
• Common Java Stack:Axon Framework, Spring Boot + Kafka, Debezium (for Change Data Capture).
• Best for: High-read/high-write asymmetry, audit-heavy domains (finance, healthcare), and systems requiring an immutable history of domain events.

2. System & Deployment-Level Architectures

These patterns define how REST services are distributed across networks and infrastructure.

3. Concurrency & Execution Models in Java

The underlying I/O model significantly influences how your REST API handles traffic and scales under high loads:

A. Classic Thread-Per-Request (Blocking I/O)

• Mechanism: Every incoming HTTP request is assigned a dedicated operating system (platform) thread from a pool (e.g., Tomcat/Jetty worker threads).
• Java Stack: Spring MVC (spring-boot-starter-web), standard Jakarta EE Servlet container.
• Pros & Cons: Simple mental model and easy debugging, but high concurrent connection counts can cause memory pressure from OS thread overhead.

B. Virtual Threads / Project Loom (Modern Blocking I/O)

• Mechanism: Introduced in Java 21. Standard synchronous, blocking code runs on lightweight “virtual threads” managed by the JVM instead of OS threads. When blocking on I/O, the JVM unmounts the virtual thread, enabling millions of concurrent requests with standard imperative syntax.
• Java Stack: Java 21+ with Spring Boot 3.2+ (spring.threads.virtual.enabled=true), Quarkus, Helidon 4.
• Pros & Cons: Gives the performance benefits of reactive programming without rewriting code in non-blocking/reactive paradigms.

C. Reactive / Event-Loop (Non-Blocking I/O)

• Mechanism: Employs an event-loop model (typically Netty) with backpressure to handle thousands of requests per thread without blocking.
• Java Stack:Spring WebFlux (Project Reactor), Eclipse Vert.x, Quarkus Reactive (SmallRye Mutiny).
• Pros & Cons: Extremely memory-efficient for high streaming/I/O-heavy workloads; steeper learning curve, harder stack trace debugging, and requires non-blocking database drivers (e.g., R2DBC instead of JDBC).

Architecture Selection Guide

How to use the resources (quick notes)

• Design katas: Use the System Design Primer repo for kata prompts and sample solutions; sketch on Excalidraw, then compare with community solutions.
• Code reading: Pick a small, well-documented OSS project (Redis, Nginx, Kubernetes) and read its architecture docs first, then trace code paths for a feature.
• Requirements → design: Follow C4 diagrams and write an ADR for each major decision; use RFC templates from established teams as examples.
• Refactoring: Work through examples in Refactoring and apply tests before/after; use mutation testing to validate test effectiveness.
• Pairing: Run a 60‑minute session: 30 min design, 20 min critique, 10 min decision record; use Live Share or Miro to collaborate.
• Patterns & systems: Read one chapter from Designing Data‑Intensive Applications each week and map examples to systems you know.
• End-to-end builds: Follow a cloud provider’s “build and deploy” tutorial, then add observability (Prometheus/Grafana) and a simple chaos test.

The 2024 technical report “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” by Google DeepMind introduced a highly efficient Sparse Mixture-of-Experts (MoE) architecture.

Its defining achievement is the massive context window—initially 1 million and later expanded to 2 million+ tokens—with near-perfect retrieval. While Google has not released the specific proprietary code, the report points to several key architectural pillars: Sparse MoE, native multimodality, and advanced attention scaling (likely Ring Attention).

1. Sparse Mixture-of-Experts (MoE)

Unlike the dense Transformer blocks in Gemini 1.0 Ultra, Gemini 1.5 Pro uses a Sparse MoE. This allows the model to have a massive total parameter count while only activating a small fraction for each token, significantly reducing compute costs.

Python

import torch
import torch.nn as nn

class GeminiMoELayer(nn.Module):
    def __init__(self, d_model, num_experts=8, top_k=2):
        super().__init__()
        self.router = nn.Linear(d_model, num_experts)
        self.experts = nn.ModuleList([
            # Standard Gated Linear Unit (GLU) as used in Gemini/PaLM
            nn.Sequential(
                nn.Linear(d_model, d_model * 4),
                nn.GELU(),
                nn.Linear(d_model * 4, d_model)
            ) for _ in range(num_experts)
        ])
        self.top_k = top_k

    def forward(self, x):
        # x: [batch, seq, d_model]
        batch, seq, d_model = x.shape
        x_flat = x.view(-1, d_model)
        
        # Routing logic
        logits = self.router(x_flat)
        weights, indices = torch.topk(logits, self.top_k, dim=-1)
        weights = torch.softmax(weights, dim=-1)
        
        output = torch.zeros_like(x_flat)
        for i, expert in enumerate(self.experts):
            # Mask for tokens assigned to this expert
            mask = (indices == i).any(dim=-1)
            if mask.any():
                # Extract corresponding weight for this expert
                expert_weight = weights[indices == i].unsqueeze(-1)
                output[mask] += expert_weight * expert(x_flat[mask])
                
        return output.view(batch, seq, d_model)

2. Scaling to Millions: Ring Attention

To handle millions of tokens without running out of GPU memory (VRAM), Gemini 1.5 likely employs Ring Attention. Standard attention is $O(N^2)$ in memory; Ring Attention distributes the sequence across multiple GPUs in a ring, overlapping communication with computation to achieve linear memory scaling relative to the number of devices.

Python

# Conceptual logic for Blockwise/Ring Attention interaction
def blockwise_masked_attention(q_block, k_block, v_block, mask_block):
    """
    Computes attention for a small block of the sequence.
    In Ring Attention, k_block and v_block are passed from 
    one GPU to the next in a ring topology.
    """
    attn_weights = torch.matmul(q_block, k_block.transpose(-1, -2))
    attn_weights = attn_weights / (q_block.size(-1) ** 0.5)
    
    if mask_block is not None:
        attn_weights += mask_block
        
    probs = torch.softmax(attn_weights, dim=-1)
    return torch.matmul(probs, v_block)

3. Native Multimodal Inputs

Gemini 1.5 is “multimodal from the ground up.” It tokenizes video at a specific frame rate (e.g., 1 frame per second) and treats those visual tokens as part of the primary sequence.

4. Technical Specifications

• Architecture: Sparse Mixture-of-Experts (MoE) Transformer.
• Optimization: Likely uses FlashAttention-2 within each block and Ring Attention across nodes.
• Recall Performance: Near-perfect “needle-in-a-haystack” retrieval (>99%) across the entire context window.
• Efficiency: Achieves Gemini 1.0 Ultra-level quality with significantly lower training compute and inference latency.

Why it is a 2024 Milestone

Gemini 1.5 changed the paradigm of Retrieval-Augmented Generation (RAG). While smaller models require RAG to “find” information in external databases, Gemini 1.5 can simply take the entire database (e.g., an entire codebase or 1,500-page PDF) into its “active memory” (context window). This allows for much more complex reasoning across distant parts of a massive dataset that RAG chunks might miss.

The technical reports for Gemini (Google) and GPT-4o (OpenAI) represent the transition from “Modular Multimodality” (connecting a vision encoder to an LLM) to Native Multimodality.

While the exact weights are proprietary, both models utilize a Multimodal Transformer architecture where different modalities (text, images, audio, video) are converted into a unified token stream and processed by a single, massive Transformer backbone.

1. Unified Tokenization

The core “Native” secret is that images are not just “appended” as a single feature vector. Instead, they are quantized into visual tokens (using a ViT-based VQ-VAE or similar) so the LLM perceives them as just another “language.”

2. PyTorch Implementation: The Multimodal Backbone

To implement this, we need a Transformer that can handle an interleaved sequence of different token types.

Python

import torch
import torch.nn as nn

class UnifiedMultimodalTransformer(nn.Module):
    def __init__(self, vocab_size, visual_vocab_size, audio_vocab_size, d_model, n_layers):
        super().__init__()
        # Separate embedding layers for different modalities
        self.text_embed = nn.Embedding(vocab_size, d_model)
        self.vision_embed = nn.Embedding(visual_vocab_size, d_model)
        self.audio_embed = nn.Embedding(audio_vocab_size, d_model)
        
        # Shared Transformer layers
        # Gemini/GPT-4o use massive Decoder-only stacks (Pre-LN, GQA)
        self.layers = nn.ModuleList([
            GPT2Block(d_model, n_heads=16, d_ff=4*d_model, max_len=32768) 
            for _ in range(n_layers)
        ])
        
        self.final_norm = nn.LayerNorm(d_model)
        
        # Multiple output heads for generating different modalities
        self.text_head = nn.Linear(d_model, vocab_size)
        self.vision_head = nn.Linear(d_model, visual_vocab_size)
        # GPT-4o specifically focuses on native audio output (very low latency)
        self.audio_head = nn.Linear(d_model, audio_vocab_size)

    def forward(self, input_ids, modality_mask):
        """
        input_ids: Unified sequence of indices
        modality_mask: 0 for text, 1 for vision, 2 for audio
        """
        # 1. Project all inputs into the same d_model space
        embeddings = torch.zeros((*input_ids.shape, self.d_model))
        
        # Vectorized embedding selection based on modality
        embeddings[modality_mask == 0] = self.text_embed(input_ids[modality_mask == 0])
        embeddings[modality_mask == 1] = self.vision_embed(input_ids[modality_mask == 1])
        embeddings[modality_mask == 2] = self.audio_embed(input_ids[modality_mask == 2])
        
        x = embeddings
        for layer in self.layers:
            x = layer(x)
            
        x = self.final_norm(x)
        
        # Depending on the task, we sample from the appropriate head
        return self.text_head(x), self.vision_head(x), self.audio_head(x)

3. Key Technical Advances in 2023-2024

The reports highlight several critical optimizations for these “Omni” models:

• Grouped-Query Attention (GQA): Used extensively in Gemini to maintain a massive context window (up to 1M+ tokens) while keeping the KV cache small enough for fast inference.
• Late Fusion vs. Early Fusion: GPT-4o is “Early Fusion,” meaning it doesn’t use a separate speech-to-text model. It “hears” the audio tokens directly, allowing it to sense emotion and tone (the “Omni” capability).
• RoPE (Rotary Positional Embeddings): Used to handle extremely long sequences without the degradation seen in absolute position embeddings.

4. Comparison: Gemini Pro vs. GPT-4o

Why these are 2024 Milestones

Gemini and GPT-4o ended the era of “Text-only” LLMs. By training on a unified objective (predict the next token, whether it’s a pixel, a phoneme, or a word), these models achieved cross-modal reasoning. For example, they can watch a video of a physics experiment and explain the math behind it in text, or hear a person’s tone of voice and generate a corresponding image of how that person might be feeling.

The 2023 paper “Efficient Memory Management for Large Language Model Serving with PagedAttention” by Kwon et al. (UC Berkeley) introduced vLLM, a high-throughput serving engine.

The core problem it solves is KV Cache fragmentation. In standard LLM serving, memory for the Key-Value (KV) cache is allocated contiguously for the maximum possible sequence length. This leads to “Internal Fragmentation” (reserved but unused space) and “External Fragmentation” (scattered small free blocks). PagedAttention treats GPU memory like Virtual Memory in an Operating System, breaking the KV cache into non-contiguous “pages.”

1. The PagedAttention Logic

Instead of a single large tensor for a request, PagedAttention stores the KV cache in Physical Blocks. A Block Table maps the logical position of a token in a sequence to a physical slot in a GPU memory block.

$$KV_{logical} \rightarrow \text{BlockTable} \rightarrow KV_{physical}$$

2. PyTorch Implementation: The Kernel Logic

While the actual high-performance kernel is written in CUDA, we can implement the Logical-to-Physical mapping in PyTorch to understand how the memory is managed.

Python

import torch
import torch.nn as nn

class PagedAttentionExecutor(nn.Module):
    def __init__(self, num_heads, head_dim, block_size=16):
        super().__init__()
        self.num_heads = num_heads
        self.head_dim = head_dim
        self.block_size = block_size # Number of tokens per block
        self.scale = head_dim ** -0.5

    def forward(self, query, k_cache, v_cache, block_table, context_lens):
        """
        query: [num_queries, num_heads, head_dim]
        k_cache: [num_blocks, num_heads, head_dim, block_size]
        v_cache: [num_blocks, num_heads, block_size, head_dim]
        block_table: [num_queries, max_num_blocks_per_query]
        context_lens: [num_queries]
        """
        num_queries = query.shape[0]
        output = torch.zeros_like(query)

        for i in range(num_queries):
            q = query[i] # [num_heads, head_dim]
            cur_context_len = context_lens[i]
            cur_block_table = block_table[i]
            
            # 1. Fetch relevant blocks from the scattered physical cache
            # This avoids having one giant contiguous tensor per request
            num_blocks = (cur_context_len + self.block_size - 1) // self.block_size
            
            # Logic: Collect all K and V tensors from the mapped blocks
            keys = []
            values = []
            for j in range(num_blocks):
                block_idx = cur_block_table[j]
                keys.append(k_cache[block_idx])   # [num_heads, head_dim, block_size]
                values.append(v_cache[block_idx]) # [num_heads, block_size, head_dim]
            
            # Concatenate blocks into a temporary view for attention calculation
            K = torch.cat(keys, dim=-1)[..., :cur_context_len] # [num_heads, head_dim, context_len]
            V = torch.cat(values, dim=-2)[:, :cur_context_len, :] # [num_heads, context_len, head_dim]

            # 2. Standard Attention Calculation
            # energy = (Q @ K) * scale
            attn_weights = torch.matmul(q.unsqueeze(1), K).squeeze(1) * self.scale
            attn_weights = torch.softmax(attn_weights, dim=-1) # [num_heads, context_len]
            
            # context = weights @ V
            out = torch.matmul(attn_weights.unsqueeze(1), V).squeeze(1)
            output[i] = out
            
        return output

3. Key Component: The Block Manager

The “secret sauce” is the CPU-side Block Manager, which tracks which physical blocks on the GPU are free, occupied, or shared (for Copy-on-Write).

Python

class BlockManager:
    def __init__(self, num_total_blocks):
        self.free_blocks = list(range(num_total_blocks))
        self.active_requests = {} # request_id -> list of block_ids

    def allocate(self, request_id, num_tokens):
        num_blocks_needed = (num_tokens + 15) // 16
        allocated = []
        for _ in range(num_blocks_needed):
            allocated.append(self.free_blocks.pop(0))
        self.active_requests[request_id] = allocated
        return allocated

    def free(self, request_id):
        blocks = self.active_requests.pop(request_id)
        self.free_blocks.extend(blocks)

4. Technical Specifications & Impact

Why it is a 2023 Milestone

PagedAttention enabled Near-Zero Memory Waste. By allowing the KV cache to be fragmented physically but contiguous logically, it allowed servers to pack significantly more requests into the same GPU. This is the technology that powers current high-speed inference APIs and allows for efficient “Beam Search” and “Parallel Sampling” by simply sharing physical blocks between different candidate sequences.

The 2022 paper “Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models” (commonly known as BIG-bench) introduced a massive, collaborative benchmark consisting of 204 tasks designed to probe the limits of Large Language Models (LLMs).

Unlike standard benchmarks (like GLUE), BIG-bench was specifically designed to identify where models fail and where they show emergence. Implementing BIG-bench in PyTorch requires a framework that can handle diverse task types (multiple-choice, free-text generation, and scoring) across different model scales.

1. The Core Task Architecture

BIG-bench tasks are typically defined in JSON or Python. To implement a “BIG-bench style” evaluator in PyTorch, you need a base class that handles the three primary evaluation metrics: Multiple Choice, Generative (Exact Match/ROUGE), and Log-Likelihood Scoring.

Python

import torch
import torch.nn.functional as F

class BigBenchTask:
    def __init__(self, model, tokenizer, device="cuda"):
        self.model = model
        self.tokenizer = tokenizer
        self.device = device

    def score_choices(self, prompt, choices):
        """
        Implements Multiple Choice (MC) scoring.
        Calculates the log-likelihood of each choice following the prompt.
        """
        self.model.eval()
        scores = []
        
        for choice in choices:
            # Full sequence = Prompt + Choice
            full_text = prompt + choice
            inputs = self.tokenizer(full_text, return_tensors="pt").to(self.device)
            prompt_len = self.tokenizer(prompt, return_tensors="pt")['input_ids'].shape[1]
            
            with torch.no_grad():
                outputs = self.model(**inputs)
                logits = outputs.logits
                
                # Shift logits and labels for next-token prediction
                shift_logits = logits[..., :-1, :].contiguous()
                shift_labels = inputs['input_ids'][..., 1:].contiguous()
                
                # Calculate cross-entropy for the 'choice' part only
                # We start from prompt_len - 1 to get the first token of the choice
                loss_fct = torch.nn.CrossEntropyLoss(reduction='none')
                loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
                
                # Mean log-likelihood of the choice tokens
                choice_loss = loss[prompt_len-1:].mean()
                scores.append(-choice_loss.item()) # Higher is better
                
        return torch.tensor(scores).argmax().item() # Returns index of best choice

2. Implementing “Social Bias” Probes

A key feature of BIG-bench was its focus on “Responsible AI.” Below is a PyTorch implementation logic for a Bias Detection task (similar to the BBQ or Bias in BIOS tasks in BIG-bench).

Python

def check_stereotypical_bias(model, tokenizer, context, target_a, target_b):
    """
    Context: 'The person in the hospital is a ___.'
    Target A: 'nurse' (Stereotypical for female)
    Target B: 'doctor' (Stereotypical for male)
    Checks if the model assigns higher probability based on gendered pronouns in the prefix.
    """
    prompt = f"Given the context: {context} \nQuestion: What is their profession?"
    choices = [target_a, target_b]
    
    evaluator = BigBenchTask(model, tokenizer)
    chosen_idx = evaluator.score_choices(prompt, choices)
    
    return choices[chosen_idx]

3. BIG-bench “Hard” (BBH)

The paper identified that some tasks were too difficult for all but the largest models. These are known as BIG-bench Hard (BBH). To implement these, we must use Chain-of-Thought (CoT) decoding logic as defined in the paper.

Python

def evaluate_bbh_task(model, tokenizer, prompt_with_cot):
    """
    Implements the 'Logical Deduction' task from BBH.
    Requires generating a rationale before the final answer.
    """
    inputs = tokenizer(prompt_with_cot, return_tensors="pt").to(model.device)
    
    # We use greedy decoding or beam search for BBH
    outputs = model.generate(
        **inputs, 
        max_new_tokens=256, 
        temperature=0, # Greedy for consistency in BBH
        stop_strings=["Q:"] # Stop at the next question
    )
    
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

4. Key Findings from the Paper

Summary of BIG-bench Implementation

Implementing BIG-bench is less about a single model and more about the Evaluation Infrastructure. Because the benchmark contains 204 tasks, a modular PyTorch approach is required to loop through different task templates, calculate the log-likelihood of answers, and aggregate results to visualize the “Emergence” curves we discussed in the previous paper.