LLM: What Interviews Test on Calculation and Data

AI engineer interviews revisit two essential areas: the computational properties of LLMs versus RNNs and data augmentation in NLP. Here are the technical benchmarks to master, from memory costs in decoding to augmentation frameworks and their risks. A preparation method centered around a concrete project complements these fundamentals.

Memory and Decoding: O(T) with KV Cache, O(T²) without

In autoregressive generation, Transformers and RNNs decode one token at a time, with a time complexity of O(T). In RNN inference, only the current hidden state is retained, resulting in a memory footprint of O(1) relative to the sequence length. For Transformers, the memory footprint depends on the use of a KV cache. Without a cache, attention is recalculated over the entire prefix, and the QKᵀ matrix can reach O(T²) in memory. With a KV cache, the K/V states of previous tokens are stored, reducing the resident memory during decoding to O(T).

Computational Costs: Self-Attention O(T²) vs RNN O(T)

Standard self-attention relates all tokens to each other through pairwise interactions. For each token, attending to all others costs O(T), leading to a total cost of O(T²) over the sequence. An RNN, on the other hand, processes tokens sequentially, summarizing the history in a hidden state, resulting in a cost of O(1) per token and O(T) in total. For a length T, the FLOPs complexity is thus O(T²) for a Transformer and O(T) for an RNN.

Training: Parallelism, Long Dependencies, and Memory

During training, Transformers parallelize the processing of existing input tokens, while RNNs remain sequential. Assuming sufficient parallel hardware, the sequential time complexity is O(1) for a Transformer compared to O(T) for an RNN. Transformers also model global dependencies between tokens, where long-term information generally degrades in an RNN, and this advantage of long dependencies persists during inference. In terms of memory during training, an RNN stores states per token for backpropagation, at O(T), while the peak memory of a Transformer comes from pairwise attention (QKᵀ matrix) at O(T²). Therefore, for a length T, the memory complexity during training is O(T²) for a Transformer and O(T) for an RNN.

Data Augmentation: Frameworks, Uses, and Trade-offs

In the absence of sufficient data, augmentation is a common lever and often evaluated in interviews. Two frameworks dominate in NLP: the rule-based approach and the generative approach. The former is less costly and more controllable in terms of quality and accuracy, at the expense of limited coverage and weaker generalization; it works well for small datasets and cold starts. The latter is less controllable, relies on a powerful model, increases costs, and complicates validation, but it better improves generalization and is used when a performant model is already available. Trade-offs revolve around the phase (start-up vs maturity), level of control, cost, and validation, with risks to monitor: overfitting and quality degradation due to injected noise.

Concrete Techniques: Paraphrases, Noise, Translation, Generation

Rule-based augmentation modifies small portions of text using explicit rules, sometimes through simple functions with dictionaries or a small LLM. In terms of paraphrasing, one can replace synonyms, change formats (dates or amounts), or insert/remove words; regarding noise, lexical errors, grammatical mistakes, or irrelevant sentences can be introduced at the document level. Paraphrasing/extending aims for generalization, while noise injection enhances robustness. The generative approach covers the production of alternative surface forms that preserve meaning, rewriting, back-translation, partial rewriting of a response to the same question, as well as generation that introduces new semantic information from textual or non-textual inputs.

Preparing through a Project and Reasoned Choices

Effective preparation involves conducting a simple project, such as a chatbot. Breaking down the steps, listing practical pitfalls, inventorying suitable techniques, and clarifying trade-offs according to scenarios form the core of the exercise. This approach helps build transferable skills and can aid in succeeding in AI engineer interviews. The proposed framework aims to be a source of inspiration in a rapidly evolving context where no guide can exhaust all possible questions.