LLM Fundamentals

• Next-token prediction

  An LLM generates text by predicting the next token, choosing one, adding it, and repeating.

• What an LLM is not

  It is not a database, a search engine or a fixed program, and it does not look up stored answers.

• The path through the model

  Text becomes tokens, tokens become numbers the model can work with, its layers refine them, and it outputs the likelihood of each possible next token.

• What parameter counts mean

  Sizes like 8B or 70B count what the model has learned; more can mean more capability, but bigger is not automatically better for your task.

• Why chat models behave as they do

  Extra training on instructions and human feedback is why assistants follow instructions, answer conversationally, refuse some requests and hedge.

Mental model · Text → Tokens → Numbers → Model layers → Next-token likelihoods

Try it · Send the same prompt five times and note how the answers vary.

Outcome · Explain in two sentences how an LLM produces an answer and why it can sound confident while being wrong.

• Tokens, not words

  Models read and charge in tokens, and the same sentence can be split differently by different models.

• Why some text costs more

  Hindi, mixed Hindi-English, code and numbers often use far more tokens than plain English for the same meaning.

• Input vs output tokens

  You pay for both; output is usually priced higher and is generated one token at a time.

• Latency you can feel

  Time to first token and generation speed decide how fast a response feels; long answers take longer than long questions.

• What fills the context window

  Instructions, conversation history, documents, examples, tool results and the model's own answer all share one limit.

• Context is not knowledge or memory

  What's in the window is temporary, the model's training knowledge is fixed, and long-term memory is something you have to build.

• Bigger windows aren't better windows

  Facts buried in the middle of long inputs are often missed, and extra text dilutes the model's focus.

Try it · Count the tokens in an English paragraph, its Hindi translation, a piece of code and a list of numbers. Then estimate the cost of sending each one 10,000 times.

Outcome · Estimate what a request will cost and roughly how long it will take, and decide whether a task needs more context or better-chosen context.

• Models that see and hear

  Many models now accept images, audio and documents as well as text, and answer in text.

• Images cost tokens too

  An image is turned into tokens, and the cost grows with its resolution, so crop or resize to what the question needs.

• PDFs come in two kinds

  Digital PDFs carry their own text; scanned ones are just pictures of pages, and the model has to read them like an image.

• Audio

  You can transcribe speech to text first and send the text, or send the audio to a model that understands it directly and keeps tone and emphasis.

• What models read well

  Text in images, the main point of a chart, and named fields on forms.

• Where they slip

  Counting objects, small or dense text, exact table values, and confidently describing details that aren't really there.

• Image or extracted text?

  When clean text is available, sending text is usually cheaper and more accurate; send images when layout, charts or handwriting matter.

Try it · Send the same chart to a model once as an image and once as text extracted from it. Ask both the same three questions and compare the answers and tokens used.

Outcome · Choose when to send images, audio or documents to a model, and predict what each will cost and where it may go wrong.

• Temperature

  Lower temperature makes answers more consistent; higher makes them more varied.

• Length limits and stop points

  Cap how long an answer can be and where it should stop, to control cost and format.

• Match randomness to the task

  Keep it low for extracting facts, classifying and factual answers; raise it for brainstorming and creative work.

• Reasoning models

  Some models think longer before answering, which helps on hard math, code, planning and multi-step problems.

• When reasoning is wasted

  For extraction, rewriting, simple classification and summaries, a fast model is usually just as good and much cheaper.

• The trade-off

  More reasoning can improve quality but costs more and takes longer; reasoning models may also ignore some settings like temperature.

Try it · Run one fact-extraction task and one creative task at low and high temperature. Then give three hard problems to a fast model and a reasoning model, and compare accuracy, speed and cost.

Outcome · Choose settings and the type of model based on the task instead of guessing.

• Common strengths

  Summarizing, rewriting and extracting information, classifying, coding, translating and handling messy text.

• Common failure modes

  Making things up, arithmetic slips, shaky reasoning, ignoring instructions, sensitivity to wording, outdated knowledge and giving different answers each time.

• Why models make things up

  They are trained to produce a likely continuation, not to check facts, so fluent wrong answers are expected behavior.

• Ask "how reliably?", not "can it?"

  The real question is how often a model gets your task right in your situation.

Try it · Write 10 questions: some the model should know, some it can't know, some built on a false premise, and some missing key information. Record each answer and how confident it sounds.

Outcome · Predict which tasks an LLM will handle well and which need safeguards such as giving it the right documents, checking its output, or a human review.

• Closed vs open models

  Models from API providers offer top capability with less control; open models you can run yourself offer control and privacy but take more work.

• No single best model

  The right model depends on your task, budget, speed needs and data rules.

• Knowledge cutoffs

  Anything newer than a model's training date has to be given to it, through documents or tools.

• Pin your version

  The model behind a familiar name can change or be retired, so choose a specific version and re-test when it changes.

• Leaderboards narrow the field; your examples decide

  Use public rankings to pick a shortlist, then test the shortlist on your own questions.

• Five things to compare

  Quality on your task, speed, cost per request, how much input it can take, and any privacy or feature needs.

• How to choose

  Define the task → shortlist → test on your own examples → compare quality, speed and cost → look at the failures → decide.

Outcome · Choose a model based on results from your own examples rather than popularity or rankings.