Base Frequency Analyzer (Cryptanalysis Teaching Tool) mechanism

Model scope

The core frequency calculation and the standard chi-squared statistic are valid under the stated assumptions, but the specification does not define preprocessing, normalization, sample-size handling, or the algorithm for top candidate mappings. The verdict is NEEDS_REVIEW until those ambiguities are resolved.

• The frequency formula is incomplete if the output is labeled percent: count/total produces a fraction, so percent requires 100*count/total.
• The chi-squared formula is underspecified. The standard statistic is sum over letters of (observed_count-expected_count)^2/expected_count, with expected_count=total*reference_probability.
• The reference frequencies shown are rounded percentages summing to 100.17%, not 100%. The implementation must explicitly normalize them before computing expected counts, or use a table whose probabilities sum exactly to 1.
• Top candidate letter mappings are not defined by a formula. Frequency ranking alone does not uniquely determine substitution mappings and can produce misleading candidates for short samples.
• The statistic needs a defined alphabet and preprocessing rule: case folding, treatment of spaces/punctuation/digits, and whether non-English or unmapped characters are excluded.

Units and conventions

• Ciphertext is case-insensitively reduced to the 26 English letters A-Z.
• total is the number of retained alphabetic characters; spaces, punctuation, digits, and other characters are excluded.
• frequency_percent is reported as 100*count/total.
• Reference values are rounded English letter frequencies in percent. For these vectors they are converted to integer weights q and normalized by Q=10017, so p(letter)=q(letter)/10017.
• Chi-squared uses expected_count=total*p(letter), includes all 26 letters including zero-observation letters, and is undefined when total=0.
• The reference convention is a widely tabulated standard English-frequency value set, rounded to the listed precision; no single universal English-frequency table exists.

Inspectable vector

One of 4 source-bound review vectors:

{
  "inputs": {
    "ciphertext_counts": {
      "E": 100
    },
    "total": 100,
    "preprocessing": "A-Z only; all non-E letters have count 0"
  },
  "expected_outputs": {
    "frequency_percent": {
      "E": 100,
      "all_other_letters": 0
    },
    "chi_squared": 688.74015748,
    "top_candidate_mappings": "not uniquely determined by the specified formulas"
  },
  "reference_data": {
    "english_frequency_percent_rounded": {
      "E": 12.7,
      "T": 9.1,
      "A": 8.2,
      "O": 7.5,
      "I": 7,
      "N": 6.7,
      "S": 6.3,
      "H": 6.1,
      "R": 6,
      "D": 4.3,
      "L": 4,
      "C": 2.8,
      "U": 2.8,
      "M": 2.4,
      "W": 2.4,
      "F": 2.2,
      "G": 2,
      "Y": 2,
      "P": 1.9,
      "B": 1.5,
      "V": 1,
      "K": 0.8,
      "J": 0.15,
      "X": 0.15,
      "Q": 0.1,
      "Z": 0.07
    },
    "normalized_integer_weights": {
      "E": 1270,
      "T": 910,
      "A": 820,
      "O": 750,
      "I": 700,
      "N": 670,
      "S": 630,
      "H": 610,
      "R": 600,
      "D": 430,
      "L": 400,
      "C": 280,
      "U": 280,
      "M": 240,
      "W": 240,
      "F": 220,
      "G": 200,
      "Y": 200,
      "P": 190,
      "B": 150,
      "V": 100,
      "K": 80,
      "J": 15,
      "X": 15,
      "Q": 10,
      "Z": 7
    },
    "normalization_sum": 10017,
    "source_convention": "widely tabulated standard English letter-frequency values; rounded table normalized for computation"
  },
  "working": "Q=10017. p_E=1270/10017. For 100 observed E characters, expected E=100*(1270/10017)=12.6785105 and expected all other letters are 100*p_i. Using the equivalent shortcut chi-squared=sum(O_i^2/E_i)-N, only E has O_i nonzero: chi-squared=100^2/(100*1270/10017)-100=100*(10017/1270)-100=688.74015748. Frequency percent for E=100*100/100=100.0%; every other letter=100*0/100=0.0%."
}

Failure boundary

BREAK_CLAIMED · SELF_FLAGGED

The specification exposes undefined-result hazards for an empty ciphertext and for zero-probability reference entries. It also leaves nonletter handling and reference-table units unspecified; different plausible conventions produce rejection or materially different chi-squared results.

Other records

Sources and evidence

These pointers bind the page to evidence. They do not grant authority or collapse verification into deployment.

• VERIFIED_LEDGER.json
• evidence-tranche7/tranche7-results.json
• BOUNDARY_HAZARD_CATALOG.json