PRINCIPLES OF MECHANISM OF A WRITTEN LANGUAGE
With a few exceptions, notably Chinese, all modern languages are
constructed of words which in turn are formed from letters. In any
given language the number of letters, and their conventional order is
fixed. Thus English is written with 26 letters and their conventional
order is A, B, C, D, E, etc. Some letters are used very frequently and
others rarely. In fact, if ten thousand consecutive letters of a text
be counted and the frequency of occurrence of each letter be noted,
the numbers found will be practically identical with those obtained
from any other text of ten thousand letters in the same language. The
relative proportion of occurrence of the various letters will also
hold approximately for even very short texts.
Such a count of a large number of letters, when it is put in the form
of a table, is known as a frequency table. Every language has its own
distinctive frequency table and, for any given language, the frequency
table is almost as fixed as the alphabet. There are minor differences
in frequency tables prepared from texts on special subjects. For
example, if the text be newspaper matter, the frequency table will
differ slightly from one prepared from military orders and will also
differ slightly from one prepared from telegraph messages. But these
differences are very slight as compared with the differences between
the frequency tables of two different languages.
Again there is a fixed ratio of occurrence of every letter with every
other for any language and this, put in table form, constitutes a
table of frequency of digraphs. In the same way a table of trigraphs,
showing the ratio of occurrence of any three letters in sequence,
could be prepared, but such a table would be very extensive and a
count of the more common three letter combinations is usually used.
Other tables, such as frequency of initial and final letters of words,
might be of value but the common practice is to put cipher text into
groups of five or ten letters each and eliminate word forms. This is
almost a necessity in telegraphic and radio communication to enable
the receiving operator to check correct receipt of a message. He
must get five letters, neither more nor less, per word or he is
sure a mistake has been made. There is little difficulty, as a rule,
in restoring word forms in the deciphered message.
We will now take up, in order, the various frequency tables and
linguistic peculiarities of English and Spanish. Frequency tables
for French, German, and Italian for single letters will follow. All
frequency tables have been re-calculated from at least ten thousand
letters of text and compared with existing tables. No marked difference
has been found in any case between the re-calculated tables and those
already in use.
Data for Solution of Ciphers in English
Table I.--Normal frequency table. Frequency for ten thousand letters
and for two hundred letters. This latter is put in graphic form and is
necessarily an approximation. Taken from military orders and reports,
English text.
10,000 Letters 200 Letters
A 778 16 1111111111111111
B 141 3 111
C 296 6 111111
D 402 8 11111111
E 1277 26 11111111111111111111111111
F 197 4 1111
G 174 3 111
H 595 12 111111111111
I 667 13 1111111111111
J 51 1 1
K 74 2 11
L 372 7 1111111
M 288 6 111111
N 686 14 11111111111111
O 807 16 1111111111111111
P 223 4 1111
Q 8
R 651 13 1111111111111
S 622 12 111111111111
T 855 17 11111111111111111
U 308 6 111111
V 112 2 11
W 176 3 111
X 27
Y 196 4 1111
Z 17
Vowels AEIOU = 38.37%; consonants LNRST = 31.86%; consonants JKQXZ
= 1.77%.
The vowels may be safely taken as 40%, consonants LNRST as 30% and
consonants JKQXZ as 2%.
Order of letters: E T O A N I R S H D L U C M P F Y W G B V K J X Z Q.
Table II.--Frequency table for telegraph messages, English text. This
table varies slightly from the standard frequency table because the
common word "the" is rarely used in telegrams and there is a tendency
to use longer and less common words in preparing telegraph messages.
10,000 Letters 200 Letters
A 813 16 1111111111111111
B 149 3 111
C 306 6 111111
D 417 8 11111111
E 1319 26 11111111111111111111111111
F 205 4 1111
G 201 4 1111
H 386 8 11111111
I 711 14 11111111111111
J 42 1 1
K 88 2 11
L 392 8 11111111
M 273 6 111111
N 718 14 11111111111111
O 844 17 11111111111111111
P 243 5 11111
Q 38 1 1
R 677 14 11111111111111
S 656 13 1111111111111
T 634 13 1111111111111
U 321 6 111111
V 136 3 111
W 166 3 111
X 51 1 1
Y 208 4 1111
Z 6
In this table the vowels AEIOU = 40.08%, consonants LNRST = 30.77%
and consonants JKQXZ = 2.25%.
Orders of letters: E O A N I R S T D L H U C M P Y F G W B V K X J Q Z.
Table III.--Table of frequency of digraphs, duals or pairs
(English). This table was prepared from 20,000 letters, but the
figures shown are on the basis of 2,000 letters. For this reason
they are, to a certain extent, approximate; that is, merely because
no figures are shown for certain combinations, we should not assume
that such combinations never occur but rather that they are rare. The
letters in the horizontal line at the top and bottom are the leading
letters; those in the vertical columns at the sides are the following
letters. Thus in two thousand letters we may expect to find AH once
and HA twenty-six times.
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
A 1 7 10 22 3 2 26 4 2 2 7 8 11 2 9 13 12 9 2 4 1 12
B 5 1 2 1 1 1 1 2 2 1 3 1
C 6 1 1 14 2 11 11 3 2 3 1 1 1 1
D 6 12 30 1 2 4 30 1 4 1 1 1 1 3
E 11 14 16 12 2 6 33 10 2 6 18 14 12 1 7 36 11 12 2 16 5 1 1
F 3 2 8 2 1 2 2 1 3 25 3 1 1 1
G 4 1 3 2 11 2 3 1
H 1 11 2 4 1 4 1 2 1 1 2 10 50 3 2
I 2 1 4 12 6 5 1 12 1 5 9 8 12 1 3 12 13 22 2 3 6 1 1
J 1
K 1 1 2 2 1 1
L 14 6 2 1 6 1 1 1 6 9 3 6 3 3 2 3 5
M 7 3 13 2 2 3 4 1 10 4 1 1 2
N 38 3 25 2 1 31 3 2 2 39 4 3 11 2
O 1 1 12 4 8 8 3 12 18 2 4 7 8 3 7 13 15 22 2 6 1 5
P 2 1 8 1 2 4 2 3 2 1 8 1 4 3 1
Q 2 1 1 1
R 16 1 3 3 40 3 6 2 6 1 2 1 25 8 2 2 8 11 2
S 16 1 3 25 1 2 17 1 2 1 12 7 2 9 11 6 11 1 6
T 25 1 3 12 13 5 2 3 20 2 1 24 8 2 16 20 11 6 2 2 7
U 1 2 1 6 1 3 2 2 3 3 1 17 1 5 3 5 5 1
V 3 1 5 5 3 2 5 1
W 1 2 8 1 1 1 1 2 4 2 3 3
X 1 4 2 1 1
Y 3 2 2 4 1 1 8 1 2 1 3 1 7
Z 1 1 1
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
Table IV.--Order of frequency of common pairs to be expected in a count
of 2,000 letters of military or semi-military English text. (Based
on a count of 20,000 letters).
TH 50 AT 25 ST 20
ER 40 EN 25 IO 18
ON 39 ES 25 LE 18
AN 38 OF 25 IS 17
RE 36 OR 25 OU 17
HE 33 NT 24 AR 16
IN 31 EA 22 AS 16
ED 30 TI 22 DE 16
ND 30 TO 22 RT 16
HA 26 IT 20 VE 16
Table V.--Table of recurrence of groups of three letters to be expected
in a count of 10,000 letters of English text.
THE 89 TIO 33 EDT 27
AND 54 FOR 33 TIS 25
THA 47 NDE 31 OFT 23
ENT 39 HAS 28 STH 21
ION 36 NCE 27 MEN 20
Table VI.--Table of frequency of occurrence of letters as initials
and finals of English words. Based on a count of 4,000 words; this
table gives the figures for an average 100 words and is necessarily
an approximation, like Table III. English words are derived from so
many sources that it is not impossible for any letter to occur as an
initial or final of a word, although Q, X and Z are rare as initials
and B, I, J, Q, V, X and Z are rare as finals.
Letters A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
Initial 9 6 6 5 2 4 2 3 3 1 1 2 4 2 10 2 - 4 5 17 2 - 7 - 3 -
Final 1 - - 10 17 6 4 2 - - 1 6 1 9 4 1 - 8 9 11 1 - 1 - 8 -
It is practically impossible to find five consecutive letters in an
English text without a vowel and we may expect from one to three with
two as the general average. In any twenty letters we may expect to
find from 6 to 9 vowels with 8 as an average. Among themselves the
relative frequency of occurrence of each of the vowels, (including
Y when a vowel) is as follows:
A, 19.5% E, 32.0% I, 16.7%
O, 20.2% U, 8.0% Y, 3.6%
The foregoing tables give all the essential facts about the mechanism
of the English language from the standpoint of the solution of
ciphers. The use to be made of these tables will be evident when the
solution of different types of ciphers is taken up.
Data for the Solution of Ciphers in Spanish
The Spanish language is written with the following alphabet:
A B C CH D E F G H I J L LL
M N Ñ O P Q R RR S T U V X Y Z
while the exact sense often depends upon the use of accents over the
vowels. However, in cipher work it is exceedingly inconvenient to use
the permanent digraphs, CH, LL and RR and they do not appear as such in
any specimens of Spanish or Mexican cipher examined. Accented vowels
and Ñ are also not found and we may, in general, say that a cipher
whose text is Spanish will be prepared with the following alphabet:
A B C D E F G H I J L M N O P Q R S T U V X Y Z
and the receiver must supply the accents and the tilde over the N to
conform to the general sense.
However, many Mexican cipher alphabets contain the letters K and
W. This is particularly true of the ciphers in use by secret service
agents who must be prepared to handle words like NEW YORK, WILSON
and WASHINGTON. The letters K and W will, however, have a negligible
frequency except in short messages where words like these occur more
than once.
In this connection, if a cipher contains Mexican geographical names
like CHIHUAHUA, MEXICO, MUZQUIZ, the letters H, X and Z will have a
somewhat exaggerated frequency.
In Spanish, the letter Q is always followed by U and the U is always
followed by one of the other vowels, A, E, I or O. As QUE or QUI
occurs not infrequently in Spanish text, particularly in telegraphic
correspondence, it is well worth noting that, if a Q occurs in a
transposition cipher, we must connect it with U and another vowel. The
clue to several transposition ciphers has been found from this simple
relation.
Table VII.--Normal frequency table for military orders and reports,
calculated on a basis of 10,000 letters of Spanish text. The graphic
form is on a basis of 200 letters.
10,000 Letters 200 Letters
A 1352 27 111111111111111111111111111
B 102 2 11
C 474 9 111111111
D 524 10 1111111111
E 1402 28 1111111111111111111111111111
F 91 2 11
G 137 3 111
H 102 2 11
I 606 12 111111111111
J 41 1 1
L 517 10 1111111111
M 300 6 111111
N 619 12 111111111111
O 818 16 1111111111111111
P 257 5 11111
Q 87 2 11
R 751 15 111111111111111
S 724 14 11111111111111
T 422 8 11111111
U 387 7 1111111
V 85 2 11
X 6
Y 103 2 11
Z 42 1 1
In this table the vowels AEIOU = 45.65%; consonants LNRST = 30.33%;
consonants JKQXZ = 1.76%.
Order of letters:
E A O R S N I D L C T U M P G Y (BH) F Q V Z J X.
Table VIII.--Table of frequency of digraphs, duals or pairs, Spanish
text. Like Table III, this table is on the basis of 2,000 letters
although prepared from a count of 20,000 letters. For this reason
it is, to a certain extent an approximation; that is, merely because
no figures are shown for certain combinations, we should not assume
that such combinations never occur but rather that they are rare. The
letters in the horizontal lines at the top and bottom are the leading
letters; those in the vertical columns at the sides are the following
letters. Thus, in two thousand letters, we may expect to find AI
twice and IA twenty-three times.
A B C D E F G H I J L M N O P Q R S T U V X Y Z
A 9 4 19 11 5 6 17 23 54 18 9 3 20 29 11 21 8 6 2 5 A
B 6 3 1 4 B
C 24 6 6 24 5 3 8 8 9 5 2 2 C
D 31 29 3 19 13 10 9 4 D
E 12 2 6 59 10 1 5 7 2 12 18 22 4 9 38 25 28 25 3 3 E
F 4 4 4 3 3 1 F
G 2 4 8 4 2 G
H 2 12 10 2 1 H
I 2 23 16 5 2 3 11 13 6 10 5 3 I
J 3 2 1 J
L 21 3 6 39 3 3 7 21 5 6 12 2 2 L
M 12 6 5 1 6 15 7 2 6 1 M
N 32 46 2 8 32 12 2 N
O 26 22 2 6 3 4 9 16 2 8 20 15 7 11 O
P 13 3 2 4 9 2 7 4 11 P
Q 11 5 1 2 3 1 Q
R 40 27 2 4 4 36 3 11 17 3 R
S 39 52 10 7 14 2 14 3 S
T 5 13 4 4 18 5 6 30 T
U 2 4 2 6 3 4 5 2 6 4 17 15 2 1 U
V 2 2 2 2 2 2 V
X X
Y 5 6 2 5 2 2 Y
Z 1 2 1 4 2 Z
A B C D E F G H I J L M N O P Q R S T U V X Y Z
Table IX.--Order of frequency of common pairs to be expected in a
count of 2,000 letters of Spanish military orders and reports. Based
on Table VIII.
DE 59 ON 32 AC 24
LA 54 AD 31 EC 24
ES 52 ST 30 CI 23
EN 46 ED 29 IA 23
AR 40 RA 29 DO 22
AS 39 TE 28 NE 22
EL 39 ER 27 AL 21
RE 38 CO 26 LL 21
OR 36 SE 25 PA 20
AN 32 UE 25 PO 20
Alphabetic Frequency Tables
(Truesdell)
Frequency of occurrence in 1,000 letters of text:
Letter French German Italian Portuguese
A 80 52 117 140
B 6 18 6 6
C 33 31 45 34
D 40 51 31 40
E 197 173 126 142
F 9 21 10 12
G 7 42 17 10
H 6 41 6 10
I 65 81 114 59
J 3 1 [1] 5
K [1] 10 [1]
L 49 28 72 32
M 31 20 30 46
N 79 120 66 48
O 57 28 93 110
P 32 8 30 28
Q 12 [1] 3 16
R 74 69 64 64
S 66 57 49 88
T 65 60 60 43
U 62 51 29 46
V 21 9 20 15
W [1] 15
X 3 [1] [1] 1
Y 2 [1] [1] 1
Z 1 14 12 4
Order of Frequency
French
E A N R S I U O L D C P M V Q F G B J Y Z
T H X
German
E N I R T S A D G H C L F M B W Z K V P J Q X Y
U O
Italian
E A I O L N R T S C D M U V G Z F B Q
P H
Portuguese
E A O S R I N M T D C L P Q V F G B J Z X Y
U H
Graphic Frequency Tables
Frequency of occurrence in 200 letters of text.
French
A 16 1111111111111111
B 2 11
C 6 111111
D 10 1111111111
E 39 111111111111111111111111111111111111111
F 2 11
G 1 1
H 1 1
I 13 1111111111111
J 1 1
K
L 10 1111111111
M 6 111111
N 16 1111111111111111
O 11 11111111111
P 6 111111
Q 2 11
R 15 111111111111111
S 13 1111111111111
T 13 1111111111111
U 12 111111111111
V 4 1111
W
X 1 1
Y
Z
Italian
A 23 11111111111111111111111
B 1 1
C 9 111111111
D 6 111111
E 25 1111111111111111111111111
F 2 11
G 3 111
H 1 1
I 23 11111111111111111111111
L 14 11111111111111
M 6 111111
N 13 1111111111111
O 19 1111111111111111111
P 6 111111
Q
R 13 1111111111111
S 10 1111111111
T 12 111111111111
U 6 111111
V 4 1111
X
Y
Z 2 11
German
A 10 1111111111
B 4 1111
C 6 111111
D 10 1111111111
E 32 11111111111111111111111111111111
F 4 1111
G 8 11111111
H 8 11111111
I 16 1111111111111111
J
K 2 11
L 6 111111
M 4 1111
N 24 111111111111111111111111
O 6 111111
P 2 11
Q
R 14 11111111111111
S 11 11111111111
T 12 111111111111
U 10 1111111111
V 2 11
W 3 111
X
Y
Z 3 111
Portuguese
A 28 1111111111111111111111111111
B 1 1
C 7 1111111
D 8 11111111
E 28 1111111111111111111111111111
F 2 11
G 2 11
H 2 11
I 12 111111111111
J 1 1
L 6 111111
M 9 111111111
N 10 1111111111
O 22 1111111111111111111111
P 6 111111
Q 3 111
R 13 1111111111111
S 18 111111111111111111
T 9 111111111
U 9 111111111
V 3 111
X
Y
Z 1 1