Readability Analysis

Assessing and Enhancing the Readability of Holocaust Historical Documents: Providing Accessibility for High School Students.

Presented by*:

  • Katie Jones, B.A., Candidate for Master’s Degree in Education, Western Washington University
  • Justin M. Thibault, B.S. Candidate for Master’s Degree in Education, Western Washington University

Under the Supervision of:

  • Dr. Ray Wolpow, Associate Professor of Secondary Education, Western Washington University

November 17, 2000

Abstract

Historical Holocaust documents are valuable instructional tools for educators who seek to explore, with their students, this significant period of our shared history. Because the character of these documents is often scholarly, densely written, and rich in detail, educators must be aware that many of their students will not be able to read and comprehend these materials.

The authors examine ten historical source documents intended for tenth grade students. These documents concerned Holocaust rescuers and survivors, and provided historical information and testimonies. Formulaic and subjective readability analysis were performed and analyzed. It was concluded that many of the documents read far above the tenth grade level, and that at least one was, in every respect, inappropriate for the intended audience. One such document was utilized to demonstrate a number of instructional tools that educators can employ to help students to access hard to read documents which contain valuable content for study.

*Data Collection Performed by the Students of Western Washington University’s Secondary Education 425: Developmental Reading, Writing, and Learning in the Secondary School, Fall Quarter, 2000.

Contents

Introduction

Before any curriculum is implemented into a classroom, educators must first be aware of the difficulty or ease of the readability of the texts students will be asked to read. In this assessment ten source documents from the Holocaust are read and analyzed. Suggestions to make several of the more difficult articles more accessible to students in a tenth grade history class are provided. The ten historical documents have been named and numbered as follows:

  1. Letter to President Truman from Guy von Dardel
  2. Alex and Mela Roslan
  3. Survivor Testimonies I
  4. Survivor Testimonies II
  5. Enigma of the Righteous Persons
  6. Flight and Rescue
  7. New York Times Article
  8. Help to Jews From "Zegota" in Pola
  9. Escape to Sweden
  10. Raoul Wallenberg's Last Report to Sweden

Each document went through a two-step process that resulted in various outcomes for each article. First, a formulaic readability assessment, known as the Fry Readability Graph, was formally conducted by Western Washington University Secondary Education students. This widely used formula estimates grade reading levels for texts. Second, the Marshall Checklist was used to analyze each text to help determine the comprehensibility of the documents for tenth grade students. Finally, the formulaic and subjective readability analyses of each text led us to make suggestions to increase text readability. This is illustrated with one of the highly valuable source documents.

The evaluators made several assumptions regarding the students who would read these texts. First, the students would have the skills to read material written between a 9th to 11th grade reading level. Second, it was assumed that students in a tenth grade World or European History class would have already had instruction to WWII and Holocaust history. These students would know basic terminology unique to this period (ex: Nazi, Auschwitz, POW, etc.). It is also assumed that many of the students have read other Holocaust literature, such as the Diary of Anne Frank and Night, in their English classes.

Definition of Readability

The term readability refers to the difficulty or ease with which a text is read. Here are some of the major factors and considerations that determine the difficulty or ease of a text:

  • Concepts- Are the major ideas presented in quick succession with adequate explanation, repetition, or examples? Are the concepts presented in a concrete or and abstract way?
  • Vocabulary Level - Longer, multi-syllabic words are generally considered to be more difficult than shorter words. How are words with multiple meanings treated? How are technical terms defined in the text? How are proper nouns referenced in the text?
  • Sentence Structure - A lengthy sentence will often be more difficult to comprehend than a short one. The longer the sentence, the longer the reader must retain the subject, verb and predicate agreement and meaning. However, a poorly written series of short sentences is not an acceptable alternative. The author's writing style and usage is taken into account along with sentence length.
  • Format/Print Factors - Graphic aids, such as maps, pictures, and charts, will usually enhance readability of a text. Clear introductions, titles, bolded headings and subheadings can ease the path to the separation and delineation of major ideas from supporting ones. Size of print and style of print may also influence readability.
  • Comprehension Questions - What level of comprehension is required to answer the questions at the end of the text? Pre-questions and post-questions can help ease the difficulty of the text.

Readability Tools Used In This Readability Evaluation

Fry Readability Formula (FRF)

This widely used readability formula estimates grade reading levels from grade 1 to grade 17. This formula compares sentence length and number or syllables in three sample passages of 100 words each. If a text is charted on the Fry Readability Graph with a score of "9" then the readability of the text is at an 8th-10th grade level. (See Figure 1 for results using this assessment tool)

Figure 1: Fry Readability Analysis of All Documents

 

 

 

 

 

Document

Subject

Mean Syllable Count

Mean Sentence Count

Fry Readability (Grade Level)

1

A Letter to President Truman from the Brother of Raoul Wallenberg

179.4

2.9

15th to 17th+

2

Alex and Mela Roslan

136.5

7.1

5th to 8th

3

Survivor Testimonies

152.0

6.5

8th to 10th

4

Rescuer Testimonies

152.7

7.1

7th to 9th

5

The Enigma of the Righteous Persons

176.2

5.2

14th to 17th+

6

Flight and Rescue

146.5

6.2

7th to 9th

7

"Sweden Offers Aid to Denmark's Jews" New York Time Article

162.7

3.1

12th to 14th

8

Help To Jews From "Zegota" in Poland

166.5

3.3

13th to 15th

9

Flight from Denmark to Sweden by Boat

135.2

6.1

6th to 8th

10

A Letter to Raoul Wallenberg from J.W. Pehle

178.3

4.7

15th to 17th+

 

 

 

 

 

 

 

 

Summary of Readability

 

 

 

Lowest

6th to 8th

 

 

 

Highest

15th to 17th+

 

 

 

"Average"

9th to 12th

Fry Readability is a formulaic numerical assessment of reading materials.

Advantages of using the Fry Scale
  • Formulas are very objective-standardized
  • Measures sentence and vocabulary difficulty
  • Ease to repeat and easy to perform
  • Can be performed without evaluator biases of the text
Disadvantages of Fry
  • Does not consider student background knowledge
  • Does not consider the context of the material
  • Is not subjective
  • Uses only samples, not full text
  • Formula can be skewed by large numerals in the text

Marshall's Readability Checklist for Comprehensibility

This checklist looks for features of the text that a readability formula does not reveal. Unlike formulae, the data is subjective. Hence, specific, readability scores are not provided. Instead, the evaluator answers "Well Done", "Average" or "Poor" to questions under the following categories: Main Ideas, Vocabulary, Concepts, Related Ideas, Referents, Audience. (See Figure 2 for results using this assessment tool.)

Figure 2: Marshal Readability Checklist for All Documents

 

 

 

 

 

 

 

 

 

 

 

 

Document Number

 

1

2

3

4

5

6

7

8

9

10

Main Ideas

 

 

 

 

 

 

 

 

 

 

Are major points stated clearly?

Good

Good

Avg

Good

Poor

Good

Poor

Avg

Avg

Good

Are chapter titles and headings meaningful?

N/A

Good

Good

Good

Poor

Good

Poor

Avg

Avg

Avg/
Poor

Do titles outline major points clearly?

N/A

Good

Good

Good

Poor

Good

Avg

Poor/ Avg

Avg

Avg/
Poor

Vocabulary

 

 

 

 

 

 

 

 

 

 

Are key vocabulary terms defined clearly when the subject is new?

Poor

Avg

Good

Good

Poor

Avg

Poor

Poor

Avg

Poor

Are there terms used in a variety of contexts meaningful to the reader?

Poor

Avg

Good

Good

Poor

Avg

Poor

Poor

Avg

Poor

Concepts

 

 

 

 

 

 

 

 

 

 

Are new concepts introduced in the context of familiar concepts?

N/A

Avg

Good

Good

Poor

Avg

Poor

Poor

Avg

N/A

Are they well-defined within the text?

Poor

Avg

Good

Good

Poor

Avg

Poor

Poor

Avg

Poor

Related Ideas

 

 

 

 

 

 

 

 

 

 

Are ideas clearly related to each other?

Avg

Good

Good

Good

Poor

Good

Poor

Avg

Good

Avg/
Poor

Will the reader be able to understand relationships among ideas?

Avg/ Good

Good

Good

Good

Poor

Good

Poor

Avg

Good

Avg/
Poor

Could the reader illustrate these graphically?

Avg

Avg

Good

Good

Poor

Avg

Poor

Poor

Avg

Poor

Referents

 

 

 

 

 

 

 

 

 

 

Are pronouns used unambiguously?

Poor

Good

Good

Good

Poor

Good

Poor

Poor

Good

Avg

Do they usually refer to referents no more than one sentence away?

Poor

Good

Good

Good

Poor

Good

Poor

Avg

Good

Avg

Audience

 

 

 

 

 

 

 

 

 

 

Has the author addressed the audience intended?

Good

Good

Good

Good

Poor

Good

N/A

N/A

Avg

Avg

The Marshal readability checklist is used to subjectively analyze texts. The results augment formulaic assessment and student feedback in effectively determining the readability of materials.

Adapted from Tonjes, M., and Wolpow, R., Integrated Content Literacy, McGraw Hill, 1999, p. 139

The Importance of Triangulation

Two tools for measuring readability are FORMULAE and CHECKLISTS. Formulae usually measure vocabulary difficulty and sentence difficulty. Checklists measure ease and difficulty of text, but do find grade levels. In the guidelines for assessment, the International Reading Association and the National Council of Teachers of English recommend that multiple sources of data be employed. Hence a third point is necessary to triangulate this data. The third point is STUDENT FEEDBACK. This third point in the triangle is necessary to assure the text writer that the text is significant for the audience for which it is intended. To gather this data, it is recommended that students be asked to complete a checklist after they have read and discussed the documents in question.

Please Note: The evaluators only examined FORMULAS and CHECKLISTS. To complete the readability triangle, student feedback is highly recommended.

(Figure 3) - triangulation diagram (.pdf file)

Evaluation of Results and Suggestions
Appropriate for students in a 10th grade class

Documents # 2,3,4,6 and 9
These remaining documents all scored at or under the 10th grade level on the Fry Readability chart. We have concluded that these document are appropriate for use at this level. Please see the sample recommendations to make text more accessible if you feel, through your own judgement, that these documents need supplements.

Problem Documents with recommendations

a) Document #1 – Letter to President Truman from Guy von Dardel

The document’s Fry Readability level was charted at 15th to 17th grade. This letter scored well on readability checklists, so the difference between the two scales may be a result of the frequent use of numerals in the text and the frequent use of proper nouns. The suggestions to help make this text more readable for the audience intended are as follows:

  • Create a gloss or reading guide to accompany the text (see samples).
  • Create a vocabulary list to be used with the text and bold difficult words (see samples).
  • Attach a map or pictures with text to help capture students’ attentions.

The text seems to be very pertinent to the subject and has excellent potential for easy access to students at the 10th grade level.

b) Document #5 – Enigma of the Righteous Persons

This document scored at the highest end of the Fry Readability graph. It bears the distinction of being the most obtuse, dense, and inaccessible document under consideration. This document, as written, appears to be wholly inappropriate for readers at the 10th grade level.

  • It is recommended that this document be either rewritten entirely or removed form the student workbook.
c) Document #7 – New York Times Article

This document scored at the 12th to 15th grade on the Fry Readability chart and is inappropriate for readers at the 10th grade level, as written. The sentences are very long, vocabulary very difficult, and lack of easy access to the material. Despite this, the article contains a wealth of information relevant to the main concepts of the documents under review. The authors have provided sample instructional materials so that this text may still be used. (PLEASE SEE SAMPLE SUPPLEMENTARY INSTRUCTIONAL MATERIALS FOR THIS ARTICLE)

d) Document # 8 – Help to Jews from “Zegota” in Poland

This document’s Fry Readability level was charted at 12th to 16th grade. This document has a good intorduction, but lacks some important and helpful tools for comprehension. The problem that seems to arise in the readability checklists is that there is vocabulary in this document that seems far above the 10th grade reading level. The suggestions to help make this text more readable for the audience intended are as follows:

  • Create a vocabulary list to accompany the text (see sample).
  • Change the title to be more captivating for 10th grade students.
  • Highlight, or bold, the introductory paragraph.
  • Use bolded headings for the different sections of the text.
  • Attach a picture or map of Poland to paint a picture for the reader.

Because this is an important and powerful document, a 10th grader should have full access to the text. This text does not necessarily need to be re-written, but should be dressed up to captivate the reader and make it more usable in a 10th grade classroom.

e) Document #10 – Raoul Wallenburg’s Last Report to Sweden

This document’s Fry Readability level was charted at 15th to 17th grade. It is possible that the results of that formula have been skewed by the frequent use of numbers in the text (see explanation of Fry Readability Graph). The text has a good use of an introduction to set up the reader for the content. The suggestions for the improvement of the text for a 10th grade classroom are as follows:

  • Highlight, or bold, titles an headings.
  • Bold the introduction.
  • Create a list of the people mentioned in the text and an explanation of them (like a vocabulary list format, but about the people).
  • Pictures can only be beneficial.

Here are four instructional tools that educators may employ to assist their students in reading difficult texts (those texts that read above the skill level of students). Following the descriptions of these tools are SAMPLES that we feel would help improve DOCUMENT #7 – the New York Times Article to be more accessible to 10th grade students.

Instructional Tools

  • Summary/ Introduction (Advanced Organizer)
    A summary or introduction to the text gives a reader a preview of what is going to be read. Students can use this information to help them gather background knowledge of the text before they read it. Summaries and introductions also help the reader prepare himself or herself to read the text. This should be given to students before they read the text.
     
  • Vocabulary List
    A vocabulary list an provide students with a means to understand difficult or unfamiliar words in the text. In this sample, students are given the word, the definition, and the use of the word in the text. This can be given to students before they read the text and can be used to accompany the text.
     
  • Reading Guides
    Reading guides function by having the student answer questions about the text. The questions serve to assist the student in understanding vocabulary and help them in recognizing the key concepts of the materials. Additionally, reading guides assist students in working through difficult reading materials without the help of the teacher, which is an important consideration for busy classrooms or for homework assignments. The guide must be given to students before they read the text.
     
  • Glossing
    Glossing involves providing students with notes, in the margin of what is being read, that assist the reader in understanding the material. These marginal notes (the Gloss) can provide the reader with a summary of the main ideas, clarification and explanations of difficult texts, synonyms or definitions of important vocabulary, and ask the student to consider questions about the text. The Gloss serves as an “assistant” to the reader and guides then through difficult texts. The Gloss must be given to the student before the read the text.

Sample Summary

Introduction

Sweden Offers Protection to Danish Jews
New York Times, October 3, 1943

On the Jewish New Year Rosh-ha-Shanah, the German police arrested 7,000 Danish Jews. Sweden says they will protect any Danish Jew that can get to Sweden. One year ago thousands of Norwegian Jews ere arrested in Norway and sent to concentration camp in Poland. Sweden tried to help last year, but could only help a few Jews by paying money to the Nazis. Sweden hopes their aid this year will keep Danish Jews safe from what happened last year. Few Danish Jews have escaped in rowboats to cross the channel to reach Sweden for protection. There are many German patrol boats in the channel. It is very dangerous for the Danish Jews who are trying to cross in rowboats. Plans have already been made to send all the arrested Danish Jews to Poland. The German news press reported that the Danish Jews can no longer be a problem in Denmark. The press wants all of Germany to know that the Jews have been “removed from public life” in Denmark. Sweden hopes that the 7,000 Danish Jews do not suffer the same way that the Norwegian Jews suffered last week.

Vocabulary List

Word Definition. Definition used in text.
Asylum protection granted by a country to a political refugee from another country. Sweden in offering protection to Danish Jews from Nazi Germany.
Communiqué an official news announcement by a person or a group of people. The official news announcement was the offer for protection to the Danish Jews.
Cynical the attitude that shows distrust or lack of sincerity about a subject. The German communiqué is being insincere about the Jews being “removed from public life”. They were not just “removed” they were arrested.
Deportation the legal act of removing someone from a country. Germany had removed the Jews from Norway last year, and is now trying to remove the Jews from Denmark.
DNB  
Foreign exchange money from other countries. Norwegian Jews paid huge amounts of Norwegian money to escape deportation. These people went to Sweden for protection.
Freemasons a men’s organization, or club, that has certain secret rituals. During this time secret clubs were dangerous to German Nazis. The Nazis believed that anyone who was meeting secretly could be planning to fight or resist them. Germany treated people in these secret clubs very badly and persecuted them like Jews.
Gantlet a line or group of armed people waiting for someone to cross in front so they can assault or shoot at him or her. It refers to the great number of armed German boats that were patrolling the ocean between Denmark and Sweden. These boats made it very dangerous for the Jewish rowboats to sneak past them.
Gestapo the German military police during World War II. The Gestapo were the German military police who arrested the Danish Jews.
Hackneyed unoriginal or overused. The German press would tell the rest of Germany hackneyed, or overused, accusations that Sweden was “infested” with Jews.
German heel German power
Humanitarian describes the act of being compassionate or caring for life of other humans. Sweden was being compassionate to human life by offering the Danish Jews protection from the Nazis.
Integral important or necessary. It was an important step for Germany to arrest Danish Jews. With Danish
Jews arrested, Germany would be closer to controlling all of Denmark.
Intercede to get involved. Sweden intercedes to help the Norwegian Jews enter Sweden during German occupation.
Jewridden a disrespectful way of saying heavily populated with Jewish people. The German press called Sweden “Jewridden” because they were protecting so many Jews. The press used this word “Jewridden”, so that the people of Nazi Germany would not respect Sweden.
Orthodox relating and practicing the conservative of traditional religion or ideology. Most of the arrested Jews were orthodox.
Anglo- Bolshevik Plutocrats wealthy politicians that influence the government very much.
Refugee one who flees their own country in search of protection by another country. Any Jew who could get to Sweden for protection would be a refugee.
Repercussions the effects or results of an event or measure. A Swedish official warns Nazi Germany, that if Germany keeps making Danish arrests Sweden will get involved. If Sweden gets involved and starts fighting against Germany, the repercussion may be a war between Sweden and Germany.
Rosh-ha-Shanah the highly celebrated Jewish New Year. The Nazi police started arresting Danish Jews on their New Year’s Day.
Sabotage destruction of property or limiting use of materials during a war or conflict. The German press says Danish Jews support the destruction, or sabotage, of German property and material. That is one reason the Germans want to arrest the Danish Jews.
Scant skimpy, small, or limited amount. In the article, Sweden had little, or scant, success helping Norwegian Jews last year.
Synagogue a place or building for Jewish worship. The German police had forced their way into synagogues to arrest Jews during worship.
Whipping Boy a scapegoat, or the one who is punished for the actions of others. In the article, the innocent Jews are being punished by the Nazis.
Wilhelmstrasse-
inspired
 

Figure 4 Sample Gloss for a Portion of "Sweden Offers Aid to Denmark’s Jews"

…The Stockholm communiqué on the subject, released this evening, reads: “In the last few days reports have reached Sweden that measures are being prepared against the Jews in Denmark similar to those already applied in Norway and other occupied countries. Acting on the Government instructions the Swedish Minister to Berlin on Oct. 1 pointed out to the German authorities concerned the serious repercussions these measures might provoke in Sweden. At the same time the minister conveyed the offer of the Swedish Government to receive in Sweden all Danish Jews”

German Reply Awaited

Stockholm political circles entertained tonight no illusion that the Swedish remarks in Berlin would receive a favorable answer if they received any at all. Stockholm was reported prepared for the first outburst of abuse of Sweden from the Wilhelmstrasse-inspired German press similar to that leveled against this country following the Swedish protest at the end or August against the sinking with all hands of two Swedish fisherman by German destroyers. The Swedes rather expect soon to read in the German press a repetition of hackneyed accusation of being Jewridden and infested with Anglo-Bolshevik plutocrats and Freemasons. The arrest of Jews in Denmark was heralded by the arrival in that country at the beginning of last week of some 1,800 German police and Gestapo agents. Definite news of the action reached Sweden in the early hours today. It was understood the measures applied were similar to those used against the Norwegian Jews a year ago when during the Rosh-ha-Shanah observation the Gestapo forced its way into homes and synagogues, sending Jews into a concentration camp and subsequent deportation. The timing and procedure was the same with the 7,000 Jews in Denmark, 6,000 of whom are orthodox, including many Germans who had sought asylum in Denmark prior to Nazi occupation.

1) A communiqué is an official announcement. What country issued the communiqué about Denmark’s Jews?

2) In the space below, paraphrase the Stockholm communiqué.

 

 

 

 

3) Yes or No: According to the communiqué, the actions of the Germans against the Jews would cause no problems in Sweden.

4) What do you think it means to “hold no illusions?”

 

5) Do you think that the German press had good or bad things to say about Sweden?

6) How did the Jews of Denmark know that the Germans were planning to arrest them?

 

 

 

7) If the Germans were planning to treat Jews in Denmark the same way that they treated the Jews in Norway, what do you think would have happened to Denmark’s Jews?

Figure 5: Sample Reading Guide for “Sweden Offers Aid to Denmark’s Jews”

Name: __________________________

Directions: This worksheet will help while you read the New York Times article “Sweden Offers Aid to Denmark’s Jews”. Read through these questions before your read the article, then answer the questions while you are reading and when you are finished.

  1. Read the title. In the space below, list three types of aid that you think Sweden could have offered to Denmark’s Jews.
  2. From reading the article, how do you think the Germans responded to the offer of aid made by Sweden to the Danish Jews?
  3. How was the German press expected to respond to Sweden’s offer of aid?
  4. How many German police and Gestapo agents were brought to Denmarl to arrest Jews?
  5. Name six countries that were named or referred to in the article. According to the article, in which of these countries did Germans arrest Jews?
  6. What explanation did German officials give for “rounding up” Danish Jews?
  7. The article mentions “countries under the German heel.” What do you think it means to be under Germany’s “heel”?
  8. Danish Jews were expected to be brought to Poland or German occupied Russia. What happened to Jews that were brought to these places?
  9. To what is the article referring when it mentions the “eliminated number”?
  10. From what you read in the article, how could the actions taken against the Danish Jews be a “largely financial operation”?

Bibliography

Fry, E. (1977). Fry’s readability graph: Clarification, validity, and extension to level 17. Journal of Reading, 21, 242-251.

Fry, E. (1989). Reading formulas—Maligned but valid. Journal of Reading, 33, 292-297.

Marshall, N. (1979). Readability and comprehension. Journal of Reading, 22, 5420544.

Tonjes, M., Wolpow, R., and Zintz, M. (1999). Integrated Content Literacy. Boston: McGraw-Hill College.

Wayne, O., White, S., and Camperell, K. (1980). Text Comprehension Research to Classroom Application. A Progress Report. Theoretical Paper No. 87, Wisconsin Research and Development, Center for Individualized Schooling, University of Wisconsin, Madison. October. p. 119.