Using Oral Reading Fluency tests to analyze Bible translation reading difficulty

UPDATE: I’ve added a sample text to the bottom of this post for those who want to try a test.

On my FutureBible blog I’ve been looking at the subject of reader fluency as a measure of comprehension. This is based two articles published in the journal of the International Reading Association:

  • Fluency: Bridge Between Decoding and Reading Comprehension (citation) (pdf)
  • Oral Reading Fluency Norms: A Valuable Assessment Tool for Reading Teachers (citation) (pdf)

It’s quite difficult to measure how well someone understands a text. You can ask content questions such as, “Where did the people build the tower?” and “How did God stop the people from building the tower?” but these types of questions often focus on details while people, when they are listening to a story like the Tower of Babel, are often focusing on the story and its main point.

According to research done on fluency, there has been shown to be a correlation between how easily a person can read a text and how well they understand it. If that were true, then we might be able to easily test comprehension simply by testing fluency.

I’ve performed one such test and published the preliminary results. My wife and I took turns reading different versions of the Tower of Babel story in Portuguese. Each text was actually a combination of two different translations. What’s interesting about the test is that I actually made more errors reading the supposedly easier text, O Livro. O Livro was meant to be an equivalent of The Living Bible in Portuguese.

There are three possible reasons for this:

  1. O Livro is more difficult.
  2. My text started with O Livro and ended with Almeida so I “warmed up” over time.
  3. I’m unfamiliar with O Livro but have used Almeida for years.

As you can see it’s really tricky to definitively prove your results one way or the other, especially with such a small sample. The focus of my study was on speakers of a second language. But this kind of test would work equally well on testing speakers of a text that is written in their mother tongue. You might consider putting together a test like this and trying it out on your family or friends. You can make note of errors by following along as the person reads and underlining where they falter or make errors. Or if you’re feeling high tech you can record the readings and then study them later.

See my post for details on the study and please let me know if you try this test.

Read my post here: Using Oral Reading Fluency tests to analyze Bible translation reading difficulty

UPDATE: Here’s a text for you to use: ORF test for English Bible versions

Instructions:

  1. Print two copies.
  2. Fold one copy in half.
  3. Ask a volunteer to read either version 1 or version 2.
  4. While they read, underline places where the reader makes a mistake or stutters.

Calculate the number of errors per paragraph.

I’ve tried to pick an unfamiliar text and also split the text evenly. Let me know if I’ve made any mistakes!

Translation ABC is Version, 1 paragraph 1 plus Version 2, paragraph 2

Translation XYZ is Version 2, paragraph 2  and Version 2, paragraph 1

I chose these two translations because they are the most common versions printed in South Africa.

Here are results for the four readers I tested:

Reader ABC errors XYZ errors
B 1 6
A 4 1
H 0 2
E 2 2

Please let us know your results if you try this test on anyone!

4 thoughts on “Using Oral Reading Fluency tests to analyze Bible translation reading difficulty

  1. Michael says:

    Anecdote alert — this proves nothing – Anecdote alert —

    Hmmmm… It might be a chicken and egg type of phenomenon. In my personal experience, I have been using oral reading practice as an aid in becoming more fluent.

    I’ve been studying Biblical Hebrew for almost 6 years. I love it and enjoy it immensely. I can hardly wait to get to the next problem. However, languages aren’t my forte and I’ve always felt that I had to work harder than the rest of my classmates.

    Two years ago I started reading the texts out-loud, even if I could only haltingly understand them. Today, I have a set of Biblical texts that I read out-loud once or twice a day (they are the same ones, over and over again). Hard as it might be to believe, I am of the opinion that I can locate an improved ability to comprehend the Hebrew to the day I began the habit of daily oral practice.

    Anyway, a fascinating subject.

    Blessings,

    Michael

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s