n a flight to the Netherlands, Dr. Hendrik Lenstra, a mathematician, was leafing through an airline magazine when a picture of a lithograph by the Dutch artist M. C. Escher caught his eye.

Titled "Print Gallery," it provides a glimpse through a row of arching windows into an art gallery, where a man is gazing at a picture on the wall. The picture depicts a row of Mediterranean-style buildings with turrets and balconies, fronting a quay on the island of Malta.

As the viewer's eye follows the line of buildings to the right, it begins to bulge outward and twist downward, until it sweeps around to include the art gallery itself. In the center of the dizzying whorl of buildings, ships and sky, is a large, circular patch that Escher left blank. His signature is scrawled across it.

As Dr. Lenstra studied the print he found his attention returning again and again to that central patch, puzzling over the reason Escher had not filled it in. "I wondered whether if you continue the lines inward, if there's a mathematical problem that cannot be solved," he said. "More generally, I also wondered what the structure is behind the picture: how would I, as a mathematician, make a picture like that?"

Most people, having thought this far, might have turned the page, content to leave the puzzle unsolved. But to Dr. Lenstra, a professor at the University of California at Berkeley and the University of Leiden in the Netherlands, solving mathematical puzzles is as natural as breathing. He has been known, when walking to a friend's house, to factor the street address into prime numbers in order to better fix it in his mind.

So Dr. Lenstra continued to mull over the mystery and, within a few days of his arrival, was able to answer the questions he had posed. Then, with students and colleagues in Leiden, he began a two-year side project, resulting in a precise mathematical version of the concept Escher seemed to be intuitively expressing in his picture.

Maurits Cornelis Escher, who died in 1972, had only a high school education in mathematics and little interest in its formalities. Still, he was fascinated by visual mathematical concepts and often featured them in his art.

One well-known print, for instance, shows a line of ants, crawling around a Moebius strip, a mathematical object with only one side. Another shows people marching around a circle of stairs that manage, through a trick of geometry, to always go up. The goal of his art, Escher once wrote in a letter, is not to create something beautiful, but to inspire wonder in his audience.

Seeking insight into Escher's creative process, Dr. Lenstra turned to "The Magic Mirror of M. C. Escher," a book written (under the pen name of Bruno Ernst) by Hans de Rijk, a friend of Escher's, who visited the artist as he created "Print Gallery."

Escher's goal, wrote Mr. de Rijk, was to create a cyclic bulge "having neither beginning nor end." To achieve this, Escher first created the desired distortion with a grid of crisscrossing lines, arranging them so that, moving clockwise around the center, they gradually spread farther apart. But the trick didn't quite work with straight lines, so he curved them.

Then, starting with an undistorted rendition of the quayside scene, he used this curved grid to distort the scene one tiny square at a time.

After examining the grid, Dr. Lenstra realized that carried to its logical extent, the process would have generated an image that continually repeats itself, a picture inside a picture and so on, like a set of nested Russian wooden dolls.

Thus, the logical extension of the undistorted picture Escher started with would have shown a man in an art gallery looking at print on the wall of a quayside scene containing a smaller copy of the art gallery with the man looking at a print on the wall, and so on. The logical extension of "Print Gallery," too, would repeat itself, but in a more complicated way. As the viewer zooms in, the picture bulges outward and twists around onto itself before it repeats.

Once Dr. Lenstra understood this basic structure, the task was clear: If he could find an exact mathematical formula for the repetitive pattern, he would have a recipe for making such a picture with the missing spot filled in.

Measuring with a ruler and protractor, he was able to estimate the bulging and twisting. But to compute the distortion exactly, he resorted to elliptic curves, the hot topic of mathematical research that was behind the proof of Fermat's last theorem.

Dr. Lenstra knew he could apply elliptic curve theory only after reading a crucial sentence in Mr. de Rijk's book. For esthetic reasons, Mr. de Rijk explains, Escher fashioned his grid in such a way that "the original small squares could better retain their square appearance." Otherwise, the distortion of the picture would become too extreme, smearing individual elements like windows and people to the point that they were no longer recognizable.

"At first, I followed many false leads, but that sentence was the key," Dr. Lenstra said. "After I read that, I knew exactly what was happening."

Escher was creating a distortion with a well-known mathematical property: if you look at small regions of the distorted picture, the angles between lines have been preserved. "Conformal maps," as such distortions are known, have been extensively studied by mathematicians.

In practice, they are used in Mercator projection maps, which spread the rounded surface of the earth onto a piece of paper in such a way that although land masses are enlarged near the poles, compass directions are preserved. Conformal principles are also used to map the surface of the human brain with all the folds flattened out.

Knowing that Escher's distortion followed this principle, Dr. Lenstra was able to use elliptic curves to convert his rough approximation of the distortion into an exact mathematical recipe. He then enlisted a Leiden colleague, Bart de Smit, to manage the project and several students to help him.

First, the mathematicians had to unravel Escher's distortion to obtain the picture he started with. A student, Joost Batenburg, wrote a computer program that took Escher's picture and grid as input and reversed Escher's tedious procedure.

Once the distortion was undone, the resulting picture was incomplete. Some of the blank patch in the center of "Print Gallery" translated into a blurred swath spiraling across the top of the picture. So, the researchers hired an artist to fill in the swath with buildings, pavement and water in the spirit of Escher.

Starting with this completed picture, Dr. de Smit and Mr. Batenburg then used their computer program in a different way, to apply Dr. Lenstra's formula for generating the distortion.

Finally, they achieved their goal: a completed, idealized version of Escher's "Print Gallery."

In the center of the mathematician's version, the mysterious blank patch is filled with another, smaller copy of the distorted quayside scene, turned almost upside-down. Within that is a still smaller copy of the scene, and so on, with the remaining infinity of tiny copies disappearing into the center.

Since Escher's distortion was not perfectly conformal, the mathematician's rendition differs slightly from his in other ways as well. Away from the center, for example, the lines of some of the buildings curve the opposite way.

The researchers also used their program to create variations on Escher's idea: one in which the center bulges in the opposite direction, and even an animated version that corkscrews outward as the viewer seemingly falls into the center. After a recent talk Dr. Lenstra gave at Berkeley, the audience remained seated for several minutes, mesmerized by the spiraling scene.

While Dr. Lenstra has solved the mystery of the blank patch and more, one question remains. Did Escher know what belonged in the center and choose not to represent it, or did he leave it blank because he didn't know what to put there?

As a man of science, Dr. Lenstra said he found it impossible to put himself inside Escher's mind. "I find it most useful to identify Escher with nature," he said, "and myself with a physicist that tries to model nature."

Mr. de Rijk, now in his 70's, said he believed Escher knew his picture could continue toward the center, but did not understand precisely what should go there. "He would be astonished to experience that his print was still much more interesting than was his intention," Mr. de Rijk said. He added that while he knew of another effort to fill in Escher's picture, it was not based on an understanding of the mathematics behind it.

"He was always interested when somebody used his prints as a base for further study and applications," Mr. de Rijk said. "When they were too mathematical, he didn't understand them, but he was always proud when mathematicians did something with his work."