AT&T Labs punishes products to find glitches and fixes

Nestled in a leafy office park in North Austin, AT&T is perfecting the present and inventing the future. The AT&T Labs Inc. office, one of the company's six major research centers nationwide, houses engineers and developers testing everything from Apple's new iPad tablet computer to the video quality on AT&T's U-verse television service.

But scores of researchers are also investigating scientific questions in areas such as mathematics, physics and computer software.

"We're one of the few remaining pure research organizations in the United States," Keith Cambron, president and chief executive of AT&T Labs, says at the facility.

A tour of the nine-floor, 150,000-square-foot building is both thrilling and intimidating.

Thrilling because the engineers and developers and industrial designers and psychologists employed here are working on upgrades and fixes to make technology more fun and intuitive.

But the lab is also overwhelming once you get a peek at the complex systems used to test and diagnose that equipment and the genius-level math required to make all the machines and networks run smoothly.

The tour begins downstairs, in the basement.

There, David Deas, executive director of video and IPTV, and his team are refining the U-verse service and set-top boxes running Microsoft software that deliver that service to the home.

Deas demonstrates an automated system that mimics a home with eight U-verse set-top boxes and stress-tests all eight machines to try to make the boxes and the service fail.

AT&T could just "hire a bunch of college students and shove pizza under the door" until they banged on remote controls long enough to find the bugs, Deas says, but automating the process is critical because it allows the company to pinpoint where flaws are.

More important, it allows AT&T to track and repeat the actions that revealed the bug.

"Our vendors don't want to pay attention to us until we can prove without a shadow of a doubt that they did something wrong," Deas says.

While the goal is to catch all bugs while hardware and software are in development, invariably customers run into problems that haven't been seen before.

The traditional response is to diagnose the problem over the phone or send a technician out to investigate.

But it's often impossible for customers to remember what they were doing when they lost a signal or had some other error.

So lab workers are designing a box that can be plugged into a customer's home network to track glitches.

But making sure U-verse works is just part of the battle.

AT&T also wants it to work well. That's where Pierre Costa comes in.

His job is to optimize the video quality of U-verse. And doing that requires cooperation from broadcasters.

Broadcasters send videos to AT&T in one of two ways: satellite or fiber-optic cable. Because fiber-optic cables carry much more data than satellite transmissions, broadcasters can send much higher-quality images over fiber to AT&T.

Although both satellite and fiber signals are dramatically compressed to fit on the limited bandwidth flowing to every U-verse-connected home, the better the source, the better the final product.

Costa shows a side-by-side demo of a baseball game with video compressed from satellite and the same game compressed from fiber.

In the satellite image, spectators are indistinct smears and players' jerseys disintegrate into chunky pixels. With a fiber-sourced video, those details are crisp and clear.

But the best technology in the world is useless unless customers can figure out how to watch it, play with it or even just turn it on.

That responsibility belongs to Jeff Brandt, principal member of technical staff in the human factors division. Brandt and his team of psychologists and industrial engineers analyze how people interact with products and services that AT&T is preparing to sell.

New cellphones, U-verse remote controls and other products all flow through his test rooms.

In one room, a research participant fiddles with a Motorola cellphone while two AT&T employees watch through a one-way mirror. Multiple high-definition cameras are trained on the subject, the phone and his hands as he goes through a checklist of tasks.

Brandt says phrasing the checklist commands in different ways can alter the tester's actions.

For example, by asking the user to "erase" a text message, the user will instinctively seek out a button or menu option labeled "erase." So in testing, the user is given vague directions, such as, "Get rid of that last message," Brandt says.

If, at the end of a test, a user is asked if he read the instruction manual beforehand, he will generally read the instructions on every subsequent test, even if he wouldn't normally do that on his own.

"I've changed his behavior by asking the question," Brandt says, and that essentially invalidates the test, since the goal is to see how the average user will respond to the device after buying it and taking it home.

Brandt employs technology such as retinal trackers that scan precisely where the user is looking during the session.

"I don't have to ask him" if he read the directions, Brandt says. "I can see if he did."

In some cases, the product will be so hard to figure out that Brandt and AT&T will tell the supplier that AT&T cannot sell the product because of the potential high cost of customer service.

Generally, the designers and engineers will look at Brandt's data and realize their original vision needs revision.

Following the advice of human factors workers can save millions of dollars.

Cambron says that when AT&T started offering DSL Internet service several years ago, the company decided to mail modems and installation discs to customers rather than have technicians drive to every house, as each visit cost the company about $300.

But only 15 percent of customers were able to figure out how to install modems on their own. So the human factors team went to work, watching how people opened the boxes, responded to directions and fiddled with cables and software.

The team recommended changes and started mailing out modems again. Presto, the success rate zoomed to 85 percent.

"They saved on the order of $40 million with that project," Cambron says.

But the human factors wizards don't get to play with everything.

Notably, Apple does not let Brandt and his crew test new iPhones before they launch.

"Apple does a pretty good job of that already," Brandt says.

But you get the feeling that Brandt would like a crack at the device.

"There's nothing we've ever brought in that we couldn't say, 'Here's how you can make it better,'" he says.

The last leg of the tour is the network-ready lab, where new wireless devices -- from smart phones to netbooks to water meter modules to even the elusive iPad -- are tested to make sure they play nice with AT&T's networks before they go on sale.

Inside copper-shielded rooms that block out any outside wireless signals, netbooks, digital photo frames, laptops and phones are dissected and connected to sophisticated monitoring equipment.

If you step inside one of the rooms, shut the vaultlike door and check your phone, it loses a connection instantly.

Another room -- the acoustic anechoic chamber -- is coated in pointy foam ridges to completely block echoes. That's where acoustic properties of phones are tested.

In the middle of the room, a rubber human torso stands on a pole, with a metallic frame to hold a phone up next to its ear. The ear contains a sophisticated microphone connected to a computer outside the chamber.

Once a phone is strapped in and the door shut, a tester calls the device with an automated program, and the dummy's ear captures the sound and relays it back for analysis.

While field teams eventually take all new devices out into the world for a final evaluation, the lab is the first proving ground.

"The lab test happens first to make sure there's no showstopper issues," says Huitao Liu, director of the network-ready lab.

The iPad seems to be something of a showstopper whenever it comes up in conversation. Liu says only, "I can't comment about that," when pressed about the highly anticipated machine that went on sale last weekend.

Even a joke about sneaking off to find an iPad is met with a nervous grin and a "no comment."

But while Apple and its shiny gadgets and obsessive secrecy loom large, the whole story of AT&T Labs is much bigger.

Cambron notes, for example, that AT&T runs the network to all of Wal-Mart's cash registers and overall handles about 17 petabytes of Internet traffic every day. One petabyte is 1 million gigabytes.

The company also routes about 7 billion phone calls a day and can run as many as 480 automated tests on any of them to track lost and stolen phones being used to make fraudulent calls.

That system saves AT&T about $100 million a year.

"If somebody commits a fraud, we'll catch it within about an hour," Cambron says.

(c) 2010, The Dallas Morning News.
Distributed by McClatchy-Tribune Information Services.