May, 2004 – From Inspiration to Action at A.G. Edwards

Jerome Nadel: Welcome to HFI's usability broadcast network. I am Jerome Nadel, Executive Managing Director and I have with me today Pat Malecek who's an Associate Vice President User Experience Manager at A.G. Edwards. Pat, welcome.

Pat Malecek: Thanks.

Jerome Nadel: This program complements an excellent white paper authored by Pat that shares perspective on the institutionalization of usability at A.G. Edwards. The paper entitled From Inspiration to Action at A.G. Edwards & Sons Inc. The Prescription and Practice for Institutionalizing Usability in this paper then we will be discussing this is the main theme of today's web cast. Pat candidly shares A.G. Edwards experience from the initial awareness of issues around usability to the creation of a program to putting matrix for results in place and then finally the continued evolution of the program. And we will spend some time discussing that and how although there is infrastructure in place some further activities continue to press. Pat, to set some perspective perhaps you can share some background on A.G. Edwards and why usability is an important business issue to you.

Pat Malecek: Okay, well, A.G. Edwards is a national brokerage firm. And like any large company we have technology products that are used by our clients, our customers, and our employees. We have roughly 16,000 employees and we have 7,000 brokers. We call them financial consultants scattered throughout hundreds of offices in the United States and two European locations. So it's important that that distributed work force have very usable applications, very usable intranet which is one of the main communication vehicles. And it's also very important that our public facing and client facing web services be very usable. Our retail customer is our bread and butter. So it is very important that we cater to their needs and we make sure that their technology experience is as pleasant as possible.

Jerome Nadel: Very good. In your paper you share that from Eric Shaffer's white paper back in 2001 that his call for institutionalizing usability struck a cord. Perhaps you can elaborate on that and share some perspective.

Pat Malecek: Right. Well, I remember reading Eric's paper and I remember being very impressed by how it was very in tune with my experience at A.G. Edwards. And how my team had grown, the things that we'd experienced, the struggles that we faced, and also set the course for the opportunities that lay ahead of us. So that paper it was pretty interesting to see that from a broader perspective that my experiences at A.G. Edwards are similar experiences throughout other industries and other companies in the United States.

Jerome Nadel: With that you also described the term that we use often as the wake up call. And you suggested that the beginning of this shift crossing the chasm if you will was a wake up call.

Pat Malecek: Right.

Jerome Nadel: Why don't you share some thoughts on that as well?

Pat Malecek: Well, we did spend a great deal of time and effort reengineering our website our public facing websites and a critical piece of that website was deemed unusable by a third party review. And after spending so much time and effort to build that and then find out very late in the game that it was considered unusable was as you say a wake up call. And suddenly we were faced with the fact that we need to go back to the drawing board, we need to redo this and oh by the way, we need some usability attention not only to this project but to projects overall. And that was really the genesis of our usability awareness at A.G. Edwards.

Jerome Nadel: So that provocative call to action if you will.

Pat Malecek: Definitely.

Jerome Nadel: With that, why don't we transition to St. Louis to the A.G. Edwards campus where we'll meet Pat there with some of his colleagues as well.

Pat Malecek: Hello! Welcome. I am Pat Malecek. I am the User Experience Manager here at A.G. Edwards. At any time during the course of the presentation today you can use the submit question button to send in a question or comment about anything you hear during the presentation. We will then answer those questions at the end of the broadcast in a live Q & A session. There's also a white paper associated with today's presentation. So you don't have to seriously take notes. A lot of the details about what we discuss today are covered in the white paper.

I want to say a little bit about A.G. Edwards. A.G. Edwards is a brokerage firm, financial services firm. We have roughly 16,000 employees. We have approximately 7,000 financial consultants in 700 branches throughout the United States and two European locations. Those financial consultants use a vast intranet and a collection of custom applications to service their clients. And my team, user experience team, lends it's services to the web based products that those financial consultants use.

Human factors has been important to me and my team as we've come together and grown over the past few years as a source of research and a great source of information. The thing I like about human factors is that they are a research aggregator. There are multiple sources of research being done on usability issues throughout the world. And Human Factors regularly collects that information and sort of digests it for you so it's easier for me to get my research in consumable chunks. So I like the fact that Human Factors does that for me.

Another great thing about Human Factors is their certification program. Myself and my team, who you'll meet later in the broadcast, are all certified usability analysts. And this has been really helpful in instilling confidence in the team, but also in furthering our mission here at the firm. People recognize that we've been certified by an outside entity. They realize that we're operating form research based principles and not just our opinion. So HFI certification has been really helpful.

One of the things that HFI produced that really had an impact on we was a white paper a few years ago called The Institutionalization of Usability by Dr. Eric Schaffer. And this was important to me. It was very interesting to me because that paper mirrored my experiences here at A.G. Edwards in starting and trying to grow a usability effort. So that paper was very important to me and it became a book, a recently published book, by the same title: Institutionalization of Usability. And again some of what you'll be hearing in today's presentation is about how our experiences at A.G. Edwards are similar to those described in Dr. Schaffer's book.

So now our first guest I am going to introduce you to today is Mike Flood. And he is my boss and my executive champion here at A.G. Edwards. I'd like to introduce Mike Flood, Vice President of Intranet Services Department here at A.G. Edwards. Mike's been with the firm for 17 years or so and he has been with us in intranet services for over 3 years. One of the principles that Eric Schaffer describes about how a firm experiences the beginning of a usability program is a wake up call. And that's certainly how we began our usability effort here at A.G. Edwards. And Mike joined us right about the same time of that wake up call and so I'd like to – Mike, do you have any recollections of that event?

Mike Flood: Oh sure. As you remember, Pat, I joined the project a little bit late in the game and we were scrambling to meet some deadlines. And we had brought a firm in to do a review of our website service the last minute. And I think to everyone's surprise and shock they came back with some pretty harsh critiques and recommendations about changes we need to make. And you know I witnessed first hand the disappointment and the shock. It became clear to me at that time that good web design was more than just a technical mastery of the tools. So the first thing that I did was designate Pat as the person responsible for usability. That was his prime job from that point moving forward. We then continued with the same firm that had done our review, got some additional reviews and advice from them. We also brought in a couple of consultants to do class room training not only for the entire department but for Pat's team as well. And then finally we put together a long-term training plan for Pat and his team with the end goal of getting Pat and several people certified as usability analysts. So it was quite an experience.

Pat Malecek: One of the critical parts of success of a usability effort is an executive champion. And Mike has definitely served as an executive champion for us, promoting usability throughout the firm. Do you feel that your counterparts and management are receptive to the notion of usability and user centered principles now?

Mike Flood: Well, they are coming around slowly. I mean it's been a long slow process. I am not going to try to kid anyone. You know there are a lot of skeptics and they don't immediately recognize the productivity benefits that come from a good web design. But as they experience poor web design first hand and we move more and more functions online and we truly do become more paperless, they start to experience the problems. And then they hear from our clients. And they hear from our financial consultants and they hear from other employees. And so slowly one by one I think they have their own wake up call of sorts. I mean even at the point now where some of the executives will start using usability without rolling their eyes. And we now have various areas of the firm coming to us and requesting from us to get reviews of their web sites or their pages. And we have actually had a number of executives participate in usability testing so it's been a long slow process but they are coming around one by one and each day we continue to win over another (inaudible).

Pat Malecek: Definitely. Part of that long range training plan that you mentioned earlier on became the notion of HFI certification. I think you noticed that sort of simultaneously as we did and it was at your urging that we pursued it. Do you feel that the certification of the team helped you promote our services throughout the firm?

Mike Flood: Absolutely, I mean it lends a lot of credibility to what we do. I mean even though people recognize that there are standards and best practices and that we have a style guide, there's also an artistic component to good web design. And consequently we ran into a lot of pride of ownership issues here in the firm particularly with distributed publishing model that we have. And so, you know, as you know you are often confronted with the question hey you know what makes your design better than my design?

Pat Malecek: Right.

Mike Flood: So having that certification makes people immediately recognize that you bring something to the table more than just another opinion.

Pat Malecek: Right.

Mike Flood: You know you have a recognized skill and experience through that certification. And it stops a lot of the arguments and actually lends a lot of credibility to what you do.

Pat Malecek: Definitely. Well, Mike, I really appreciate your (inaudible) time today.

Mike Flood: You are quite welcome.

Pat Malecek: With me now is the team that came together after that wake up call. This is the user experience team for A.G. Edwards including Lisa Holder, Heather (inaudible), and Leslie Howery. All of us are HFI certified usability analysts. And the team shares experience in psychology, marketing research, web and application design, and customer research. Our department has two primary products: our public site A.G. Edwards.com and our intranet called A.G.E net. We work and consult with the site managers and the product managers to build and improve the site. I want to talk first about the intranet redesign that we did. To fuel that effort, we did significant research out in the branches. Leslie and Heather did some primary research with card sorting exercises and prototype testing. Can you tell us a little bit about that?

Leslie Howery: Sure. We started with home office employees just to kind of get a good feel of what we needed to change and how we needed to reorganize the navigation. So we started with card sorting with a level one and level two of navigation and basically just put all the cards out on the table, ask them to group them, and also we gave them a little sticky and asked them to name the groups. So that way we got a better idea of may be how things should be arranged as opposed to have it work currently. And after we did that with home office employees, we thought it would be a good idea to actually go out in our branches where our main audience were our financial consultants. Financial consultants and our FAs were and we did the same exercise at several different branches just to make sure that we weren't thinking from a home office perspective as far as how things were labeled or grouped.

Pat Malecek: Right.

Leslie Howery: And Heather and I just started going out to several St. Louis area branches to do this exercise. And then Heather did most of those visits. So do you want to talk about...

Pat Malecek: These were card when you went to the branches these were more card sorting exercises or were you by this time showing them prototypes of what the navigation would be?

Heather: Well, with this redesign we did several phases and the first phase was card sorting. And this as Leslie explained we took, I think there were actually 124 cards that we sat a participant down with, gave them an hour to look at each card and group them. And then, as Leslie mentioned, after they had them grouped, they actually labeled them.

Pat Malecek: Okay.

Heather: So that was the first phase. And then the second phase, we actually took what we had found in the field and with our home office participants and marked that up in paper form. And then we took that out to several branches.

Pat Malecek: Okay.

Heather: And showed them. We then tweaked the design and then for the third phase we went with html prototypes.

Pat Malecek: Okay. Was the resulting navigation significantly different than what it was in the prior AGE net?

Leslie Howery: Yes.

Pat Malecek: Yeah.

Leslie Howery: Yes.

Heather: Yeah. I think the old AGE net version or the...

Pat Malecek: The prior site.

Heather: The prior site, yeah, I guess I don't know exactly how many but I would say there were 20 to 25 different directories or categories global navigation wise. After the redesign, we had it narrowed down to 9.

Pat Malecek: Okay.

Heather: That

Pat Malecek: And we put the mouse over menus in there as well to kind of help them peak into those directories too, okay.

Heather: Right. With such a – we had what was it – 35,000 pages in AEG net.

Pat Malecek: Pages.

Heather: So it is a pretty extensive site and with only having 9 categories giving the user a glimpse into what was under each of the groupings was key.

Pat Malecek: Was pretty helpful.

Heather: Yes.

Pat Malecek: Now we also do obviously a lot of work on our public and client facing sites. Since the site was re-launched in 2001 it's been an ever evolving thing. Lisa, can you talk a little bit about how we do a lot of informal testing with the employees and what not when we are rolling out a new enhancement or something?

Lisa Holder: Right. Basically we do several forms of in house testing with home office employees either we have html mark ups in place or we have basic functionality working or we will actually have paper prototypes where we would just one of us would act as the computer and another one would act as the facilitator and just have users point to things. And then we would show them the resulting page or the resulting display. Another thing that we do with our home office employees is that we conduct internal focus groups like if have questions about different things and we just have more discussion type exercises where we say have them look at different definitions of terms and say is this a logical term to you does this make sense to you. And then get the basic formulation for the design from those results and then take it out to if we have time and the resources available to us take it out to external client types.

Pat Malecek: Sure.

Lisa Holder: Or out to the FCs to even test further. We also test our client facing sites with our FCs and FAs and branch personnel just to see if it's intuitive to them. Because they are kind of an external audience as well.

Pat Malecek: Sure. Is there, do you see, what are the pros and cons of internal test participants and external test participants? Any of you.

Lisa Holder: Internal participants are easier to obtain part time and are cheaper (laughter).

Pat Malecek: Yeah, definitely.

Lisa Holder: We've used external sources to also gain participants for testing and it's been a pretty seamless process working with the outside marketing firms to recruit those. But internal participants, it's a quicker turn around because you can just pick up the phone or email somebody and ask them if they can participate and get it done in a day or two.

Pat Malecek: Right.

Lisa Holder: Whereas external participants you have to do more arranging and more paper work.

Pat Malecek: Lot of logistical...

Lisa Holder: Logistical work.

Pat Malecek: What are some of the tools that you all use to do mark ups and prototypes?

Heather: I do most of my prototypes in html. So we currently use Dreamweaver and max to do that.

Pat Malecek: Okay, but sometimes we do pretty simple stuff.

Leslie Howery: Like a software (inaudible) draw function in Microsoft Word

Lisa Holder: Or even we just do it on paper in a real quick test.

Pat Malecek: And do you feel that those paper prototype tests are I mean beneficial? It helps to figure things out quickly cheaply.

Lisa Holder: I believe so. I think it gives a good initial impression of what somebody thinks about the design or if they can find something.

Pat Malecek: Sure.

Lisa Holder: They are you know you always need to if you can take it to the next level which is marking it up into html.

Pat Malecek: Right.

Lisa Holder: And actually having them view it on screen because when you look at a paper prototype a lot of times you can see the whole page.

Pat Malecek: Right.

Lisa Holder: But that's not how a viewer would see it.

Pat Malecek: Right. I want to talk also about we recently re tooled the sites again to coincide with the firms' national branding effort. We had a fairly accelerated time frame to get the sites re tooled into the new branding guidelines and the style guidelines. And Lisa and Leslie were actually all three of you had a separate piece of that effort. Could you talk a little bit about working in that accelerated time line? How that helped? How that hindered? And how the results came out.

Lisa Holder: Well, with the accelerated time frame we primarily worked with internal participants because of that. And then post lunch went to the external participants.

Pat Malecek: Right.

Lisa Holder: And that's obviously probably less than ideal. We probably would have liked the time to take it out to our client type people within the St. Louis community prior to the lunch. But because we couldn't, we worked with our parameters of paper prototypes, still images on the computer, focus groups with me, and internal participants and things like that. And it is just also cutting down the amount of reporting, just keeping the reports quick bulleted just so that the developers could get the results the next day.

Pat Malecek: Right.

Lisa Holder: And then make the appropriate enhancements or changes to the design, so.

Pat Malecek: Let's talk about those focus groups for a second. We did an internal focus group or a set of a couple and then we also did some external focus groups. Can you guys talk about what we found out from those?

Leslie Howery: Well the internal focus groups were actually I think they kind of emulated the same thing as the external focus group did.

Pat Malecek: Right.

Leslie Howery: They all kind of had the same general reaction and we kind of did a little bit different things with both groups, internal and external. So it's hard to really gauge how they totally compared. The external focus groups were very helpful.

Pat Malecek: Yeah.

Leslie Howery: There were some issues that the usability team thought might be issues in the design that not everybody was possibly listening to at the moment.

Pat Malecek: Right.

Leslie Howery: So it was nice to have an external focus group really...

Pat Malecek: Validate that.

Leslie Howery: Validate that. And that was an external focus group too that we didn't facilitate.

Pat Malecek: That's right.

Leslie Howery: So it was even nicer to have you know somebody else leading it and people validating what we had been trying to express to the team.

Pat Malecek: Definitely.

Leslie Howery: But it was an eye opening experience too, because there were some things that we thought were going to work well.

Pat Malecek: Right.

Leslie Howery: And when you put them out there in the external land out there and even we had some eye opening experience.

Pat Malecek: Sure. Yeah, we have to be objective recipients about (inaudible)

Leslie Howery: Exactly.

Pat Malecek: We have to respond correctly.

Lisa Holder: And I think that's a good point to make because in our environment I don't know if that's true amongst all usability groups that may be hearing this broadcast, but really we are very heavily involved.

Pat Malecek: Right.

Lisa Holder: In the design process and trying to move away from that and just becoming more of a testing role but still to stay with all the redesign that we've done being very married to our work and trained to keep that objectivity and being...

Pat Malecek: Right.

Lisa Holder: Totally separate you know good recipients of our own you know criticism (laughter)

Pat Malecek: Right.

Lisa Holder: Somewhat challenging at times because there is such a close you know we are doing the testing. We are doing the design work. We are doing architecture. So I mean even though we aren't you know trained to remove ourselves more from design and just focus on the testing or you know even to improve to have you know like when Leslie did some of the designs for the redesign.

Pat Malecek: Right.

Lisa Holder: You know and was really heavily involved in that process and then Heather and I...

Pat Malecek: Right.

Lisa Holder: To conduct the testing. I mean that's even something that we've moved toward.

Pat Malecek: So at the very least we try to detach the person doing the design from the testing effort. Yeah, and that follow-up testing was I thought incredibly beneficial, really eye opening, and quite fun. We had 8 non clients come in to sort of look at the newly branded web sites from the perspective of I am not a client of this firm yet. Will these new websites help me become a client. Talk a little bit about that effort.

Lisa Holder: Well, I'd like to interject something.

Pat Malecek: Yeah.

Lisa Holder: It was not only the websites though. It was our whole branding campaign. Because these people were seeing it all for the first time and I think Leslie did an excellent job of showing them you know our print ads our web pages, our banner ads.

Pat Malecek: Right.

Lisa Holder: All of that. And it's just not the website experience, but it is the whole experience with the new A.G. Edwards brand.

Pat Malecek: The pieces that they are mailed, the pieces that they are emailed, the things that they see in magazines it was all looked at.

Lisa Holder: You know Leslie had some really good insights for our clever communications department to make improvements not only to you know the banner ads but also to the print pieces as well. Because there were you know there were some things some key messages and some key cause to action that these clients that we brought in or these client types that we brought in weren't getting.

Pat Malecek: Right.

Lisa Holder: From what we were putting out there. So...

Leslie Howery: And these were things that we didn't have any idea we were going to happen. We just started trying to test the flow of the user experience and you know and we'd hadn't seen some of these materials before and we hadn't...

Pat Malecek: Didn't bother us, right.

Leslie Howery: (Inaudible) some of these things too. So it was very fruitful to have somebody actually see how they actually would use all of that and not just the website but what leads them to the website. Were we you know losing people before we even get them.

Pat Malecek: Even gotten them, sure. Well that testing took place in one of our new test rooms. And that's been there's a couple of recent developments I think that are really going to help us get some traction and build our activity here. And one of them is the development of those testing rooms. And the other is the development of our relationship with training to get this steady flow of our participants and branches. I think Heather was really took the lead on both those efforts, developing and relationship with training and also getting our labs together. We call them labs but they are really just they are sort of informal observation rooms. But can you talk a little bit about the struggles that we had in getting those rooms together, but also the successes that we had since then.

Heather: Right. We had previously had just used some conference rooms on a floor that was available at the time. And we had several times where there were difficulties trying to get the equipment set up or get to the right test environment that we needed. And we actually did testing before market hours and after close which made it even more of an obstacle to get things set up and done correctly the first time. Lot of times we didn't have the correct people that we needed because it was after hours.

Pat Malecek: Sure.

Heather: So these test rooms allowed us to have all of our equipments set up in one place at all times and helped up to the environment that we used as often. So that was a good...

Leslie Howery: And also now we have access to the workstations that our financial consultants actually use back at their office.

Lisa Holder: Right, and the conference rooms that we used before, most often we would have to bring a laptop or have one of our PCs moved down to that conference room. In our new test labs, we actually have a replica of what the workstation is out at the branches.

Pat Malecek: At the branches, great.

Lisa Holder: Which is very different from what we have here at the home office.

Pat Malecek: Right. So do you think we are done with setting up the rooms or are there more things you'd like to see? More technology, more hardware, is there anything else that you'd like to see?

Leslie Howery: There's always a wish list (laughter).

Pat Malecek: Yeah.

Heather: I think we are getting closer.

Pat Malecek: Yeah.

Heather: I think we finally have them set up correctly for the right environment.

Pat Malecek: Sure.

Heather: For what we need to test. I think certainly down the line there may be additional equipment or software that might be nice over there.

Pat Malecek: Okay.

Heather: But right now it's certainly adequate.

Lisa Holder: Yeah, it's really adequate. It's really been a great stride for this team to have that. Before we had it you know we would come in very early in the morning and panic quite often because people hadn't moved the machine or they'd taken the laptop and we had it reserved and oh my God! (inaudible).

Heather: Our projector was missing.

Pat Malecek: Right.

Lisa Holder: You know even like we just did some testing earlier this week and it was nice that everything was there. But when we actually had a lot of people where they want to come as observers and it was a little more difficult to get them all around. You know it's a big screen a big computer screen, monitor that they're looking at. And when you have 5 or 6 people who want to be an observer, it was kind of difficult to get everybody crowded around that so that they could actually see what was going on.

Heather: I guess we should have mentioned in our test lab right now we actually have two monitors, one monitor that the user and the facilitator looks at and then the other monitor is strictly for the observers.

Pat Malecek: Right.

Heather: So it's a 21 inch monitor.

Pat Malecek: But still.

Heather: But still if you have 4 or 5 observers in there, it's difficult for the five of them to squeeze in and look at it.

Pat Malecek: Hopefully down the line some sort of remote viewing opportunity or may be even the one way glass thing still has some benefit.

Lisa Holder: Yeah, I think it does have some benefit but I know we are downplaying that we are saying that it was just a conference room. But I think in a way it is a little less intimidating for participants to come in to that. That if they come in and see you know that one way glass and wonder (inaudible)

Pat Malecek: Our former lab.

Lisa Holder: Who's on the other side of that.

Pat Malecek: Right.

Lisa Holder: So I think it is, even though it is a conference room I think it is a little less intimidating when they walk into the environment and they can see us in the room and they can introduce the people who are observing and things like that. So I think there is still a little bit...

Leslie Howery: More friendly.

Lisa Holder: Trouble there.

Pat Malecek: Right. Oh that's an excellent point because we bring in a lot of our brokers, a lot of our financial consultants to do testing on the applications and websites that they use day in and day out and I think they find it encouraging to see a number of people from the home office there and ready to hear their input and clearly expressing their interest in what the site has to say. I think that's an interesting side benefit to having your observers in the room. Like I said, we set up this process to get the steady stream of financial consultants in. How often do we test? Are we doing about once or twice a month based on that or...

Heather: It's like 2 to 3 times a month.

Pat Malecek: Okay. And is that recruiting effort fairly easy now that we have this system set up?

Heather: Well, it's definitely easier than it was.

Pat Malecek: Yeah.

Heather: (Laughter) What we have set up now is an arrangement with the training department where they have access to all of the visiting FCs, FAs, branch managers, all the users of our applications or...

Pat Malecek: Right.

Heather: Our websites and they send out a brief class questionnaire that asks them about the logistical information about their hotel and travel arrangements. And on this questionnaire is a little information about usability testing and asking them if they want to participate. And for the ones that select yes, then we receive all of the information and we contact them. So they've already made the first step.

Pat Malecek: They've indicated something to us.

Heather: They've indicated that they are interested which makes our job much easier.

Pat Malecek: A lot easier.

Heather: So when we call them on the phone we can you know go directly into what we are (inaudible) looking at.

Pat Malecek: There has already been some information shared with us.

Heather: Right, exactly.

Pat Malecek: Okay, and we get what's our success rate there about 50% of the folks or more that indicate some interest. Do we get them actually in the room testing?

Heather: Yeah, I think it varies based upon the conference. We certainly have more luck getting branch managers.

Pat Malecek: Right.

Heather: Or very opinionated individuals.

Pat Malecek: Yeah.

Heather: There's been a few times where there's been a few conferences that we haven't gotten as many. But I would say about 50% of the time we get about half of them.

Pat Malecek: So with each test event we have 5 to 8 or so ..

Lisa Holder: And we get people who are interested too that they just don't have time in their schedule. Their flight's leaving early or you know there's been more people who want to do it who just couldn't because of their schedule as well.

Leslie Howery: And that's why ideally I think we would like to of course have some sort of remote testing software. Because we have kind of the wish list is to have that remote testing software so that we could you know we do get the people that we can visit once here at the home office when they are here for training. But there's all that you know vast you know our 7,000 financial consultants across the country where some of those pockets we could have looked I think it would be greatly beneficial for us to tap into because financial consultants may do business different regionally and you know that sort of thing. That sort of areas where there's may be different areas of interest for certain products that we have so they might use one application more so over others.

Pat Malecek: Right.

Leslie Howery: And I think there's a whole large vast audience that I think we could have our wish list that perhaps that that would be something that we would wish for so that they on their own time table on their own time schedule time frame and such could sit down and complete you know a short 30 to 40 minute test.

Pat Malecek: Right.

Leslie Howery: Give us some of that really great feedback. We do try to collect feedback you know through the internet through the existing surveying tools that we already have we do get some of that feedback and get some of that specific tasks and having them run through specific things that would be nice to have that remote...

Pat Malecek: Yeah (inaudible).

Leslie Howery: They are actually using our own system too. I think sometimes when we are sitting in a room and they know they have this task, they may focus more on something than if they were sitting in their actual office.

Pat Malecek: At their desk, sure.

Leslie Howery: Sitting there, doing it. They might, we might be able to see for instance what applications would they jump to for that.

Pat Malecek: Right.

Leslie Howery: You know, do they stick in one application or do they jump around in different applications.

Pat Malecek: Right. I think another nice benefit of having the FCs come through and having the test rooms is now we have this whole new population of observers who have come down and experienced for the first time they have that ah ha moment where they watch a broker struggle with the thing that they are responsible for. I think that's really fostering a lot of interest in what we do.

Heather: Right. Because before we had to go out and visit the St. Louis area branches and while that was certainly acceptable, it was sometimes difficult to get some of the upper level management and executives to actually take time out of their day to drive down to the branch and spend you know 3 or 4 hours there.

Pat Malecek: Right.

Heather: And so it's been definitely beneficial in that way.

Lisa Holder: And that's one of the reasons why my room was so crowded in our testing earlier this week.

Pat Malecek: Because of a lot of visitors.

Lisa Holder: A lot of visitors had the ah ha moment in another round of testing and really wanted to be there for this as well.

Pat Malecek: Right. And we're spreading that information we solicit the comments from those observers and we use those to sort of build business. We say oh well here's our other folks who have experienced our services you might benefit as well. With the labs in place and with the stream of participants in place, I am confident that we can really branch out and start hitting a lot of stuff, but sort of outside our normal domain. Would you guys agree? What would you like to see? What do you want to do?

Heather: Well, right now we are focused mainly on web based sites.

Pat Malecek: Right.

Heather: But we have a whole workstation which we call client one and our brokers use that has gosh I don't know, 30 applications on it. And I think those could use quite a bit of testing.

Pat Malecek: Right.

Heather: And redesigning.

Lisa Holder: Especially now since we are in the process of a refreshing those workstations and getting new workstations on and getting you know different and new applications out to the you know to across the firm in November.

Pat Malecek: It's a great opportunity.

Lisa Holder: That this is the time now you know over the next few months that we could be potentially very busy in there with hands on doing lots and lots of testing before they have this major system overhaul.

Pat Malecek: Right. Well, I think the time is now and we're here for some growth I think. I thank you guys for coming today. And thanks.

Heather: Thank you.

Pat Malecek: As you can see, A.G. Edwards has a pretty well established usability program going on. It's pretty well localized to the intranet services department right now, but we're seeing some signs of growth. One of the important things that we do in intranet services is that we ensure that we can measure the effects of what we are doing. A gauge of success is very important to know that the investment the firm is making in usability practices and in our web efforts in general is paying off. So for over 2 years or so we've had surveying schools in place. We systematically survey our brokers and our clients about their web usage satisfaction. For our client facing websites we send out a survey monthly and we get that data back and it measures their satisfaction across dozens of matrix. The same sort of thing is done on AGE net. We survey our users out in the branches about their satisfaction with dozens of matrix across for AGE net. What I do, and I would recommend this to anyone who is in my position is, since there are so many things you can measure, pick just a few that pertain to your team. You don't want to have the situation where there is so much to look at that you can't really make sense of it. So I look at about 6 or 8 key matrix across these two sites. And what we do is we ask the survey respondent to ascribe their importance for a particular aspect like navigation and then also to ascribe their satisfaction to it. And then we look at the gap between that importance and satisfaction and that's what I look at for a snapshot of how well we're doing. And the survey tool is built such that we keep our goal is to keep that gap below 1. So that's what I look at across a number of matrix. And then I also have a sort of a snap shot view in which I can see all of my matrix across both websites and I can see how they are moving in relation to each other and what that helps me see is you can trends of where we are doing particularly well and where we might need to do further work. So having some tools in place to measure success is very important for the success of our usability effort and also for the firm to know that they are making a wise investment.

I'd like to thank you all for watching today. I also want to thank Mike Flood, Lisa Holder, Heather (inaudible), and Leslie Howery or joining us. And now we are going to move to the question and answer part of the broadcast.

Jerome Nadel: Welcome back. Well, apparently everybody has found that the submit question button as we have several questions. We'll try to get through each of them as systematically and completely as we can. Let me begin first if I may. So clearly we've seen much progress in the organization from building a program into leveraging the program that you built. What would be the most significant achievement that you made today?

Pat Malecek: I think the thing that we are most pleased about is the inclusion in Gomez's benchmark score cards for our full service brokerage site. Gomez was the company that early on said hey this piece of your website is unusable. And so here we are a few years later and we find ourselves on their list for top 10 full service brokerage websites with special distinction for ease of use. I think that was twice being recognized like tat was really a nice feather in our cap.

Jerome Nadel: Excellent. Well, I guess the counter side of that of this follow up would be what would be one area that you'd like to see some improvement or continued improvement?

Pat Malecek: Oh I think we are making the in roads in to the other areas of development at A.G. Edwards that is not within the internet services domain, but I think that that's a set of work that could be greatly expanded. So not as fairly downside but a side that needs more growth.

Jerome Nadel: Continued effort.

Pat Malecek: Sure.

Jerome Nadel: Superb. We will begin taking some questions from the audience. I begin with this first one. Mike Flood seems to be a great executive champion for you. Are you planning to develop more champions in different departments to further your cause or is one enough?

Pat Malecek: (Laughter) well, Mike is definitely enough, but I would love to have as many champions as I could throughout the firm. Mike does a great job of promoting usability concepts throughout the (inaudible) of business management. I do it on my level of peers at the firm and Mike's doing in his level. Obviously if we can get more folks throughout the firm promoting usability practices, the more the merrier.

Jerome Nadel: Excellent. We had some conversations about this next question. Do you typically establish matrix or measurements of success for each project? How do you measure your impact?

Pat Malecek: We do and we don't. It's a struggle for us. We have tools in place that keep a measure on the things that we've built. We measure user satisfaction across a number of matrix, navigation, content, design, things of that nature. We do not necessarily set forth a pre stated matrix when we are building a project or starting a project. It's actually something that we are working hard right now to initiate. In other words, every time you start the development, start out by saying we want to hit this metric when we're done or we want to see this increase or this decrease when we're done. So if we can get that instilled in our work, I think we'll be able to really have a much more focused measure of our success.

Jerome Nadel: We find that to be common place effect as we kick off a design project we are encouraging our client at a strategic business level to look at success criteria that enable us to accurately measure improvement. And thinking in that way associated with usability is not always so straightforward. Let's move on. What types of projects do you get involved with as usability specialists and then carrying on? Do you build any web applications? So content versus application.

Pat Malecek: Right. Well, we're part of the internet services department. So we are officially tasked to anything that the internet services department owns or manages. So majority of that is web based projects and the majority of our web based projects are the delivery of content via our public web site and out intranet. Now there are also interactive portions of our public facing website and also some interactivity on our intranet. We're involved in all of that, in some things more than others. But we touch all of that. The other question do we build web applications? My group does not. We don't build anything. We perform usability testing. We give usability reviews. We help develop prototypes and mark ups and that sort of things to get the ball rolling. But we don't per say build anything.

Jerome Nadel: Excellent. Next question. What is, clearly on that I can answer here, what is the CUA and how many do you have at A.G. Edwards? How many do you want to have and then finally, do you plan to spread them throughout your organization?

Pat Malecek: Well, a CUA as you all know, is a certified usability analyst. How many do we have? We have 4, myself and my three team mates at A.G. Edwards are all certified. And I think it is interesting to note that I believe that we were among the very first exam class for CUA some time ago.

Jerome Nadel: A legacy.

Pat Malecek: Yeah, right. How many do we want to have. I'd love to have a dozen. I think again the more that we can spread usability practices and usability awareness throughout the firm, the better. And if all those folks are certified that's a nice way to sort of guarantee their work. Do I plan to spread them throughout the organization? Well, again I am in intranet services. There's a lot of work being done elsewhere in the firm that I don't necessarily touch. I would very much like to see usability awareness elsewhere in the firm. I'd also like to see the I'd like my team to be recognized as the focal point for that information. So that's actually discussed by Eric in his material a great deal about having a central force and then also distributing the force throughout the firm. And I think that's something you just have to approach sort of organically as it grows.

Jerome Nadel: But the notion of a matrix organization to have a consolidated expertise but kind of delegate it out by project or function. Very good. Last question of this batch. Do you recommend building multiple versions of the website for UI comparative testing?

Pat Malecek: Well, sure. That can be expensive and time consuming. I would definitely recommend it like if the paper phase and we do that. We'll build a couple of different markups a couple of different interfaces for something and see which one tests better. I think it would be remarkable if you could do it in live sites or in electronic capacity. But I think that would be a tall order.

Jerome Nadel: Often measuring once you're post the differences between old and new and doing some split surveys.

Pat Malecek: Oh yeah. But (inaudible) that would be as well doing it (inaudible) beginning as you develop a life cycle.

Jerome Nadel: Very good, continuing on. How can one convince managers that the quality assurance phase if a project is too late for usability testing. I'm sure many of our viewing audience can relate to that. Exactly what phase of a project is ideal for such testing? Obviously some development has to be done in order to have a product to test. So that seems to be somewhat of a statement. To rephrase, how do you convince managers of quality assurance that it's too late at the end of the development life cycle and then where within this life cycle would you advocate testing to rectify issues?

Pat Malecek: Let me speak to the statement first. Obviously some development has to be done in order to have a product to test, not necessarily true. You can fare it out a lot of really worthwhile information just with paper screens. I mean even as simple as pencil and paper figuring out just basic controls and basic navigation.

How do you convince managers that they QA process is too late? I think it's a fairly common understanding that the later you get in development the more expensive and difficult it becomes to correct things. I think if you can promote that concept, you could get some buy in. And what phase of a project is ideal for such testing? Well, the earlier the better. I think there is a lot of development that many usability teams you know throughout the world don't get to see everything. So I think any testing at any time is a great idea. The earlier the better because that will give the entire team time to react to whatever you find.

Jerome Nadel: Clearly we would agree on that. Moving on. For your intranet redesign, did you hold focus groups with employees or did you do more one on one testing and / or both?

Pat Malecek: Right. If memory serves, it was a great deal of one on one and not so much focus group. We had probably some anecdotal evidence from the field just phone calls, emails etc. from our FCs or financial consultants around the country saying I like this, I don't like this, I'm struggling with this. But there was significant testing in one on one sessions for card sorting, for the navigation to make it more intuitive for the folks in the field. And also just the general presentation of the home page and the secondary pages as well. Significant usability testing and not so much focus groups.

Jerome Nadel: And you mentioned in the major part of the program the notion of card sorting and looking at information architecture, labeling, categorization. Clearly these are done early without product development.

Pat Malecek: Right.

Jerome Nadel: Moving on. Have you initiated any style guide web standards project within your organization and then continuing? If you do, then can you throw some more light on how you are keeping such efforts going?

Pat Malecek: Okay.

Jerome Nadel: May be I can add to that. Separate the methodological component of standards from the detailed design aspects of it.

Pat Malecek: Standards or style guides just in my perspective is often kind of a chicken or the egg kind of thing. It's been my experience that often you build something and then you document what you've built as opposed to let's document something and then build everything per that document. For our public facing websites we have a very rigorous style guide in place. The folks that are contributing to it all know the fonts, the colors, the style guidelines, the presentation standards that we use. And that is basically controlled through the few folks that are sort of in the final inch of creating a public site page.

On our intranet site, the question asked have we initiated any style guide projects. Yes, we have. On our intranet site we have many publishers throughout the firm who create pages for our intranet and we have created a style guide to help them know what works, what doesn't work in the web space. How do we keep the efforts going? On both of those on the public facing site and on the intranet we have a small group of people put together who regularly review the style guide, edit it, change it, communicate those changes out to the field, and we get buy in from the publishing community, from the technical community etc. So it takes buy in and communication of those style guides.

Jerome Nadel: And you reinforce within the major part of the program and in your white paper the idea the advertising, the impact of your group and collecting people to the style guide, it is available online.

Pat Malecek: Absolutely, yeah. If you are going to create the document you have to make it easy to find and easy to use itself.

Jerome Nadel: And promote it adequately as well.

Pat Malecek: Definitely. Very good. The other thing you mentioned is that much of what you are focusing on has more access to and display of content, but our experience is as an organization is embarking on more applications development it is equally important to look at templates for design that a content management system might not support. So templates for navigation, templates for presentation, templates for interaction. And all of the detailed design rules are rolled up and represented in these core templates.

Jerome Nadel: Very good. Is upper management satisfied with the ROI of your usability efforts?

Pat Malecek: I would say in general yes. Because we are still there and it's a growing effort. I think the general consciousness of usability concepts is growing and therefore I would say management is buying in. We aren't asked for statements of ROI, but we definitely demonstrate through user satisfaction how our work is important. And you have to remember that a good portion of our users a very important portion of our users is our brokers, is our financial consultants and if we keep them happy we definitely keep management happy.

Jerome Nadel: And you seem to have a good level of contact and communication with them. You spoke about the training program and soliciting people's involvement as they come in here in the campus.

Pat Malecek: Right.

Jerome Nadel: Very good. This is a lengthy question over here. Do you feel that the user experience function belongs within the e-business intranet department, market research, project management, business systems analysis, or in the business operational areas?

Pat Malecek: Yes. (Laughter) again, if you don't have a user experience or usability group yet, establish one and worry about where it is housed later. Most of what I read and what I've learnt through contact with my peers is that you try to keep the usability group separate from IT so that you are not so stitched into the development process or like you are so roped in by schedules. I have seen it located in marketing departments. I've seen it close to the catalog departments. I've never heard of the project management or business systems analysis area. Again I think if you can get one established anywhere, great. I think you need to basically, most important thing you can do is communicate its importance to the business side. So don't worry so much about where you are organizationally. Worry that you're getting buying from the folks that are making decisions in your firm.

Jerome Nadel: Which comes back to the notion of executive champion and being able to transcend these boundaries of a given group. So you've shared that the challenges the growing pains you have is how to get past ISD.

Pat Malecek: Right.

Jerome Nadel: So to the extent that the group is positioned at a high enough and transversal enough level to support varying activities associated with the development that would be good. How do you address "pride of ownership" issues with designers and publishers?

Pat Malecek: I think the best way to handle that is to have that individual or those individuals witness a usability test. I've certainly designed pages that fell flat on their face and does that irk me? Sure. Because I should be able to do this pretty well, but if I get a room of users together and they are not succeeding with it, that to me is the most objective way to handle those issues.

Jerome Nadel: So objectivity and being empirical.

Pat Malecek: Definitely, yeah. You don't you want to make it a question of what's right and not who's right.

Jerome Nadel: Which also deals with some issues of compliance with your standards and style guides as well.

Pat Malecek: Definitely.

Jerome Nadel: Well done. How can one convince managers that the quality assurance phase of a project – we're back to the beginning – let's continue on as we cycle through? How do you integrate user centered design and usability into your project management methodology for IT development and then more specifically from an ISO perspective have you adopted standards such as 13407 user centered design, ISO 9241 high level usability, and then issues of accessibility.

Pat Malecek: Right. Okay, well, I'll answer the second one first. We have not really put any attention into the ISO issues. We address accessibility by striving to comply with the priority one ADA standards. We're not a government entity. So basically we're trying to be a good corporate citizen and accommodate all of our users as best we can. How we integrated user centered principles into project management, we have attempted to insert ourselves into the methodology that IT ascribes to. I personally feel that you want to try to not over complicate it and basically just make your services very publicly available. Right now there's a lot of projects that don't get much usability attention at all. So my first strike is just to get some usability attention at some place. I wouldn't try to over sell it right now by saying you have to contact us at 0.379 and 11, rather just give us a ring and we will help you.

Jerome Nadel: Sure. More gag if you will.

Pat Malecek: Definitely.

Jerome Nadel: And you shared and we speak to the notion of a showcase project that people will follow with successful end results.

Pat Malecek: Sure.

Jerome Nadel: And ask how did you get to that point. And if that methodology drove you to that point others will likely follow and jump on board.

Pat Malecek: The most critical thing is the earlier the better. Do it before you've built anything. Do it before you've spent a lot of money and you'll be happy.

Jerome Nadel: How did you secure funding for a program? Do you have UI designers in house?

Pat Malecek: Securing funding for the program is frankly is just part of our departmental operating costs. There are four of us in the department and we just we don't have a budget per say. We're just salaried employees and that's how we work. Do we have UI designers in house? In the intranet services department we have web publishers and web designers. We assist in that effort. Over on the IT side I think there's plenty of folks who do UI design day in and day out. I don't know if that's their official job title, but there's definitely folks with those skills in house.

Jerome Nadel: So your group is a cost center not a profit center.

Pat Malecek: Yes.

Jerome Nadel: Do you feel that you would have more influence or impact if you charge for your services?

Pat Malecek: I have peers in St. Louis who charge for their services and I am aware of others who charge for their services. I think there are pros and cons on both sides. I think if you are charging for your services, you are probably very often spending a lot of time drumming up business. And should that be where you are spending your time, I don't know.

Jerome Nadel: Okay. What suggestions do you have for introducing usability testing to a graphics design firm that knows nothing about usability concepts? You can have the hay day with (inaudible).

Pat Malecek: Well, I'll answer from the perspective of how we did it at our firm. We had the wake up call that we described earlier. A pretty, a fairly critical issue that needed addressing. So how did we address that? We brought in knowledgeable consultants to help us out. So if you are a firm that's just starting out, bring in a consultant that can bring you the skills, the know how, the research materials that will get you started. It can be a fairly low impact kind of thing. I would say send some people from the graphic design firm to a usability test. And let them witness that ah ha moment of when a user does or doesn't succeed with what they've built. I think that would create buying and get that process started.

Jerome Nadel: I would only supplement that with the notion of training. Make them aware of the issues and aware that there is a structured approach, a methodology that allows for a formed validated design. And that's really the beginning of the model. And secondly, don't rely only on external consultants.

Pat Malecek: Sure.

Jerome Nadel: Build that expertise in house and use that extra level of expertise as needed.

Do you have any suggestions on how to get an executive champion or buy in from upper management without having the pain of a wake up call?

Pat Malecek: Very good question. Sure. Have one or more of the individuals in the appropriate position there in that managerial position that you are targeting have them come to a usability test. There is nothing more eye opening than watching someone that is your target customer, your target client, or your employee user struggling with the thing that the firm has just spent money on or is about to spend money on. I can't over state it. That it's that ah ha moment of everything that you think is totally intuitive, you watch your standard users struggle with it. That's going to set to create buying for those services. They'll realize the value of usability practices.

Jerome Nadel: And even as well perhaps include senior management in the testing.

Pat Malecek: Absolutely.

Jerome Nadel: Because often I think that the perspective is if I say I like it, it goes. But if they understand that it's about things like protocol simulation and you are going to complete a set of tasks and you are successful or not it's not about subjective likes or dislikes. Then they start feeling more comfortable with the structure of the approach.

Pat Malecek: Right. I would also say that there is far greater value in having that individual witness a test than there is in them reading a report.

Jerome Nadel: Clearly.

Pat Malecek: There is a huge benefit to having them watch the thing happen.

Jerome Nadel: Or at least kind of except (inaudible)

Pat Malecek: Sure.

Jerome Nadel: Last question of this bundle. What kind of survey management tools do you use?

Pat Malecek: The survey management tools that are mentioned in the white paper is a company called Set Matrix and I believe we have over 2 years of data with them. And it was very handy for me. We are currently actually pursuing or seeking a new survey vendor. We just have different needs that have changed over time and it's time for us to pursue something.

Jerome Nadel: This gets back to bit of matrix from a cost savings perspective. The white paper mentions your intranet site summary of cost savings and ROI for usability. Could you describe how you are gathering that data? And also do you own most of the data or do you have to work with other project managers to get that data?

Pat Malecek: Well, the intranet, the items that I have posted to the intranet are publicly available articles and research papers about the ROI of usability. What I collect from personal experience is the testimonials of folks who have utilized our services and testimony of the folks who have been our participants to say this was a very worthwhile experience. We've already made a lot of changes and we're seeing great benefit from that. So do we own that data? I wouldn't call it data, again it's sort of testimonials and anecdotal evidence and the other stuff was publicly available research.

Jerome Nadel: Okay. What methods would you use to move your usability to move usability outside your current intranet services department something that we discussed at length and we address in the white paper as well. Do you need an awareness campaign to grow or use your executive champion or other methods?

Pat Malecek: Okay. Basically right now we have had over the past year or so a trickle of interest for projects outside of intranet services. So other IT projects have come to us and said we are aware of what you do and we'd like to engage your services. From that we are developing basically a word of mouth type reputation and also use our intranet pages to promote our services a great deal. I have thought about an awareness campaign. We have a marketing group that could definitely help me with that. And my executive champion is also doing that for me in his conversations with the management groups that are outside of our normal reach. So again it's a fairly organic thing.

Jerome Nadel: Good happening. We will see some other questions related to that momentarily. Here we have a solo practitioner. I am the only usability person in my company. Where do you recommend I focus my time? On the front end of the project (observing current user experience, developing personas, assisting in the design process etc.) or at the back end conducting usability tests? And then there's a thanks with an exclamation point. Let's do a good job on this one.

Pat Malecek: Well, I would sort of turn your question outside and say spend a great deal of time on usability testing on the front end of the project. The book that Eric Schaffer has written and I believe I get to it in my paper is that the biggest bang for you buck is in usability testing. That's when you really get to not only see your work being done. But you also get to convince other people of its value. The other notions of the front end of the project, the current user experience developing personas etc. definitely assist the design process by communicating best practices that you can gleam from any publicly available research. Developing personas can be a pretty time intensive activity and that's really down to a company by company basis if you need to do that or not. But I would say conducting usability testing and doing it early in the project.

Jerome Nadel: To add to that, I would suggest that it's an issue of problem identification versus problem solution. So to the extent that you need that wake up call, perhaps engaging in usability test towards the end can demonstrate the pain of not being more proactive. But then it's important to quickly turn it on its head as you suggest and go back to a process that in a preventative health way as you suggest, and we speak in that language as well doesn't let the problems occur.

Pat Malecek: Right.

Jerome Nadel: Because the problems are mitigated before they in fact manifest.

Pat Malecek: I agree.

Jerome Nadel: We come, okay one more question and then we have our last question. This is very pointing to you. Can usability survive in your organization in the long term?

Pat Malecek: Oh I think so. I think for the sole reason that there is a lot of technology development that's not receiving usability attention right now I think it can definitely survive in the long term and only grow.

Jerome Nadel: Okay. This brings us to the last question and somehow I have the feeling I know the author of this question. It says where did you find such a well qualified team. Everybody looks very professional, knowledgeable, and friendly. Don't we Pat? Do you have any idea who might have written this question?

Pat Malecek: Well, Heather. Where did I find such a qualified team? Basically we found folks that had a lot of great skill with regards to application development, website building, online banking, lot of customer research, and online marketing. And those three very qualified professional individuals plus myself, we spend everyday doing usability testing, usability reviews, and communicating best practices, and helping develop standards. So we're in it everyday. So we have those core skills that I described just a second ago. And it's been refined and improved over time to get where we are today.

Jerome Nadel: Superb. Well, Pat, thank you very much for being with us on the program today.

Pat Malecek: Thank you. Thanks a lot.

Jerome Nadel: Thank you all.

Top