Monday, March 9, 2015

Evidence-Based Management



It is a scientific fact that decision-making requires reasoning using research, facts and details, combined with emotion or gut feel. For example, Antonio and Hanna Damasio’s work demonstrated that when a person only has the capability of reasoning, it diminishes his/her aptitude for planning and deciding on actions.1 In translation, emotion integrated with reason is an essential building block to decision-making – removing emotion reduces the capacity to evaluate effects, which renders decision-making inert.

There are two perspectives about decision-making that are important from a leadership perspective. First, only rational reasoning is necessary for decisions, a fact rebuffed by the research described above. This bias may be been exacerbated by the logically thinking Spock character in Star Trek. Alternatively is the belief that emotions are dominant in decision-making and that, as long as a decision has intuitive appeal, confirming evidence is not required.

In this publication, the focus will be on the latter problem. When gut feel becomes sole support for a decision, it usually happens at the higher ranks of the organization. Unfortunately, unchecked power can lead to flawed decisions with disastrous consequences. For example, Ron Johnson’s decision, as CEO of JC Penney, to discontinue promotional sales and discount coupons turned out to be very costly. His decision relied on a strategy from his former employer, Apple (“If it worked at Apple, it will work at JCP.”). The strength of his conviction led him to implement without a market test. Sales plummeted from $17 billion when he arrived in late 2011 to under $12 billion when he left in 2013. JCP has been teetering on the brink of bankruptcy ever since.

There is research supporting the importance of experience and intuition, but few recommend it as a stand-alone decision mechanism. Evidence-based decisions are essential in business today, and confirmation of a decision’s veracity should be discovered in advance to support what one emotionally or intuitively believes might be a good decision. Leaders can energize this issue in organizations by making sure decision-making bodies always appoint someone to play devil’s advocate or by asking someone to deliberately present opposing views using evidence.5 Without a process like this, humans tend to filter information that is contrary to ingrained beliefs. The Groupthink literature (see Irving Janus, 1971)6 provides helpful suggestions to prevent bias from entering the decision-making process.

What is evidence-based management? “Evidence-based management  (EBMgt) involves thoughtfully and explicitly gathering, evaluating, and integrating the best available scholarly research evidence, local evidence (i.e., facts of the situation, organizational characteristics), viewpoints of affected parties, ethical considerations, and the practitioner’s knowledge and judgment in the process of managerial decision-making.”2 (p. 322)

One question that often arises is how much evidence is necessary? I recently worked on a consulting project where the charge was to develop an initial cost benefit analysis for a series of strategic projects. Calibrating the right amount of effort to get a preliminary look is not always easy. Thus, an initial effort to find evidence may only lead to a 40% confidence level that the project is viable. Once the initial round of evidence is gathered, and the metrics look good, the project can accelerate toward a 70% level. Why not 100%? It is too expensive to attempt to locate perfect information – instead, all decision-making is under a condition of ambiguity.

Generally, the objective for evidence-based management is to reduce the variation between what business leaders think they know versus what is supported in the research.3 The first steps of acquiring evidence is to do a search using a search engine.3 This is how the 40% confidence level is attained. In parallel, one can conduct the same research on an academic database; many college libraries are open to the public, and some larger public libraries also may provide access. I have also found Google Scholar TM helpful.  The benefit of using scholarly searches is that many of the journal articles are peer-reviewed; this means scholars in the field have previewed and accepted the research as valid and consistent with current knowledge. 

Beyond just finding the evidence, a manager must also assess its credibility. One way to do this is to find more than one piece of research to support a research conclusion.3

Even with the soundness supporting evidence-based management, leaders remain hesitant for a number of reasons. Here are a few outlined on Slideshare:4

  • It takes longer to make a decision; managers want things to move fast – do it now rather than wait.

  •  A leader may have found recent information that suggests an easy solution to a problem – why wait if it is simple?

  • Organizational rewards may be aligned with quick action rather than tied to outcomes.


In addition, hesitancy may exist for other reasons:


  • The sense that the leader looks impotent if he or she does not have the answers.

  • A misunderstanding about leadership – leadership is co-sponsored by followers and leaders.

  • Loss of control. If the leader needs evidence for decisions, what’s to stop all significant decisions from using evidence and diminishing the leader’s value to the company? Finally, what if workers start to challenge the leader based on evidence?

 I can see all the points made, but frankly, too many decisions bypass the appropriateness of finding evidence, and I propose it has pushed organizations toward mediocrity or failure.

Evidence is important for all significant decisions in organizations. Leaders must redefine their role as facilitators of decision-making rather than the decision-makers.

Please feel free to leave comments.

References

1 Damasio, A.R. (1994). Descartes' Error: Emotion, Reason, and the Human Brain. New York, NY: Avon Books, Inc.

2 Goodman, J.S., Gary, M.S., & Wood, R.E. (2014). “Bibliographic Search Training for Evidence-Based Education: A Review of Relevant Literatures.” Academy of Management Learning & Education, 13(3), pp. 322 - 353.

3 Kepes, S., Bennett, A.A., & McDaniel, M.A. (2014). “Evidence-Based Management and the Trustworthiness of Our Cumulative Scientific Knowledge: Implications for Teaching, Research and Practice.” Academy of Management Learning & Education, 13(3), pp. 446-466.

4 CEBMa Center for Evidence-Based Management, Slideshare, EBGMgt Course Module 4.  “Evidence-based Management: What is Stopping Us?” Retrieved 2-24-15 from: http://www.slideshare.net/barene/4-eb-mgtwhatisstoppingus/1

5 Janis, I. (1995). Groupthink. In J.T. Wren (Ed.), “The Leader's Companion” (pp. 360-373). New York: Free Press.

6 Janis, I.L. (1971). Groupthink. Psychology Today, pp. 43–46 and 74-75.

Wednesday, January 28, 2015

Leaders Don’t Always Get It

It is surprising, given the thousands of articles on leadership, how many leaders still do not seem to get it. Leading is not about the leader but instead is about followers and their coproduction of organizational outcomes – profits, revenues and social causes.

Part of the problem may be the perspective conveyed by books and articles on leading.  For example, consider these three books: (1) Moments of Impact: How to Design Strategic Conversations That Accelerate Change;1 (2) Open Leadership: How Social Technology Can Transform the Way you Lead;2 and (3) Transforming Your Leadership Culture.3  

Moments of Impact1 has valuable ideas on the process of strategy development and the plethora of meetings that are often part of the process. It offers a range of ideas and uses scholarly research to explain – this is good. However, it lacks details about the followers – it tells the leaders to engage followers but not how. Their sound advice of “active learning, engaging participants as whole people and following a narrative arc”1 (p. 137) is sound but falls short of how a leader can make it happen via his or her behavior. As a result, the book feels more like a to-do list, an approach that seems to dominate the leadership literature. 
 
Similarly, Open Leadership2 offers guidance to protect the organization in times of nearly complete transparency – it provides leaders advice on dealing with openness. Some of its ideas address the concerns I have about the portrayal of leading in the literature. That is, leadership (and decision-making) is a shared or distributed activity – the book is a move away from the leader-centric mantra of most publications. Practitioner leaders will find this book interesting, valuable and worthy of the time to read it, but as with other writings, it favors a checklist of action items.
 
Finally, Transforming Your Leadership Culture3 is a stronger example on how leader scholars are thinking about leadership. It advocates collective leadership and promotes an inclusive culture where leaders and followers are partners. It also drives home the message that “human systems [in organizations exist] first” followed by “operational systems.”3 (p. 4) I enjoyed reading the survey questions found at the back of some of the chapters – these questions highlight the importance of a topic while helping a reader consider actual behaviors versus the ideal. However, as with other writings, it too has the feel of a to-do list.
 
These brief descriptions are not for the purpose of trashing the books but instead provide examples of how even valuable books can provide incomplete messages – that is, effective leading is not just about following a formulaic list of action items (doing behaviors). Having a solid set of action items is necessary but insufficient to be an effective leader. Recent scholarly work reveals a combination of doing behavior as suggested by these lists in conjunction with making it possible to work with and through followers/workers – the concept of being a leader 

It is common for leaders in the early stages of their development to default to action lists because it guides and structures actions in the new role.5 However, a leader must continue to grow – that is, he or she must integrate leading as a component of his/her identity.5 This means the role of leader becomes part of the person.  

Unfortunately, most books and writings on leadership seem to align with the early stages in a leader’s development, leaving out altogether the complexity of incorporating the role of leader into identity. In other words, leader books and articles focus on what leaders should do.  Identity work, on the other hand, is a more gut-wrenching, reflective process whereby the leader learns from experience until it becomes automatic and intuitive. The problem that evolves from the literature is that it gives the false impression that if one takes the recommended actions, he or she will be a success. 

For example, a recent article in the New York Times Magazine4 portrayed Marissa Mayer’s impact on Yahoo during her first two years at the helm; her results have not been stellar. Some believe Yahoo’s results evolved from its ownership of Alibaba and, without its inclusion, would leave Yahoo with a negative valuation.4   

What I found interesting about the article is how negative it was about her behaviors as a leader; I was surprised. I wonder if she has fallen victim to some of the perspectives found in the literature described above. Here are some excerpts describing Mayer’s behaviors. My comments are in brackets. 

“…Mayer, who has a tendency to compare herself with Steve Jobs, wasn’t about to abandon her turnaround plan.” 4 (p. 24) [She is not Steve Jobs and should instead craft her own leader style.]

“Hours after entering Yahoo’s complex on the morning of July 17, 2012 [second day on the job], she set up her computer to log into the company’s code base so she could personally make changes, much like the founder of a tiny tech firm might do.” 4 (p. 26) [Sounds like micromanagement.]

“Mayer…immersed herself in the redesign of the [smartphone mail app]. Mayer would regularly interrogate designers about the minutest details of display and user experience.” 4 (p. 26) [Should hire quality people she can trust.]               

A NewFronts “event seemed to ignite Mayer’s interest in content, and within a month she asked that all programming decisions be run by her.” 4 (p. 27) [Sounds like micromanagement.] 

“…Mayer tended to require countless tests about user preferences before making an important product decision. But when it came to media strategy, she seemed perfectly comfortable going with her gut.”4 (p. 27) [May confuse staff.]

“Mayer spent as much time deliberating Yahoo’s parking policies as she did strategizing over the sale of its Alibaba stock.”4 (p. 44) [This is a management activity; leaders don’t need to be involved in everything.]

“Mayer also had a habit of operating on her own time. Every Monday at 3 p.m. Pacific, she asked her direct reports to gather for a three-hour meeting….Invariably, Mayer would be at least 45 minutes late.”4 (p. 44) [Could be thought of as rude.]

“This delinquency eventually became a problem outside Yahoo…Mayer was scheduled for dinner with executives from the ad agency IPG. The 8:30 p.m. meal was inconvenient for the firm’s C.E.O., Michael Roth, but he shuffled his calendar so he could accommodate it. Mayer didn’t show up until 10.”4 (p. 44) [This may lead to long-term relationship problems.] 

In reading this article, assuming its accuracy, Mayer seems to be doing a leader role rather than being a leader. It is not unlike some of the general press about the leader actions described in the literature. For example, her adoption of Steve Jobs’ habits could be from the literature (even the movie) on his behavior and the results he accomplished. Like all leaders, she needs to construct her own leader-self; too much mimicry could lead people to view here as inauthentic. Authenticity is an important part of a leader’s success.  

I was also surprised that she feels the need to be involved in so many things – even parking. This may not be coming from the literature and instead from fear. New leaders need to get their “sea legs.” Her need to sign off on programming decisions would suggest she is a “go-it-alone leader” and she does not trust the people she has working for her. Most press places the leader at the center of everything, but this is not wise or good.  It fails to recognize the role of followers as producers of outcomes in partnership with the leader. 

Even making people wait suggests an attitude she has about leadership – “I am in charge, and they will just have to wait.” When I first read that she had required those from working at home to start working from the office, I thought her reasoning was sound – collaboration is more difficult at a distance. However, this article has made me think differently. While she was forcing employees back into the office, she built a baby nursery near her office so that she could work at the office. The nursery was at her expense, but were any other employees given this option, and how many could afford it?

There is a disconnect somewhere. While the literature keeps providing a list of actions for leaders to take, successful leadership transcends a set of actions. Mayer may or may not have been influenced by this barrage of recommendations found in the literature, but her behaviors seem to suggest she has been influenced. While I have hopes that Mayer will be a success – and judging by her work at Google, I think she will eventually do well – it appears she has some development work to achieve. Nevertheless, she has not yet learned how to lead effectively – she is doing rather than being a leader. She also needs to hire the right people so that she can trust in what they do. 

I fear that part of the problem with leadership literature today is we have heard too much about actions that leaders need to take and not enough about how a leader should be.  

Feel free to make comments or add your thoughts on this matter. 

References

1Ertel, C., & Solomon, L.K. (2014). Moments of Impact: How to Design Strategic Conversations That Accelerate Change. New York, NY: Simon & Schuster.

2Li, C. (2010). Open Leadership: How Social Technology Can Transform the Way You Lead. San Francisco, CA: Jossey-Bass.
 
3McGuire, J.B., & Rhodes, G.B. (2009). Transforming Your Leadership Culture. San Francisco, CA: Jossey-Bass and The Center for Creative Leadership.
 
4Carlson, N. (2014, Dec. 21). “No Results,” New York Times, New York Times Magazine, pp. 22-27 & 44-46. 

5Ely, R.J., & Rhode, D.L. (2010). “Women and Leadership: Defining the Challenges,” N. Nohria & R. Khurana (Eds.), Handbook of Leadership Theory and Practice (pp. 377-410). Boston, Massachusetts: Harvard Business Press.

 

 

 

Monday, July 21, 2014

Renewal


Everyone needs downtime and renewal. In his timeless book, The 7 Habits of Highly Influential People1, Steven Covey identified renewal as one of seven vital habits; he called it “Sharpen the Saw.”  The habit comprises four dimensions: “Physical (Exercise, Nutrition, Stress Management), Social/Emotional (Service, Empathy, Synergy, Intrinsic Security), Spiritual (Value Clarification & Commitment, Study & Meditation) and Mental (Reading, Visualizing, Planning and Writing).” 1 (p. 288)

Leaders need to institute some or all of these practices in their routines in order to be effective and to avoid the breakdowns related to stress. It is common for the daily grind of work life to consume leaders in a succession of meetings, deadlines, challenges and problems with no apparent means of escape. And, as time passes, it becomes more difficult to rise above these ongoing and evolving work issues.

Stress, “an unpleasant psychological process that occurs in response to environmental pressures,” 2 (p. 595) can have positive and/or negative consequences in organizations. Some believe that lower levels of stress are best, but this is untrue in organizational situations. In fact, if job stress is too low, motivation is reduced. Alternatively, if stress is too high, performance deteriorates because it can overwhelm and dominate.2 & 6 As such, a balanced stress level, when stress is not too high or low, can lead to higher performance.  “As the leader’s stress level increases, performance level also increases until she reaches the optimal performance level relative to stress.” 6 (p. 134) Because the leader’s role is by definition stressful, it is necessary for him or her to occasionally detach and find an equilibrium in order to maintain a top level of performance.

Excessive stress can have both personal and organizational costs. Physiologically, stress can lead to “headaches, high blood pressure and heart disease.” 2 (p. 597)  Psychologically, stress can produce “anxiety and depression.” 2 (p. 597) Stress also restrains organizational results by reducing job satisfaction and, in turn, causing negative effects on “productivity, absenteeism and turnover,” 2 (p. 597) including the possibility of undesirable trends in organizational citizenship behavior, customer satisfaction, loyalty, and workplace deviance.

Stress does not have the same effect on everyone. How a given person reacts can be influenced by temperament, indoctrination to stress as a child, perception of control over situations, and self-confidence.2   Leaders need to gain an awareness of how he/she personally responds as well as how assigned workers might respond. This information enables the leader to craft an appropriate set of actions and preventative measures.

Leaders have a degree of control over the organizational causes of stress: task demands (type and set-up of work), role demands (pressures associated with the job itself) and interpersonal demands (the organization’s climate of relationships and support).2  For example, the leader may be able to install a more supportive team, creating the conditions for reduced personal stress.  Leaders can also alter stress levels for subordinates. For example, leaders can increase freedom and independence, variables that can benefit motivation.5 A leader can also balance role demands – all too often leaders pile work on subordinates without considering the consequences. In one situation, a team of leaders was asked to prioritize the many projects in progress, but the end result was that management forced several #1 priorities – thus, more than one project was on top. The final message was to get it all done, rendering prioritization pointless.

As described, there are real consequences to excessive stress and some may push people to devote vast amounts of energy to their job, especially for those who are obsessive. For example, one leader felt the need to work from 6 a.m. until 10 or 11 p.m. daily and was also consumed on the weekends; the leader applied this same expectation on subordinates. This flies in the face of research that states an optimal day for sustained performance is about 8 hours.3 Nevertheless, organizations tend to drive knowledge workers to at least 10-hour days.

So what is the solution? One step is to follow Covey’s formula of “sharpening the saw” along with its prescribed activities.1 In addition, developing work habits that are more efficient and effective can also reduce stress. For example, research confirms that a messy desk creates a sense of disorganization.3 Meditation and other relaxation techniques can be beneficial as well. 2 

Prioritization is also a key issue. Unfortunately, most organizations attempt to do everything, even when focus and restraint is more appropriate for the long-term. Sustained success requires pinpoint attention to a chosen strategy for not only the economic health of the organization but also the human factors. “Strategy involves focus and, therefore, choice. And choice means setting aside some goals in favor of others.” 4 (p. 59)

Further, without the people, nothing would be happening, and it is up to the leaders to optimize the stress levels in the organization such that it is not too high or low.

Renewal is an important habit. Without it, stress can overwhelm, causing performance to be suboptimal. Leaders need to renew to stay fresh and in the game, and they need to cause their organizations and those that work there to keep stress under control. An organization cannot do everything; pick your battles.

Comments are welcome.

References

1 Covey, S.R. (1989). The 7 Habits of Highly Effective People. New York: Simon & Schuster.

2 Robbins, S., & Judge, T. (2013). Organizational Behavior (15th ed.). Boston: Pearson.

3 Mackenzie, R.A. (1972). The Time Trap: How to Get More Done in Less Time. McGraw-Hill.

4 Rumelt, R. (2011). Good Strategy/Bad Strategy: The Difference and Why It Matters. New York: Crown Business.

5 Pink, D.H. (2009). Drive: The Surprising Truth About What Motivates Us. New York: Riverhead Books.

6 Thompson, H.L. (2010). The Stress Effect: Why Smart Leaders Make Dumb Decisions – And What to Do About It. San Francisco: Jossey-Bass.


Monday, June 9, 2014

Selecting Grads



Back in the early 1970s, two papers were published that still have significance for leaders. The Spence1 and Akerlof2 Nobel prize-winning articles described decisions under conditions of ambiguity such as employee selection. Michael Spence’s article describes how difficult it is to make employee selection decisions because “the employer cannot directly observe the marginal product prior to hiring. What he does observe is personal data” 1 (p. 357) in addition to supporting evidence, like educational credentials. This means that it is only after the hire that the employer can confirm or deny the original assumption of the quality of the candidate.1 A related concept is found in George Akerlof’s article, which describes the concept of information asymmetries. Information asymmetries exist when the buyer (i.e. employer) and seller (i.e. graduating student and new hire candidate) hold different knowledge of the candidate’s potential value.2 The employer will attempt to plug the gap in information by using “institutional guarantees,” 2 (p. 499) which serve as an external “endorsement” of the candidate’s worth. “The high school diploma, the baccalaureate degree, the Ph.D., even the Nobel Prize, to some degree, serve this function and certification.” 2 (p. 500) Attending a brand name university also gives the candidate a leg up because of its reputation.2

The challenge for leaders, as described in previous posts, is getting the “right people on the bus,”3 and it begins not just with senior management but entry-level positions in which newly minted university grads are hired.

Today’s post is not based on a scientific study but rather my observations as a corporate executive, consultant and teacher in higher education for 12 years. While Spence’s concept of education as a signal of quality and Akerlof’s solution to information asymmetries remain cogent, there is a problem that has been evolving in higher education. Somehow, somewhere along the way, higher education has been reinterpreted as the earning of a piece of paper rather than what it is meant to represent. Students frequently say that “barely passing gets degrees,” and they have decided the documentation of a degree now exceeds the need for learning and mastery of reading, writing and critical thinking. For example, some students, when they find themselves in a demanding class, simply refuse to do the work or even quit. My observation is that only about 20% - 30% of students recognize a strong positive correlation between energy devoted to college studies and future career prospects. One student, lamenting about the papers he had to write, said, “I should not have to do this.”

Unfortunately, I think that universities, perhaps unintentionally, have been part of the problem. Some of the fault rests with the economics of universities and the need to keep enrollment from falling. In a Forbes ranking of the financial health of 925 four-year private not-for-profit colleges and universities, half have a “financial fitness [that] would be considered ‘C’ students or worse,” 4 (p. 86) and 107 earned a “D.” Lowering standards is more frequently used to maintain attendance. Combine this with the way faculty are assessed for promotion and tenure, and the causal chain continues. Faculty should receive feedback from and students and peers, but when the university makes student evaluations prominent in the decision for promotion or tenure, faculty adjust classroom requirements to maintain and increase scores. “The problem is for the vast majority of colleges and universities, student opinion is the only means by which administrators evaluate teaching.”5 From a faculty perspective, there is no secret on how to increase ratings – relax grading, reduce rigor and treat students as customers. “There’s also a natural tendency to avoid delivering bad news if you don’t have to. So the prospect of end-of-term student reviews, which are increasingly tied to job security and salary increases, is another upward pressure on professors to relax standards.”5

Thus, the problem is compounded for businesses, because unless the student is from a top brand university the interpretation of the quality of the candidate becomes difficult. University administrations can solve this problem by establishing standards of rigor – a nearly impossible undertaking given the need for academic freedom. Further, any movement could cause the university to find itself in a prisoner’s dilemma; if one university raises standards while peer institutions remain the same, the peer schools will become more attractive. Likewise, faculty – who own curriculum – can unilaterally raise the standards, but arriving at a consensus among independent and autonomous individuals with the shield of academic freedom is probably unrealistic.

Therefore, I believe that a component of the solution of this complicated problem rests with leaders and employers. That is, beyond the signal from candidates of having a degree, employers need to start asking employee candidates different questions. If there is a collective change in how selection is conducted and what is asked of future employees, then backward in the chain universities will be required to react. Here are some ideas in addition to other standard hiring practices:

  • Always close the information gap by seeking candidate transcripts. Look for evidence of excessive withdrawals and class failures; most students are afforded the chance to replace a “D” or “F” on a transcript by retaking a class.
  • Only hire students with grade point averages (GPA) at or above 3.0; while this may be adjusted depending on the discipline, I have witnessed that when students are below a 3.0, they appear to be calibrating their effort toward getting by rather than devoting energy. Some may not have the capability, but this is an exception.
  • Ask students what papers they have written during their academic life, as well as how many, and perhaps ask for a sample. "Lecture and memorization do not lead to long-term knowledge or the ability to apply that knowledge to new situations." 7 (pp. 91-92) Writing assignments, essay questions on tests, case studies and the receipt of directive feedback are methods for increasing critical thinking.7 This is also a dilemma for faculty; writing is more difficult, time consuming and subjective to grade and can lead to claims of unfairness from the student, which can negatively impact evaluation scores. When students are asked to think through a problem, they often complain of how difficult it is – and it should be. Alternatively, multiple choice tests are easier to grade and not subjective; students may actually take more personal blame for failing a test – “I did not study enough” – preserving the faculty member’s evaluation scores.
  • Look for at least one internship for credit and a project outcome from that experience that demonstrates a student’s ability. A strong Q&A during the candidate’s interview can be very informative about the student’s future potential.
  • Ascertain if the focus of the program is on teaching content versus use of content. "[Faculty] must teach content through thinking, not content and then thinking."6  
  • Look for voluntary community service (charity or on campus) and other extracurricular activities such as sports or clubs. However, the latter should not be in place of learning; a student who focuses on athletics to the detriment of academics may not be in college for an education.

Today, in my view, the quality of a candidate with a bachelor’s degree is more difficult to interpret by leaders and employers. That is, the state of higher education has succumbed to financial pressures, and the belief system behind the purpose of getting a degree has shifted; it is now about the documentation rather than learning and developing competencies in reading, writing, and critical thinking. Universities and faculty are caught in a prisoner’s dilemma. Therefore, a transformation must come from the very organizations that need the new talent from universities. Leaders and employers must ask the right questions, some of which are identified above; change can be forced backward in the educational system when employers demand different things.

The problems identified do not apply to all students and universities, but my observations suggest it is increasing. There is no single or easy solution, but this could be a beginning.

Feel free to respond.

References

 1 Spence, M. (1973). “Job Market Signaling.” The Quarterly Journal of Economics, 87(3), p. 355-374.

2 Akerlof, G. A. (1970). “The Market for ‘Lemons’: Quality Uncertainty and the Market Mechanism.” The Quarterly Journal of Economics, 84(3), p. 488-500.

3 Collins, J. (2001). Good to Great. New York, New York: HarperCollins.

4 Schifrin, M. (2013). “Colleges at Risk.” Forbes, 192, p. 84-90.

5 Asher, L. (2013). When Students Rate Teachers, Standards Drop. Journal, (October 27, 2013). Retrieved from http://online.wsj.com/news/articles/SB10001424052702304176904579115971990673400

6 Paul, R. (2004). The State of Critical Thinking.   Retrieved December 29, 2013, from www.criticalthinking.org

7 Snyder, L. G., & Snyder, M. (2008). Teaching Critical Thinking and Problem Solving Skills.   Retrieved December 29, 2013