By Steve Creech | October 17, 2011
Several of my clients, and their committee members have had some misunderstandings about the use of parametric statistics with ordinal data, so I decided to write this article.
Many statistical procedures such as Pearson’s correlation and Linear regression analysis require certain assumptions about the data in order for the procedure to be valid. One of those assumptions is regarding the measurement scale of the variables. There are two main types of measurement scales, categorical and continuous. Examples of categorical variables are: 1) gender; 2) what is your favorite color, and; 3) level of agreement, strongly disagree, disagree, neutral, agree, strongly agree. Examples of continuous measurements are: 1) height in inches (measured to the nearest 10th of an inch); weight (measured to the nearest 10th of a pound).
Within categorical measurement scales there are two types, nominal and ordinal. Gender and favorite color are nominal whereas level of agreement is ordinal. The only difference between nominal and ordinal is that the order of the categories for a nominal variable has no meaning whereas an ordinal variable has a meaningful order from smallest to largest. For example, you can’t put favorite color, red, green, blue, yellow in order from least to most, but strongly disagree is clearly less than disagree, which is less than neutral and so on.
It is pretty unusual to have an ordinal dependent variable in research. When ordinal variables are involved in a dependent variable it is far more common to have an instrument consisting of numerous questions, each of which is measured on an ordinal measurement scale. The instrument is designed to produce a scale score, which is an aggregate of the individual survey questions (usually the average of the questions). So, if your survey has 10 Likert-type (i.e. ordinal) questions, you usually aren’t going to be analyzing the individual survey questions themselves. The survey is designed such that you take the average of the 10 questions, which produces a number that can take on fractional values like 1.17, 2.73, 3.47 etc. So, the resultant scale score is the actual measurement (i.e. independent or dependent variable) that you are analyzing, and it is measured on a continuous measurement scale, so it meets that particular requirement of the parametric test, like Pearson’s correlation or linear regression.
By Steve Creech | February 26, 2011
Suppose you want to find out if there is a correlation between job satisfaction and the perception of the supervisor’s leadership style among non-supervisory employees. Suppose you use the Multifactor Leadership Style Questionnaire (MLQ) to measure five transformational leadership styles (the MLQ measures other leadership styles too but we don’t need them for this explanation). Let’s call the five leadership styles: L1; L2; L3; L4, and; L5. Let’s call job satisfaction “JS”.
So, why not just do one multiple linear regression analysis, put all five leadership styles (independent variables) into the model and whichever ones are statistically significant, those are the leadership styles that are correlated with job satisfaction? The answer is, it is possible for several independent variables to be individually correlated with a dependent variable, but not all of them will be statistically significant in the same multiple linear regression model.
Here is the idea, suppose the Pearson correlation statistic for comparing the five leadership styles with job satisfaction are: .29; .31, .38, .28, and .44, for L1, L2, L3, L4, and L5, respectively. Suppose that all five correlations are statistically significant. Now, when we put all five leadership styles in the same “multiple linear regression model”, the analysis shows that only L3, and L5 are statistically significant.
This would be important to know because it means that if all you have is one of the leadership style scores, that can be used to predict the dependent variable (job satisfaction). We can also see that if we had to choose, we would prefer to know L5, because it had the strongest correlation with job satisfaction. But if we wanted the best prediction model possible for predicting job satisfaction, we would know that we only care about L3, and L5. Once we know how much L3 and L5 leadership style a person has, knowing the amount of L1, L2, and L4 leadership style they have will not alter our prediction of the dependent variable.
In other words, all five leadership styles are correlated with job satisfaction, but not all five add up to collectively better predict the dependent variable. Only L3 and L5 “add independent information” about job satisfaction.
By Steve Creech | February 26, 2011
One-sided alternative hypotheses are rarely used and I usually discourage their use. The point is, why limit yourself to a one-sided alternative hypothesis? If the results should happen to be statistically significant, but in the opposite direction you expected, you will not be able to reject the null hypothesis.
It is my understanding there are mainly two circumstances under which you would use a one-tailed test: 1) when a statistically significant result in one of the directions would be of no interest (e.g. a pharmaceutical company developing a new drug and they only care if it is better than the standard drug), or; 2) when it is physically impossible for the relationship to go in one of the directions.
I have worked with many doctoral students that have had a committee member that felt very strongly that one-tailed hypothesis tests should be used. I think the advice is a misguided attempt to get the doctoral student to state their “expected finding”. In other words, sometimes a committee member will say that you should have a “research hypothesis” that specifies which direction the correlation will go.
I think in this context, “research hypothesis” is synonymous with “expected findings”, but not synonymous with “alternative hypothesis”. In other words, what I think you should do is, use two-tailed alternative hypotheses but then add a section called “expected findings”. That way, you can have the full benefit of two-tailed tests and still satisfy the committee member’s request for you to state a research hypothesis that specifies a particular direction.
By Steve Creech | February 26, 2011
This is actually a tougher question than you might think. The text book way to determine a sample size is to conduct a literature review to determine what effect size you are looking for. For example, suppose you wanted to know if there is a correlation between job satisfaction and the perceived leadership style of the supervisor among non-supervisory employees.
Suppose there are 10 published articles on this very subject, all used the exact same instruments you are planning to use and all used rigorous methodology. Suppose the strength of correlation between job satisfaction and leadership style ranged from .20 to .35 and the average among the 10 papers was .27. Then, your best guess as to how strong the correlation will be in your study is .27. Then, you would conduct a statistical power analysis to determine how large a sample size you need in order to detect a correlation of .27.
In most cases the above scenario won’t work. Most doctoral students don’t replicate previous studies. Therefore, most doctoral students do not have any articles to refer to in order to estimate the expected effect size. The next best thing is to find several articles that reported results of studies that were very similar to yours. For example, they studied the correlation between job satisfaction and leadership style, they used the same leadership style questionnaire, but a different job satisfaction instrument.
Again, in most cases, you will not find articles that come close enough to what you are studying to be of any use. Therefore, in this case, you have no idea what the effect size will be. On the one hand, you want to use a very large sample size, just in case the effect size is small. On the other hand, for all you know, the effect size will be large, so you don’t want to waste your time and money on a large sample when a small sample would do. The logical thing to do is select a sample size large enough to detect a medium effect size.
In the real world (as opposed to the text book approach), doctoral students are usually restricted by time, cost and other constraints that limit their sample size. The sample size is determined in part based upon what is doable given the constraints, and then the power analysis can demonstrate what effect size can be detected with that sample size, and in effect, that justifies the sample size.
For example, when selecting a sample, you have to think about how you can get access to the target population. For example, if you want to study job satisfaction and perceived leadership style among non-supervisory employees in the fast food industry, you might go door-to-door to a number of fast food restaurants, ask to speak to the manager, and ask permission to conduct a survey of his/her employees. This would be a lot of work and would not be a practical approach. So, you think of a better way, perhaps you could contact corporate headquarters for one particular fast food chain and get permission that way. Then, the corporate administrator could assist with disseminating your survey among the employees. These practical considerations will often determine the target population you are “able” to study. This may not be the ideal target population, but it may be a necessary limitation of the study to make it doable.
Now that you have selected a “doable” target population, you need to estimate how many members of the target population are eligible for your study (e.g. non-supervisory employees). You might be able to find that out from the corporate administrator that is assisting you. Let’s say it is 1,000 employees. Now, since your survey is voluntary, your actual sample size is going to be a function of how many people agree to participate, sign informed consent, and complete the survey.
You can do a literature review for typical survey response rates. They vary according to a number of factors but over the years the most common response rate I have seen cited is 20%. So, for your sample size justification section, you basically say: I have access to a target population of approximately 1,000 non-supervisory employees. All 1,000 members of the target population will be invited to participate in the study. Based upon papers by …., typical survey response rates are approximately 20%. Thus, the anticipated sample size is approximately 200.
Now, perform your statistical power analysis using a sample size of 200, 80% power, and an alpha level of .05 and solve for the effect size. Now you will be able to say… a convenience sample of approximately 200 will be used, based upon a power analysis, a sample size of 200 will produce 80% power at the .05 level of significance to detect an effect size of ….
By Steve Creech | January 22, 2011
1) It starts with personal interest in a “general topic” within your area of specialization. Most likely this topic is something you are familiar with from personal or professional experience.
2) Conduct some preliminary literature review to insure the general topic you have in mind has not already been thoroughly researched and published (i.e. don’t reinvent the wheel).
3) Once you have identified a general topic and you have done enough literature review to know you are not reinventing the wheel, hire a statistician to help with the statistical aspects of your proposal.
Aside: I charge on a fixed price basis. Whether you get me on-board from day one, or after attempting to get your entire proposal accepted, only to have it rejected one or more times, the price is the same. It is almost always “more” work for me to help a doctoral student that has gotten very far into the proposal than to help a doctoral student that is just starting out. The sooner you start working with a statistical consultant, the smoother things will go for you. With my services, since it is the same price, why not get me on-board early?
4) Consult with the statistician about your topic and share your ideas about what sort of data you want to collect (e.g. maybe you have a particular survey in mind, or an archived data set). The statistician can advise you on methodological considerations relating to your planned approach. Most likely the statistician will point out a variety of options, each of which has pros and cons, and the choices you make have implications for your problem statement, purpose of the study, research questions, instrumentation, population and more.
5) After the consultation with the statistician, you should have a rough draft of your problem statement, purpose of the study, research questions, independent and dependent variables, research questions, instrumentation, population and data collection strategy.
Aside: I almost always send a rough draft of that information after the first collaborative phone consultation. Then, within a matter of 7 days or less, I will have completed all of the statistical considerations for your proposal (e.g. data analysis plan, sample size justification).
6) Once you receive the write-up of the statistical considerations from the statistician (just a cleaned up, technically written documentation of what we collaborated on during the initial consultation), then it is just a matter of incorporating that information into the current draft of your proposal.
7) Actually writing the proposal from this point on is largely an organizational challenge. I believe you should use the following process to “construct” the proposal:
1. Start with a blank Word Document.
2. Insert the title on page 1
3. Copy the Table of Contents from the rubric onto pages 2 through however many pages it takes.
4. On the very next page, insert the chapter heading (e.g. CHAPTER 1: INTRODUCTION), and underneath that, insert each of the section headings that go in that chapter. Make sure the section headings match the rubric. Don’t add additional headings, don’t leave any out, and don’t change the order.
5. Repeat step four for chapters 2 (Literature review) and 3 (Methodology).
6. Make a copy of the document you just created and save it as something like “Disseration Proposal Shell”.
7. Using a copy of the “Shell” from step 6, start with chapter 1. Skim through the sub-headings and pick the one you feel the most confident in, the one you think you could write off the top of your head (i.e. the low hanging fruit).
8. Write as little as you possibly can in that section, while capturing all of the main points that you think should be in that section. Try to keep it to one paragraph or less if possible.
9. Pick the next section in chapter 1 that you feel most comfortable with and repeat step 8.
10. Continue steps 8 and 9 until you get stuck. If you are stuck, if it has anything to do with statistics, consult with your statistician for advice. If it is a subject matter issue, maybe you need to stop and do more literature review.
11. Follow steps 8-10 until you have 1 paragraph or less written in every section of chapter 1. You might find that while doing this, you can fill in some of the sections in chapters 2 and 3. Go ahead and do that also. After all, you should have the research questions, hypotheses, data analysis plan, power analysis and other statistical considerations from your statistician by this point, and that all goes in chapter 3.
12. Once you have written 1 paragraph or less in every section of chapters 1, 2 and 3, stop and save a copy. That is your “bird’s eye view” of the entire proposal.
13. Work on cleaning up the “Bird’s Eye View” of your proposal until it flows naturally, with smooth transitions from one section to the next.
14. Review each section again and ask yourself, did I “mention each core idea” that needs to go into this section (check the rubric for what should be there)? If you missed an idea, add it. Do this for all of the sections and save this as Bird’s Eye View Revision 1.
15. Run the Bird’s Eye View draft past your statistician. He or she should be able to tell you if there are any inconsistencies with what you wrote, and the statistical aspects of your study. The statistician should also be able to give you critical input about how to make the paper flow logically.
16. Now that you have this Bird’s eye view of the proposal finished, it is just a matter of “fleshing out” each section. What I would do is take each main “idea” within each section, determine how long I want that section to be, and then decide how much I want to expand on each main idea until that section is the desired length.
17. At this point you should have a well organized and nearly complete dissertation proposal. Read through it carefully, correct as many grammar and punctuation mistakes as possible and try to make the transitions from one section to the next as smooth as possible. Then, submit that draft to your statistician for another review.
18. The statistician will likely recommend a number of revisions to help organize the proposal and to insure what you wrote is consistent with the statistical aspects of the study. Work with the statistician back and forth until there are no further revisions from the statistician’s perspective.
19. Submit the proposal to your mentor. The mentor will likely have several comments, questions and suggested revisions. Share those comments with your statistician. You want to make sure you understand any comments and questions that relate to the statistical aspects of your study. Work with your statistician to develop responses to the reviewer’s comments.
Aside: Just because the mentor suggested a revision doesn’t mean you should make that revision. Remember, you probably know 10 times more about your study than your mentor does at this point. You have probably spent several weeks if not months of intensive work on just this one study, whereas your mentor has to teach and probably mentor several other doctoral students as well.
20. Once you have developed a response to every comment from the mentor, making revisions where you and your statistician felt they were appropriate, send the revised draft back to the mentor.
21. Repeat steps 19 and 20 until the mentor has no further comments and passes it on to the other committee members.
22. Repeat steps 19-21 for the other committee members until they are satisfied and they submit the proposal to the ARB, IRB, external reviewer or whatever the next step is at your university.
23. By this point, you should be very close to having an accepted proposal.
By Steve Creech | December 2, 2010
Recruiting participants for a research study can be challenging. I often advise my clients to consider inviting members of a professional association to participate in their survey research. There are many advantages to this approach:
1. Very little red-tape. You are not targeting one or more companies individually, so you don’t have to get permission from many different people. You only need permission from the appropriate administrator of the professional association.
2. Convenience – Instead of mailing letters through the U.S. Postal service, or sending emails to a massive list of email addresses, you simply ask the appropriate administrator of the association to send your (email) letter of invitation on your behalf. This insures anonymity of the study participants since you are not requesting names, addresses, phone numbers, email address etc.
3. You can reach a broader audience. Rather than recruiting study participants from a single organization, located in a single geographic location, a national professional association has members located across many organizations and geographic locations, which goes to the ability to generalize your study sample to a larger population.
One question that is often asked of me is, how do I write a letter to the professional association to get permission to include the members of their association to participate in the study? I decided to post this article to benefit all doctoral students that have the same question.
First of all, a disclaimer: As a professional statistician, it is really outside my area of expertise to give you specific advice about how to craft the letter asking for permission. You might want to contact your university writing center to see if they can advise you, or maybe your mentor/committee chair. Having said that, I have worked with hundreds of doctoral students that needed help with this issue, so I know a little about it.
Here is my advice:
1) Introduce yourself, mention you are working on a doctoral dissertation at “fill in your university”, mention the topic of your study.
2) Make it very clear that you are not requesting email addresses, phone numbers, mailing addresses or any personally identifying information about the members of the association. You are asking that they email your letter of invitation to complete your online survey (note, I always recommend an online survey when possible, check out www.SurveyMonkey.com), on your behalf, by simply sending a group email to all of the members of their association. Make it clear that your survey does not ask for any personally identifying information, that the study participant’s identification will be completely anonymous.
3) Either include a copy of the questionnaire(s) you intend to use, or state that you will share the survey with them as soon as you have permission from your school and authors of the instrument(s) to do so, to assist with their making a final decision to agree to allowing you to study the members of their association.
4) I’m not sure of the job title of the person at the association that you should send the letter to. I “do not” recommend starting at the top like the CEO, president or vice president. I would look for a “director of communications” or something like that.
My name is ____, I am developing a research proposal for my doctoral dissertation at the University of ____ titled _____. I am requesting your permission to invite members of your association to participate in my study by completing an online survey. Please find attached a copy of the survey that I plan to use for my research.
I am not requesting email addresses, phone numbers, mailing addresses or any personally identifying information about the members of the association. Instead, I would like you to email my letter of invitation to complete the online survey, on my behalf, to all of the members of your association. My survey does not ask for any personally identifying information, the study participant’s identification will be completely anonymous.
I am not asking you to send the letter of invitation at this time. I must first obtain official approvals from my university and your organization. The intent of this email is to request your permission to invite members of your association to complete my survey. Once I have all of the appropriate permission letters, then I will forward to you the actual letter of invitation and ask you to email the letter on my behalf at that time.
If you are not the person in charge of approving this type of request I would very much appreciate if you would forward the name and contact information of the person I should communicate with. I would welcome the opportunity to discuss this with you by phone if that would be helpful. In addition, I would be happy to provide any further information you may require in order to make a decision.
Thank you for your time.
By Boyd | June 17, 2009
ALL BUT DISSERTATION (ABD)
Are you at the ABD destination in your program?
There are two types of Ph.D. candidates that fall into this category:
1) The “just arrived” and anxious to move forward.
2) The “been there for awhile” and think they will never move forward.
While both types might require help to move on, it is the latter that is likely to derive the most benefit from this article and become motivated to complete, perhaps, the most important event in their life.
You are intelligent enough to have come this far, there is no reason (from an academic stand point) to linger in the “ABD Zone.” The longer you are there, the more difficult it becomes to pick up the pieces and move forward.
Many Ph.D. candidates seem to hit a brick wall and feel disarmed when called upon to work on the “methods” and “results” section of their dissertation. This is the point where many students diligently search for help calling on their mentor, peers, university assistance and even Google. This is also the time when the student may ask themselves the question “HOW MUCH HELP IS TOO MUCH”?
Surely no one will deny that having your dissertation written for you is very wrong. On the other hand, it is not unusual for doctoral students to get help on specific aspects of their dissertation (e.g. APA formatting and editing). It is also not unusual for advisers to encourage students to seek outside help with the statistical aspects of their dissertation.
A qualified and experienced statistical consultant who works with Ph.D candidates understands the special circumstances that can lead to ABD status (e.g. hectic fulltime job, family, and other personal issues). The question is how do you find a qualified statistician?
The best way to get started is with a phone call to a statistical consultant and ask the question: “How can you help me move beyond the ABD level and complete my Ph.D. program”? This is also the time you need to evaluate the consultant and answer the following questions:
1) Does the consultant have an advanced degree in statistics.
2) Will this person answer the phone and personally talk to you every time you call.
3) Can you reach this person when you need to. (Evenings and weekends).
4) Will this person be available all the way through the defense.
5) Will the consultant give you a fixed price quote up front so you will know your costs ahead of time?
6) Will the consultant give you a money-back guarantee the results will be correct, you will fully understand them,
and they will be accepted by your committee?
7) Will you get unlimited email and phone support until the day you graduate?
Caution: Make sure the consultant actually has an advanced degree in statistics, rather than having a degree in some other subject and is simply “good at statistics”. Where the difference really shows up is when a committee member asks lots of technical questions or you have a problem defending.
For many doctoral students, the most rigorous parts of a quantitative or mixed-methods dissertation are:
1) Methods Section
* Study Design
* Research questions and hypothesis formulation
* Development of instrumentation
* Describing the independent and dependent variables
* Writing the data analysis plan
* Performing a Power Analysis to justify the sample size and writing about it
2) Results Section
* Performing the Data Analysis
* Understanding the analysis results
* Reporting the results.
If you are a distance learning student it is almost essential you seek outside assistance for the methods and results section of your dissertation. The very nature of distance learning suggests the need for not only outside help but help from someone gifted in explaining highly technical concepts in understandable language by telephone and e-mail.
The ideal time to begin working with a statistical consultant is once you have a topic and you have done some preliminary literature review. Otherwise, you run the risk of unnecessarily complicating your study. This could result in the consultant being unable to help you, unless you are willing to start over with the problem statement, purpose of the study, research questions, instrumentation and data analysis plan.
As stated above, many students hit their dissertation “brick wall” when they encounter the statistical considerations. Frequently, a student will struggle for months before they seek a statistician’s help. This often leads to additional tuition costs and missed graduation dates. The number of Ph.D candidates not completing their program is staggering. If I were to name a single reason why a Ph.D candidate, doing a quantitative or mixed-methods study gets off track in their program, it is the statistics and their fear of statistics. So, the question is whether or not it is ethical to get help at all. If so, how much help is too much?
I don’t know if there has ever been a survey of dissertation committee members who were asked this question, however, I know many advisers take the following position when they suggest or approve outside help: To a large extent the process is self controlling. If the student relies too much on a consultant, the product may look good; however, the student will be unable to defend their dissertation.
It takes a committed effort on the part of the student and the consultant (resulting in a collaborative/teaching exchange) to have the student responsible for the data and thoroughly understand the statistics. This is not accomplished in just one or two emails or a single telephone conversation. It is a dynamic process; one that calls for unending patience on the consultant’s part and perseverance on the student’s part.
The day the student walks in front of the committee to defend, there should be no question as to their understanding of statistics. It is the consultant’s job to see to it this occurs.
When their defense is successful, the question ”was the help too much” is answered.
If you are a Ph.D candidate and would like additional information, you may wish to review the referenced sites below:
By Steve Creech | October 9, 2008
I would like to emphasize that the sooner you start working with a statistician during the development of your proposal, the smoother things will go for you. Once you have a topic and you have done some preliminary literature review is an ideal time to start working with a statistician.
So many of my clients come to me only after multiple rewrites of the problem statement, research questions, data analysis plan etc. They could have saved themselves a lot of time, money and frustration by contacting a statistician sooner.
In addition, many of my clients come to me only after they fully approved proposal. Often times, a proposal is accepted even though the statistics are not clearly written and sometimes, their statistics are just plain wrong. This can happen if you don’t have someone with an advanced degree in statistics on your committee. When that happens, I am unable to do the analysis for them unless we first redo the statistics in the methods chapter.
Another thing that happens is, the student comes to me with a fully approved proposal and they want help with the analysis. The statistical aspects of their methodology are correct, but unnecessarily complicated that I am unable to help them due to the scope of the project being so large I could not fit the work into my schedule, nor could I do the work for an affordable price.
These are some of the reasons why you want to work with a statistician early in the development of your proposal.
By Steve Creech | December 12, 2007
Probably, but many doctoral students do not have an “actual statistician” on their committee. The methodologist on your committee surely has more experience with statistics than your other committee members, but that is very different than having someone with an advanced degree in statistics and 14 years or more of experience as a statistical consultant on your committee. The point is, I have seen many committee-approved dissertation proposals that have research questions that do not lend themselves well to statistical analysis.
By Steve Creech | December 5, 2007
I have had a number of doctoral students call me with a question something like this: I purchased the student version of SPSS. I was able to calculate the mean and the frequency and percent for my variables, and I even tried to compare my independent and dependent variables with an ANOVA, but I don’t know if I did it right. Would you please review my work and see if I did it right?
By Steve Creech | December 2, 2007
Past clients who have something they would like to share with others about their experiences working with me.
By Steve Creech | December 2, 2007
Many of my clients have reported to me that their advisor recommended they hire a statistician to help with their dissertation. I am curious to know how many of you out there have had this happen. In my view, statistical consulting benefits both the doctoral student and the mentor. Often times the mentor has limited experience with statistics.
By Steve Creech | December 2, 2007
Once you got more heavily into the statistical aspects of your study (e.g. development of your methods chapter), what was the first thing you did? Many of my clients have told me they didn’t even know statisticians existed. They came to me only after many rewrites as a result of criticisms from their committee regarding their statistical considerations. I also have heard comments from clients that they had a statistics class or two several years ago and since have forgotten everything.
By Steve Creech | December 2, 2007
At what point during the development of your dissertation did you begin to struggle with statistics? In my view, statistical considerations come into play almost as soon as you have developed a topic. For all practical purposes, a statistical consultant is probably not necessary until you have spent time developing the topic and doing some literature review.
By Steve Creech | December 1, 2007
I have helped hundreds of doctoral students in developing their research questions, hypotheses, survey design, data analysis plan, power analysis and sample size justification, and performing the statistical analysis of their data.
My clients receive a clearly written report that demonstrates how to interpret and report the results “AND” they receive unlimited email and phone support to answer any questions they might have, to ensure that they completely understand their statistics.
By Steve Creech | December 1, 2007
In choosing a statistical consultant, ask yourself the following questions:
1) Does my consultant have a graduate degree in statistics?
2) Is statistical consulting for doctoral students their fulltime job, or is this something they do in the evenings and the weekends when they have time?
By Steve Creech | November 30, 2007
When I am asked this question, I like to respond with the following analogy. Surgeons do not usually perform their own anesthesia, because anesthesia is a highly technical and specialized field, and the surgeon would prefer to leave that to an expert. By analogy, most researchers do not perform their own statistical analyses, because statistics is a highly technical and specialized field, and they would prefer to leave the statistics to an expert. So, if writing a dissertation is about learning how to do research, then by working with a statistician, you are gaining real-world experience in how to do research. Read the rest of this entry »