It's that time of year again. Last night we honored the new SPA fellows and otherĀ awardĀ winners at theĀ American Geophysical Union (AGU. These awards allow us to publicly recognize those in our field who have significantly impacted many areas. The awards span things likeĀ early career,Ā outreach, or aĀ named lectureĀ (I will note here that most of the named lectures are named after men - but that's a whole other blog post topic).
In the recent past, I had the honor of being on theĀ SPA Fellows Nomination Committee. For two years, I was the chair. The process can be intense, but we take our job very seriously. At times, what happens on our committee and then on the Union Fellows Nomination Committee seems like a black box. Today I want to make that box more transparent. You can also read our new paper on this at Frontiers. Before going further, one thing to note is that each committee is different. However, each set of people finds their way toward trying to reach a consensus. What I'll describe below is how our committee was run. As I am no longer the chair or even on the committee, I no longer know what format is used. In all the years that I have served, and with all the other committee chairs I have talked to, everyone has taken this process very thoughtfully.Ā
The process for the 2023 nominations is starting now (so reach out to the Nomination Task Force https://connect.agu.org/spa/committees/ntf). A few years ago, it was called out and acknowledged that our fellows were not representative of our community. Even today, there is a disproportionate overrepresentation of white men and people from the US/Europe. Once the committee meets, we can only consider the nominations that have been put forward. If no women are nominated (as happened the first year I was on the committee), then no woman can become a fellow that year.Ā Liz MacDonald and others started a new committeeĀ to make sure qualified women, minorities, and other underrepresented groups we nominated through the development of theĀ nomination task force. While I am highly disappointed in my field for this stark bias, I am also proud of the steps we have taken to fix it. Climate and culture are, unfortunately, slow to change, but it is good that we are starting the process. And yes, we should do more.
So, what achievements can someone be nominated for to become an AGU fellow? We have three selection criteria.Ā
1. breakthrough or discovery;
2. innovation in disciplinary science, cross-disciplinary science, instrument development, or methods development; and/or
3.Ā sustained scientific impact
The nominee only needs to have one of these selection criteria to become a fellow. However, a nomination package often shows how the nominee fits two, if not all, three selection criteria. The committee does not consider the number of breakthroughs or innovations etc. But there is some possible implicit bias that may occur.Ā
The nomination package consists of a two-page nomination letter, a two-page CV, a two-page bibliography, and three support letters, each two pages long. These all help the committee understand the nominee's impact on the field. A good guide for a top nomination package is 1) The nomination letter provides a high-level overview of the nominee's selection criteria and the supporting info supplied in the rest of the package. 2) The CV is a full two pages and highlights the work towards the selection criteria, any service activities, and potentially any other awards won (see identified biases below). 3) The bibliography includes the works which showcase how the nominee's work has changed and impacted the field. A few really great ones highlight these papers and underneath the citation, provide a sentence or two stating how it changed the thinking in the field. 4) The supporting letters each cover in more detail one or a couple of aspects highlighted in the nomination letter. Making sure the nomination package is easy to digest is incredibly important, as you will see.Ā
So, once all the packages have been submitted, AGU goes through and ensures they are all compliant. Usually, by this time, the committee has been formed.Ā
While as chair, I couldn't do anything about the demographics of the nomination packages we received, I could do something about the committee itself. The president of SPA and I (and many others, too) felt it necessary to attempt complete and comprehensive representation. We did not quite achieve this, but we did okay for a 10 - 12 person committee. We had a 50/50 split of women and men. We had 50/50 white and non-white members on the committee. Most of the committee members either live or had lived for extended periods outside of the US. We had a decent distribution of more senior and middle/early-career individuals.Ā
You might think that we had covered all of our basis, but there was another area where representation was necessary. One area of bias that we also found in the fellows was with respect to their subfield. Specifically, our ionospheric and atmospheric colleagues were disproportionately underrepresented. Therefore we ensured that there were an equal number of committee members from each subfield.Ā
In short, ensuring representation on our committee had an impact - as did the nomination task force.Ā
Now we have our committee. We started with an unusual first step. During our initial telecon and periodically after, we discussed potential biases we may bring to the meeting and, eventually, the ranking of the nomination packages. The list of preferences that our committee identified and strived to mitigate and call out when observed were:Ā
Gender
Career level (retired/senior/expert vs. mid or even mid/expert/senior )
nationality/race
Extrovert vs. introvert (speaks more at conferences vs. doesn't speak up at conferences)Ā
Well-funded institute/country vs. not (e.g., able to be seen at conferences and visit other scientists vs. can't afford to travel as much)Ā
Large Mission participation vs. smaller projects such as CubeSats, rockets, balloons, ect. (more time, funding, and potential collaborations leading to more papers and co-authorships)
Experimentalists vs. theorist
Dependance on shortcut metrics (e.g. h-index, which moves away from discussing the substance of the publications). - sometimes, indices or data sets aren't appropriately cited once they become standards and are "always there" and "owned by the community."Ā
Bias towards our own subfields (this was mitigated by having equal representation on our committee)
People who publish/work in a small group and/or are often the first author vs. those who work more in large collaborative groups and/or mentor others to be first authors/PIs.
The Matthew Effect (A paper or result being attributed to the most prominent name, not the person who necessarily did the work or the first author. e.g.,Ā http://www.garfield.library.upenn.edu/merton/matthew1.pdf)Ā
The Matthew/Matilda effect (Where men tend to get the credit or more credit than women who did just as much or more of the work. E.g.,Ā https://journals.sagepub.com/doi/10.1177/030631293023002004Ā )Ā
This step was taken and found to be essential for the evaluation process as it brought forward and made present the biased issues we have faced in past committees. It led to acknowledgments and mitigations throughout the evaluation process about biases within the nomination packages and our discussions. This step has led us to put forward our most deserving and strongest nominees to become AGU fellows over the past few years.Ā
The discussion about our biases was followed by the discussion and creation of our evaluation criteria. This also led to further talks and mitigations on the biases that may be within our evaluation criteria. As you saw above, there is a bit of subjectiveness to them. We tried to identify and come to a consensus on what these criteria meant to each of us. The selection criteria were predetermined before looking at the nominations. We also routinely discussed the definitions and what accomplishments counted in the different categories.Ā +No predetermined order or weight to the bulleted definitions/criteria.Ā
What constitutes making a breakthrough and/or discovery:Ā
Breakthrough: an idea that, once accepted, allows others to frame ideas/approach problems differently and more effectively than before.Ā
Discovery: This seemed self-evident, so no clear definition or clarification was given.Ā
What constitutes innovation in disciplinary science, cross-disciplinary science, instrument development, or methods development:Ā
Enabling collaborations across many subfieldsĀ
Development of new instruments that have been successful in the field and lead to new* understandingsĀ
Development of new methods that other scientists have adopted and have led to new* understandings within the field.
Produced a data product or a method that is used on a routine basis, even if not correctly cited. (It is so routine that people have forgotten that this is either produced by someone or was not a standard product previously.)
*New: something that deviates enough from 'standard understandings' in any one field in the presented form, even if the process to arrive at 'new' happened through a series of gradual improvements/advancements
What constitutes sustained scientific impact:Ā
Something that has changed how other scientists approach a problem, perhaps on a smaller scope but cumulatively changes people's perceptions over time.
Enabled long-lasting collaborations leading to a significant impact within the field (one of the above criteria)Ā
Mentor a significant number of collaborators/scientists/students, enabling their development as researchers.
Produced continued excellent research throughout their career (a lifetime achievement award, so to speak)
Now that we have our selection criteria better defined, we can start looking at the packages. This year we took the following steps. Step 1) Discuss the different types of biases we may have, which were identified in the previous year's committees. Step 2) Agree on evaluation criteria and process. Step 3) Rank packages into the top, middle, and bottom thirds and discuss the average order. Step 4) Initial discussion about the first few packages - talk about how the evaluation process applies to the package. Make any changes to the evaluation definitions/process. Step 5) Finish discussing packages. Step 6) Rank the packages 1 - N. Look to see if there were any explicit biases we didn't catch or account for (were all the women rated lower, were all the solar nominees ranked at the bottom, etc. ). Step 7) Discuss and finalize rankings. Step 8) Go through and write up the top X selections. Step 9) Go through and write up supplemental selections. (the number of selections and supplemental selections is determined by AGU and based on the proportional number of members in AGU.) Step 10) Finalize criticism/write-ups of the rest of the packages so they can be improved for next year. All of this had to be done in about a month and a half. In SPA, we often have 20 - 30 nominations. Thus, we had approximately two 2-hour meetings a week to go through all of the packages.Ā
āSo what happens then. Our committee writes up a report. In this report, we provide summaries of the nomination packages and our support of them. This report goes up to the union level. We also provide summaries and comments to the nomination writers who wrote the packages we do not push forward to the union committee. At that point, it's a bit of a black box to me. I believe they follow a similar process to what we do. Then, a few months later, AGU announces who will be inducted into that year's class of fellows.Ā
Each year there are only a handful of spots for SPA members to become fellows. However, we have way more than a handful of incredible candidates each year. Therefore, if your nominee is not selected, the best advice is to keep submitting the nomination package.Ā
Hopefully, this has given you a glance into how the fellow's nomination committee works. Of course, we're always happy to answer questions.Ā