Some thoughts on the Research to Operations (R2O) path and the AULs

Conversations from those in the trenches experiencing growing pains.

I just got back from the UK space weather space environment workshop, and it was a fantastic meeting. I presented on the AULs and if you want to learn more about them- check out the paper at the Journal of Space Weather Space Climate or in my recent substack post. There were many talks, but somehow, it felt like there were also plenty of really important discussions. It may have been because the meeting started at 8 am most mornings, with coffee and tea, and ended officially at around 6 pm, leading into networking sessions until 8 pm. Then, everyone went to the hotel bar, where work discussions continued until 1 or 2 in the morning. It was about 15 - 17 hour days of all things space weather and space environment. It was fantastic. It was draining. I am tired. But I am also inspired!

While the days were long, several great conversations must be brought further into the public sphere and continued. And these can be summarized, at least partially, by something Colin Forsyth stated. (I'm paraphrasing, so Colin, I apologize if I get it wrong)... "We know how to do the R [research] , we know how to fund the R, but we don't seem to fund or do well with the '2O' [to operations]". We still have persistent issues with understanding the transition process, even when the people involved have been a part of operations or very close to the processes - and yes, this is a grand, overarching, and perhaps unfairly broad characterization, as there are clear successes within the R2O [Research to Operations] realm in Space Weather.

However, I think we can learn some critical things from these conversations—no matter where we are on the R2O pathway or how experienced you are with this process. Heck, every new project and new team learns new things and provides important lessons for future projects and teams. In this post, I’m stating things very firmly. But this is what I heard loud and clear from those at the UK Space Weather Space Environment meeting, and I hope we continue to hear more from others across the space weather community. However, I genuinely believe that for the R2O process to improve and become smoother/more routine, this must be a conversation within the entire R2O community.

So, the first theme of the week: 1) Know your end user (i.e., where your product will be housed) and make sure you are specific to your end user and the folks and processes along the way towards operations. Unlike research, most R2O projects are highly specific. For instance, I may be trying to predict when a solar flare will happen - sounds general and something that could be useful to so many people. But that isn't really what you are trying to do. You are trying to predict solar flares with a model their forecasters will use at NOAA Space Weather Prediction Center (SWPC). If you wanted to use the same model but at the UK Met Office, that's a different application. Now, you might ask why when both are operational centers. Very simply, there are different computers and different forecasters at SWPC and the Met Office. That means you will have a different operational environment and perhaps even different end visualizations or representations of the predictions that their forecasters want to use. Beyond having different compilers (I'm guessing - I don't really know), one may want it to come out in a table with confidence intervals using the quartiles. Perhaps the other may want a solar map and contours of the active region and confidence intervals at the 97th percentile, 50th, and 25th. Very similar, seemingly easy to change, but still very different products. Even if they wanted the exact same thing, had installed the same compilers, and worked on the same exact computer, there may still be differences - even though we all speak English, you may still need different training documents. And, perhaps more importantly, each country has different users of their weather services, different types of space weather due to their geology, magnetic latitude, different culture, and the like, so no matter how close to copy and paste you might get, you still need to be mindful of what their individual needs are and that there will almost certainly be differences.

Theme 2: You must be aware of the requirements throughout the entire process. Suppose you are working with a company that doesn't want to host and run the product. In that case, you need to meet both the company’s requirements and the requirements needed to run in the operational environment where the product will sit. So if the company doesn't care if it's written in IDL, Python, Matlab, or Oook (yes that is a coding language from Terry Pratchett’s disc world) - but the operational environment needs it to be written in C++, then you better write it in C++. If the company doesn't care what data is used as long as it provides them actionable information - but the operational environment can only use operational data from ground arrays in Scotland (don't ask me why - it might be the case), you better make sure your product only needs operational data from Scotland ground arrays to run. Now, suppose the company needs maps of the Moon, but the operational center you are working with can't produce or provide maps to this company. In that case, you need to have a different discussion. You have now found that your project and idea are really at only an AUL of 2 or 3 - i.e., you have a great idea and an interested end user, but at the moment, your project is not feasible or viable. Instead of working with that operational center, you may find a different pathway or spend the time and effort to work with the operational center to change their requirements and figure out the hurdles and roadblocks that need to be overcome to produce the maps.

Theme 3: Not everyone will reach the final destination, which is okay. It's not that everyone can't - it's that often not all are needed, and with limited resources, we can't afford for all to be transitioned. It might seem like a case of the rich getting richer - and perhaps the process of deciding which models are transitions can be improved... but currently, I don't believe that any operational center can afford to run all research models - and I don't know if there is any person or forecaster/decision-maker who would want to try to look at all of them in a high-stress situation and try to make sense of them. There is a point where more information causes confusion and no longer provides timely, helpful, and actionable information. Yes, that means most of us will not be chosen to have our models and products running in operational centers. But that doesn't mean my or anyone else's work doesn't matter. I've still added value to the scientific and operational community - if only because I keep finding ways not to do things - something I feel I am an expert in. But this work also pushes forward our fundamental understanding of what works and what doesn't, what is helpful in high-pressure situations and what doesn’t, what works in a timely manner and what doesn’t; this is all so important! Trying new methods and seeing how well they work is essential! Pushing out new ways of thinking and new methodologies is so important. So, it's okay if most of our community’s products and tools are not used in an operational setting.

Theme 4: We must work as a team during the '2O' process and really through the whole R2O process. If you are the ultimate end user but the project will run elsewhere, or you are the independent validator, or you have multiple groups working together - support each other. The most restrictive requirements for all points along the process should be the final set of requirements for the project. And no, you shouldn't ignore or try to get around them. Sure, ask why they're there. They might be able to be changed or that people will work to find funding if they understand why it might hurt the rest of the project and that it is valuable and worth the time, energy, and funds to remove these limitations. However, ultimately, the requirements are the requirements. If that makes the project no longer viable, then the project is no longer viable. That doesn't mean there is no time when the project becomes viable - it just isn't at this moment. And if the project isn’t viable, why continue wasting money, time, and effort on it when money and time are in very short supply these days? Use that time to transition viable projects, improve the current project to have better precision and time resolution, or run more efficiently; move on to another project and a new idea! Write up the current project, and include why it currently isn't feasible. Others may have solutions that will help push it forward or have a different customer who needs this tool as it is currently being developed. But ultimately, we have to support each other, especially if the product is for operations.

For example, you wouldn't go to a surgeon and say, ‘I know you need a knife, so I brought you a meat clever - that’s a knife, right? Why are you complaining?’ You might say, 'Hey, I know you need a knife, and you stated that you need a knife that allows you to be very precise. We think this laser knife might fit your needs; let me show you how it's more efficient- oh, it would have too many wires creating trip hazards and need its own generator to run, which you don’t have. Okay, that's good to know; I'll return a battery-operated laser knife for you.’ (Here you are in Phase 1 of the AULs; effectively, you are trying to complete AUL 2 and 3, where you determine the project’s requirements and if it is viable and feasible for your end user needs) Five months later, you return with, "So we made it smaller but couldn't make it workable with a battery. However, we talked with others, and they suggested the wires run along the ceiling instead of the floor. Does that help? Yes, but you don't have the funding to build the infrastructure for hanging the wires from the ceiling? Okay, let us know when you do, and we'll keep working on new solutions that will meet your needs." In the context of the AULs, you are still at an AUL of 2 or 3. You may have determined all the requirements that would put you at an AUL 2, but you haven’t yet determined that it is a viable and feasible solution for your end user. The project is still not viable, but there is progress to celebrate towards finding a solution that works for your end user. You have a new incredible product, but there are limitations towards implementation at that specific surgeon’s operating room. You and the end user can work on those solutions so your identified end user can eventually prosper from this new, better laser knife. As you advertise and document your work on this new laser knife, you may find that other hospitals can install and use the new device with minimal work.

Theme 5: Don't oversell your progress. If you say you are further along than you really are, that will only anger and frustrate people and ruin the trust between the different parties in your R2O processes. If you think you're at an AUL 8 because you run all the time on your system. Then, if you tell your new clients it's also at an AUL 8 for them; they expect it to run consistently on their systems starting day one. When they suddenly discover that the code must be rewritten entirely from Python to Java, the documentation and training documents must be translated from Greek to Spanish. It needs to be re-validated because you have to use new input data and new compilers - they are going to be sorely disapointed... they expected it was ready to be running on their systems immediately. Instead, they will be lucky if it is running in a few months to years. You have now hurt a potentially great relationship, and they are less likely to want to work with you or trust your work again in the future.

Theme 6: This one might be more for the customer, user, program officer, or reviewer - An AUL of X on one project with one end user might differ from an AUL of X on another project. If I have a forecasting tool for the radiation belts - one for researchers and one for NOAA SWPC - they are ultimately going to two very different places with very different requirements along the entirety of the AUL path. You can't then take the one headed to the CCMC or another researchers computing system and expect it to be ready to be transistioned to an operating center like NOAA. If you want to compare projects directly, you need to make sure they are projects attempting to fill the same need for the same user with all the exact same requirements.

I'm sure there are other lessons learned out there. It would be wonderful to hear them! Reach out, and let's continue this conversation!