hopefully solved merge conflict

paper/coverletter.tex

@@ -75,7 +75,7 @@ I also have a number of specific comments that should be addressed in a revision
 
   \verena{we need to think about this. Maybe: keep this here, but repeat it less often in the paper?.}
 
-The noise resulting from the measurement uncertainty inhibits the accuracy of any solar wind categorization. This is not intended as a critic on the available instruments, but rather to continue the trend of improvement towards higher wquality measurement already implemented by the respective modern instruments in Parker Solar Probe and Solar Orbiter.  
+The noise resulting from the measurement uncertainty inhibits the accuracy of any solar wind categorization. This is not intended as a critic on the available instruments, but rather to continue the trend of improvement towards higher wquality measurement already implemented by the respective modern instruments in Parker Solar Probe and Solar Orbiter.
  }
 
 \commentB{Line 16}{Suggest something like: based on properties that are, however, affected by processes during transport from the Sun.}
@@ -155,10 +155,10 @@ Line 185: This paragraph could be moved later. It seems unnecessary here - just
 \reply{The main reason is the SWICS dying. The data set provided by the SWEPAM team actually combines SWICS and SWEPAM data to mellow the effects of SWEPAM degrading. This also the the data set we used. }
 
 \commentB{Line 204}{ The study mentions multiple times that certain parameters are subject to transport effects, but on the other hand, all the observations are made at 1 AU, so the sampled plasma has suffered similar effects (such as expansion) such that signatures of the solar source may still be evident.}
-\reply{Thank you for pointing that out. We agree that a transport effect like expansion is not playing a role in our data set since it is all measured at 1AU. This is not true for SIRs. Not every plasma stream undergoes a SIR on the way to 1AU. \sophie{TO DO: andere transport effekte} }
+\reply{Thank you for pointing that out. We agree that a transport effect like expansion does not play a role in our data set since it is all measured at 1AU. Since we consider SIRs as a transport effect this argument does not hold here. Not every plasma stream undergoes a SIR on the way to 1AU. So we see the compression regions that a SIR might induce. For wave particle interaction we have the temperature as a proxy for those interactions. Another transport effect visible in our data set is collisions in the plasma. We do not observe a lot wave activity for slow solar wind at 1AU but we see that the plasma has undergone of collisions. For that we actively decided to include the proton-proton collisonal age (also called coulomb number ) as a parameter. The collisonal age has also been used to differntiate between solar wind types so it is not the same for all solar wind parcels.   }
 
 \commentA{What are the averaging times of the data sets used? This is mentioned later, but it would be natural to include this information in the data description.}
-  
+
 \commentB{Line 217}{Brackets around references.}
 \reply{Thank you for the helpful comment}
 
@@ -168,9 +168,11 @@ Line 185: This paragraph could be moved later. It seems unnecessary here - just
 
 Equation 1: Define the parameters in the equation. It's strange to have units in the denominator. There's a gap in the divisor line.
 
-Line 230: I don't really understand from this brief comment why ICMEs are excluded. In particular, why don't the same arguments (no temporal information and variable properties) apply say to high speed streams, which are also variable from event to event, and, as Figure 3 shows, even with the lack of temporal information, continuous high speed stream intervals can be identified.
+\commentB{Line 230:}{ I don't really understand from this brief comment why ICMEs are excluded. In particular, why don't the same arguments (no temporal information and variable properties) apply say to high speed streams, which are also variable from event to event, and, as Figure 3 shows, even with the lack of temporal information, continuous high speed stream intervals can be identified.}
+\reply{Thank you for the comment. We decided to exclude ICMEs because the main factor on recognizing ICMEs is the magnetic clouds, which cannot be detected with the features that we are currently using. The low temperature could be an indicator but this might not be a strong enough hint for kmeans to actually detect those regions. Furthermore within the ICME there are times where the ICME looks like regular solar find \sophie{TODO quelle hier für finden}. Thus we do not expect to be able to identify this regions as they are not fully understood yet. This is why exclude this from our work. \sophie{TODO Beispeil plot basteln wie das missclassifizieren passieren kann} We see why this can be quiet irritating for the reader so we diceded to emphisize this in our text.  }
 
-Line 234: Are ICME sheath regions also removed? i.e., does the preceding 6 hour buffer extend before the shock (not really necessary as there's no signal of the ICME ahead of the shock, except maybe some waves) or the ICME leading edge (in which case, prior to 6 hours could still include the sheath)?
+\commentB{Line 234:}{ Are ICME sheath regions also removed? i.e., does the preceding 6 hour buffer extend before the shock (not really necessary as there's no signal of the ICME ahead of the shock, except maybe some waves) or the ICME leading edge (in which case, prior to 6 hours could still include the sheath)?}
+\reply{The lists we use are not consistent with removing the sheath regions. So we diceided to try our best by including the six hour window to exclude this from our data set without loosing too much data. We cannot be certain that there is no sheath is always completly removed.  }
 
 Line 239: "To estimate their impact on our results..." - the details of the analysis haven't yet been presented, and some related knowledge is needed to understand many comments in this paragraph. I suggest moving this detailed discussion to later.
 
@@ -386,17 +388,17 @@ The study is potentially interesting but, in my opinion, the methodology, result
 
  \reply{
 
-   Yes, indeed we a treating the 7-parameter classification as a stand-in for the ground truth. We emphasized this more strongly in the revised manuscript in Section 2.1 and 3.1. 
+   Yes, indeed we a treating the 7-parameter classification as a stand-in for the ground truth. We emphasized this more strongly in the revised manuscript in Section 2.1 and 3.1.
 
   Disregarding the Monte Carlo computations for the moment, we require 127 runs of $k$-means, which takes on a \verena{add computer and running time}. This would increase considerably, if we added more than seven features.
 
   The additional (12700, i.e. 100 runs per input parameter combination) runs are solely for the purpose to derive errors that indicate the effect of the measurement uncertainty.
-  
+
   We are interested in a situation where the results of the solar wind classification depends on several solar wind parameters that are not independent from each other. We aim to understand how including or omitting certain solar wind parameters affects the resulting solar wind classification. In addition, we intend to analyse how the measurement uncertainty of the respective input parameters impacts the respective solar wind clustering.
 
    To this end, we require an approach with the following properties:
    \begin{enumerate}
-   \item\label{item:1} A quantification of the impact of different subsets of the input parameters on the final result, i.e. the solar wind classification, is needed. 
+   \item\label{item:1} A quantification of the impact of different subsets of the input parameters on the final result, i.e. the solar wind classification, is needed.
    \item\label{item:2} The input parameters are not assumed to be independent from each other. Combinations of two and more input parameters can be dependent of each other.
    \item\label{item:3} Uncertainties in the input parameters are propagated to the results while also allowing dependencies between input paameters.
    \end{enumerate}
@@ -404,7 +406,7 @@ The study is potentially interesting but, in my opinion, the methodology, result
    These requirements are related to several different fields of research:
    \begin{itemize}
    \item Feature selection. Feature selection aims to identify which features, i.e. input parameters, are most suitable for a given machine learning task. Here, the focus lies usually in reducing the number of features.  Feature selection approaches \citep{chandrashekar2014survey,lin2017selecting} can be put into three different approaches: Filtering methods, that regards each input parameter separatel and assign a feature raking based on measures like the correlation or the mutual information shared with the output signal. These methods are insufficient for our use case, since they assume independence of features (or replace single features with linear-cmobinations of individual features). The second group are wrapper methods \citep{javed2010feature,battiti1994using}, that systematically test each combination of input features and evaluate (again with measures like, for example correlation coefficients, mutual information or, in the case of classification tasks measures for the classification accuracy like the Rand score or Fowlkes-Mallows score). Wrapper methods are  very similar to our approach with the difference that in the context of feature selection the aim is to discard features that are less influential on the results whereas here we are interested in understanding the physical reasons why each features is more or less important for solar wind classification. The third group of feature selection method, embedded or search based-approaches \citet{khoshgoftaar2013survey, law2004simultaneous} are designed for situations where wrapper methods are not feasible, that is for large numbers of features in the order of several dozens or hundreds of features. For this situation more elaborate approaches are suggested that attempt to guide the search of the feature space with for example, evolutionary algorithms  \citep{al2005feature}. These loose information compared to the wrapper methods since only some of the subsets of features are explored and evaluated but are thereby applicable to much larger feature sets. We avoid the need for such more involved approaches by restricting our feature space to a number of features that is still small enough to be analyzed with a wrapper approach.
-   \item Sensitivity analysis. Sensitivity analysis deals with a different task than feature selection. Instead of quantifying which features are important for accurate results, sensitivity analysis quantifies how the uncertainties of the individual input parameters affect the results. Thus, in principle a sensitivity analysis approach can replace our Monte Carlo simulations which fulfil the same role. Most sensitivity analysis methods focus on local effects, that is, they are only interested of the influence of the uncertainties of single input parameters and assume that all input parameters are independent from each other. This is not applicable in pur case since solar wind parameters are known not to be independent from each other. Even most global sensitivity analysis methods assume independence of the different features from each other. \citet{lamboni2021multivariate} proposed a sensitivity analysis approach for dependent variables. This approach requires the assumption that each variable is multivariate Gaussian distributed which, as Fig. \verena{1d hists per parameter} show, this is not a fitting assumption in our case. The approach provides then cross-covariances between pairs of variables. This is also not sufficient for our purpose since we are also interested in combinations of more than two variables. Global sensitivity analysis approaches that attempt to quantify the effect of uncertainty of all input parameters are more appropriate for our use case. For example, \citet{decarlo2018efficient} approaches global sensitivity analysis of dependent variables based on importance sampling-based kernel regression and generalizes to dependent variables with a Bayesian approach. This work focuses on first order Sobol's indices. First order Sobol's indices again only consider the effect of varying each parameter individually (averaged over the uncertainties of the other parameters.) Thus, this approach - which is one of the most general available in the literature is not yet sufficient to replace the Monte Carlo simulations in our study. In the sensitivity analysis literature this is best resolved by global measures, like for example total Sobol's indices. These condense the effect of the uncertainties of all input parameters into a single value which is then intended to be interpreted in comparison to first-order (single input variable effects) and second-order (effects of pairs of input parameters) Sobol's indices. Sobol's indices are calculated based on randomizing all input parameters multiple times to assess the effect of their uncertainties. Thus, these approaches are very similar to our Monte Carlo simulations. However, the guidelines in the literature suggest to use several hundreds or thousands of randomized input parameter combinations. Thus, allowing these guidelines would considerable increase the computational cost of our method. 
+   \item Sensitivity analysis. Sensitivity analysis deals with a different task than feature selection. Instead of quantifying which features are important for accurate results, sensitivity analysis quantifies how the uncertainties of the individual input parameters affect the results. Thus, in principle a sensitivity analysis approach can replace our Monte Carlo simulations which fulfil the same role. Most sensitivity analysis methods focus on local effects, that is, they are only interested of the influence of the uncertainties of single input parameters and assume that all input parameters are independent from each other. This is not applicable in pur case since solar wind parameters are known not to be independent from each other. Even most global sensitivity analysis methods assume independence of the different features from each other. \citet{lamboni2021multivariate} proposed a sensitivity analysis approach for dependent variables. This approach requires the assumption that each variable is multivariate Gaussian distributed which, as Fig. \verena{1d hists per parameter} show, this is not a fitting assumption in our case. The approach provides then cross-covariances between pairs of variables. This is also not sufficient for our purpose since we are also interested in combinations of more than two variables. Global sensitivity analysis approaches that attempt to quantify the effect of uncertainty of all input parameters are more appropriate for our use case. For example, \citet{decarlo2018efficient} approaches global sensitivity analysis of dependent variables based on importance sampling-based kernel regression and generalizes to dependent variables with a Bayesian approach. This work focuses on first order Sobol's indices. First order Sobol's indices again only consider the effect of varying each parameter individually (averaged over the uncertainties of the other parameters.) Thus, this approach - which is one of the most general available in the literature is not yet sufficient to replace the Monte Carlo simulations in our study. In the sensitivity analysis literature this is best resolved by global measures, like for example total Sobol's indices. These condense the effect of the uncertainties of all input parameters into a single value which is then intended to be interpreted in comparison to first-order (single input variable effects) and second-order (effects of pairs of input parameters) Sobol's indices. Sobol's indices are calculated based on randomizing all input parameters multiple times to assess the effect of their uncertainties. Thus, these approaches are very similar to our Monte Carlo simulations. However, the guidelines in the literature suggest to use several hundreds or thousands of randomized input parameter combinations. Thus, allowing these guidelines would considerable increase the computational cost of our method.
      We computed  total Sobol's indices based on the same randomized input parameters as used in out Monte Carlo simulations. The results are shown in Figure ... \verena{todo}
      Thus, we can replace one part of our approach, the Monte Carlo simulations for each input parameter combination, with a standard sensitivity analysis tool. However, this would not decrease but rather increase the computational effort of our study without much gaining additional information. The core part of our study, the importance of each input parameter for solar wind classification, cannot be addressed by sensitivity analysis.
      \item Feature importance and explainability methods \citep{solorio2020review,kumar2014feature,alelyani2018feature}. The third research area related to our approach is feature importance (which aims at evaluating the importance of individual features on the final result necessarily without eliminating the features) and explainability In particular explainability is a rather young research field that has gained importance with the rise of successful deep learning applications. As a result, these methods are usually tailored to considerably more complex cases than $k$-means and focus on large numbers of features. Thus, again many approaches focus on explaining individual features (for example, \citet{zhuang2019decision}, or  with SHAP values \citet{kumar2020problems, pudil1995feature}). Similar to embedded or search-based feature selection methods, the available explainability approaches attempt to deal with the situation where wrapper-like approaches are not feasible. Thus, they provide only explainability for a subset of features. This again is not helpful for our purposes. These methods tend to be less informative than our wrapper-like approach and have a higher  computational cost.
@@ -422,9 +424,9 @@ The study is potentially interesting but, in my opinion, the methodology, result
 %such as sequential forward or backward searches, floating
 %search, beam search, bidirectional search, and genetic
 %search have been suggested [9], [33], [47], [63]''
- 
+
  % using unsupervised clustering as feature selection: \citet{mitra2002unsupervised}
- % promising reviews: 
+ % promising reviews:
 
 \verena{todo:  exaplainable AI for k-means: \citet{frost2020exkmc}}