The performance of neural networks has dominantly been measured using metrics such as classification or segmentation accuracy, precision, recall, and the area under the ROC curve. However, recent studies have shown the dangers of only considering such globally-averaged metrics (Shit et al. 2020; Mosinska et al. 2018; da Silva et al. 2022) that provide only an aggregated, summarized, view of the performance of machine learning algorithms on datasets with sometimes millions of images. Such an approach may hide important biases of the model (da Silva et al. 2022). For instance, for medical images, a 95% accuracy is usually considered a good performance. But what about the remaining 5%? It is usually unrealistic to expect models to reach 100% accuracy, but the samples that are not correctly processed by a neural network may hide important biases of the model. These concerns led to the definition of new approaches and metrics that can aid the interpretation of black box models (Chaddad et al. 2023).

For medical image segmentation, the detection of relevant structures is usually only the first step of a more elaborate procedure for measuring relevant properties such as size (de P. Mendes et al. 2021), regularity (Pérez-Beteta et al. 2018), length (Paetzold et al. 2021; Freitas-Andrade et al. 2022), and curvature (Krestanova et al. 2020; Freitas-Andrade et al. 2022) of the imaged structures. Therefore, systematic segmentation mistakes might lead to undetected errors when characterizing samples for clinical diagnoses (Reinke et al. 2021) and research purposes (Shit et al. 2020). An important cause of such systematic errors can be the presence of samples with characteristics that occur with low frequency in a dataset. This can happen due to additional, unexpected, noise during image acquisition, variations in tissue staining, image artifacts, or even the presence of structures that are anatomically different than what was expected. Assuming for illustration purposes that the data is normally distributed, a machine learning model having good performance around the peak of the distribution will tend to have good average accuracy measured for the whole dataset, even if it cannot correctly classify or segment images that are around the tail of the distribution (Gupta et al. 2019), which might be important for downstream analyses.

Segmenting the vasculature in tissue samples tends to be particularly challenging since the appearance of blood vessels can change significantly depending on tissue preparation and imaging protocols. In addition, in most cases, the global shape of the vasculature can be very different among the samples. We argue that blood vessel segmentation methodologies should have good performance, or even be directly optimized, on both prototypical and atypical samples. This focus can lead to models that are more robust to samples located in a sparsely populated region of the feature space of the dataset. In addition, it might also lead to models that generalize better to out-of-distribution samples as well as to new datasets. With these aspects in mind, we create a new dataset which we call the Feature-Mapped Cortex Vasculature Dataset (VessMAP). The dataset is designed to be as heterogeneous as possible by including samples having very different characteristics from each other. To this end, we use a simple and intuitive sampling methodology to select a subset of 100 images from a non-annotated base dataset containing 18279 image patches. The selected samples were then manually annotated with pixel-wise accuracy.

The dataset allows the creation of training and validation splits with images having different characteristics, such as contrast and blood vessel density. We show that different splits of the VessMAP dataset lead to very different training and validation results. As illustrated in Figure 1, the performance on the validation set can be dissimilar depending on the samples used for training a neural network. Thus, we expect the dataset to be useful for the development of new segmentation algorithms that are robust under distribution shifts of the data as well as for the validation of novel few-shot and active learning approaches.

The main contributions of this work are listed as follows:

• A new dataset, VessMAP, is made available to aid the development of blood vessel segmentation algorithms;

• It is shown that VessMAP has high variability compared to other popular datasets in the literature, leading to respective large variations of performance when training data is scarce;

• Specific splits of the dataset are provided for testing techniques that improve the generalizability of blood vessel segmentation algorithms.

Related works

Table 1 shows a summary of the main blood vessel datasets used in the literature as well as some recently published datasets. Most of the datasets have images from the retina. Few datasets are associated with microscopy images. More importantly, to our knowledge, none of the datasets were specifically designed to maximize the diversity of the samples. The diversity on some datasets tends to come as a proxy from the inclusion of healthy and abnormal tissue. For instance, samples in the DRIVE dataset contain diabetic retinopathy, which generates abnormal characteristics in blood vessels and image artifacts such as exudates that are not related to blood vessels. Still, most blood vessels tend to have a well-defined geometry and texture in all samples of the dataset. Thus, it becomes a simple task for a segmentation algorithm to generalize to new unseen samples from the same dataset. It is not surprising that many methods can reach an accuracy larger than 0.94 on the DRIVE dataset (Kovács and Fazekas 2022).

Summary of important blood vessel datasets on the literature. *CTA: Computed tomography angiography. MRA: Magnetic resonance angiography. $\mu$CTA: Micro-computed tomography angiography. {#tab:datasets}

Regarding microscopy images, all datasets found by our survey include very few samples. Usually, very large 3D volumes are annotated in a semi-supervised fashion. They contain large amounts of vessels, but represent a single individual and image acquisition procedure. Therefore, most vessels have similar appearance and it becomes difficult to measure the generalization capability of segmentation methods. With these limitations in mind, we created a dataset that was specifically designed to include blood vessels having very different characteristics.

The creation of the dataset involved the application of a method for selecting relevant samples for annotation. A concept that is similar to the used methodology is the so-called coreset (Yu et al. 2024). The aim of a coreset is to select a subset of samples that can optimally represent the whole dataset. Many different methodologies and criteria were developed for defining relevant coresets (Zheng et al. 2019; Adhikari et al. 2021; Guo et al. 2022). Indeed, the subset defined by our methodology can be associated with a coreset, but in our case, the aim of the methodology and the approach used differs markedly from the usual definition of a coreset. The aim of our methodology is not focused on accurately representing the underlying distribution of the data or preserving the accuracy of a machine learning algorithm, but on providing a relevant dataset for training machine learning algorithms while avoiding the underrepresentation of atypical samples. In addition, many coreset methodologies use a surrogate neural network to estimate latent features or to estimate a degree of uncertainty about each sample, while our methodology is more general in the sense that any set of features obtained from the samples can be used. Furthermore, many related studies consider a similarity metric for selecting relevant samples (Zheng et al. 2019; Adhikari et al. 2021), which is a degenerate metric and, therefore cannot provide a full representation of the data distribution.

Materials and methods

In the following, we describe the methodology used for creating the VessMAP dataset. The methodology is illustrated in Figure 2 and can be divided into three steps: (a) acquisition of the base data from different experiments; (b) characterization of the base dataset according to important morphometry features; (c) selection of samples that uniformly covers the mapped feature space. The base non-annotated dataset used for selecting relevant samples for VessMAP is described in Section 3.1. The sampling methodology used for selecting appropriate samples from the base dataset is described in Section 3.2.

Blood vessel microscopy base images

We start from a collection of 2637 confocal microscopy images of mouse brain vasculature. The images were acquired under different experimental conditions in different works published in the literature (Lacoste et al. 2014; Gouveia et al. 2017; Ouellette et al. 2020). Conditions include control animals, animals that have suffered a deletion of chromosome 16p11.12, animals that have experienced sense deprivation or sense hyperarousal, samples from stroke regions, and also from different stages of mouse development. The images have sizes from $1376\times 1104$ to $2499\times 2005$ pixels, totaling around 3.8GB of data.

The dataset is interesting because it has a considerable variety of characteristics of blood vessels. In addition, the images represent samples obtained from hundreds of different animals and experimental conditions. This makes it an excellent dataset for training machine learning algorithms for blood vessel segmentation. However, training supervised algorithms requires the manual annotation of the blood vessels.

After annotating a few samples, we estimated that each image in the dataset takes roughly 12 hours to fully annotate. Therefore, it is unfeasible to annotate the whole dataset, and a subset of samples needs to be selected. Our objective was to select a diverse set of samples containing both prototypical and atypical samples, so that it would be possible to create useful training and validation splits for quantifying the performance of segmentation algorithms under challenging distribution shifts between the splits. To this end, a sampling methodology was developed to select appropriate samples.

Sampling methodology

Each image in the base dataset may include illumination inhomogeneities, changes in contrast, different levels of noise, as well as blood vessels having distinct characteristics (e.g., caliber, tortuosity, etc). Thus, from the original dataset, we generated a new set of images, each having a size of 256$\times$256 pixels. These smaller images were generated by extracting 256$\times$256 patches from the original images. As shown in Figure 3, seven regions were extracted from each image. The seven regions were extracted in key areas of each image, with four windows in each of the corners of the image, a central window, and two windows at random positions. The latter two may overlap with the other windows. Windows that did not contain a satisfactory number of blood vessel segments were removed. The total size of the resulting dataset is 18279 images. This new dataset was used in the remainder of the sampling procedure.

The methodology developed to sample relevant images has three steps: 1) dataset mapping to a feature space, 2) generation of a discrete representation of the feature space, and 3) selection of points from the feature space representation. We explain each of these steps in the following sections.

Dataset mapping

We represent the dataset as $D = \{\delta_1, \delta_2, ..., \delta_n\}$ where $n$ is the number of samples. Given a function $f: \delta_i \to \vec{p_i}$ that maps a sample $\delta_i$ to a vector $\vec{p_i}$ with dimension $d$, the dataset is mapped to a feature space as a $n \times d$ matrix, which we call $D_{\textrm{mapped}}$. The function $f$ represents a set of characteristics measured from the samples. Each line of matrix $D_{\textrm{mapped}}$ therefore represents the features of a sample $f(\delta_i)$.

Given that the images from our base dataset were used in previous works, each sample has a respective segmentation that was obtained using a semi-supervised methodology. This methodology is based on the adaptive thresholding of the original images, where the threshold was selected manually for each image. The full details of the segmentation procedure are described in (Freitas-Andrade et al. 2022). Using the semi-supervised segmentation, the following features were used to characterize the samples: blood vessel contrast, level of Gaussian noise, blood vessel density, and medial line heterogeneity.

The blood vessel contrast is related to the average difference in intensity between the vessels and the background of the image. The greater the contrast, the easier it is to detect the vessels. It can be measured using the original image of the vessels and the respective semi-supervised segmentation containing the pixels belonging to the vessels. The contrast was calculated as

\begin{equation}
    C = \frac{\bar{I}_v}{\bar{I}_f},
    
\end{equation}

where $\bar{I}_v$ and $\bar{I}_f$ are the mean intensities of, respectively, the pixels belonging to the blood vessels and the background of the image.

The signal-to-noise level of the images can be estimated in different ways. We investigated different definitions and used the method that was the most compatible with a visual inspection of the images. The method proposed in (Donoho and Johnstone 1994) was used. It assumes a noise with normal distribution and uses wavelets to identify the most likely standard deviation of the noise component. To prevent the method from capturing vessel variation, only the background of the image was used for the estimation.

Blood vessel density is defined as the total length of blood vessels in an image divided by the image area. To do this, we first applied a skeletonization algorithm to extract the medial lines of the vessels (Palàgyi and Kuba 1998). The total length of vessels was then calculated as the sum of the arc-lengths of all vessel segments.

The last metric, which we call medial line heterogeneity, measures the illumination changes in the vessel lumen. To calculate this metric, we first blurred the image using a Gaussian filter with unit standard deviation to remove extreme values. The medial line heterogeneity was then calculated as the standard deviation of the pixel values along the medial lines of this blurred image. The medial lines considered are the same ones used for the blood vessel density metric.

We observed that the medial line heterogeneity tended to be correlated with the average intensity of the blood vessels. In order to remove this dependency, the medial line heterogeneity, as well as the average intensity of the medial lines, were calculated for all images in the dataset. Then, a straight line fit $h_m = a*m + b$ was applied to the calculated values, where $m$ is the average intensity and $h_m$ is the expected medial line heterogeneity associated with $m$. Next, a normalized medial line heterogeneity was defined as $\tilde{h} = h - h_m$, where $h$ is the medial line heterogeneity calculated for an image.

These specific features were used because they can significantly impact the quality of the morphometry assessment of the cortex vasculature. For example, images with low contrast and/or high noise levels are expected to be more challenging to be accurately segmented. Conversely, images with a larger amount of blood vessels and high medial line heterogeneity provide intricate topology and texture to segmentation algorithms. Other features, such as blood vessel tortuosity, the density of branching points (bifurcations), and additional noise estimators, were considered. However, we disregarded strongly correlated features for our final data selection. The four remaining metrics mapped the base dataset to a 4-D feature space. As mentioned before, the dataset contains 18279 images. Hence, the whole dataset was mapped to a matrix $D_{\textrm{mapped}}$ having size $18279\times 4$.

Feature space discretization

A regular grid was defined in the 4-D feature space, and each data point was mapped to the nearest point in this grid. Figure 4 illustrates this procedure. For creating the grid, it is useful to first normalize the values of $D_{\textrm{mapped}}$ to remove differences in the scale of the features. Thus, each feature was normalized to have zero mean and unit variance. Then, the discretization was done by defining a scale $\nu$ that sets the size of each grid cell, and calculating

\begin{equation}
    
    \displaystyle D_{\textrm{grid}} = \left\lfloor\frac{D_{\textrm{mapped}}}{\nu}\right\rfloor,
\end{equation}

where $\lfloor . \rfloor$ represents the floor function. As shown in Figure 4(c), this operation ensures that each value of $D_{\textrm{grid}}$ lies within a regular grid. Note that, as a consequence of undersampling, we expect multiple data points to fall in the same grid position; this is one of the key properties of the method that will allow a uniform sampling of the data. A value of $\nu=10$ was used since we observed that it provided a good balance between grid sparsity and variability.

After the feature space discretization, we generated a sparse set of points representing an estimation of the possible values that can be obtained in the feature space. We call this set the sampling set of the feature space. This procedure works as follows. A 4-dimensional discrete hypersphere $S$ with radius $r$ (in grid units) centered on each data point is defined. This hypersphere is translated to each data point position. The union of the calculated hypersphere positions of all points defines the sampling set $D_{\textrm{sset}}$. The general appearance of $D_{\textrm{sset}}$ is depicted by the blue points of Figure 5. The hypersphere radius used for creating the VessMAP dataset was $r=4$.

Uniform selection of points

The final step of the method is to select the samples to be manually annotated. The samples are selected by first drawing a set of points from the sampling set $D_{\textrm{sset}}$. As illustrated in Figure 5, we draw from $D_{\textrm{sset}}$ $k$ points with uniform probability (green dots in Figure 5). For each point drawn, the closest data sample is identified using the Euclidean distance. If the same data sample is obtained more than once, a new point is drawn from $D_{\textrm{sset}}$ until $k$ unique data samples are obtained. The final set of data samples (orange stars in Figure 5) is represented as $D_{\textrm{sampled}}$.

A uniform sampling of $D_{\textrm{sset}}$ allows the selection of prototypical and atypical samples from the dataset with equal probability. Nevertheless, a single realization of the sampling may lead to distortions, such as the selection of many samples at similar regions of the space or the creation of large regions with no samples selected. This is due to random fluctuations in the sampling process. To amend this, we define a metric called Farthest Unselected Point (FUS) that punishes sampled subsets with large gaps between the selected points.

Let $D_{\textrm{sampled}}$ be the set of sampled data points from $D_{\textrm{grid}}$, and $\neg D_{\textrm{sampled}}$ the set of points from $D_{\textrm{grid}}$ that were not selected in the sampled subset. For each data point in $\neg D_{\textrm{sampled}}$, the Euclidean distance to the closest point in $D_{\textrm{sampled}}$ is obtained. The FUS metric is defined as the largest calculated distance among all points in $\neg D_{\textrm{sampled}}$. Sampled subsets leading to low values of the FUS metric should be preferred since it avoids the creation of large regions of the feature space with no samples. In our experiments, we found that minimizing FUS for 1000 different subsets covered a good amount of subset possibilities.

We decided to select $k = 100$ images for annotation. Also, to avoid data leakage, an additional restriction that prevented the selection of samples from the same image was used.

Results

Dataset heterogeneity

The sampling approach used to generate the VessMAP dataset should lead to a heterogeneous set of samples. It is difficult to properly measure the heterogeneity of the dataset because it would involve the estimation of the probability density function of the original data, which is not a trivial task and can be influenced by the choice of parameter values. However, it is clear that the method should naturally lead to a uniform selection of the samples. This is so because the set $D_{\textrm{grid}}$ (defined in Section 3.2.2) represents an estimation of the domain of the probability density function of the data, and this domain is sampled uniformly.

One approach to illustrate the characteristics of the sampled images is displayed in Figure 6, which shows histograms of the four considered features for both the full dataset and the sampled subset. The histograms of individual features are not expected to be uniform since they represent a projection of the original data into one dimension. Still, it can be seen that the histograms of the sampled set tend to represent a slightly flattened version of the histograms of the original data, indicating that a larger priority is being given to atypical samples when compared to the original distribution.

A more robust way of visually checking the sampled subset is to visualize the data using Principal Component Analysis (PCA). Using PCA, the original 4-D data can be projected into 2-d with optimal preservation of the variance (Gewers et al. 2021). Figure 7 shows the PCA projection of the data. The four plots included in the figure represent the same projection, but the points are colored according to the different features used to characterize the images. The selected samples are shown in red. It can be noticed that the sampling methodology selects a subset of images that uniformly covers the distribution of the data. Furthermore, as also suggested by the histograms in Figure 6, the sampling was capable of covering the full range of values of every considered feature.

The subset of images selected by the method (the VessMAP dataset) is shown in Figure 8. The subset indeed contains a heterogeneous set of images covering many different values of the considered features (e.g., low contrast, high vessel density, etc). For instance, some of the samples in the dataset come from animals who suffered hemorrhagic strokes. These samples are very different from the typical samples contained in the base dataset, and they would be largely underrepresented if a sampling following the data distribution was performed.

We manually labeled each of the 100 images and made the dataset publicly available (Silva et al. 2023). To account for inter-annotator variability, 20 samples were labeled by two annotators. The Dice similarity score between the two annotators is 0.8780. We identified that most disagreement between annotators lies in delineating the vessel borders, resulting in mildly different blood vessel calibers. With that in mind, we also calculated the centerline Dice (clDice) (Shit et al. 2020) between both annotations, which provides a metric of how well the annotators agreed about the topology of the blood vessels. A clDice of 0.9556 was obtained, indicating a good agreement between annotators regarding the preservation of continuities and bifurcations. As a comparison, the annotations of the DRIVE dataset’s (one of the most used blood vessel segmentation datasets) test set have a Dice similarity of 0.7881 and a clDice of 0.7634.

The VessMAP repository includes manually annotated binary labels, their skeletons (calculated by the Palágyi-Kuba algorithm (Palàgyi and Kuba 1998)), and the metrics for each sample (as described in Section 3.2.1) – which were calculated using the manual annotations. We verified that the metrics calculated from the manual annotations have a strong correlation with the metrics calculated using the labels obtained from the semi-supervised segmentation algorithm. This evidences the quality of the algorithm in providing useful metrics to map the dataset into a feature space. We expect the VessMAP dataset to be useful for future studies regarding the influence of image and tissue characteristics on the generalization capability of segmentation algorithms.

Neural network performance on VessMAP splits

Many current methods for semantic segmentation of biological images involve neural networks (Xu et al. 2024). Convolutional Neural Networks (CNNs) have been successfully used for segmenting biological structures for many years, especially after the adoption of encoder/decoder architectures, such as the original U-Net (Ronneberger et al. 2015) and its variants (Isensee et al. 2021; Galdran et al. 2022b; Fhima et al. 2024). Recently, the emergence of the Transformer (Vaswani et al. 2017) architecture allowed remarkable performance in tasks such as the development of Large Language Models (OpenAI2024?; Grattafiori et al. 2024), speech processing (Radford et al. 2022), multimodal learning (Zhang et al. 2024), and drug discovery (Raza et al. 2023). Regarding image processing, Transformers don’t make assumptions about the relationship between the pixels of an image. This lack of inductive bias makes it harder to train a model from scratch in scenarios of scarce data, and it is usually necessary to pre-train a Vision Transformer (ViT) (Dosovitskiy et al. 2021) in large datasets such as ImageNet (Deng et al. 2009). Since we aim to evaluate the VessMAP performance using small training sets, we chose to use CNNs, which tend to perform better than ViTs for medical image segmentation with limited data (Shamshad et al. 2023).

To evaluate the potential of VessMAP to generate data splits that are challenging for neural networks, we generated eight different splits based on the features used for creating the dataset: blood vessel density, contrast, medial line heterogeneity, and noise estimation. For each feature, we selected 20 of the samples with the lowest and highest values and trained a segmentation CNN using two configurations: (i) training with samples that have the lowest feature values –lowest split– and evaluating with samples that have the highest feature values –highest split–, and (ii) training with the highest split and evaluating with the lowest split. We chose to use 20 images because it is a similar number to common split sizes used for well-known blood vessel datasets, such as DRIVE (Staal et al. 2004), STARE (Hoover et al. 2000), and CHASEDB1 (Fraz et al. 2012b). The idea behind this experiment is to test whether we can use VessMAP to generate splits that challenge the generalization capability of CNNs.

For this experiment, we used the CNN architecture illustrated in Figure 9. This architecture encodes the input data through a series of residual blocks (He et al. 2016), concatenates the resulted feature vector with the activations from the first convolution operation (similar to a U-Net (Ronneberger et al. 2015)), and decodes the feature vector with a single residual block. For each training/evaluation split, we trained the network for 1000 epochs. The training was carried out using the Cross-Entropy as the loss function, the Adam optimizer (Kingma 2014), and a polynomial learning rate scheduler (power = 0.9) – which decays the initial learning rate (0.01) almost linearly.

Figure 10 presents the loss curves of the eight training setups (two split configurations for each metric). We evaluate the distance between the training and validation loss curves, $\delta$, as a metric of how well the CNN generalized for out-of-distribution data. Only the first 200 epochs are plotted because $\delta$ did not change significantly during the remaining epochs. We define $\delta$ as the difference between the training loss and the validation loss at a specific epoch. It is also worth noting that the loss curves were smoothed using an exponentially weighted moving average in order to reduce the natural variance of the loss values and obtain a more precise $\delta$ value. Here, we calculate $\delta$ at epoch 150.

For the splits using the contrast feature, when training with samples having low contrast (Figure 10(b)), the network generalizes well for new data having high contrast. This behavior can be attributed to the fact that high-contrast images are less challenging and, if the network learns how to properly segment low-contrast images, it tends to handle well high-contrast images. The opposite behavior occurs when we invert the training and validation sets. When training with high-contrast images (Figure 10(a)), the CNN could not generalize towards low-contrast data, yielding a negative $\delta$. The same behavior can be observed for the blood vessel density splits. Notice that a negative $\delta$ indicates that using specifically low-density samples as the training set yields low generalization towards more dense images. The exact opposite happens when training with denser samples. It can also be noticed that the training splits using the highest noise and medial line heterogeneity values resulted in similar training and validation loss curves. This indicates that, although these splits are unbalanced regarding feature values, the samples are still diverse enough to allow good generalization. The results of the experiments with different splits are summarized in Table 2.

$\delta$ values for the experiments with different splits generated using the VessMAP metadata. Each experiment name depicts the feature used to split the dataset, followed by (h) if the training set had the largest values of the feature or (l) if the training set had the lowest values. Negative $\delta$ values suggest poor generalization and were marked as bold. *Skel. het: skeleton heterogeneity. {#tab:deltas}

Considering that the VessMAP images are diverse, another approach for generating challenging training and validation splits is to select samples that are far apart in the feature space. To do so, it is first necessary to identify a distance threshold above which the training and validation sets can be considered to be adequately separated in the feature space. We calculated this threshold by generating 10000 random splits of 20 training and validation images and obtaining the smallest Euclidean distance between all pairs of points of the two sets for each split. Then, we analyzed the histogram of the calculated distances and considered that two sets are far apart if their distance is larger than a threshold of $t=0.7$, which corresponded to approximately $2.4\%$ of the randomly drawn splits.

One of the identified splits containing highly distinct samples is highlighted in blue and red in Figure 8. By using the same CNN and hyperparameters as the previous experiments, an average validation Dice score (Dice 1945) of $0.824\pm0.008$ was obtained for 100 training runs using the identified split. When the training and validation sets were swapped, a Dice score of $0.892\pm0.006$ was obtained. This result is in agreement with our previous experiment depicted in Figure 10, as the images of the training set present high contrast, low density, and low medial line heterogeneity.

For reference, we ran a similar experiment on the retinography images from the DRIVE, STARE, and CHASEDB1 datasets. For the DRIVE dataset, a Dice of $0.803\pm0.002$ was obtained using the official split of the dataset, and a Dice of $0.791\pm0.003$ was obtained when the training and validation sets were swapped. Since the STARE dataset does not have an official training and validation split, we trained the network for 100 randomly drawn splits and calculated the average performance difference between each split and its swapped counterpart. An average Dice difference of $0.029\pm0.024$ was obtained, with a maximum difference of 0.12. This same approach was applied to the CHASEDB1 dataset, where an average Dice difference of $0.01\pm0.008$ was obtained, with a maximum difference of 0.03. Note that the CHASEDB1 and DRIVE datasets got similar results regarding the performance difference between splits. The difference in Dice values obtained for the splits of the VessMAP dataset, $0.892-0.824=0.068$, was significantly larger than the maximum values obtained for the CHASEDB1 and DRIVE datasets.

Interestingly, our experiments show that the STARE dataset contains the split with the largest performance difference among all datasets. Indeed, some samples of the STARE dataset have very distinct appearances when compared to the typical characteristics of the dataset. Thus, in addition to VessMAP, STARE also seems to be a suitable dataset for evaluating network generalizability on blood vessel segmentation tasks. Nevertheless, the higher number of images in VessMAP compared to STARE and the feature metadata enables the definition of training setups with a greater number of training/validation splits.

It is worth mentioning that the Dice values obtained in our experiments with the fundus images are slightly lower than the state-of-the-art results for these datasets (Cervantes et al. 2023). This is mainly because no preprocessing and data augmentation were applied in order to match the training setup applied to VessMAP.

The heterogeneity of the VessMAP dataset is particularly useful for developing robust segmentation models when the training data is scarce. To show this, we ran a series of experiments using only four images for the training set. This replicates situations where, for instance, an active learning method suggests a small set of images for annotation, or on interactive segmentation scenarios where only a small set of blood vessels might be annotated. A neural network was trained on 4 randomly selected samples from the VessMAP dataset and the performance was measured on the remaining 96 samples. The same process was repeated 100 times using different sets of samples. To show that the overall results of our analyses are not dependent on a specific network architecture or training parameters, we replicated the same model and training approach used in (Galdran et al. 2022a). Specifically, the $\phi_{3,8}$ U-Net model containing 6 convolution layers in the encoder was used. The training protocol was also replicated with the exception of the cyclical learning rate scheduler, which was replaced by a polynomial scheduler. The batch size was also changed from 4 to 2 since the training set has only four samples.

For each image of the dataset, we measured the Dice scores obtained for trainings runs in which the image was not included in the training set. The result for all images is shown in Figure 11(a). It is clear that, for most images, the samples used for training the network have a large influence on the quality of the segmentation. The training set can lead to either very good segmentations or to segmentations that are of very poor quality.

For comparison, we repeated the same experiments for the DRIVE, STARE, and CHASEDB1 datasets. The results are shown in Figures 11(b)-(d). The variation observed for these datasets is much smaller compared to the VessMAP dataset. That is, four training samples are usually enough to obtain good and robust performance on the remaining samples. Thus, methods developed to work on scarce data annotation regimes might trivially result in good, low-biased performance when tested on these datasets. The same trend was observed for the area under the ROC curve (AUC) and average precision performance metrics (Figures 13 and 14 of the supplementary material).

To quantify the performance variation observed, the difference between the highest and lowest Dice score obtained for each sample was calculated. The average difference for all samples was then calculated for each dataset. The values are shown in Table 3 and confirm the high performance variations observed for VessMAP.

Influence of the training set on the generalizability of a model. For each dataset, models were trained on 100 different training sets containing 4 images each and were validated on the remaining images. The difference between the maximum and minimum Dice scores obtained for each sample across all runs was calculated and averaged over all samples. The standard deviation of the differences is also shown to provide a reference regarding the degree of variation observed among samples. {#tab:dice_diff}

In Figure 12 we show example segmentations obtained for the sample having the median standard deviation of Dice scores among all runs, that is, a sample with a typical variation of Dice scores observed in the dataset. The training set can lead to many missing blood vessels, to an oversegmentation of the vessels, to the presence of spurious holes as well as to discontinuities on the vessels. For many other samples, we also observed a large number of false positives.

With these analyses, we suggest two applications of the VessMAP metadata. First, one can generate splits that challenge the generalization capacity of a neural network, yielding negative $\delta$ during training. This kind of split can be used to test or develop new approaches to handle datasets having very distinct samples. In a similar fashion, splits that have positive $\delta$ can be used for developing new active learning methods, where it is useful to identify challenging samples for training networks so as to obtain low validation loss. In both situations, an ideal model should converge the training and validation loss curves, resulting in $\delta \approx 0$.

To aid the development of such methods, we provide official splits of the dataset containing training sets that lead to vastly distinct inference performances. The splits are shown in Table 4. For calculating the performance of each split, each of the 100 training runs was repeated 5 times using different seeds for the random number generator used during training.

Relevant training splits of the VessMAP dataset. Each row shows the average Dice score obtained on the remaining 96 samples when training a neural network using the four images indicated in the training set column. The standard deviation obtained across five repetitions of the training runs is also shown. {#tab:splits}

Conclusion

Annotating appropriate images from a larger dataset for training machine learning algorithms is an important task. This is because the usual approach is to use as many images as possible. While this approach is relevant for general classification problems, for medical image segmentation, where image annotation can be very costly, the images used must be carefully selected in order to ensure good coverage of different tissue appearances and imaging variations. In addition, it is important that the annotated images do not lead to biases in downstream tasks related to tissue characterization. For instance, training segmentation algorithms mostly on prototypical images can lead to incorrect measurements on samples having unusual properties (e.g., very bright or very noisy).

We used an intuitive sampling methodology that evenly selects, as best as possible, both typical and atypical vascular image samples for creating VessMAP, a dataset containing a heterogeneous set of samples representing many possible variations of image noise and contrast as well as blood vessel density and intensity variance. One important characteristic of the dataset is that it provides an intuitive uniform grid in the feature space that can be used for further analyses. For example, one can study the accuracy of a segmentation model on different regions of the grid to identify regions where samples are not being correctly segmented. A robust algorithm should provide good segmentation no matter if a sample is too noisy, bright or dark, if it has low or high contrast, or any other variation on relevant image properties. The dataset is being made available together with the metadata containing the features used for creating the dataset.

We showed that different splits of the dataset can lead to largely distinct validation performances. The heterogeneity is particularly noticeable when training data is scarce. For many popular blood vessel datasets, the vasculature has similar characteristics throughout all samples. Thus, while they can be used for testing novel approaches for segmenting blood vessels, they are not ideal for quantifying the robustness of methods under small distribution shifts regarding sample characteristics and vessel geometry. Our analyses showed that VessMAP displays stronger appearance changes, with an average Dice score change of $0.55$, depending on the samples used for training. This result contrasts with the average Dice score difference of $0.29$ observed for the STARE dataset, the most heterogeneous dataset identified in the experiments after VessMAP.

One drawback of VessMAP is that the samples are relatively easy to segment. When training with more than 20 samples, the validation performance tends to be good and has little dependence on the training set. Thus, the usefulness of the dataset lies mostly in tasks with very limited training data. Another important consideration is that the features used to create the dataset are not necessarily related to the underlying conditions affecting the tissue samples (e.g., wild type, mutations, stroke, development stage) or to the acquisition process of the samples. Thus, obtaining good performance on the VessMAP dataset is important but not sufficient to conclude that a model is not biased on downstream tasks.

We expect that the dataset will be useful for studies regarding data distribution shifts as well as few-shot, interactive segmentation and active learning methods. We suggest two specific applications. Observing the official splits shown in Table 4, it is clear that among the 100 samples, training on samples 4404, 11828, 16295, and 7344 (split 1) led to robust models displaying good segmentation accuracy on the remaining samples (Dice score of 0.846). The same is not true for most of the other samples in the dataset. An active learning method should be able to automatically identify these four samples since they lead to a very low annotation effort to segment the whole dataset with good accuracy.

Another interesting application is the automatic identification of segmentation mistakes. A common scenario in real applications is the following. A new dataset is provided and needs to be segmented for downstream analyses. Since manually annotating blood vessel samples is time-consuming, only a fraction of the samples are manually annotated. A segmentation model is then trained on the annotated samples and applied to the remaining images. But how do we verify that the annotated samples were enough to provide good accuracy on downstream analyses for the remaining data? Looking back at Table 4, if the manually annotated samples are those of split 6, the performance of the model is known to be poor (Dice score of 0.589). Thus, one can develop additional heuristics to identify where the model is making mistakes. For instance, an interesting prospect is to analyze the topology of the vasculature and automatically identify missing segments, spurious branches and unrealistic connectivity patterns. The official splits of the vessMAP dataset allow a systematic comparison between methods developed by different research groups.

Interestingly, the splits in Table 4 represent different degrees of difficult for such methods. Split 6, with a Dice score of 0.589, should lead to clearly unrealistic connectivity patterns. However, the difference between splits 1 and 2 is likely more subtle, and automatically identifying segmentation mistakes in split 2 that are not on split 1 should be more challenging.

The VessMAP dataset might also be used for testing the performance of more general methods that were not developed specifically for segmenting blood vessels. Many large-scale biomedical datasets have been created in recent years (Yang et al. 2023; Cheng et al. 2023; Wang et al. 2023; Wasserthal et al. 2023; D’Antonoli et al. 2024). Fluorescence microscopy samples are relatively uncommon in such datasets. Thus, the VessMAP dataset can be useful as an additional imaging modality for quantifying the performance of general methods.

Supplementary material: Additional boxplots.

Generating accurate segmentations of blood vessels from digital images is an important task for diagnosis (Mookiah et al. 2021; Eladawi et al. 2018; Sangeethaa and Uma Maheswari 2018; Li et al. 2021; Almotiri et al. 2018; Nair and Muthuvel 2020) as well as for developing precise measurements to support novel studies regarding the role of blood vessels on disease evolution (Roda et al. 2021; Ouellette et al. 2020; Wong et al. 2019; Dolati et al. 2015). Current state-of-the-art approaches for segmentation involve training neural networks on manually annotated samples (Mookiah et al. 2021). The samples used for training must represent well the distribution of the whole dataset so that the neural network can generalize to unseen samples. An important difficulty with such an approach is that the pixel-wise annotation of blood vessels is very costly.

One common approach to reduce the amount of samples that need to be manually annotated is to pre-train the network on artificially generated blood vessels (Mou et al. 2021; Todorov et al. 2020; Chen et al. 2023; Wijethilake et al. 2023). Given that blood vessels tend to have a tubular structure, a simple methodology for creating artificial vasculature is to randomly generate tubes with varying thickness and to blur the tubes with a smoothing filter (Tetteh et al. 2020; Sule et al. 2020). More advanced techniques involve simulating the growth of vascular trees (Schneider et al. 2012) to generate images with accurate distribution of vessel geometries. Common to many approaches that have been developed is the focus on the geometry of the vessels, and not on the appearance of the vessels. One line of work that does focuses on appearance consists of different approaches for style transfer and image generation (Tmenova et al. 2019; Ma et al. 2019; Wu et al. 2022), but these techniques do not allow precise control of the generated vessels regarding properties such as vessel density, caliber, tortuosity, and local intensity changes. Furthermore, they require the development of an additional neural network training procedure for the generative model, which in practice requires hyperparameter tuning and can have poor performance for samples at the tail of the data distribution.

In this work, we develop a methodology for extracting the texture of manually annotated blood vessels from real samples and generating artificial images using the extracted textures. The method only requires the annotation of a few vessel segments of a dataset. Given a dataset containing blood vessel images with different appearances (example images are shown in Figure 1), a common practice would be to annotate all blood vessels in some of the samples to train a segmentation algorithm. The developed method involves the annotation of one or a few vessel segments from some of the images, shown in green in Figure 1. The annotations are used for generating artificial images containing blood vessels with the same texture as the annotated segments, which can then be used for pre-training neural networks. This allows the network to be adjusted to as many artificial images as desired, each image containing a rich set of blood vessel geometries and appearances. Notably, the annotations can be done in seconds, or a few minutes if a more diverse dataset is desired. The neural network may then be refined on a few fully annotated real samples.

The methodology involves creating standardized representations of the appearance of the vessels by mapping the annotated segments into a rectilinear grid. The result is called a texture map since it shares some similarities with texture maps in computer graphics (Hughes et al. 2013). A procedure is then developed for associating a given texture map to a randomly generated Bézier curve. This is done by defining two corresponding sets of points, one for the texture map and one for the Bézier curve. A Delaunay triangulation is generated for each set of points, and linear transformations are estimated between pairs of corresponding triangles of the two sets of points. This procedure allows the generation of artificial vessels with varying geometries and realistic textures that can be systematically controlled.

We show that artificial images generated using the methodology are effective for pre-training neural networks. On a fluorescence microscopy dataset, images generated from only 5 annotated vessel segments, which corresponds to annotating 0.03% of the vessels in the dataset, lead to an average Dice score of 0.87$\pm$0.02, which is close to the baseline value of 0.92 obtained when training using the fully annotated dataset. For a fundus photography dataset, an average Dice score of 0.74$\pm$0.02 is achieved by annotating 10 maps (0.29% of the vessels), which is also close to the baseline value of 0.81 obtained when all vessels are annotated. While the manual annotation of the entire datasets can take weeks or months, 10 vessel segments can be annotated in under 5 minutes.

The code of the methodology and the obtained results are publicly available online¹.

Related works

In (Galarreta-Valverde et al. 2013) the authors proposed a methodology that extended the traditional L-systems grammar (Lindenmayer 1968), which was used for synthesizing angiographic images simulating computed tomography and magnetic resonance images. The generation of synthetic vessels was aimed at assisting in validating segmentation algorithms and to be used in virtual surgeries.

Vessel tree simulators were used to compare artificially created data with real vascular trees (Kociński et al. 2012). The authors approached the comparison using the texture properties of the trees’ geometry (not the vessels’ surface). They showed that the models had important correlations with different types of confocal brain tissue images. Vascular tree parameter changes were also observed between healthy and tumorous tissue.

A few recent works (Mou et al. 2021; Todorov et al. 2020; Wijethilake et al. 2023) used biologically simulated vascular trees for training neural networks using the method of Schneider et al. (Schneider et al. 2012). The method consists of an iterative generation of arterial trees through two main processes: constructive (growth) and destructive (degeneration). Each vessel segment is modeled as a cylindrical tube and tissue metabolism demands drive the angiogenic process. For instance, in (Tetteh et al. 2020) the model was used to simulate magnetic resonance angiography images, which were then used to train different neural network architectures. It was shown that the synthetic data greatly contributed to network pre-training.

The aforementioned works focused mostly on generating plausible geometries for the vessels. Usually, the texture is added using simple procedures such as blurring and Gaussian or Poisson noise. Another line of work considers the generation of vasculature with plausible geometry and appearance using generative adversarial networks (Tmenova et al. 2019; Ma et al. 2019; Wu et al. 2022; Popescu et al. 2021; Zhao et al. 2018). But the generated vessels cannot be tightly controlled, for instance, one cannot easily generate samples containing blood vessels with large tortuosity or with low-contrast segments (i.e., discontinuities). In addition, a new training procedure needs to be applied when there is a distribution shift in the data, such as when changing microscope parameters. Our model allows the precise control of the geometry of the vessels as well as the selection of specific texture patterns for the generated images.

Methodology

The method can be divided into five main steps. They are: (a) manual vessel delineation; (b) creation of vessel models; (c) generation of vessel texture maps, (d) transformation of the maps to follow randomly generated curves, and (e) insertion of the transformed maps into an artificially generated background. In the following sections, we describe each of the steps².

Manual vessel delineation

The first step consists of manually annotating a few blood vessel segments. Segments are annotated by delineating their borders, as shown in Figure 1. There is no need to annotate the whole segment, only a portion of the segment is enough to obtain its texture pattern. A simple graphical user interface was created for this task, which stores the pixel coordinates of the two delineated borders as a .json file, but the segments can be annotated on any software, provided a file containing the coordinates is generated. The sets of points from the annotated borders are represented as $\tilde{P}_{b1}$ and $\tilde{P}_{b2}$.

Vessels annotated from different images generate a dataset of vessel segments. The method can work with a single annotated segment, but additional segments can increase the diversity of the textures of the vessels on the artificial images. Thus, it is useful to annotate vessels from different images and having distinct appearances. In our experiments, five segments usually led to a good segmentation performance.

Generation of vessel models

The manual delineations $\tilde{P}_{b1}$ and $\tilde{P}_{b2}$ are used for obtaining additional information about the vessels such as their medial lines and normal vectors at each border position. The set of all such information for a given vessel segment is henceforth called the vessel model of the segment. An example of a vessel model is shown in Figure 2. The procedure used to generate a vessel model is explained next.

Depending on the annotation tool used, the points in $\tilde{P}_{b1}$ and $\tilde{P}_{b2}$ might not be equally spaced, and the annotations might be noisy due to the precision required to trace vessel borders. Thus, for creating the vessel model, the first step is to interpolate the manual delineation of the vessel borders, which is done using a two-stage procedure. A linear interpolation of the points is first performed to uniformly sample the manual annotations. Then, a cubic spline interpolation is performed on the result of the first interpolation. This two-stage interpolation is useful to make sure that the interpolated curves pass close to all manually marked points. In our implementation, the functions splprep and splev of the SciPy Python package were used.

Both interpolations have a parameter $\delta$ setting the distance between interpolated points, which equivalently sets the total number of points used in the interpolated curves. A value of $\delta=1$ pixel was used in all experiments since smaller values lead to redundant information due to the finite size of the image grid. The result of the interpolation process are two sets of points $P_{b1}$ and $P_{b2}$ representing high-resolution smooth curves describing the borders of the vessel segment.

For each point in $P_{b1}$ and $P_{b2}$, a normal vector to the curve at the position of the point is calculated using the spline representation of the curve. The two respective sets of normal vectors are represented as $N_{b1}$ and $N_{b2}$. Figure 2 shows in green the interpolated curves and in orange the normal vectors at each point.

Next, $P_{b1}$ and $P_{b2}$ are used to calculate the medial axis of the blood vessel, which is obtained using a Voronoi tessellation of the border points (Tagliasacchi et al. 2016). In our implementation, the Voronoi tessellation is calculated using the Voronoi class from the SciPy Python package. Figure 3 shows an example of Voronoi tessellation of a vessel segment. The tessellation provides the medial axis with sub-pixel accuracy, which makes the following procedures more accurate. However, the points defining the medial axis are not equally spaced. Thus, the same two-stage interpolation applied to the manual delineations is applied to the medial axis using the same value of $\delta$. Respective normal vectors are also similarly calculated. The resulting sets of medial axis points and respective normal vectors are represented, respectively, as $P_m$ and $N_m$. They are illustrated in Figure 2.

The three generated curves (two borders and one medial axis) represented by $P_{b1}$, $P_{b2}$ and $P_m$ as well as the normal vectors $N_{b1}$, $N_{b2}$ and $N_m$ are stored and define the vessel model of the segment.

Vessel texture map creation

Vessel models are used for generating vessel texture maps, which are images containing only the texture of the vessels. To do so, a coordinate system following the model geometry is defined. The medial axis defines one coordinate, while perpendicular lines to the medial axis define the second coordinate. The intensities of the vessel are then mapped into a new rectilinear grid. Figure 4 shows an illustration of the procedure. However, some important details need to be taken into account when generating the coordinates.

First, a set of sampling lines perpendicular to the medial axis are created. Figure 4(a) shows in yellow and blue examples of perpendicular lines. The normal vectors of the medial axis are used for defining a first guess for the direction of the lines. However, the normal of an axis point and the normals of corresponding border points might not be aligned. For instance, on high-curvature segments or when the border at one side significantly changes, the normal vector at the medial axis might lead to very distinct points at the opposing borders of the vessel.

To obtain a precise cross-section of the vessel at a medial axis point $\mathbf{p}_m$, we find the line that best aligns with the normal vector $\mathbf{n}_m$ at the point as well as the normals of the corresponding points at the border of the segment. The procedure used for finding the best-fitting line is described in Algorithm 5 and is illustrated in Figure 6. The algorithm receives as input the medial axis point $\mathbf{p}_m$ with respective normal $\mathbf{n}_m$ and the complete sets of border points $P_{b1}$ and $P_{b2}$ with their respective normals $N_{b1}$ and $N_{b2}$. A set of angles $\Theta$ is defined in the range $[-45^\circ,45^\circ]$ with steps of $\Delta\theta=3^\circ$ (line [acp_angles] of Algorithm 5).

The medial point $\mathbf{p}_m$ and respective vector $\mathbf{n}_m$ define an initial candidate line for a cross-section of the vessel at point $\mathbf{p}_m$ (black line in Figure 6). For each angle $\theta$ in $\Theta$, the vector $\mathbf{n}_m$ is rotated by $\theta$ (line [acp_rot] of Algorithm 5), thus defining a new candidate line for the cross-section of the vessel. Candidate lines are shown as dashed lines in Figure 6. The function get_closest($\mathbf{p}_m$, $\mathbf{\tilde{n}}_m$, $P_{b1}$) (line [acp_closest] of Algorithm 5) returns the point $\mathbf{p}_{b1}$ in $P_{b1}$ with the smallest point-to-line distance to the line defined by $\mathbf{p}_m$ and $\mathbf{\tilde{n}}_m$. The same calculation is done for the other border, defining point $\mathbf{p}_{b2}$. The respective normals are also obtained (lines [acp_normal1] and [acp_normal2] of Algorithm 5). Points $\mathbf{p}_{b1}$ and $\mathbf{p}_{b2}$ of each candidate line are shown as small brown dots in Figure 6 and respective normals are shown as purple arrows.

The alignment between vectors $\mathbf{\tilde{n}}_m$, $\mathbf{n}_m$, $\mathbf{n}_{b1}$ and $\mathbf{n}_{b2}$ (line [acp_align] of Algorithm 5) is then calculated as

\begin{equation}
    e = 2\mathbf{n}_m\cdot\mathbf{\tilde{n}}_m + \mathbf{n}_{b1}\cdot\mathbf{\tilde{n}}_m + \mathbf{n}_{b2}\cdot\mathbf{\tilde{n}}_m,
\end{equation}

where $\mathbf{x}\cdot \mathbf{y}$ represents the dot product between $\mathbf{x}$ and $\mathbf{y}$. The equation represents the degree of alignment between the candidate line with direction represented by $\mathbf{\tilde{n}}_m$ and the normals of the medial and border points. A weight of 2 is used on $\mathbf{n}_m\cdot\mathbf{\tilde{n}}_m$ to slightly favor the alignment with the medial axis. The angle $\theta_o$ with the largest value of $e$ (line [acp_max] of Algorithm 5) defines the direction of the cross-section at point $\mathbf{p}_m$. This direction is illustrated as a blue line in Figure 6.

The identified cross-sectional lines together with the medial axis define a 2D coordinate system based on the vessel’s geometry. The medial axis represents a coordinate along the blood vessel, while the perpendicular lines represent a coordinate along cross-sections of the vessel. Sampling points are then created along each perpendicular line. Given a parameter $r$ setting the range of the perpendicular lines, that is, the distance between each endpoint of a line and the medial axis, the sampling points $S=[-r, -r+\delta, \dots,r-\delta,r]$ are created. The parameter $\delta$ sets the distance between points and is the same used when interpolating the border and medial axis points. The sampling points are then oriented according to the perpendicular lines found using Algorithm 5. Examples of sampling points are shown as yellow dots in Figure 4(a).

Each cross-sectional line can be mapped to a column of an output image. The number of sampling points in $S$ defines the number of rows of the image. An illustration is shown in Figure 4(b). Thus, for each pixel of the output image, the corresponding intensity of the original annotated segment can be calculated using bilinear interpolation. In our implementation, this is done using the map_coordinates function of the SciPy Python package. The output image defines the vessel texture map $M$.

The main parameter of the map creation procedure is the range $r$ used for generating the cross-section sampling points. If $r$ is too small, the cross-section lines might not capture the whole vessel. The transformation also needs to capture some portion of the background of the image for the artificial image generation, but if $r$ is too large, other vessels might be included in the texture map, which will lead to artifacts. A good compromise is to use the largest diameter of the vessel segment together with an additional range of around 10 pixels to capture the background.

The sets of border points $P_{b1}$ and $P_{b2}$ are also projected onto the map. They are used for generating a segmentation mask for identifying pixels belonging to the vessel and to the background. Notice that it is not necessary to project the medial axis coordinates since it will always correspond to the middle row of the map.

Association of vessel textures to arbitrary curves

The texture maps can be used for generating textures for arbitrary curves. In our methodology we focused on Bézier curves, which are simple to implement and can faithfully represent the curvilinear structure of a vessel. A Bézier curve is defined by the initial and final points, $\mathbf{p}_i$ and $\mathbf{p}_f$, as well as a set $P_c$ of $q$ intermediate control points that control the geometry and the smoothness of the curve. The curve is created using Algorithm 7. $q$ linearly spaced values are created in the range $[0.2, 0.8]$ (line [ab_ts] of Algorithm 7). These values are used for defining $q$ respective points along a straight line between $\mathbf{p}_i$ and $\mathbf{p}_f$ (line [ab_pl] of Algorithm 7). Each point is then displaced along a normal vector $\mathbf{n}_l$ to the straight line (line [ab_disp] of Algorithm 7). The normal vector is a unit vector having a dot product with vector $\mathbf{p}_f-\mathbf{p}_i$ equal to zero. The translation amount is randomly drawn with uniform probability in the range $[-d_{m},d_{m}]$, where $d_{m}$ is a parameter of the method. Lower values of $d_{m}$ lead to more straight curves.

A Bézier curve with order $q+1$ can then be defined using control points $\mathbf{p}_i$, $\mathbf{p}_f$ and $P_c$. The application of the curve for transforming the maps requires a set of sampling points along the curve. Thus, for any given curve, $n_p=100$ uniformly spaced points between the initial and final points are created along the curve. The generated points of the curve are represented as $P_r$.

Given a vessel map $M$ and the points $P_r$, a procedure is defined to transform the map to follow the geometry of the curve. To do so, we first create two new curves by shifting points $P_r$ to the left-hand and right-hand sides of the original curve. Figure 8 shows an example of the resulting geometry. The curves are created by finding all points with a point-to-line distance of $d_b$ to the original Bézier curve, where $d_b$ is given by half the number of rows of the vessel map. Then, all points having the endpoints of the Bézier curve as the closest points to them are removed. This defines two curves, which are then linearly interpolated to have the same number of points $n_p$ as $P_r$. The set of the three generated curves is called the vessel geometry. The points defining the three curves, and respective vessel geometry, are all represented by the set $P_g$.

Having obtained points $P_g$, a respective set of points $P_v$ needs to be defined for the vessel map with a one-to-one correspondence with points in $P_g$ so that the intensities can be transformed from the map to the curve. Before defining $P_v$, it is important to observe that the length of the Bézier curve, measured as the path length along the curve, and the length of the vessel map, measured as the number of columns of the map, are usually distinct. Thus, the vessel map is expanded to have the same length as the Bézier curve. This is done by repeating the map column-wise the number of times required for the lengths to be compatible.

Representing as $c$ the number of columns of the vessel map, $n_p$ sampling points are defined at row 0 of the map and at the column positions $[0, \Delta_s, 2\Delta_s,\dots,(n_p-1)\Delta_s]$, where $\Delta_s=(c-1)/(n_p-1)$. That is, $n_p$ points are created along the first row of the map with equal spacing between them from column positions 0 to $c-1$ (the last column of the map). The same procedure is repeated for the middle and last row of the map. Set $P_v$ is composed of the created points. Figure 9(a) shows in red, yellow, and blue the points in $P_v$.

Having points $P_g$ and $P_v$, with a one-to-one correspondence between them, a Delaunay triangulation of each set of points is calculated. This defines two sets of triangles, also with a one-to-one correspondence. For each pair of triangles (one from the map and the other from the curve), an affine transform is estimated and the intensities are mapped using bilinear interpolation. In our implementation, this procedure is applied using the class PiecewiseAffineTransform and the warp function of the scikit-image Python package. Figure 9 shows an example triangulation of the original map and the Bézier curve together with the transformed image.

The Delaunay triangulation might present some artifacts because points that are on different parts of the border might generate triangles that are outside the curve. These triangles can be seen on the right side of Figure 9(b). These artifacts were removed by zeroing all pixels outside the vessel geometry.

Besides the vessel texture, it is also necessary to map the location of vessel pixels so that a corresponding binary image can be created and used as a target for segmentation algorithms. This is done by creating a binary vessel image from the map image. This is simple since we have the position of the vessel borders on the map. The binary image is then transformed using the same transformation applied to the grayscale image, but using a nearest neighbor interpolation. Figure 10 shows an example of a transformed map and transformed binary vessel.

Generation of artificial images using the transformed vessel maps

Having a collection of transformed maps, it is possible to insert them into an image to generate a realistic vasculature tissue image. One possibility is to generate a vascular tree and place the maps on each vessel segment of the tree. However, blood vessel identification is usually implemented as a semantic segmentation task. This is because digital samples usually contain only a portion of the vasculature of a tissue. In addition, for most biological tissues, the vasculature does not have a standard geometry at the typical scales captured by most imaging modalities. Therefore, we consider that the global geometry of the generated artificial images is not particularly relevant for vessel segmentation methods. Local properties such as the curvature, caliber, border definition, and texture are likely more relevant. This is particularly true for Convolutional Neural Networks, which are known to be more biased towards local features (Islam et al. 2021; Geirhos et al. 2018). Thus, we generate artificial images by randomly placing texture maps into an image, without following a global vascular structure.

The only remaining step is to generate an artificial background image $B$ to insert the maps into. This can be done in several ways. For instance, it is possible to generate a random image with values drawn from a normal distribution. One can also annotate a few background regions from a sample and replicate these regions multiple times to generate a full image.

Note that since the transformed maps include some portion of the background of the samples they were extracted from, the artificial background image does not need to be realistic. The main challenge of the segmentation algorithm will be to segment the vessel from the texture map background, and not from the artificial background. Thus, we do not consider the artificial background generation as part of our methodology. In the experiments, we used a simple methodology to create $B$, consisting of dividing a real sample into small windows and replacing windows containing blood vessels with windows that did not contain blood vessels. It is important to notice that the size of $B$ defines the size of the final artificial blood vessel image.

To generate images without a sharp change in intensity between the artificial background and the map background, before insertion onto the background the maps are normalized as

\begin{equation}
    \tilde{M} = M - \mu_m + \mu_b,
\end{equation}

where $\mu_m$ and $\mu_b$ are the average intensity of, respectively, the map’s background and the artificial background image. A normalization using the standard deviation of the intensities may also be used in case the intensities of the vessel maps have a larger standard deviation than the artificial background.

The procedure used for generating an artificial image is summarized in Algorithm 11. The algorithm receives as input a database of texture maps $\mathcal{M}$, the number of maps $m$ to use, an initial background image $B$, the number of vessels $n_v$ to insert, the minimum length $s_l$ of the artificial vessels and the $q$ and $d_m$ parameters for the Bézier curve generation. First, a subset $\tilde{\mathcal{M}}$ of $m$ texture maps is randomly drawn from $\mathcal{M}$ (line [gen_sub] of Algorithm 11). A value $s_m$ setting the maximum length of the vessel geometries is then calculated (line [gen_s] of Algorithm 11). It is given by the number of rows or columns of the image, whichever is larger.

For each vessel to be inserted into the image, the length $d_e$ of the vessel is drawn with uniform probability in the range $[s_l,s_m]$. The function get_endpoints (line [gen_ends] of Algorithm 11) randomly generates two points $\mathbf{p}_i$ and $\mathbf{p}_f$ with distance $d_e$ between them. Next, the geometry of the vessel $P_g$ is generated by the function generate_geometry using the methodology presented in Section 3.4. One texture map is then randomly drawn from the set of available maps. The map is transformed into the artificial geometry by the transform function using the method presented in Section 3.4 and inserted into the image $I$.

The result is an artificial image containing the inserted blood vessel textures. Parameters $n_v$, $s_l$, $q$, and $d_m$ can be set according to the density and typical curvatures of the vessels from the real dataset. Parameter $m$ should be as large as possible since the diversity of the textures is increased. However, larger $m$ requires more annotated vessel segments.

Results and Discussion

In this section, we present the results of the vessel model and texture map creation methods. We also show that the generated artificial images can be used to efficiently train neural networks. The experiments were conducted on an AMD Ryzen 5 5600G 3.90 GHz CPU with 16.0GiB RAM, except for the training and validation of neural networks, which were executed on an Intel I9 12900K 3.20 GHz CPU with an RTX 3090 GPU. Versions 1.10 and 0.21 of, respectively the SciPy and scikit-image Python packages were used.

Datasets

Three datasets are used to show the potential of the methodology. The first dataset is composed of 50 confocal microscopy images of the mouse cortex. The images have a size of 1376 $\times$ 1104 pixels, with each pixel representing 0.908 $\times$ 0.908 $\mu$m. Details about the image acquisition procedure can be found in (Freitas-Andrade et al. 2022). This dataset is challenging because the blood vessels do not have well-defined borders and some samples have low contrast. Figure 12 contains examples of images from the dataset. This dataset is henceforth called CORTEX.

We also use the DRIVE dataset (Staal et al. 2004), one of the most popular datasets for evaluating blood vessel segmentation algorithms. The dataset contains 20 training and 20 test color fundus photographs, each image having a size of 584 $\times$ 565 pixels. Following common practice in the literature, only the green channel of the images was used. The fundus images from the DRIVE dataset are markedly distinct from the microscopy images from the CORTEX dataset. Thick vessels tend to have better contrast with the background when compared to the CORTEX dataset, but the DRIVE images contain a larger number of very thin vessels, which tend to be difficult to segment.

As discussed above, one of the motivations for our method is to improve upon the common pre-training procedure of generating artificial vessels with trivial textures. Thus, we also compare our method with a technique used by recent works to simulate artificial vessels (Mou et al. 2021; Todorov et al. 2020; Wijethilake et al. 2023), which we refer to as TREE. The technique consists of using the method of Schneider et al. (Schneider et al. 2012) to generate realistic artificial vascular trees. A Gaussian blur followed by Gaussian noise is then applied to the images. In (Tetteh et al. 2020) a dataset containing 136 artificial samples was generated using the artificial tree method and made available online. We use this dataset in the experiments.

The CORTEX dataset is used in the following sections to provide examples of artificial images generated using the proposed methodology. All three datasets are used in Section 4.6 to quantify the potential of the method for pre-training neural networks.

Vessel models

To measure the typical time spent on manual annotation of vessel segments, six annotators were asked to delineate 3 vessel segments in a given image from the CORTEX dataset, and we recorded the time taken for the annotation of the vessels. The results are shown in Table 1. Overall, the annotation of a vessel segment takes an average of 27 seconds, with the time varying according to the characteristics of the vessel. Vessels with larger dimensions or that are very tortuous tend to take more time to be annotated.

For the following experiments, a total of 343 vessel segments were annotated from the 50 images in the CORTEX dataset by a single annotator. On average, between 5 and 8 vessels were annotated for each image. The minimum and maximum caliber of the vessels were, respectively 4.1 and 25.59 pixels, with an overall average caliber of 8.65 pixels.

A vessel model was created for each annotated segment. Figure 13 shows examples of created models. It is important to have a rich set of vessel models. Thus, we annotated vessels with large variations of intensity, contrast, tortuosity, caliber, and noise amount.

Vessel texture maps

The vessel models were used for generating respective vessel texture maps. The database of 343 vessel texture maps obtained is represented as $\mathcal{M}$. Figure 14 shows examples of vessel maps from $\mathcal{M}$. Each vessel map contains the intensities of the vessel segment and, by design, the background of the sample which the vessel was extracted from. The manual annotation indicating the borders of the vessel is also stored together with the texture map. Again, it is important to have maps containing vessels with varying characteristics since this allows the generation of a diverse set of artificial images.

Vessel geometries

Vessel geometries were generated by randomly drawing the initial and final points $\mathbf{p}_i$ and $\mathbf{p}_f$ of the Bézier curves such that the Euclidean distance $d_e$ between the points was in the range $[500,1376]$ pixels. Figure 15 shows examples of geometries that can be generated. $q=30$ control points were used for the curve in Figure 15(a), while $q=2$ was used for the curve in Figure 15(b). The number of control points has a large impact on tortuosity. Figures 15(c), 15(d), and 15(e) show curves with $q=6$ and different values of $d_m$, which controls the maximum distance between control points and a straight line between the initial and final points. Lower $d_m$ values lead to straighter vessels.

For the remainder of the experiments, we used $q=6$ and $d_m=500$ for generating the geometries.

Artificial image generation

Several artificial images with realistic textures were generated using Algorithm 11. We considered different situations according to the number of annotations $m$ (and respective vessel maps) required for generating the images. We considered situations where only a single annotation is used ($m=1$), as well as situations where $m=5$, $m=10$, and $m=160$ annotations are used. The annotations used were randomly drawn from the $\mathcal{M}$ set of 343 annotations. The number of vessels $n_v$ inserted into each image was randomly drawn with uniform probability in the range $[20,50]$ and the minimum vessel length used was $s_l=500$.

The results are images containing $n_v$ artificial vessels. Figure 16 shows examples of images generated from a single annotated vessel segment ($m=1$). In this case, all vessels in the image have the same texture but different geometries. Interestingly, these images can be used for investigating the performance of segmentation algorithms when only the geometry of the blood vessels changes throughout the images.

Figure 17 shows images generated from different numbers of annotated vessels. Figures 17(a), 17(b), 17(c) and 17(d) were generated from, respectively, $m=1$, $m=5$, $m=10$ and $m=160$ vessel annotations. The images show a large diversity of geometries and vessel textures, as well as different degrees of vessel density.

It is possible to use additional data augmentation transforms to further diversify the images. For instance, the images can have the size, brightness, and contrast randomly changed. Gaussian noise can also be added to the images (an example is shown in Figure 18). Since there are many distinct approaches for data augmentation, our method is defined independently of any specific data augmentation pipeline.

Validation using neural networks

The artificial images were used to verify if the proposed methodology can assist in training neural networks. The neural network used is based on the U-net architecture (Ronneberger et al. 2015) using residual blocks. Figure 19 illustrates the architecture. The network was trained using a learning rate of 0.01 with a polynomial learning rate scheduler with an exponent of 0.9 and a batch size of 8. The performance was quantified using the Dice and centerlineDice (clDice) (Paetzold et al. 2019) metrics.

The first experiment concerns the CORTEX dataset. The dataset was randomly split into 40 training images and 10 validation images. A set of vessel texture maps, $\mathcal{M}$, was extracted from the 40 training images. A respective set $\mathcal{I}$ of 100 artificial images was generated using a fixed subset of $m$ texture maps from $\mathcal{M}$. To make sure that the results are not due to the specific maps used, $n_d$ different sets were generated. The specific procedure used to generate the artificial datasets is described in Algorithm 20. The results presented in this section are always averaged over the $n_d$ datasets.

We considered cases where $m=1$, $m=5$, $m=10$, and $m=160$ vessels were annotated. For $m=1$, $m=5$ and $m=10$, $n_d=10$ datasets were generated. Only one dataset of 100 images was generated for $m=160$ maps since this set already has a large number of maps and random fluctuations due to the specific maps used are not expected.

For comparison, a baseline model was trained on the 40 original training images using all available vessels. This model represents the best result that can be obtained with the training parameters considered since all pixels are annotated and used for training. As an additional comparison, the network was also trained on 100 randomly selected images generated by the TREE method, and the training was repeated 10 times on different sets of randomly selected images to asses the performance variability when using different images. This dataset is used as a baseline comparison with artificial images containing realistic vessel geometries without real textures.

In all experiments the networks were trained for 300 epochs. The performance of all trained models was measured on the validation set consisting of 10 real images. The models with the lowest validation loss found during training were used. The only data augmentation used was the inclusion of Gaussian noise and the darkening of the image according to the distance from the center of the sample to simulate microscope illumination.

The results are shown in Table 2. In the table, average and standard deviation values among the 10 sets of 100 images are shown for $m=1$, $m=5$, and $m=10$ annotated maps. No statistics were calculated for the 160 maps and baseline images because they have a single set of images. The results show that the performance difference between 5 and 160 maps is small. Thus, 5 maps seem to be enough to reach the best accuracy for this dataset when using artificial images. Interestingly, while we observe a 0.05 difference in Dice score between the artificial images using 5 maps and the baseline, the clDice difference is only around 0.03. Thus, the centerlines of the vessels are being segmented with an accuracy that is very close to the baseline. Training on the TREE images results in a Dice score of 0.79. Thus, our method achieves a 10.1% Dice score improvement over the usual pre-training strategy using artificial tree generation.

Average and standard deviation of the Dice and clDice metrics obtained for neural networks trained on artificial vessel images and evaluated on the CORTEX dataset. {#t:result_table1_cortex}

Example segmentations of a network trained with artificial images generated from 1 annotation are shown in Figure 21. The images indicate the reason for the difference observed between the Dice and clDice metrics. The network trained on the artificial images tends to generate segmented vessels with the same caliber. Thus, while the centerline of the vessels is correctly identified, the caliber is not always correct. Adding more annotations with high-caliber vessels may mitigate this problem.

To verify if the method is general enough to be applied to different datasets, the experiments done for the CORTEX dataset were repeated on the DRIVE dataset. The artificial images were created using exactly the same procedure used for the CORTEX dataset. In short, a set of vessel texture maps was extracted from the 20 training images of the original train/test split of the DRIVE dataset. Algorithm 20 was then applied to create artificial images using $m=1$, $m=5$, $m=10$, and $m=160$ maps. $n_d=10$ sets of 100 images were created for the $m=1$, $m=5$, and $m=10$ cases, and one set of 100 images was created for $m=160$. A baseline model was trained on the 20 training images of the dataset for comparison. An additional model was trained 10 times on different sets of 100 randomly selected images generated by the TREE method. The performance of all trained models was evaluated on the original test split of the dataset.

There were two changes in network training compared to the CORTEX dataset. The networks were trained for 1000 epochs, which was necessary for convergence. A lightweight data augmentation consisting of random flipping, random rotation, and random resized crops was necessary to avoid overfitting³.

The results of the experiments are shown in Table 3. For the DRIVE dataset, 10 maps resulted in a Dice score of 0.74, which is close to the baseline score of 0.81. The TREE images led to poor results since the method used to generate the images was originally developed for microscopy images. In contrast, our method can naturally adapt to different imaging modalities.

Average and standard deviation of the Dice and clDice metrics obtained for neural networks trained on artificial vessel images and evaluated on the DRIVE dataset. {#t:result_table1_drive}

The amount of manual annotation required by our methodology was measured for the CORTEX and DRIVE datasets. For each map used in the experiments, the length of the vessel represented by the map was measured. This is given by the length of the curve represented by the set of medial axis points $P_m$ used to create the map. For the CORTEX dataset, 5 maps required, on average, annotating a total vessel length of 513 pixels. For comparison, the vasculature of the 40 training images has a total length of 1555520 pixels. Thus, the Dice score of 0.87 is reached using 0.03% of the annotation effort used on the full dataset.

For the DRIVE dataset, the creation of 10 maps required on average annotating a total vessel length of 510 pixels. The number is similar to annotating 5 maps on the CORTEX dataset because the annotations on the DRIVE dataset for each map were usually shorter since vessel segments on the DRIVE dataset tend to have shorter lengths. The 20 training images have a total vessel length of 174080 pixels. Thus, the Dice score of 0.74 is reached using 0.29% of the vessels annotated on the full dataset. Note that the method is used only for pre-training the network. After pre-training, the network can be refined on the real samples.

Conclusion

We presented a methodology for generating artificial blood vessel images with realistic textures. The method is built upon the fact that vessel segments can be used as a reference for a coordinate system, which allows the definition of texture maps. The method is fast, being able to generate hundreds of images from a single vessel segment annotation, and can significantly decrease the manual annotation effort when segmenting novel datasets.

We showed that by annotating only 5 vessel segments of the CORTEX dataset, corresponding to only 0.03% of the vessels, the generated artificial images can be used for training a neural network with a Dice score of 0.87$\pm$0.02 and a clDice score of 0.91$\pm$0.02, which are very close to the baseline values of, respectively, 0.92 and 0.94 obtained when annotating the whole dataset. The annotation can be done in under 2 minutes by a trained annotator. For the DRIVE dataset, annotating only around 0.29% of the vessels, or 10 segments, led to a Dice score of 0.74$\pm$0.02 and a clDice score of 0.73$\pm$0.02, which are also close to the baseline values of, respectively, 0.81 and 0.8.

The results were compared with those obtained by a common procedure used for pre-training neural networks in recent works (Tetteh et al. 2020; Mou et al. 2021; Todorov et al. 2020; Wijethilake et al. 2023), which we called the TREE method. For the CORTEX dataset, the TREE method resulted in a Dice score of 0.79$\pm$0.02 and a clDice score of 0.85$\pm$0.04, which are lower than the values obtained by our method. For the DRIVE dataset, a Dice score of 0.21$\pm$0.01 and a clDice score of 0.23$\pm$0.06 were obtained using the TREE method. The reason for the weak performance is that the method was developed with a focus on microscopy images. The advantage of our method is that it can adapt to different imaging modalities since the appearance of the vessels is extracted directly from small segments of the real images. Thus, the method worked well for both fluorescence microscopy and fundus photography images.

The developed methodology also allows to easily add new annotations to the dataset. For instance, one can annotate a specific blood vessel that was not segmented correctly by the network and generate a new set of artificial images containing blood vessels with the same appearance but different geometries. The network can then be refined on the images.

The analysis was done with lightweight pre-processing and augmentation procedures to avoid biasing the results. That is, it is important to verify that the performance of the method does not come from a complex augmentation pipeline, but from the representation capability of the textures captured from the real images. For future analyses, more elaborate pre-processing steps, such as histogram equalization (Vijayalakshmi and Nath 2023, 2022), and augmentation procedures may be used to improve the results of the method.

The generation of vessel texture maps also opens new possibilities for disentangling the influence of shape and texture on neural network training. It allows identifying the importance of each aspect and to apply data augmentation and style transfer techniques separately for them. This is a promising research direction with some recent interesting results regarding the texture bias of CNNs (Geirhos et al. 2018).

Funding

C. H. Comin thanks FAPESP (grant no. 21/12354-8) for financial support. M. V. da Silva thanks FAPESP (grant no. 23/03975-4) and the Google PhD Fellowship Program for financial support.

CRediT author statement

Adriano dos Reis Carvalho: Methodology, Software, Validation, Investigation, Visualization, Data Curation, Writing - Original Draft. Matheus Viana da Silva: Software, Methodology, Writing – review and editing. Cesar H. Comin: Conceptualization, Investigation, Methodology, Software, Supervision, Project Administration, Writing - Original Draft, Writing – review and editing.

Blood vessels take part in many physiological processes in humans and animals and can be found almost anywhere in an organism. Therefore, many ailments and developmental disorders may be caused by abnormal blood vessels (Potente et al. 2011). Thus, characterizing blood vessels is an important matter not only for diagnosis but also to help answer important research questions regarding angiogenesis (Guzel et al. 2020; Fernandez-Klett et al. 2020), blood vessel related ailments (Canton et al. 2021) and the blood-brain barrier (Fernandez-Klett et al. 2020; Frías-Anaya et al. 2021). Common metrics for characterizing blood vessels are density and tortuosity since they have been shown to influence neuronal activation (Lacoste et al. 2014) and blood flow (Han 2012). Acquiring precise values for those metrics usually requires annotating blood vessels in images, which is time-consuming and error-prone. Consequently, many image processing techniques have been developed for automatically characterizing blood vessels (Ma et al. 2019; Tongpob et al. 2019). Usually, the most challenging step of the developed techniques lies in segmenting the blood vessels (Lesage et al. 2009).

More recently, owing to the recent advancements of Deep Learning methods on many computer vision tasks (Goodfellow et al. 2016), Convolutional Neural Networks (CNN) have been applied with great success for segmenting blood vessels. Among the most common types of images where CNNs have been applied for blood vessel analysis are retina fundus and magnetic resonance images (Li et al. 2021; Moccia et al. 2018). Recent works have also explored the use of CNNs for blood vessel microscopy images (Tetteh et al. 2020; Kirst et al. 2020; Todorov et al. 2020). Many CNN architectures have been defined for blood vessel segmentation, U-Net-based architectures (Li et al. 2021) being the most common. Given that annotating blood vessel images, which can contain thousands of blood vessel segments, is a demanding task, most approaches for training CNNs involve transfer learning methods (Tetteh et al. 2020; Kirst et al. 2020; Todorov et al. 2020). This is especially true for 3D images, since tracing blood vessels in volumetric images is a particularly difficult task. Most commonly, transfer learning consists in obtaining some off-the-shelf network, that may or may not have been trained on blood vessel images, and fine-tuning the network for segmenting blood vessels for the type of image under study.

In this work, we investigate a possible pitfall when using transfer learning for blood vessel segmentation. The following situation, illustrated in Figure 1, is considered. Suppose that a CNN was trained based on a set of images that were obtained and manually annotated from some clinical trial or biological experiment. Most likely, the objective of the training was to maximize some segmentation performance metric such as the Intersection over Union (IoU) or the Dice coefficient. The trained network was then made available on some repository or software. Then, a new set of images obtained under distinct experimental conditions was acquired during some other experiment. Notice that the experiment might even be carried out by a different group of researchers. Suppose that the overall objective of this new experiment is to study the tortuosity of the blood vessels. For this task, the off-the-shelf network trained using the data from the previous experiment is applied for segmenting the blood vessels, followed by a calculation of the tortuosity.

The above situation has two important possible drawbacks that need to be taken into account. First, the objective of the original experiment where the network was initially trained was solely the segmentation of the blood vessels, there is no guarantee that the values obtained for the tortuosity are correct. Second, the data used in the two situations might have distinct characteristics. Here we do not focus on possible changes on the image characteristics, such as brightness or contrast, or the type of blood vessel under study, but on more subtle differences that might affect the blood vessels. Specifically, we consider two sets of images that have similar statistics but contain blood vessels having distinct degrees of tortuosity. The idea being that a CNN trained on blood vessels having low tortuosity, when used for segmenting highly-tortuous blood vessels, might lead to an overestimation or underestimation of the tortuosity values. The same situation is studied for the opposite problem, that is, CNNs trained on highly-tortuous blood vessels being used for segmenting blood vessels with low tortuosity.

It is important to point out that the possible drawback of transfer learning considered in this work happens mainly on segmentation tasks. For instance, for classification tasks, labeling the data is usually the end goal of digital image processing and computer vision methodologies. Contrariwise, for many analyses in biology and medicine involving digital images, the segmentation of the samples in a dataset tends to be the first step of an overall procedure aimed at characterizing the objects in the images (e.g., cells, organs and blood vessels). Thus, the results of the segmentation are used in downstream tasks involving, for instance, measuring different properties regarding the shape of the objects in the images. In the case of blood vessels, the tortuosity is a commonly analyzed property.

The problem considered in this study is particularly relevant for current researches involving blood vessels, where the impact of an experimental condition might lead to subtle changes in the morphology of the blood vessels, which need to be quantified using robust and systematic approaches. We specifically focus on the tortuosity in this study because it is a difficult metric to be measured and compared by visual inspection, and thus requires particularly reliable means for automatic quantification.

Two main investigations are considered. The first is to assess if there are indeed possible biases that may appear on the calculated tortuosity when using off-the-shelf CNNs. Provided this is true, we verify to which extent such biases can be avoided by fine-tuning the network on the new dataset. This requires the manual annotation of additional images, and thus we also verify how many images need to be annotated in order to reduce the observed biases. In addition, we also investigate a possible data augmentation technique that can be used when training the networks in order to better generalize them for segmenting blood vessels possessing different degrees of tortuosity. All in all, our analysis involves training a CNN over 7000 times in different experimental conditions.

The analysis is focused on a large dataset of confocal microscopy images, but the results should be general for other types of images having blood vessels with a similar appearance.

Literature Review

Recently, many Deep Learning methodologies have been developed for blood vessel segmentation. Much of the effort has been focused on the segmentation of 2D retinographies. This is likely due to the availability of many annotated public datasets. For instance,  Liskowski and Krawiec (2016) evaluated six methodologies for retinal vessel segmentation and found that correcting class imbalances tends to be more critical for performance than applying data augmentation techniques. In the last few years, several retinal vessel segmentation analyses involving different adaptations of the U-Net architecture have been published (Xiao et al. 2018; Zhang and Chung 2018; Jin et al. 2019; Gu et al. 2019).

One disadvantage of public retinography datasets is that they are usually small. For instance, the DRIVE dataset (Staal et al. 2004), one of the most popular retinography datasets in the literature, contains only 40 images. Thus, several works in the literature use transfer learning for improving the convergence of the networks, which is done by either pre-training with extensive datasets such as the ImageNet ¹ (Jiang et al. 2018; Martinez-Murcia et al. 2021; Maninis et al. 2016), or by generating additional synthetic data (Zhao et al. 2018, 2019; Andreini et al. 2022).

Besides retinography analysis, there have been some important recent developments regarding the segmentation of blood vessels in the brain, such as the DeepVesselNet pipeline (Tetteh et al. 2020). This pipeline addresses the issue of collecting and annotating structures in large volumetric data. The authors suggested applying a pre-training step using a database composed of artificially generated blood vessels. This transfer learning approach allowed a faster network convergence while maintaining a high-quality segmentation. Todorov et al. (2020) presented a methodology for analyzing important characteristics of whole mouse brain vasculature. The network architecture used was similar to DeepVesselNet, except for the input layer, which used two-channel images. A transfer learning approach involving artificially generated data was also used. With only 0.02% of the collected mouse brain being annotated, the authors were able to segment the vasculature with high accuracy. Kirst et al. (2020) also defined a methodology for the segmentation of whole mouse brain vasculature using a combination of traditional methods and CNNs. While capillaries could be segmented by image filtering, thicker vessels appeared as hollow tubes. Thus, a CNN trained with synthetic data was used to fill hollow regions.

The majority of the developed methodologies were aimed mainly at improving the segmentation accuracy of blood vessels. Despite the development of additional performance metrics such as clDice (Shit et al. 2021), the influence of segmentation on downstream tasks, such as blood vessel morphometry, has rarely been addressed.

Methods

Training and Fine-Tuning Off-The-Shelf Segmentation CNNs

As stated above, transfer learning is an important technique for training CNNs. This is particularly true when the size of the dataset is small or it is costly to annotate the images. Figure 2 illustrates in more detail the common approach for transfer learning depicted in Figure 1. First, a group of researchers interested in segmenting blood vessels obtains a set of training images and manually annotates them to optimize a neural network (Figure 2A). After choosing a proper network architecture and optimizing it, a trained neural network is obtained. This neural network is usually evaluated through pixelwise accuracy metrics, such as IoU and Dice. Next, the researchers might make the trained network available on a public source-code repository or an image processing software.

The pre-trained network can then be used by other research groups in future studies aimed at measuring the morphology of blood vessels. The network might be used as is, or an additional fine-tuning step can be applied to adjust the network to the new data (Figure 2B). For fine-tuning the network, a subset of the new data must be manually annotated. Here, it is interesting to highlight two main points. First, the fine-tuning applied to the pre-trained network requires an amount of manual work proportional to the number of additional annotations used. Second, it is usually not possible to directly optimize the network using the morphological metrics being studied since their calculation is not fully differentiable. Thus, the metric that is optimized is usually the cross-entropy of the class probabilities or the segmentation accuracy, and the morphological metrics are calculated based on the resulting segmentations. As a consequence, there is no guarantee that the original or fine-tuned network will provide segmentations that are unbiased with respect to the desired morphometric analyses.

To investigate the situation described above, we conducted two main analyses: (i) the identification of possible biases when performing the morphometry of a blood vessel dataset using an off-the-shelf network, initially optimized to segment a different dataset; (ii) if biases are identified, to which extent they can be mitigated by fine-tuning the off-the-shelf network to the new dataset. We divided our experiments into two steps. First, we optimized two neural networks, each specialized in segmenting blood vessels with either high or low tortuosity. Next, we measured the reliability of each specialized network when applied to blood vessels having distinct tortuosity than the blood vessels used during training.

Quantifying the Influence of Transfer Learning on Blood Vessel Tortuosity

Optimizing a CNN for Blood Vessel Segmentation

The first part of our experiment (illustrated in Figure 2A) consists in training a CNN capable of segmenting blood vessels with state-of-the-art accuracy. For the sake of generality, we should ideally use methodologies that are well-adopted in the literature. Therefore, our chosen network architecture was the U-Net (Ronneberger et al. 2015), as it is widely used for segmentation tasks based on small datasets and it performs well on blood vessels (Zhang and Chung 2018; Jin et al. 2019; Livne et al. 2019).

The U-Net is composed of an encoder and a decoder. The encoder maps the input image into feature maps having progressively lower spatial resolutions, while the decoder progressively upsamples the output of the encoder and incorporates feature maps from intermediate stages of the encoder. The idea behind this methodology is to generate pixelwise probabilities for image segmentation while taking into account image features at different spatial resolutions. For our experiments, we use a U-Net with a ResNet-34 encoder (He et al. 2016).

Recall that the objective of our analysis is to investigate possible biases when segmenting blood vessels with high or low tortuosity. Therefore, we optimized two distinct networks. One network is trained on a dataset composed predominantly of blood vessels with high tortuosity, while the other is trained on a dataset where the blood vessels typically have low tortuosity. Section 2.5 describes the procedure used for generating these two datasets. This initial training step is depicted in Figure 3A for the case of blood vessels with low tortuosity. This training is performed for 30 epochs. 80% of the images containing blood vessels with low tortuosity were used in the training set, 10% in the validation set, and 10% in the test set. The training is carried out using the Adam optimizer (Kingma and Ba 2017), with the cross-entropy as a loss function. The learning rate is scheduled using the 1cycle policy (Smith 2018), with a maximum value of 0.0005. The procedure used for training a network on the dataset containing blood vessels with high tortuosity is identical. Provided the training is successful, the resulting networks will be able to segment blood vessels having similar characteristics as those in the respective training sets. Thus, we henceforth refer to those networks as being specialized on blood vessels with high or low tortuosity.

As illustrated in Figure 3B, after training, the network is applied to the test set in order to generate a set of segmented images $S_{sl}$, where $sl$ stands for specialized in low tortuosity vessels. From $S_{sl}$, we calculate the average tortuosity value of each image using the methodology described in Section 2.4. The set of calculated tortuosity values is represented as $T_{sl}$. For the network specialized on blood vessels with high tortuosity, we represent the obtained segmented images as $S_{sh}$ and average tortuosity values as $T_{sh}$, where $sh$ stands for specialized in high tortuosity vessels.

Fine-Tuning an Off-The-Shelf Network for Blood Vessel Morphometry

The second part of our analysis consists in implementing the process illustrated in Figure 2B. Thus, the networks trained using the procedure described in the previous section are fine-tuned to a new dataset. The approach used for fine-tuning the networks is illustrated in Figure 3C. The figure shows the methodology for the network specialized in low tortuosity blood vessels, but the procedure is identical for the other network. As depicted in the figure, the network is fine-tuned using a subset of the training set containing images of blood vessels with high tortuosity. The fine-tuning is done for 15 epochs. Moreover, we use the same methodology and hyperparameters employed when creating the specialized networks.

The fine-tuned network is then applied to the test data of the dataset containing vessels with high tortuosity (Figure 3D). A new set of segmented images $S_{fh}$ and average tortuosity values $T_{fh}$ are generated. The subscript $fh$ stands for fine-tuned with high tortuosity vessels. As mentioned above, this fine-tuning procedure is also performed for the network specialized in blood vessels with high tortuosity. In this case, we represent the new set of segmented images as $S_{fl}$ and average tortuosity as $T_{fl}$, where $fl$ stands for fine-tuned with low tortuosity vessels.

Quantifying Morphometry Biases

The segmented images and average tortuosity values obtained from the specialized and fine-tuned networks can be used to assess the segmentation quality and identify possible biases generated by the training procedures regarding the tortuosity of the blood vessels. We define the segmentation quality as the IoU between the segmentations obtained by the network fine-tuned on a dataset and those obtained by the network specialized on the same dataset. So, for the blood vessels with low tortuosity:

\begin{equation}
    IoU_l = \frac{1}{N}\sum_{i=1}^{N} IoU(S_{fl}(i), S_{sl}(i)),
\end{equation}

where $S_{fl}(i)$ and $S_{sl}(i)$ represent, respectively, the i-th image from the sets $S_{fl}$ and $S_{sl}$, and $N$ is the number of images. The term $IoU(S_{fl}(i), S_{sl}(i))$ in Equation [eq:IoUs] quantifies the IoU between the segmentations obtained by a network that was initially trained on blood vessels with high tortuosity and then fine-tuned on blood vessels with low tortuosity ($S_{fl}(i)$), and the segmentations obtained by a network that was specifically trained for segmenting blood vessels with low tortuosity ($S_{sl}(i)$). Ideally, $IoU_l$ should be close to 1. Notice that this can happen even if the networks cannot segment the blood vessels with good accuracy.

Equivalently, for blood vessels with high tortuosity:

\begin{equation}
     IoU_h = \frac{1}{N}\sum_{i=1}^{N} IoU(S_{fh}(i), S_{sh}(i)).
\end{equation}

The quality of the tortuosity values obtained by each network can also be measured. For the low tortuosity dataset, we define

\begin{equation}

    R_l = \frac{1}{N}\sum_{i=1}^{N}\frac{T_{fl}(i)}{T_{sl}(i)}.
\end{equation}

The term $T_{fl}(i)/T_{sl}(i)$ quantifies the average tortuosity obtained using the network that was initially trained on blood vessels with high tortuosity and then fine-tuned on low tortuosity vessels ($T_{fl}(i)$) with respect to the tortuosity value obtained using the network specialized on low tortuosity blood vessels ($T_{sl}(i)$).

Similarly, for blood vessels with high tortuosity:

\begin{equation}

    R_h = \frac{1}{N}\sum_{i=1}^{N}\frac{T_{fh}(i)}{T_{sh}(i)}.
\end{equation}

Values $R_l$ and $R_h$ quantify the changes in tortuosity obtained when using an off-the-shelf network instead of training a neural network from scratch. It is important to notice that the off-the-shelf network was not trained on completely unrelated data. The high and low tortuosity datasets have similar characteristics, with their main distinction being the tortuosity of the blood vessels contained in the images. Also, given the natural sinuosity of the blood vessels in our images, we observed in our experiments that the values in $T_{sl}$ and $T_{sh}$, being averages calculated over the entire images, did not reach values close to zero.

Having defined these quality metrics, we search for biases of segmentation and tortuosity values by calculating the values of relative tortuosity and IoU as we increase the number of annotated images used in the fine-tuning step. The number of images was varied from 0 (no fine-tuning) to 40. Since the annotated images are randomly sampled, the fine-tuning is repeated $K$ times, and each quality metric is calculated as the average for all repetitions. Since using less annotated images leads to large fluctuations in the results, we define $K$ as

\begin{equation}
    K = max\left(5, \frac{t}{2n}\right),
    
\end{equation}

where $t$ is the total size of the training set and $n$ the number of images used in the refinement. Thus, the smaller the number of images used in the refinement, the larger is the number of repetitions used in the evaluation. The average value of a quality metric $X$ calculated over $K$ repetitions is henceforth referred to as $\bar{X}$.

A summary of the metrics defined above is presented in Table 1 of the Appendix.

Data Augmentation and Transfer Learning

Manually labeling blood vessels is a burdensome task. Thus, in many situations, annotating a large number of blood vessels may be prohibitive. Another difficulty with the fine-tuning approach happens when the off-the-shelf network needs to generalize better toward vessels with high tortuosity. In this regard, high tortuosity vessels are usually disease markers. Therefore, it becomes harder to optimize a network to increase its generalization capability since more effort is needed to develop, image, and label samples containing vessels with high tortuosity. Thus, it is beneficial to develop approaches for increasing the generalizability of a network using as few manual labelings as possible.

Training CNNs usually requires large amounts of labeled data. This problem motivated the development of several data augmentation techniques over the years. Data augmentation techniques allow a network to learn patterns that are not present in the original data and, consequently, improve its generalizability. We propose a methodology for increasing the generalization capability of a network specialized in segmenting blood vessels with low tortuosity. This is done by applying data augmentation to the set of low tortuosity vessels using elastic transformations (Simard et al. 2003), a type of transformation capable of distorting the blood vessels and giving them a more tortuous appearance.

The elastic transformation deforms an image by drawing random displacement fields. Thus, for each image, horizontal ($\Delta_x$) and vertical ($\Delta_y$) displacement fields are defined by drawing random numbers between -1 and +1, generated from a uniform distribution. Then, the fields $\Delta_x$ and $\Delta_y$ are convolved with a gaussian with standard deviation $\sigma$. In this case, $\sigma$ works as an elasticity coefficient. When $\sigma \approx 0$, the result is an uncorrelated displacement field. For non-zero $\sigma$, an elastic deformation effect is obtained. Lastly, the displacement fields $\Delta_x$ and $\Delta_y$ are multiplied by a scale factor $\alpha$, which controls the transformation intensity. The elastic transformation can generate realistic-looking highly-tortuous vessels based on images containing vessels with low tortuosity. An example of the application of elastic transformations in our dataset is depicted in Figure [fig:ex_elastic_transformation] of the Supplementary Material. Also, Figure [fig:tort_over_et] of the Supplementary Material shows typical tortuosity values obtained for different values of parameter $\alpha$.

It is important to notice that data augmentation can be applied in two distinct situations. The first involves applying data augmentation when training the network from scratch. The second situation concerns applying data augmentation for fine-tuning a pre-trained network.

Data Augmentation When Training From Scratch

To evaluate the influence of data augmentation when training a network from scratch, we define a new set of images based on the application of the elastic transformation to the set of vessels with low tortuosity, which we henceforth refer to as false highly-tortuous vessels. Next, we optimize a neural network using the false highly-tortuous vessels and assess the capability of this network in performing the morphometry of the real set of high tortuosity vessels. The training methodology employed is the same as the one used in the optimization of the specialized networks (see Section 2.2.1), including the architecture, hyperparameters, and the number of epochs. After training, the set of blood vessels with high tortuosity is segmented and the set of tortuosity values $T_{sf}$ is calculated. In this case, $sf$ stands for specialized in false highly-tortuous vessels. Similarly to Equations [eq:R_hort_s] and [eq:R_hort_t], the relative tortuosity in this experiment ($R_{sf}$) is calculated as the average ratio between each element of $T_{sf}$ and the respective element of $T_{sh}$ obtained by the network specialized in high tortuosity vessels (NSHV), that is,

\begin{equation}
    R_{sf} = \frac{1}{N}\sum_{i=1}^{N}\frac{T_{sf}(i)}{T_{sh}(i)}.
\end{equation}

Similarly, the relative IoU, $IoU_{sf}$, is calculated as

\begin{equation}
    IoU_{sf} = \frac{1}{N} \sum_{i=1}^N IoU(S_{sf}(i), S_{sh}(i)).
\end{equation}

Data Augmentation During Fine-Tuning

As mentioned above, data augmentation may also be used when fine-tuning a CNN. Thus, it is interesting to verify if the generalizability of a pre-trained network can be increased by fine-tuning the network using false highly-tortuous vessels. This can be investigated by fine-tuning the network specialized in low tortuosity vessels (NSLV) on false highly-tortuous vessels generated using the elastic transformation with $\alpha=64$ and $\sigma=4$. These parameters were chosen because they provided the best performance in our experiments. All other parameters were kept the same as those used in the fine-tuning applied to the NSLV (Section 2.2.2).

Measuring the Tortuosity of Blood Vessels

An approach for quantifying the tortuosity of blood vessels is required for implementing our experiments. Many different metrics have been defined in the literature (Ramos et al. 2018; Grisan et al. 2008; Wilson et al. 2008). Ramos et al. (2018) compared the tortuosity scores provided by five experts regarding blood vessels in retina fundus images. They also compared the scores of the experts with those obtained from different automated approaches for measuring tortuosity. They found a high inter-expert variability as well as different degrees of agreement between the experts and the considered automated measurements. Their results demonstrate that there is no optimal approach for defining tortuosity. Indeed, as discussed in (Bullitt et al. 2003), different types of tortuosity can be considered. Therefore, any given tortuosity measurement will have advantages and drawbacks. In this work, we calculate the tortuosity of blood vessels using linear regression residuals ².

The tortuosity is obtained as follows. First, the centerlines of the blood vessels are calculated using the Palàgyi-Kuba topological thinning algorithm (Palàgyi and Kuba 1998). Next, we associate a tortuosity value to each pixel of the generated centerlines. As illustrated in Figure 4, given a reference centerline pixel $p_c(i)$ belonging to a vessel segment, we define a circle of radius $r$ centered at $p_c(i)$. The centerline pixels inside this circle define the neighborhood of $p_c(i)$. Then, a line $l$ is fitted to the set of neighboring pixels using least-squares linear regression. The tortuosity of $p_c(i)$ is then defined as the average of the point-to-line distances between the neighboring pixels of $p_c(i)$ and $l$ (dashed lines in Figure 4). In other words, the tortuosity is given by the root mean squared error of the least-squares regression of the neighborhood of $p_c(i)$. If the blood vessel segment around $p_c(i)$ is relatively straight, the respective tortuosity will be small. Contrariwise, if the segment cannot be well represented by a straight line, the tortuosity will be large. The overall tortuosity of the blood vessels in an image is then defined as the average tortuosity of the centerline pixels, that is

\begin{equation}
T = \frac{1}{q}\sum_{i=1}^{q} p_c(i),
\end{equation}

where $q$ is the number of centerline pixels. Notice that the radius $r$ controls the size of the detected tortuous structures. For smaller values of $r$, small sinuous structures will have larger tortuosity values. For larger values of $r$, longer vessels with smoother curvatures will be recognized as more tortuous. Thus, parameter $r$ controls the scale of the analysis. The computational cost of calculating the tortuosity is dominated by the linear regression for large $r$, which has an asymptotic time complexity of $O(r)$. Therefore, the calculation of the average tortuosity for a single image has a time complexity of $O(rq)$. For the experiments in this work, we set $r=10$ since it allows the detection of sharp blood vessel turns in the mouse cortex. The calculation of the average tortuosity of all blood vessels in an image with a resolution of $1376\times 1104$ pixels takes around 9 seconds using the computer described in Section 2.6. An example of tortuosity values obtained for a sample can be found in Figure [fig:tort_example] of the Supplementary Material.

Dataset

To apply the methodology proposed in this work, we use a dataset composed of 1838 3D volumes, all obtained from the cerebral cortices of mice. All volumes were collected in 3D using confocal microscopy and have a resolution of $1376 \times 1104 \times 51$ voxels, each voxel representing $0.908\mu m \times 0.908\mu m \times 1\mu m$. The procedure used for obtaining the volumes is described in Ouellette et al. (2020).

To obtain the ground truth labels, we used a semi-supervised segmentation approach. This same strategy was previously used in other works (Ouellette et al. 2020; Lacoste et al. 2014). First, a Gaussian filter with unit standard deviation is applied to each 3D volume. Then, vessel regions are identified by an adaptive thresholding algorithm with a window size of $100 \mu m \times 100\mu m$ applied to each image along the depth of the volume (z-direction). Next, connected components smaller than $500 \mu m^3$ are removed. Since we are interested in evaluating the performance of 2D CNNs, the maximum intensity projections (MIP) of the volumes (along with their respective segmentations) were used as input to the networks – i.e., 2D images with a resolution of $1376 \times 1104$ pixels. Please refer to Ouellette et al. (2020) for a more detailed description of the methodology. An example of labeled cortical vasculature is depicted in Figure 5.

In order to separate the images between sets of high and low tortuosity blood vessels, the average tortuosity value is calculated for each image in the dataset. The 100 images resulting in the highest average tortuosity were labeled as samples containing blood vessels with high tortuosity. Similarly, the 100 images with the lowest average tortuosity were labeled as low tortuosity vessel samples. For performance reasons, each image was divided into 16 windows resulting in 1600 samples for each class with a resolution of $344 \times 276$ pixels. We were careful not to include windows from the same image in both training and validation or test sets. All analyses presented in Section 3 were done using these 3200 samples.

Implementation Details

All experiments were implemented using PyTorch ³. The experiments ran on a desktop computer equipped with an Intel i5-10400f 6 core and 12 threads CPU, 16 GB of RAM, and a Nvidia RTX 2060 6GB GPU. The total processing time of all experiments was approximately 34 days.

Results and Discussion

Generation of the Specialized Networks

Two networks were optimized using the methodology described in Section 2.2.1, each specialized in segmenting blood vessels with either low or high tortuosity. After training, the NSLV obtained an average IoU of 0.8882 ⁴. The NSHV had an average IoU of 0.9158 ⁵. These results are comparable to the performance obtained by other works in the literature focused on the segmentation of blood vessels imaged by confocal microscopy (Todorov et al. 2020; Tahir et al. 2021). Figure 6 shows examples of results obtained by the CNNs when compared to the ground truth labels.

It is worth mentioning that even though our experiments evaluate the segmentation performance using IoU, we optimized our networks using the cross-entropy as loss function. We observed that using the IoU or Dice as loss functions led to a similar performance compared to the cross-entropy, but the cross-entropy led to better convergence.

Fine-Tuning the Specialized Networks

After optimizing the specialized networks, we examined their ability in segmenting data that they originally were not trained on. When used for segmenting blood vessels with high tortuosity, the NSLV obtained an average IoU of 0.6825. Similarly, when segmenting low tortuosity vessels, the NSHV had an average IoU of 0.6516. Figure 7 shows examples of typical segmentation results. Figure 7B shows the segmentation of a sample containing blood vessels with high tortuosity, obtained using the NSLV. In this case, small connected components that are not observed in the ground truth (Figure 7A) are created. These components represent false positives and thus lead to a high recall and low precision. Such artifacts seem to be associated with the fact that this network tends to classify regions of constant intensity as blood vessels since low tortuosity vessels usually do not present longitudinal intensity discontinuities. Figure 7D shows a typical result obtained by the NSHV when applied to images containing blood vessels that predominantly have low tortuosity. By comparing this segmentation with the ground truth (Figure 7C), a considerable caliber underestimation is observed, that is, the segmented blood vessels are typically thinner than the ground truth. Since the result contains many false negatives, the NSHV leads to high precision and low recall when segmenting images with mostly low tortuosity vessels.

The observed segmentation problems can be considered a typical consequence of applying a pre-trained network to a dataset having different statistics than the dataset used for training. Nevertheless, we emphasize that in the present study, the dataset differences are not caused by changes in the sample preparation or imaging protocol but by differences in the morphology of the underlying biological system.

Having made the initial characterization of the segmentation problems, we proceed to try to revert the observed biases by fine-tuning the specialized networks. Thus, as presented in Section 2.2.2, each specialized network undergoes a refinement step using samples from the dataset where the morphometry analysis is to be performed. Figure 8 shows the performance of the NSHV when segmenting vessels with low tortuosity as a function of the number of images used for fine-tuning. The result shows that the values of $IoU_l$ increase with the number of samples. Still, $R_l$ does not display the same behavior. The tortuosity is overestimated when the fine-tuning step is performed with up to approximately 13 images. When more images are used, $R_l$ stabilizes around 1, while $IoU_l$ slowly increases. It is also interesting to note that $IoU_l$ and $R_l$ are not correlated. Therefore, improvements in a pixelwise accuracy metric, such as the IoU, do not necessarily improve the tortuosity estimation.

Figure 9 shows the relationship between the performance of the NSLV when segmenting vessels with high tortuosity and the number of images used for fine-tuning the network. As refinement progresses and more high tortuosity examples are presented to the network, both values of $R_h$ and $IoU_h$ increase. Also, there is a considerable improvement in the segmentation quality ($IoU_h$) when just a few samples are used for fine-tuning. In this case, it was possible to achieve an average IoU of 0.9 with approximately 10 images, in contrast to the 40 images used for achieving the same performance in the previous experiment (Figure 8). Low tortuosity vessels usually have simple morphologies. By fine-tuning the network to just a few additional samples containing high tortuosity vessels, the network quickly adapted to the new morphological characteristics of the samples. Therefore, it was easier to optimize a network for recognizing more complex vessels, starting from a simpler representation, than to optimize it to simplify a more complex vessel representation.

The results obtained from the two experiments considered in this section show that both networks segmented the out-of-distribution samples with similar accuracy. However, each experiment resulted in different degrees of tortuosity underestimation or overestimation depending on the number of images used for fine-tuning. As observed in Figure 8, without fine-tuning, the NSHV overestimated the average tortuosity of the low tortuosity data by approximately $5\%$ compared to the NSLV performance, given that a value of $R_l=1.05$ was obtained when no fine-tuning was performed. Similarly, the NSLV underestimated the average tortuosity value of the high tortuosity data by approximately $8\%$ compared to the NSHV performance, as shown in Figure 9. It was observed that these errors could be mitigated by applying a fine-tuning step with 10 to 40 additional images (0.625% to 2.5% of the original training set), depending on the tortuosity of the blood vessels, which led to a more accurate estimation of the tortuosity.

Improving the Tortuosity Estimation Using Data Augmentation

We first verified if data augmentation can improve the estimation of the tortuosity when training the network from scratch. For this experiment, the procedure described in Section 2.3.1 was used. Figure 10 displays $R_{sf}$ and $IoU_{sf}$ as a function of the parameter $\alpha$ used in the elastic transformation. The value of $\sigma$ was set to 4. We observed that keeping $\sigma=4$ and varying $\alpha$ between 0 and 70 covered several examples of tortuous blood vessels found in our dataset. There is a considerable drop in $R_{sf}$ when $\alpha$ is between 1 and 10. On the other hand, for $\alpha \geq 12$, training with the set of false highly-tortuous vessels improves the tortuosity estimation. The best performance is observed when $\alpha$ is around 64, where $\bar{R}_{sf} \approx 1$ even though the segmentation performance is relatively low, having an average value of $\bar{IoU}_{sf}=0.78$. Another important result is the lack of a strong correlation between $R_{sf}$ and $IoU_{sf}$ for different values of $\alpha$. This reinforces the idea that improvements in vessel morphometry may be caused by effects that are not properly quantified by pixelwise accuracy metrics.

We also investigate if data augmentation can lead to a better estimation of the tortuosity during fine-tuning. The results of applying the methodology described in Section 2.3.2 are depicted in Figure 11. Contrariwise to what was observed in the results shown in Figures 8, 9 and 10, there was no noticeable improvement on the values of $IoU_h$ and $R_h$. Even after fine-tuning the network using 40 additional images, the performance of the network was, on average, $\bar{R}_h \approx 0.9$ and $\bar{IoU}_h \approx 0.8$. Thus, data augmentation was not enough to avoid an underestimation of the tortuosity. This result contrasts with the improvement obtained when applying data augmentation for training the network from scratch.

Some considerations can be made from the obtained result. First, data augmentation was unable to correct the bias of the NSLV after fine-tuning the network. In addition, $\bar{R}_h$ values have large standard deviations, which indicates that the quality of the morphometry obtained by the network is not reliable. Additional approaches could be investigated for fine-tuning the network. Some possibilities include: modifying the learning rate scheduler for finding a solution that better generalizes the data, increasing the number of images used in the fine-tuning step, and using different data augmentation techniques.

Discussion

The segmentation and characterization of blood vessels is an important task for diagnosing many types of diseases as well as for studying the development of neurovascular systems. Therefore, defining robust and unbiased approaches for identifying blood vessels in digital images is of great importance. Usually, most of the focus is placed on obtaining good segmentation accuracy, commonly measured using the Dice or IoU metrics. We focused on identifying to which extent segmentation accuracy is related to morphometric precision when measuring the tortuosity of blood vessels.

Two main investigations were proposed. The first was to identify if the tortuosity calculated for blood vessels may change depending on the CNN used for segmenting the blood vessels, even in situations were the dataset used and the segmentation accuracy is similar for the CNNs. A procedure was developed for comparing a CNN that was trained only on images containing mostly high tortuosity blood vessels with a CNN that was trained on low tortuosity vessels and fine-tuned on vessels with high tortuosity. The opposite situation was also investigated. We found that the tortuosity values obtained from a CNN trained from scratch may not agree with those obtained by a CNN that was pre-trained on data acquired from a different experimental condition, and consequently, with slightly different tortuosity statistics. CNNs trained on blood vessels with high tortuosity tended to lead to an overestimation of the tortuosity, while CNNs trained on blood vessels with low tortuosity resulted in an underestimation of the tortuosity.

The second investigation involved the definition of possible procedures for reducing the observed biases. A fine-tuning approach on the dataset used for inference and a data augmentation methodology were considered. We identified that the tortuosity biases could be corrected by applying a fine-tuning step using additional annotated images. On average, around 13 new annotated images were required for removing the biases. Interestingly, the improvement in tortuosity estimation was not correlated with changes in segmentation accuracy when analyzing the low tortuosity dataset.

Regarding data augmentation, we investigated a possible bias-reducing data augmentation approach using elastic transformations of the training set. This strategy can be helpful for small datasets and also requires fewer manually annotated images. We observed that training the network using the proposed data augmentation approach did not lead to a significant improvement of the IoU metric. However, this approach made the network less biased when used for morphometric quantifications. This means that adding proper data augmentation techniques when training CNNs might be useful even if they do not improve segmentation accuracy. Conversely, when using off-the-shelf networks, it might be useful to give preference to networks that were trained with appropriate data augmentation techniques, even if its accuracy is lower than other state-of-the-art architectures. This result also highlights the importance of correctly optimizing a blood vessel morphometry pipeline for prospective studies regarding vascular systems. The optimization should not consider only pointwise accuracy metrics but also the performance of metrics associated with morphological properties of the blood vessels.

To our knowledge, a similar investigation regarding tortuosity biases has not been performed before in the literature. On the other hand, analyses involving the influence of segmentation on the morphometry of bone trabeculae (Parkinson et al. 2008) and brain structures (Callaert et al. 2014; Rajagopalan et al. 2014; Katuwal et al. 2016) as well as radiotherapy planning (Poel et al. 2021) have been performed before, where similar results were obtained. For instance, established pipelines for voxel-based morphometry of brain MRIs (Magnetic Resonance Images) can lead to inconsistent results regarding the estimation of age-related gray matter volume decrease (Callaert et al. 2014), brain volume in autism spectrum disorder (Katuwal et al. 2016), and gray matter volume atrophy for amyotrophic lateral sclerosis patients (Rajagopalan et al. 2014). Furthermore, in (Poel et al. 2021) the authors report that during radiotherapy planning, geometrical similarity scores (such as the Dice coefficient) used to evaluate the segmentation of peripheral (not-targeted) organs, do not correlate well with the changes in the radiation dosage they receive. To mitigate possible morphometry biases, the authors suggested that new evaluation metrics must be developed, in the sense that they should reflect more accurately the underlying clinical problem – in their case, radiation dosage on peripheral organs. In (Rebsamen et al. 2020) and (Cruz et al. 2021), deep neural networks were trained to directly optimize the morphological features of brain MRIs, rather than the segmentation accuracy. Still, little attention has been given to the influence of network training on morphometry.

For future studies, it would be interesting to consider the influence of segmentation on other morphological quantities, such as blood vessel length and the number of branching points. It would also be interesting to verify if the same results are obtained for other types of blood vessel images such as those acquired by retinal imaging and by angiography. Actually, the same analysis considered in this study can be applied to any system composed of a segmentation step and a downstream quantification task.

Conclusion

Going forward, we argue that more focus should be given to the impact of neural network segmentation on downstream tasks related to blood vessel morphometry. Specifically, when developing computerized systems, it is important to take into account that a high, but not perfect, segmentation performance does not necessarily lead to properly segmented images for morphometric analyses. This is especially true for neural networks given that explaining the results provided by such systems is still an active area of research (Rudin 2019; Adebayo et al. 2018). As shown in our analysis, fine-tuning a pre-trained neural network on new data might not completely remove morphometric biases that may be present in the original network. In such cases, we argue that the network should be retrained from scratch using new data and image augmentation techniques that can directly perturb the morphological measurements that will be calculated.

Funding

Cesar H. Comin thanks FAPESP (grants no. 18/09125-4 and 21/12354-8) for financial support. The authors acknowledge the support of the Government of Canada’s New Frontiers in Research Fund (NFRF) (NFRFE-2019-00641) and the Latin America Research Awards (LARA 2021). This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix: Performance Metrics Used in This Work

Symbols and corresponding definitions of the metrics used for quantifying the performance of blood vessel segmentation and tortuosity calculation. {#tab:metrics}