Migrating Spring Sleuth to Micrometer Observability in Spring Boot 3

Spring Sleuth was removed in Spring Boot version 3 in favor of Micrometer and I recently migrated many services and want to share a bit of the learning and things that we need to change to make everything work with Micrometer as they were working before.

We were using mostly automatic instrumentation and in some places, we were using @NewSpan or even injecting the Tracer and starting a scoped span to track some specific operations. We had distributed tracing in HTTP requests, RabbitMQ messages, Scheduled Jobs, and Async executions.

Under the hood, our Spring Sleuth was using Brave as the implementation and we were exporting the context in the Zipkin B3 format with the multiple headers options (X-B3-TraceId, X-B3-ParentSpanId, and X-B3-SpanId).

I am not going to go over all the things that require micrometer work from scratch, as there are already very good guides on how to do it, I will mostly focus on what we need to change from Sleuth to Micrometer.

Let’s start with the dependencies.

Dependencies

After upgrading the Spring Boot version to something like 3.2.4 (the latest when this article was written), we have to remove the sleuth dependencies, in our case we had two:

implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin"
implementation "org.springframework.cloud:spring-cloud-starter-sleuth"

And we need to add the new ones.

implementation 'io.micrometer:micrometer-tracing-bridge-brave'  
implementation 'io.zipkin.reporter2:zipkin-reporter-brave'

These are the basic ones, which should allow you to trace and export to Zipkin.

HTTP request

By default, Micrometer uses W3C propagation type. In the Migration Page they recommend setting Sleuth & Boot 2.x application

spring.sleuth.propagation.type=w3c,b3

so it exports both and you can use the default in the newly migrated application. But we cannot do this as we don’t only have Spring Services, we also have Node.JS which receives expects the B3 format for now.

To continue being compatible with the whole system, we just changed the default propagation type in Micrometer to b3_multi by adding the following property.

management.tracing.propagation.type=b3_multi

Sampling

Learned this the really hard way (by having it in production for a few days) but by default, (not sure why they thought that it was a good idea), but it only exports 10% of the traces (source) and you have to change a property to send all.

management.tracing.sampling.probability=1

I was really pissed about this, it should always export everything and IF YOU WANT to change, you can, but not the other way around. Also, the naming is very confusing as setting it to 100% probability of sampling, means that it sends all and not discard all.

@NewSpan

To enable support for the “old” annotations like @NewSpan, we needed to add the following property (source)

management.observations.annotations.enabled=true

You also have to make sure that you have Spring AOP in the classpath, as it is required by what I understood of the documentation.

implementation "org.springframework.boot:spring-boot-starter-aop"

RabbitMQ

For RabbitMQ, we were injecting

private final SpringRabbitTracing springRabbitTracing;

and setting the decorateSimpleRabbitListenerContainerFactory

SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();  
springRabbitTracing.decorateSimpleRabbitListenerContainerFactory(factory);

With our container factory which was used to create our consumer factory.

As we don’t have the SpringRabbitTracing anymore in Micrometer, what we did was to remove these lines and simply enable tracing in the RabbitTemplate.

rabbitTemplate.setObservationEnabled(true);

We also have to set the enabled in the ConnectionFactory based on this to enable the tracing for the consumers

factory.setObservationEnabled(true);

Database

For databases, we had P6Spy via

implementation "com.github.gavlyukovskiy:p6spy-spring-boot-starter:1.8.1"

but we were not really using all the functionalities of P6Spy and only the JDBC tracing so we changed the library to use the

implementation 'net.ttddyy.observation:datasource-micrometer-spring-boot:1.0.3'` 

in which instruments the Database with Micrometer. Maybe com.github.gavlyukovskiy still works, I haven’t fully checked that as I wanted to get rid of P6Spy logging as we use the span duration to detect slow queries now.

One adjustment I had to make was to rename the service name in the spans for the Database spans. Without doing anything, it was using the Hikari Pool name (as our pool didn’t start with HikariPool-) but otherwise, it would use the schema name (as resolved by DefaultDataSourceNameResolver. I wanted to change it to the service name, so I created a custom DataSourceNameResolver that returned the service name from the application properties.

@Configuration  
public class ObservabilityConfig  
{  
    @Value("${spring.application.name}")  
    private String serviceName;  
  

    @Bean  
    public DataSourceNameResolver dataSourceNameResolver()  
    {  
        return (beanName, dataSource) -> serviceName;  
    }  
}

Async

For our async configuration, we were using LazyTraceExecutor in something like this

@EnableAsync  
@Configuration  
public class AsyncConfiguration implements AsyncConfigurer  
{  
    @Inject  
    private BeanFactory beanFactory;  

  
    @Override  
   public Executor getAsyncExecutor()  
    {  
        return new LazyTraceExecutor(beanFactory, new OurCusomThreadPool("async-task-"));  
    }  
}

And we changed it to the code below based on this StackOverflow answer

@EnableAsync  
@Configuration  
public class AsyncConfiguration implements AsyncConfigurer  
{  

  
    @Override  
    public Executor getAsyncExecutor()  
    {  
        ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();  
        threadPoolTaskExecutor.setCorePoolSize(5);  
        threadPoolTaskExecutor.setMaxPoolSize(5);  
        threadPoolTaskExecutor.setThreadNamePrefix("async-task-");  
  
        threadPoolTaskExecutor.initialize();  
  
  
        return ContextExecutorService.wrap(Executors.newCachedThreadPool(threadPoolTaskExecutor), ContextSnapshot::captureAll);  
    }  
}

Scheduled

For the scheduled tasks, we did something a bit different, in the configuration that implemented SchedulingConfigurer, we injected the ObservationRegistry and set it in the ScheduledTaskRegistrar based on the docs

@EnableScheduling  
@EnableSchedulerLock(defaultLockAtMostFor = "PT05M")  
@RequiredArgsConstructor  
@Configuration  
public class SchedulingConfiguration implements SchedulingConfigurer  
{  
    private final ObservationRegistry observationRegistry;  
  
    @Override  
    public void configureTasks(ScheduledTaskRegistrar taskRegistrar)  
    {  
        taskRegistrar.setTaskScheduler(new OurTaskScheduler("scheduled-task-")); 
        taskRegistrar.setObservationRegistry(observationRegistry);  
    }  
}

Differences

I notice some differences in the tracing and tags. These are not all the changes, but just the ones that I have noticed.

HTTP spans

In Sleuth, we have the following tags http.method, http.path, http.status_code, mvc.controller.class, mvc.controller.method and Client Address.

In micrometer, http.method became method, http.path became http.url and http.status_code became status and we have a new uri which shows the URL without interpolation (a nice add).

Also, a good improvement is that database spans (select, insert, result-set) are not duplicated anymore. In Sleuth, it generated two for each operation, one for the server and one client, with the same data. In my opinion that was not necessary and only polluted (making us have to add sampling). Now we have only one span.

These were some of the changes that we made. I hope this helps you in some way.

Cheers.