It seems to be a wide-spread idea that doing Agile means that there is no need for planning or preparation. Planning is seen as 'waterfall' and the feeling seems to be that the uncertainty inherent to complexity means it is a waste of time to even try. People are genuinely convinced that everything will become apparent as sprints progress.
With this disposition it is not surprising backlog refinement ends up as an afterthought with a very limited role.

However, I think backlog refinement is one of the most essential parts of scrum, as the preparation it catalyses is critical to running successful sprints. In software development, it is how we manage the information that gives our work its name: Information Technology. Developers, certainly in a Scrum team, are no longer production workers responsible for writing lines of code, but information managers; experts in finding ways to organise information.

With these thoughts in mind, it becomes apparent that without some kind of rough backlog to allow a team to set priorities and oversee what they are getting themselves into, running sprints is like headless chickens running around in a yard - your velocity may be very high but you have no idea where you are going. The chance you will end up smashing against a wall is very high.

In practise, I find that backlog refinement frequently is a major challenge, especially during the transition to Agile. For me it has always been one of the major hurdles for implementing Scrum, symbolic of the negative things people see in Scrum: too many meetings, too much work leaving too little time for productivity, too hard, unclear results.

So how can we improve this situation? Over the years, working with different kinds of teams and trying various approaches, I have collected the following insights.

It has different goals

Ever notice that often those very long discussions in backlog refinement have to do with different perspectives? The Product Owner is thinking about priorities, one developer is thinking business value, another is thinking of the context within which it will be developed, another one sees all the risks, the last one can't wait to dive in with the newest tech.
Basically, people are mixing different goals, so the conversation ends up going all over the place.

These are the different goals which I have determined together with various teams, in no particular order:

• Why are we building something?
• How are we going to build it (rough idea - enough for estimation)?
• How much effort will it take?

Successful backlog refinement depends very much on synergy between the Product Owner, who clearly plans and communicates the goal of each session, and the Scrum Master, who chooses the best way to facilitate the event to achieve the given goal.

I have found that it also helps to split backlog refinement into different parts, each part focusing on one of the goals. For example, I've had teams where backlog refinement evolved into 3 distinct activities:

1. plenary brainstorm by the team, to create a common understanding and gather collective insights,
2. story preparation, in which missing information is collected and assumptions are validated,
3. estimation.

It is important to realise that these goals can be approached at different levels of detail and not all of them are always required. For example, initially the Product Owner may only want a rough indication of the effort needed to build something, just to be able to decide with stakeholders whether it is worthwhile. He may only want to dive into the details after that, to improve the final estimate.

It's an Event

Like all the other "meetings" in Scrum, backlog refinement is called an "event" in the scrum guide. This is not an attempt at being elitist or at making things unnecessarily vague. The choice of the word "event" has the purpose of leaving the implementation open, allowing us to channel the empiricism characteristic of Scrum towards evolving the event into a method that fits our particular situation. A meeting may seem like the obvious first choice, but gradual improvements will often see that change.

Considering the three separate activities I described above, I tend to replace meetings with the following:

1. collective insights - I'll usually use a 1-2-4-all liberating structure for this one, sometimes followed by a min specs to determine what the must-haves are,
2. story preparation - assigning user stories for individual research or in pairs is very efficient. Of course, only if a user story is actually missing information. I'll leave people to figure out the details of the user story, using INVEST as guidelines for writing a good user story, The Product Owner usually determines a time-cap.
3. estimation - After reading each user story the team play their poker cards silently and only discuss the user story shortly (2-3 minutes) if the estimates vary. If no consensus is reached within the timebox we move the user story back to step one.

It stands above the sprint level

Too often I see backlog refinement as a last minute scramble to prepare work for the upcoming sprint. "We have to estimate these user stories so we have enough work for the sprint!". And if the estimates turn out to be lower than expected, the unreal situation whereby there is no joy, but instead the frustration and stress of finding other work on time to start the next sprint!

Changing the focus of the backlog refinement from the upcoming sprint to a higher level encompassing the foreseeable future allows the development of an extensive backlog of work waiting to be refined.
If we then use backlog refinement as an iterative activity, in which we gradually sharpen the level of detail of all that work, it is easy to always have a ready buffer of work for a team. Furthermore, it also helps the team to keep sight of the overlying context of the individual chunks of work.

To help determine a rhythm for this iterative process I find the rule of having roughly 3 sprints of work ready in the backlog surprisingly valuable. It feels like a rather artificial rule, but it is very effective, as it creates a very good balance between having a work buffer and not investing too much time on work that may have to be reworked due to changing requirements and new insights..

It's an investment

It is important to realise that backlog refinement does not just work out of the box. It is a complex event which requires investment at two different levels.
First of all, at the temporal level it involves making an investment which only bears fruit later, during a sprint, when the work gets picked up. This feedback delay often makes it difficult to improve backlog refinement.
Second, at the competency level it involves investing in various skills that are not necessarily the typical skills developers get hired for. Teams therefore need room to develop these skills.

It is therefore very important as a scrum master to approach the improvement of the backlog refinement with extra attention, simply because due to the temporal effect, the effect of backlog refinement may be obscured.
For example, when discussing a user story that wasn't finished in the sprint, I often find that teams will focus on events during the sprint planning and the sprint itself. The fact that the user story in question might not have been well prepared is often something I need to point out before the team becomes aware of it.

Also, patience! Scrum makes it possible to fix problems very effectively. Often, a problem arising in a sprint is fixed in the following sprint. But with backlog refinement everyone, and especially stakeholders, need to be made aware that fixes take longer before they have the desired effect.
Estimation is perhaps the best example of this. We all know estimating is really, really hard, and something that takes practise and experience to develop. Yet I often find that there is little patience for inaccurate estimation, especially in companies still very much attached to their fixed scope & time approach.

It's a planning tool

The scrum guide says "Product Backlog refinement is the act of adding detail, estimates, and order to items in the Product Backlog". The way I interpret this is that the main focus of the backlog refinement is on refining user stories in order to provide accurate estimates, which in turn allows work to be prioritised better.
Figuring out how to build something - the implementation - is something that should be worked out in the sprint planning and the sprint itself. This is very important because it leaves the implementation to the last possible moment - right at the beginning of the sprint - the most efficient moment to figure it out.

Of course, in order to give accurate estimates, a scrum team needs to have some general idea of the implementation. Like most things Agile, it's a grey area. But it helps to realise that the implementation should only be discussed in so far as it helps to improve estimates.

I find that realising that the backlog refinement is about estimates takes a lot of pressure off the event, helping to make it a more relaxed, fast-moving activity.
The team no longer feel they have to figure everything out perfectly. Gradually, as their experience increases, the team begins to see the backlog refinement as a continuous routine of attention to the backlog, with the event itself being only the highlight of the process, where everything comes together.

With these insights I have seen backlog refinement grow in my teams to a shared and valued part of the scrum process. It has helped my teams to really understand the event and wield it as an effective tool, to the extent that it becomes second nature.
But it has even more far-reaching results. I have seen it bring developers closer to the Product Owner, resulting in a better cooperation to manage the backlog. In one team, due to the fact that they could suddenly choose user stories from a large buffer, the use of sprint goals was quite naturally adopted.

I think my favourite is seeing the Product Owner describing some general requirements after the daily scrum and getting estimates in minutes. That's the kind of agility we are all aiming at!