Unable to override agent per stage

I have a pipeline with top-level agent but want to override single stage to run on different agent:

pipeline {
  agent { label 'agent-pool' }
  stages {
    stage('checkout') {}
    stage('prepare data') {
      agent { label 'special-agent' }
      steps {
        // do what's needed
        stash name:'data', include: 'data/**'
    }
    stage('build') {
      unstash 'data'
    }
    // more stages that should execute on the same agent from agent-pool
    stage('deploy') {}
  }
} 

The idea is there are a lot of stages that execute on single node but there is one special stage that has to execute on special agent that is not in the pool (and move results around with stash).

My reading of https://www.jenkins.io/doc/book/pipeline/syntax/#agent and JENKINS-37779 is that this should just work however when I tried this, Jenkins attempted to use just about any agent except the one matching special-agent:

'agent-1': doesn't have label 'special-agent'
'agent-2': doesn't have label 'special-agent'
# etc...
# infinite spinner 

I've found one post on stackoverflow claiming this is actually not supported: https://stackoverflow.com/questions/46630168/in-a-declarative-jenkins-pipeline-can-i-set-the-agent-label-dynamically#comment116214727_46631354 however this goes against referenced documentation.