Migrating a Programs Data Accounts
How can you migrate a program's data accounts?
When you create a program, each data account associated with that program will have a specific data structure. If you ever need to upgrade a program derived account, you end up having a bunch of leftover program derived accounts with the old structure.
With account versioning, you can upgrade your old accounts to the new structure.
Note
This is only one of many ways to migrate data in Program Owned Accounts (POA).
Scenario
To version and migrate our account data, we will be providing an id for each account. This id will allow us to identify the account version when we pass it to the program, and thus handle the account correctly.
Take the following account state and program:
//! @brief account_state manages account data
use arrayref::{array_ref, array_refs};
use borsh::{BorshDeserialize, BorshSerialize};
use solana_program::{
msg,
program_error::ProgramError,
program_pack::{IsInitialized, Pack, Sealed},
};
use std::{io::BufWriter, mem};
/// Currently using state. If version changes occur, this
/// should be copied to another serializable backlevel one
/// before adding new fields here
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct AccountContentCurrent {
pub somevalue: u64,
}
/// Maintains account data
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct ProgramAccountState {
is_initialized: bool,
data_version: u8,
account_data: AccountContentCurrent,
}
impl ProgramAccountState {
/// Signal initialized
pub fn set_initialized(&mut self) {
self.is_initialized = true;
}
/// Get the initialized flag
pub fn initialized(&self) -> bool {
self.is_initialized
}
/// Gets the current data version
pub fn version(&self) -> u8 {
self.data_version
}
/// Get the reference to content structure
pub fn content(&self) -> &AccountContentCurrent {
&self.account_data
}
/// Get the mutable reference to content structure
pub fn content_mut(&mut self) -> &mut AccountContentCurrent {
&mut self.account_data
}
}
/// Declaration of the current data version.
pub const DATA_VERSION: u8 = 0;
/// Account allocated size
pub const ACCOUNT_ALLOCATION_SIZE: usize = 1024;
/// Initialized flag is 1st byte of data block
const IS_INITIALIZED: usize = 1;
/// Data version (current) is 2nd byte of data block
const DATA_VERSION_ID: usize = 1;
/// Previous content data size (before changing this is equal to current)
pub const PREVIOUS_VERSION_DATA_SIZE: usize = mem::size_of::<AccountContentCurrent>();
/// Total space occupied by previous account data
pub const PREVIOUS_ACCOUNT_SPACE: usize =
IS_INITIALIZED + DATA_VERSION_ID + PREVIOUS_VERSION_DATA_SIZE;
/// Current content data size
pub const CURRENT_VERSION_DATA_SIZE: usize = mem::size_of::<AccountContentCurrent>();
/// Total usage for data only
pub const CURRENT_USED_SIZE: usize = IS_INITIALIZED + DATA_VERSION_ID + CURRENT_VERSION_DATA_SIZE;
/// How much of 1024 is used
pub const CURRENT_UNUSED_SIZE: usize = ACCOUNT_ALLOCATION_SIZE - CURRENT_USED_SIZE;
/// Current space used by header (initialized, data version and Content)
pub const ACCOUNT_STATE_SPACE: usize = CURRENT_USED_SIZE + CURRENT_UNUSED_SIZE;
/// Future data migration logic that converts prior state of data
/// to current state of data
fn conversion_logic(src: &[u8]) -> Result<ProgramAccountState, ProgramError> {
let past = array_ref![src, 0, PREVIOUS_ACCOUNT_SPACE];
let (initialized, _, _account_space) = array_refs![
past,
IS_INITIALIZED,
DATA_VERSION_ID,
PREVIOUS_VERSION_DATA_SIZE
];
// Logic to uplift from previous version
// GOES HERE
// Give back
Ok(ProgramAccountState {
is_initialized: initialized[0] != 0u8,
data_version: DATA_VERSION,
account_data: AccountContentCurrent::default(),
})
}
impl Sealed for ProgramAccountState {}
impl IsInitialized for ProgramAccountState {
fn is_initialized(&self) -> bool {
self.is_initialized
}
}
impl Pack for ProgramAccountState {
const LEN: usize = ACCOUNT_STATE_SPACE;
/// Store 'state' of account to its data area
fn pack_into_slice(&self, dst: &mut [u8]) {
let mut bw = BufWriter::new(dst);
self.serialize(&mut bw).unwrap();
}
/// Retrieve 'state' of account from account data area
fn unpack_from_slice(src: &[u8]) -> Result<Self, ProgramError> {
let initialized = src[0] != 0;
// Check initialized
if initialized {
// Version check
if src[1] == DATA_VERSION {
msg!("Processing consistent data");
Ok(
ProgramAccountState::try_from_slice(array_ref![src, 0, CURRENT_USED_SIZE])
.unwrap(),
)
} else {
msg!("Processing backlevel data");
conversion_logic(src)
}
} else {
msg!("Processing pre-initialized data");
Ok(ProgramAccountState {
is_initialized: false,
data_version: DATA_VERSION,
account_data: AccountContentCurrent::default(),
})
}
}
}
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct AccountContentCurrent {
pub somevalue: u64,
}
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct ProgramAccountState {
is_initialized: bool,
data_version: u8,
account_data: AccountContentCurrent,
}
//! instruction Contains the main ProgramInstruction enum
use {
crate::error::DataVersionError,
borsh::{BorshDeserialize, BorshSerialize},
solana_program::program_error::ProgramError,
};
#[derive(BorshDeserialize, BorshSerialize, Debug, PartialEq)]
/// All custom program instructions
pub enum ProgramInstruction {
InitializeAccount,
SetU64Value(u64),
FailInstruction,
}
impl ProgramInstruction {
/// Unpack inbound buffer to associated Instruction
/// The expected format for input is a Borsh serialized vector
pub fn unpack(input: &[u8]) -> Result<Self, ProgramError> {
let payload = ProgramInstruction::try_from_slice(input).unwrap();
match payload {
ProgramInstruction::InitializeAccount => Ok(payload),
ProgramInstruction::SetU64Value(_) => Ok(payload),
_ => Err(DataVersionError::InvalidInstruction.into()),
}
}
}
impl ProgramInstruction {
/// Unpack inbound buffer to associated Instruction
/// The expected format for input is a Borsh serialized vector
pub fn unpack(input: &[u8]) -> Result<Self, ProgramError> {
let payload = ProgramInstruction::try_from_slice(input).unwrap();
match payload {
ProgramInstruction::InitializeAccount => Ok(payload),
ProgramInstruction::SetU64Value(_) => Ok(payload),
_ => Err(DataVersionError::InvalidInstruction.into()),
}
}
}
//! Resolve instruction and execute
use crate::{
account_state::ProgramAccountState, error::DataVersionError, instruction::ProgramInstruction,
};
use solana_program::{
account_info::{next_account_info, AccountInfo},
entrypoint::ProgramResult,
msg,
program_error::ProgramError,
program_pack::{IsInitialized, Pack},
pubkey::Pubkey,
};
/// Checks each tracking account to confirm it is owned by our program
/// This function assumes that the program account is always the last
/// in the array
fn check_account_ownership(program_id: &Pubkey, accounts: &[AccountInfo]) -> ProgramResult {
// Accounts must be owned by the program.
for account in accounts.iter().take(accounts.len() - 1) {
if account.owner != program_id {
msg!(
"Fail: The tracking account owner is {} and it should be {}.",
account.owner,
program_id
);
return Err(ProgramError::IncorrectProgramId);
}
}
Ok(())
}
/// Initialize the programs account, which is the first in accounts
fn initialize_account(accounts: &[AccountInfo]) -> ProgramResult {
msg!("Initialize account");
let account_info_iter = &mut accounts.iter();
let program_account = next_account_info(account_info_iter)?;
let mut account_data = program_account.data.borrow_mut();
// Just using unpack will check to see if initialized and will
// fail if not
let mut account_state = ProgramAccountState::unpack_unchecked(&account_data)?;
// Where this is a logic error in trying to initialize the same account more than once
if account_state.is_initialized() {
return Err(DataVersionError::AlreadyInitializedState.into());
} else {
account_state.set_initialized();
account_state.content_mut().somevalue = 1;
}
msg!("Account Initialized");
// Serialize
ProgramAccountState::pack(account_state, &mut account_data)
}
/// Sets the u64 in the content structure
fn set_u64_value(accounts: &[AccountInfo], value: u64) -> ProgramResult {
msg!("Set new value {}", value);
let account_info_iter = &mut accounts.iter();
let program_account = next_account_info(account_info_iter)?;
let mut account_data = program_account.data.borrow_mut();
let mut account_state = ProgramAccountState::unpack(&account_data)?;
account_state.content_mut().somevalue = value;
// Serialize
ProgramAccountState::pack(account_state, &mut account_data)
}
/// Main processing entry point dispatches to specific
/// instruction handlers
pub fn process(
program_id: &Pubkey,
accounts: &[AccountInfo],
instruction_data: &[u8],
) -> ProgramResult {
msg!("Received process request");
// Check the account for program relationship
if let Err(error) = check_account_ownership(program_id, accounts) {
return Err(error);
};
// Unpack the inbound data, mapping instruction to appropriate structure
let instruction = ProgramInstruction::unpack(instruction_data)?;
match instruction {
ProgramInstruction::InitializeAccount => initialize_account(accounts),
ProgramInstruction::SetU64Value(value) => set_u64_value(accounts, value),
_ => {
msg!("Received unknown instruction");
Err(DataVersionError::InvalidInstruction.into())
}
}
}
fn check_account_ownership(program_id: &Pubkey, accounts: &[AccountInfo]) -> ProgramResult {
// Accounts must be owned by the program.
for account in accounts.iter().take(accounts.len() - 1) {
if account.owner != program_id {
msg!(
"Fail: The tracking account owner is {} and it should be {}.",
account.owner,
program_id
);
return Err(ProgramError::IncorrectProgramId);
}
}
Ok(())
}
/// Initialize the programs account, which is the first in accounts
fn initialize_account(accounts: &[AccountInfo]) -> ProgramResult {
msg!("Initialize account");
let account_info_iter = &mut accounts.iter();
let program_account = next_account_info(account_info_iter)?;
let mut account_data = program_account.data.borrow_mut();
// Just using unpack will check to see if initialized and will
// fail if not
let mut account_state = ProgramAccountState::unpack_unchecked(&account_data)?;
// Where this is a logic error in trying to initialize the same account more than once
if account_state.is_initialized() {
return Err(DataVersionError::AlreadyInitializedState.into());
} else {
account_state.set_initialized();
account_state.content_mut().somevalue = 1;
}
msg!("Account Initialized");
// Serialize
ProgramAccountState::pack(account_state, &mut account_data)
}
/// Sets the u64 in the content structure
fn set_u64_value(accounts: &[AccountInfo], value: u64) -> ProgramResult {
msg!("Set new value {}", value);
let account_info_iter = &mut accounts.iter();
let program_account = next_account_info(account_info_iter)?;
let mut account_data = program_account.data.borrow_mut();
let mut account_state = ProgramAccountState::unpack(&account_data)?;
account_state.content_mut().somevalue = value;
// Serialize
ProgramAccountState::pack(account_state, &mut account_data)
}
/// Main processing entry point dispatches to specific
/// instruction handlers
pub fn process(
program_id: &Pubkey,
accounts: &[AccountInfo],
instruction_data: &[u8],
) -> ProgramResult {
msg!("Received process request");
// Check the account for program relationship
if let Err(error) = check_account_ownership(program_id, accounts) {
return Err(error);
};
// Unpack the inbound data, mapping instruction to appropriate structure
let instruction = ProgramInstruction::unpack(instruction_data)?;
match instruction {
ProgramInstruction::InitializeAccount => initialize_account(accounts),
ProgramInstruction::SetU64Value(value) => set_u64_value(accounts, value),
_ => {
msg!("Received unknown instruction");
Err(DataVersionError::InvalidInstruction.into())
}
}
}
In our first version of an account, we are doing the following:
ID | Action |
---|---|
1 | Include a 'data version' field in your data. It can be a simple incrementing ordinal (e.g. u8) or something more sophisticated |
2 | Allocating enough space for data growth |
3 | Initializing a number of constants to be used across program versions |
4 | Add an update account function under fn conversion_logic for future upgrades |
Let's say we want to upgrade our program's accounts now to include a new required field, the somestring
field.
If we didn't allocate extra space on the previous account, we could not upgrade the account and be stuck.
Upgrading the Account
In our new program we want to add a new property for the content state. The changes that follow are how we leveraged the initial program constructs as they come into use now.
1. Add account conversion logic
//! @brief account_state manages account data
use arrayref::{array_ref, array_refs};
use borsh::{BorshDeserialize, BorshSerialize};
use solana_program::{
borsh::try_from_slice_unchecked,
msg,
program_error::ProgramError,
program_pack::{IsInitialized, Pack, Sealed},
};
use std::{io::BufWriter, mem};
/// Current state (DATA_VERSION 1). If version changes occur, this
/// should be copied to another (see AccountContentOld below)
/// We've added a new field: 'somestring'
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct AccountContentCurrent {
pub somevalue: u64,
pub somestring: String,
}
/// Old content state (DATA_VERSION 0).
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct AccountContentOld {
pub somevalue: u64,
}
/// Maintains account data
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct ProgramAccountState {
is_initialized: bool,
data_version: u8,
account_data: AccountContentCurrent,
}
impl ProgramAccountState {
/// Signal initialized
pub fn set_initialized(&mut self) {
self.is_initialized = true;
}
/// Get the initialized flag
pub fn initialized(&self) -> bool {
self.is_initialized
}
/// Gets the current data version
pub fn version(&self) -> u8 {
self.data_version
}
/// Get the reference to content structure
pub fn content(&self) -> &AccountContentCurrent {
&self.account_data
}
/// Get the mutable reference to content structure
pub fn content_mut(&mut self) -> &mut AccountContentCurrent {
&mut self.account_data
}
}
/// Declaration of the current data version.
const DATA_VERSION: u8 = 1; // Adding string to content
// Previous const DATA_VERSION: u8 = 0;
/// Account allocated size
const ACCOUNT_ALLOCATION_SIZE: usize = 1024;
/// Initialized flag is 1st byte of data block
const IS_INITIALIZED: usize = 1;
/// Data version (current) is 2nd byte of data block
const DATA_VERSION_ID: usize = 1;
/// Previous content data size (before changing this is equal to current)
const PREVIOUS_VERSION_DATA_SIZE: usize = mem::size_of::<AccountContentOld>();
/// Total space occupied by previous account data
const PREVIOUS_ACCOUNT_SPACE: usize = IS_INITIALIZED + DATA_VERSION_ID + PREVIOUS_VERSION_DATA_SIZE;
/// Current content data size
const CURRENT_VERSION_DATA_SIZE: usize = mem::size_of::<AccountContentCurrent>();
/// Total usage for data only
const CURRENT_USED_SIZE: usize = IS_INITIALIZED + DATA_VERSION_ID + CURRENT_VERSION_DATA_SIZE;
/// How much of 1024 is used
const CURRENT_UNUSED_SIZE: usize = ACCOUNT_ALLOCATION_SIZE - CURRENT_USED_SIZE;
/// Current space used by header (initialized, data version and Content)
pub const ACCOUNT_STATE_SPACE: usize = CURRENT_USED_SIZE + CURRENT_UNUSED_SIZE;
/// Future data migration logic that converts prior state of data
/// to current state of data
fn conversion_logic(src: &[u8]) -> Result<ProgramAccountState, ProgramError> {
let past = array_ref![src, 0, PREVIOUS_ACCOUNT_SPACE];
let (initialized, _, account_space) = array_refs![
past,
IS_INITIALIZED,
DATA_VERSION_ID,
PREVIOUS_VERSION_DATA_SIZE
];
// Logic to upgrade from previous version
// GOES HERE
let old = try_from_slice_unchecked::<AccountContentOld>(account_space).unwrap();
// Default sets 'somevalue' to 0 and somestring to default ""
let mut new_content = AccountContentCurrent::default();
// We copy the existing 'somevalue', the program instructions will read/update 'somestring' without fail
new_content.somevalue = old.somevalue;
// Give back
Ok(ProgramAccountState {
is_initialized: initialized[0] != 0u8,
data_version: DATA_VERSION,
account_data: new_content,
})
}
impl Sealed for ProgramAccountState {}
impl IsInitialized for ProgramAccountState {
fn is_initialized(&self) -> bool {
self.is_initialized
}
}
impl Pack for ProgramAccountState {
const LEN: usize = ACCOUNT_STATE_SPACE;
/// Store 'state' of account to its data area
fn pack_into_slice(&self, dst: &mut [u8]) {
let mut bw = BufWriter::new(dst);
self.serialize(&mut bw).unwrap();
}
/// Retrieve 'state' of account from account data area
fn unpack_from_slice(src: &[u8]) -> Result<Self, ProgramError> {
let initialized = src[0] != 0;
// Check initialized
if initialized {
// Version check
if src[1] == DATA_VERSION {
msg!("Processing consistent version data");
Ok(try_from_slice_unchecked::<ProgramAccountState>(src).unwrap())
} else {
msg!("Processing backlevel data");
conversion_logic(src)
}
} else {
msg!("Processing pre-initialized data");
Ok(ProgramAccountState {
is_initialized: false,
data_version: DATA_VERSION,
account_data: AccountContentCurrent::default(),
})
}
}
}
/// Current state (DATA_VERSION 1). If version changes occur, this
/// should be copied to another (see AccountContentOld below)
/// We've added a new field: 'somestring'
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct AccountContentCurrent {
pub somevalue: u64,
pub somestring: String,
}
/// Old content state (DATA_VERSION 0).
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct AccountContentOld {
pub somevalue: u64,
}
/// Maintains account data
#[derive(BorshDeserialize, BorshSerialize, Debug, Default, PartialEq)]
pub struct ProgramAccountState {
is_initialized: bool,
data_version: u8,
account_data: AccountContentCurrent,
}
Line(s) | Note |
---|---|
6 | We've added Solana's solana_program::borsh::try_from_slice_unchecked to simplify reading subsets of data from the larger data block |
13-26 | Here we've preserved the old content structure, AccountContentOld line 24, before extending the AccountContentCurrent starting in line 17. |
60 | We bump the DATA_VERSION constant |
71 | We now have a 'previous' version and we want to know it's size |
86 | The Coup de grâce is adding the plumbing to upgrade the previous content state to the new (current) content state |
We then update our instructions, to add a new one for updating somestring
, and processor for handling the new instruction. Note that the 'upgrading' the data structure is encapsulated behind pack/unpack
//! instruction Contains the main VersionProgramInstruction enum
use {
crate::error::DataVersionError,
borsh::{BorshDeserialize, BorshSerialize},
solana_program::{borsh::try_from_slice_unchecked, msg, program_error::ProgramError},
};
#[derive(BorshDeserialize, BorshSerialize, Debug, PartialEq)]
/// All custom program instructions
pub enum VersionProgramInstruction {
InitializeAccount,
SetU64Value(u64),
SetString(String), // Added with data version change
FailInstruction,
}
impl VersionProgramInstruction {
/// Unpack inbound buffer to associated Instruction
/// The expected format for input is a Borsh serialized vector
pub fn unpack(input: &[u8]) -> Result<Self, ProgramError> {
let payload = try_from_slice_unchecked::<VersionProgramInstruction>(input).unwrap();
// let payload = VersionProgramInstruction::try_from_slice(input).unwrap();
match payload {
VersionProgramInstruction::InitializeAccount => Ok(payload),
VersionProgramInstruction::SetU64Value(_) => Ok(payload),
VersionProgramInstruction::SetString(_) => Ok(payload), // Added with data version change
_ => Err(DataVersionError::InvalidInstruction.into()),
}
}
}
//! Resolve instruction and execute
use crate::{
account_state::ProgramAccountState, error::DataVersionError,
instruction::VersionProgramInstruction,
};
use solana_program::{
account_info::{next_account_info, AccountInfo},
entrypoint::ProgramResult,
msg,
program_error::ProgramError,
program_pack::{IsInitialized, Pack},
pubkey::Pubkey,
};
/// Checks each tracking account to confirm it is owned by our program
/// This function assumes that the program account is always the last
/// in the array
fn check_account_ownership(program_id: &Pubkey, accounts: &[AccountInfo]) -> ProgramResult {
// Accounts must be owned by the program.
for account in accounts.iter().take(accounts.len() - 1) {
if account.owner != program_id {
msg!(
"Fail: The tracking account owner is {} and it should be {}.",
account.owner,
program_id
);
return Err(ProgramError::IncorrectProgramId);
}
}
Ok(())
}
/// Initialize the programs account, which is the first in accounts
fn initialize_account(accounts: &[AccountInfo]) -> ProgramResult {
msg!("Initialize account");
let account_info_iter = &mut accounts.iter();
let program_account = next_account_info(account_info_iter)?;
let mut account_data = program_account.data.borrow_mut();
// Just using unpack will check to see if initialized and will
// fail if not
let mut account_state = ProgramAccountState::unpack_unchecked(&account_data)?;
// Where this is a logic error in trying to initialize the same account more than once
if account_state.is_initialized() {
return Err(DataVersionError::AlreadyInitializedState.into());
} else {
account_state.set_initialized();
account_state.content_mut().somevalue = 1;
}
msg!("Account Initialized");
// Serialize
ProgramAccountState::pack(account_state, &mut account_data)
}
/// Sets the u64 in the content structure
fn set_u64_value(accounts: &[AccountInfo], value: u64) -> ProgramResult {
msg!("Set new value {}", value);
let account_info_iter = &mut accounts.iter();
let program_account = next_account_info(account_info_iter)?;
let mut account_data = program_account.data.borrow_mut();
let mut account_state = ProgramAccountState::unpack(&account_data)?;
account_state.content_mut().somevalue = value;
// Serialize
ProgramAccountState::pack(account_state, &mut account_data)
}
/// Sets the string in the content structure
fn set_string_value(accounts: &[AccountInfo], value: String) -> ProgramResult {
msg!("Set new string {}", value);
let account_info_iter = &mut accounts.iter();
let program_account = next_account_info(account_info_iter)?;
let mut account_data = program_account.data.borrow_mut();
let mut account_state = ProgramAccountState::unpack(&account_data)?;
account_state.content_mut().somestring = value;
// Serialize
ProgramAccountState::pack(account_state, &mut account_data)
}
/// Main processing entry point dispatches to specific
/// instruction handlers
pub fn process(
program_id: &Pubkey,
accounts: &[AccountInfo],
instruction_data: &[u8],
) -> ProgramResult {
msg!("Received process request 0.2.0");
// Check the account for program relationship
if let Err(error) = check_account_ownership(program_id, accounts) {
return Err(error);
};
// Unpack the inbound data, mapping instruction to appropriate structure
msg!("Attempting to unpack");
let instruction = VersionProgramInstruction::unpack(instruction_data)?;
match instruction {
VersionProgramInstruction::InitializeAccount => initialize_account(accounts),
VersionProgramInstruction::SetU64Value(value) => set_u64_value(accounts, value),
VersionProgramInstruction::SetString(value) => set_string_value(accounts, value),
_ => {
msg!("Received unknown instruction");
Err(DataVersionError::InvalidInstruction.into())
}
}
}
After building and submitting an instruction: VersionProgramInstruction::SetString(String)
we now have the following 'upgraded' account data layout