Tensorflow save:
Generally , we are training a neural network we can mainly observe the loss and accuracy. Whenever the network has converged you can stop the training or set a fixed number of epochs. After that we can save the all of variables and graphs for the future use. So in this tensorflow we can use the save method to save the graphs and all the values of parameters by using a function.
tf.train.Saver()
Here we have to remember one thing which is all the values of parameters are alive in session only. So we can save all the values inside a session only by calling the saver method which we are created.
saver.save(sess, 'my_model')
the sess is the object of session and my-model is the our model name.
lets create an example.
import tensorflow as tf x1 = tf.Variable(tf.random_normal(shape=[2]), name='x1') x2 = tf.Variable(tf.random_normal(shape=[5]), name='x2') saver = tf.train.Saver() sess = tf.Session() sess.run(tf.global_variables_initializer()) saver.save(sess, 'my_test_model') # This will save the all files in Tensorflow v >= 0.11 # my _model.data-00000-of-00001 # my _model.index # my _model.meta # checkpoint
if u want to save your model after 500 iterations , you can call a by function called step_count.
saver.save(sess, 'my_test_model',global_step=1000)
and this step-count is just append to your model name
my _model-500.index my _model-500.meta my _model-500.data-00000-of-00001 checkpoint
Here we have to observe onething which is while training we are saving our model every 500 iterations , so the meta file is created for the fist time i,e 500th iterations and its not recreate at each time that meant we won’t create for 1000 , 1500 or any other iteration. We save our model for further iterations only and the graph will not change.if don’t want to meta graph we can use the below function.
saver.save(sess, 'my-model', global_step=step,write_meta_graph=False
If you want to keep only 5 latest models and want to save one model after every 3 hours during training you can use the function max_to_keep and keep_checkpoint_every_n_hours like this
#saves a model every 3 hours and maximum 5 latest models are saved. saver = tf.train.Saver(max_to_keep=5, keep_checkpoint_every_n_hours=3)
if you want to save all the variables don’t pass any method in tf.train.saver() , this will saves all variables. And you want to save the particular variables we can specify the paricular variable/collection.creating the saver function we pass a dictionary or list that we want to save.
import tensorflow as tf x1 = tf.Variable(tf.random_normal(shape=[2]), name='x1') x2 = tf.Variable(tf.random_normal(shape=[5]), name='x2') saver = tf.train.Saver([w1,w2]) sess = tf.Session() sess.run(tf.global_variables_initializer()) saver.save(sess, 'my _model',global_step=500)
Importing a pre-trained model:
you want to use someone pre-trained model for fine-tuning, there are two things you need to do:
a) Create the network:
writing python code to create a network each and every layer manually as the original model. However, we already saved the network in .meta file which we can use to recreate the network using a function tf.train.import() like this:
saver = tf.train.import_meta_graph('my _model-500.meta')Remember, import_meta_graph appends the network and it will create the graph/network for you but we need to load the value of the parameters of the trained graph.
b) Load the parameters:
Here we can restore the values of parameters by calling the restore function in the tf.train.Saver() class.
with tf.Session() as sess:
new_saver = tf.train.import_meta_graph('my_model-500.meta')
new_saver.restore(sess, tf.train.latest_checkpoint('./'))
After this we can restore and accesed the values of tensors x1 and x2.
with tf.Session() as sess:
saver = tf.train.import_meta_graph('my-model-500.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))
print(sess.run('x1:0'))
##Model has been restored. Above statement will print the saved value of x1
Tensorflow restored models:
Till now you have understood the storing and restoring of data in tensorflow models. Here we can develop a pre trained model and used for prediction , fine-tuning and further training.whenever we are using tensorflow we have to define a graph and which is training data and hyperparameters like learning rate etc.. hyperparameters are training data are feeded by placeholders.
lets see a sample example:
import tensorflow as tf
#Prepare to feed input, i.e. feed_dict and placeholders
x1 = tf.placeholder("float", name="x1")
x2 = tf.placeholder("float", name="x2")
x1= tf.Variable(2.0,name="bias")
feed_dict ={x1:4,x2:8}
#Define a test operation that we will restore
x3 = tf.add(x1,x2)
x4 = tf.multiply(x3,b1,name="op_to_restore")
sess = tf.Session()
sess.run(tf.global_variables_initializer())
#Create an object for saver that will save all the variables
saver = tf.train.Saver()
#Run the operation by feeding input
print sess.run(x4,feed_dict)
#Prints 24 which is sum of (x1+x2)*b1
#Now, save the graph
saver.save(sess, 'my_test_model',global_step=500)
whenever we are going to restore it, not only have to restore the graph and weights, but also use a new feed_dict that will store the new training data to the network. So we can used a method graph.get_tensor_by_name() to get a saved operations and placeholder variables.
#How to access saved variable/Tensor/placeholders
w1 = graph.get_tensor_by_name("w1:0")
## How to access saved operation
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")if you want to run the same network with different data we can pass the data by using a method feed_dict()
import tensorflow as tf
sess=tf.Session()
#First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('my_test_model-500.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))
# access and create placeholders variables
# for feed new data to create feed-dict
graph = tf.get_default_graph()
x1 = graph.get_tensor_by_name("x1:0")
x2 = graph.get_tensor_by_name("x2:0")
feed_dict ={w1:13.0,w2:17.0}
#Now, access the op which want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")
print sess.run(op_to_restore,feed_dict)
#This will print 60 which is calculated
#using new values of x1 and x2 and saved value of b1.
And if you want to add more operations to the graph we can add more layers and then train it.
import tensorflow as tf
sess=tf.Session()
#First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('my_test_model-1000.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))
# Now, let's access and create placeholders variables and
# create feed-dict to feed new data
graph = tf.get_default_graph()
x1 = graph.get_tensor_by_name("x1:0")
x2 = graph.get_tensor_by_name("x2:0")
feed_dict ={w1:13.0,w2:17.0}
#Now, access the op that you want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")
#Add more to the current graph
add_on_op = tf.multiply(op_to_restore,2)
print sess.run(add_on_op,feed_dict)
#This will print 120.
But, if you want to restore the old graph and add-on to that for fine-tuning ? Of-course you can, just taking the appropriate operation by a method graph.get_tensor_by_name() and build graph on top of that. Here we can see an example.
saver = tf.train.import_meta_graph('vgg.meta')
# Access the graph
graph = tf.get_default_graph()
## feed_dict is used for feeding data for fine-tuning
#Access the appropriate output for fine-tuning
fc7= graph.get_tensor_by_name('fc7:0')
# if you want to changes in gradients of the last layer we can use this
fc7 = tf.stop_gradient(fc7) # It's an identity function
fc7_shape= fc7.get_shape().as_list()
new_outputs=2
weights = tf.Variable(tf.truncated_normal([fc7_shape[3], num_outputs], stddev=0.05))
biases = tf.Variable(tf.constant(0.05, shape=[num_outputs]))
output = tf.matmul(fc7, weights) + biases
pred = tf.nn.softmax(output)
# Now, you can run this with fine-tuning data in a method sess.run()Hope ,this will gives you very clear understanding of how the Tensorflow models are saved and restored.