On windows 10 with miniconda enviroment tensorflow 2.5 gives error on centralized_gradients.py file.

the solution is change import keras.backend as K with import tensorflow.keras.backend as K

Describe the bug The results in your colab ipython notebook are misleading: https://colab.research.google.com/github/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/main/examples/gctf_mnist.ipynb

In this example, the model is first trained with a normal Adam optimizer:

model.compile(optimizer = tf.keras.optimizers.Adam(),
              loss = 'sparse_categorical_crossentropy',
              metrics = ['accuracy'])

history_no_gctf = model.fit(training_images, training_labels, epochs=5, callbacks = [time_callback_no_gctf])

And afterwards the same model is recompiled with the gctf.optimizers.adam(). However, recompiling a keras model does not reset the weights. This means that in the first fit call the model is trained and then in the second fit call with the new optimizer the same model is used and of course then the results are better.

This can be fixed, by recreating the model for the second run, by just adding these few lines:

import gctf #import gctf

time_callback_gctf = TimeHistory()

# Model architecture
model = tf.keras.models.Sequential([
                                    tf.keras.layers.Flatten(), 
                                    tf.keras.layers.Dense(512, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(256, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(64, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(512, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(256, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(64, activation=tf.nn.relu), 
                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])

model.compile(optimizer = gctf.optimizers.adam(),
              loss = 'sparse_categorical_crossentropy',
              metrics=['accuracy'])

history_gctf = model.fit(training_images, training_labels, epochs=5, callbacks=[time_callback_gctf])

However, then the results are not better than without gctf:

Type                   Execution time    Accuracy      Loss
-------------------  ----------------  ----------  --------
Model without gctf:           24.7659    0.88825   0.305801
Model with gctf               24.7881    0.889567  0.30812

Could you please clarify what happens here. I tried this gctf.optimizers.adam() optimizer in my own research and it didn't change the results at all and now after seeing it doesn't work in the example which was constructed here. Makes me question the results of this paper.

To Reproduce Execute the colab file given in the repository: https://colab.research.google.com/github/Rishit-dagli/Gradient-Centralization-TensorFlow/blob/main/examples/gctf_mnist.ipynb

Expected behavior The right comparison would be if both models start from a random initialization, not that the second model can start with the already pre-trained weights.

Looking forward to a fast a swift explanation.

Best, Max

The package as of now to be installed requires tensorflow ~= 2.4.0 and keras ~= 2.4.0. It turns out that this is sometimes problematic for folks who have custom installations of TensorFlow and a winder requirement could be set up.

This release includes some fixes and improvements

✅ Bug Fixes / Improvements

• Allow wider versions for TensorFlow and Keras while installing the package (#14 )
• Fixed incorrect usage example in docstrings and description for centralized_gradients_for_optimizer (#13 )
• Add clear aims for each of the examples of using gctf (#15 )
• Updates PyPi classifiers to clearly show the aims of this project. This should have no changes in the way you use this package (#18 )
• Add clear instructions for using this with custom optimizers i.e. directly use get_centralized_gradients however a complete example has not been pushed due to the reasons mentioned in the issue (#16 )

Add an "About The Examples" section which contains a summary of the usage example notebooks and links to run it on Binder and Colab.

Close #15

Add PyPI classifiers for:

• Development status
• Intended Audience
• Topic

Further also added the Programming Language :: Python :: 3 :: Only classifer

Closes #18

I am specifically thinking of adding three more categories of pypi classifiers:

• Development status
• Intended Audience
• Topic

Apart from this I also think it would be great to add the Programming Language :: Python :: 3 :: Only to make sure the audience to know that this package is intended for Python 3 only.

It would be great to write an "About the example" section which could demonstrate in short what the example notebooks aim to achieve and show.

I noticed that the docstrings for gctf.centralized_gradients_for_optimizer have an error in the example usage section. The example creates an Adam optimizer instance and saves it to opt however the centralized_gradients_for_optimizer is applied on optimizer which ideally does not exist and running the example would result in an error.

Beep boop. Your images are optimized!

Your image file size has been reduced by 19% 🎉

Black Lives Matter | 💰 donate | 🎓 learn | ✍🏾 sign

📝 docs | :octocat: repo | 🙋🏾 issues | 🏅 swag | 🏪 marketplace

Beep boop. Your images are optimized!

Your image file size has been reduced by 19% 🎉

Black Lives Matter | 💰 donate | 🎓 learn | ✍🏾 sign

📝 docs | :octocat: repo | 🙋🏾 issues | 🏅 swag | 🏪 marketplace