minor-mode disable font weight and set to original value

[goncholden]

I have a minor mode to which I want the font-lock-comment-face to be normal 
because I have
used (set-face-attribute 'default nil :height 160 :weight 'ultra-bold) in my 
init file.

How can I include normal weight for comments when rich-minor-mode is enabled, 
but revert back to
the original setting (:weight 'ultra-bold) when the minor-mode is disabled?

(defun rich-annotation-font-weight ()
"todo"
(set-face-attribute 'font-lock-comment-face nil :weight 'normal))

;;;###autoload
(define-minor-mode rich-minor-mode "todo" nil "" nil

(font-lock-remove-keywords nil rich-font-lock)

(when rich-minor-mode
(font-lock-add-keywords nil rich-font-lock 'append)
(set (make-local-variable 'jit-lock-contextually) t) )

(when font-lock-mode
(if (fboundp 'font-lock-flush)
(font-lock-flush)
(with-no-warnings (font-lock-fontify-buffer)) ))

(rich-annotation-font-weight) )