11
Got roasted by a data scientist for using the same learning rate on every AI model I trained
He said I was treating every dataset like a flat straight road instead of a mountain trail, so I swapped to a cyclic learning rate schedule and my validation loss dropped 30% in half the epochs has anyone else had their training assumptions blown up by someone pointing out something that simple?
2 comments
Log in to join the discussion
Log In2 Comments
james_ross19h ago
Yeah it's wild how one small change can fix everything, like finally using the right key for a lock.
2
paul_lane8016h ago
Three days of tuning hyperparameters down the drain because I had the learning rate cranked to the same number for image data and time series data. The data scientist who pointed it out compared it to trying to parallel park a truck on a dirt road with the same pedal pressure you'd use on the highway. I felt pretty dumb, but then my validation loss dropped from 0.45 to 0.31 in about half the training time after switching to a cyclical schedule. Sometimes the simplest advice is the most humbling.
0