Skip to content

FloatingPointError: Loss became infinite or NaN at iteration=5593 #42

@MishimaCrs

Description

@MishimaCrs

It seems that the loss is too large. I used only 1 gpu to train and set batchsize to 1, and I did not change any other configs. I have tried to set learning rate to 1e-3 and 1e-4, but this error still happen.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions